IDEAS home Printed from https://ideas.repec.org/a/aea/apandp/v110y2020p75-79.html
   My bibliography  Save this article

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Author

Listed:
  • Stefano DellaVigna
  • Nicholas Otis
  • Eva Vivalt

Abstract

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values, (2) whether predictions are in raw units or standard deviations, (3) text-entry versus slider responses, and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered.

Suggested Citation

  • Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    • Handle: RePEc:aea:apandp:v:110:y:2020:p:75-79
    DOI: 10.1257/pandp.20201080
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080
    Download Restriction: no
    File URL: http://doi.org/10.3886/E117601V1
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080.appx
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080.ds
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    File URL: https://libkey.io/10.1257/pandp.20201080?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dirk Bergemann & Marco Ottaviani, 2021. "Information Markets and Nonmarkets," Cowles Foundation Discussion Papers 2296, Cowles Foundation for Research in Economics, Yale University.
    2. Tamim, Abdulrazzak & Smith, Emma & Palmer, I. Bailey & Miguel, Edward & Oparina, Ekaterina & Rozo, Sandra V. & Stillman, Sarah, 2025. "Housing Subsidies for Refugees: Experimental Evidence on Life Outcomes and Social Integration in Jordan," IZA Discussion Papers 17622, Institute of Labor Economics (IZA).
    3. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    4. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    5. Alston, Mackenzie & Deryugina, Tatyana & Shurchkov, Olga, 2025. "Leaving Money on the Table," IZA Discussion Papers 17812, Institute of Labor Economics (IZA).
    6. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    7. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    8. Henning Schaak & Jens Rommel & Julian Sagebiel & Jesus Barreiro-Hurlé & Douadia Bougherara & Luigi Cemablo & Marija Cerjak & Tajana Čop & Mikołaj Czajkowski & María Espinosa-Goded & Julia Höhler & Car, 2022. "How Well Can Experts Predict Farmers' Risk Preferences ?," Post-Print hal-03738351, HAL.
    9. Tamim,Abdulrazzak & Smith,Emma & Palmer,I. Bailey & Miguel,Edward & Leone,Samuel & Rozo, Sandra & Stillman,Sarah, 2025. "Housing Subsidies for Refugees : Experimental Evidence on Life Outcomes and Social Integration in Jordan," Policy Research Working Paper Series 11042, The World Bank.
    10. Olckers, Matthew, 2021. "On track for retirement?," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 76-88.
    11. Matias Giaccobasso & Brad C. Nathan & Ricardo Perez-Truglia & Alejandro Zentner, 2022. "Where Do My Tax Dollars Go? Tax Morale Effects of Perceived Government Spending," NBER Working Papers 29789, National Bureau of Economic Research, Inc.
    12. Yang, Dean & Allen, James & Mahumane, Arlete & Riddell, James & Yu, Hang, 2023. "Knowledge, stigma, and HIV testing: An analysis of a widespread HIV/AIDS program," Journal of Development Economics, Elsevier, vol. 160(C).
    13. Iacovone,Leonardo & Mckenzie,David J. & Meager,Rachael, 2023. "Bayesian Impact Evaluation with Informative Priors : An Application to a ColombianManagement and Export Improvement Program," Policy Research Working Paper Series 10274, The World Bank.
    14. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    15. Mackenzie Alston & Tatyana Deryugina & Olga Shurchkov, 2025. "Leaving Money on the Table," CESifo Working Paper Series 11788, CESifo.
    16. Del Carmen,Giselle & Espinal Hernandez,Edgardo Enrique & De Gouvea Scot De Arruda,Thiago, 2022. "Targeting in Tax Compliance Interventions : Experimental Evidence from Honduras," Policy Research Working Paper Series 9967, The World Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    3. Iacovone,Leonardo & Mckenzie,David J. & Meager,Rachael, 2023. "Bayesian Impact Evaluation with Informative Priors : An Application to a ColombianManagement and Export Improvement Program," Policy Research Working Paper Series 10274, The World Bank.
    4. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    5. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).
    6. Mackenzie Alston & Tatyana Deryugina & Olga Shurchkov, 2025. "Leaving Money on the Table," CESifo Working Paper Series 11788, CESifo.
    7. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    8. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    11. Alston, Mackenzie & Deryugina, Tatyana & Shurchkov, Olga, 2025. "Leaving Money on the Table," IZA Discussion Papers 17812, Institute of Labor Economics (IZA).
    12. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    13. Schaerer, Michael & du Plessis, Christilene & Nguyen, My Hoang Bao & van Aert, Robbie C.M. & Tiokhin, Leo & Lakens, Daniël & Giulia Clemente, Elena & Pfeiffer, Thomas & Dreber, Anna & Johannesson, Mag, 2023. "On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 179(C).
    14. Merl, Robert & Stöckl, Thomas & Palan, Stefan, 2023. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Journal of Banking & Finance, Elsevier, vol. 154(C).
    15. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    16. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    17. Chávez, Carlos A. & Murphy, James J. & Stranlund, John K., 2018. "Managing and defending the commons: Experimental evidence from TURFs in Chile," Journal of Environmental Economics and Management, Elsevier, vol. 91(C), pages 229-246.
    18. Stefano Della & Jörg Heining & Johannes F Schmieder & Simon Trenkle, 2023. "Evidence on Job Search Models from a Survey of Unemployed Workers in Germany," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 137(2), pages 1181-1232.
    19. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    20. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.

    More about this item

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • O10 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:apandp:v:110:y:2020:p:75-79. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.