IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26716.html
   My bibliography  Save this paper

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Author

Listed:
  • Stefano DellaVigna
  • Nicholas Otis
  • Eva Vivalt

Abstract

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values; (2) whether predictions are in raw units or standard deviations; (3) text-entry versus slider responses; and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered.

Suggested Citation

  • Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," NBER Working Papers 26716, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:26716
    Note: DEV LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26716.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    2. Alston, Mackenzie & Deryugina, Tatyana & Shurchkov, Olga, 2025. "Leaving Money on the Table," IZA Discussion Papers 17812, Institute of Labor Economics (IZA).
    3. Tamim,Abdulrazzak & Smith,Emma & Palmer,I. Bailey & Miguel,Edward & Leone,Samuel & Sandra Rozo & Stillman,Sarah, 2025. "Housing Subsidies for Refugees : Experimental Evidence on Life Outcomes and Social Integration in Jordan," Policy Research Working Paper Series 11042, The World Bank.
    4. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    5. Olckers, Matthew, 2021. "On track for retirement?," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 76-88.
    6. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    7. Del Carmen,Giselle & Espinal Hernandez,Edgardo Enrique & De Gouvea Scot De Arruda,Thiago, 2022. "Targeting in Tax Compliance Interventions : Experimental Evidence from Honduras," Policy Research Working Paper Series 9967, The World Bank.
    8. Tamim, Abdulrazzak & Smith, Emma & Palmer, I. Bailey & Miguel, Edward & Oparina, Ekaterina & Rozo, Sandra V. & Stillman, Sarah, 2025. "Housing Subsidies for Refugees: Experimental Evidence on Life Outcomes and Social Integration in Jordan," IZA Discussion Papers 17622, Institute of Labor Economics (IZA).
    9. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    10. Iacovone,Leonardo & Mckenzie,David J. & Meager,Rachael, 2023. "Bayesian Impact Evaluation with Informative Priors : An Application to a ColombianManagement and Export Improvement Program," Policy Research Working Paper Series 10274, The World Bank.
    11. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    12. Mackenzie Alston & Tatyana Deryugina & Olga Shurchkov, 2025. "Leaving Money on the Table," CESifo Working Paper Series 11788, CESifo.
    13. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    14. Matias Giaccobasso & Brad C. Nathan & Ricardo Perez-Truglia & Alejandro Zentner, 2022. "Where Do My Tax Dollars Go? Tax Morale Effects of Perceived Government Spending," NBER Working Papers 29789, National Bureau of Economic Research, Inc.
    15. Yang, Dean & Allen, James & Mahumane, Arlete & Riddell, James & Yu, Hang, 2023. "Knowledge, stigma, and HIV testing: An analysis of a widespread HIV/AIDS program," Journal of Development Economics, Elsevier, vol. 160(C).
    16. Henning Schaak & Jens Rommel & Julian Sagebiel & Jesus Barreiro-Hurlé & Douadia Bougherara & Luigi Cemablo & Marija Cerjak & Tajana Čop & Mikołaj Czajkowski & María Espinosa-Goded & Julia Höhler & Car, 2022. "How Well Can Experts Predict Farmers' Risk Preferences ?," Post-Print hal-03738351, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    3. Mackenzie Alston & Tatyana Deryugina & Olga Shurchkov, 2025. "Leaving Money on the Table," CESifo Working Paper Series 11788, CESifo.
    4. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    5. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    6. Iacovone,Leonardo & Mckenzie,David J. & Meager,Rachael, 2023. "Bayesian Impact Evaluation with Informative Priors : An Application to a ColombianManagement and Export Improvement Program," Policy Research Working Paper Series 10274, The World Bank.
    7. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    8. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).
    9. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.
    10. Alston, Mackenzie & Deryugina, Tatyana & Shurchkov, Olga, 2025. "Leaving Money on the Table," IZA Discussion Papers 17812, Institute of Labor Economics (IZA).
    11. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    12. Schaerer, Michael & du Plessis, Christilene & Nguyen, My Hoang Bao & van Aert, Robbie C.M. & Tiokhin, Leo & Lakens, Daniël & Giulia Clemente, Elena & Pfeiffer, Thomas & Dreber, Anna & Johannesson, Mag, 2023. "On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 179(C).
    13. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    14. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    15. Alexandru Marcoci & David P. Wilkinson & Ans Vercammen & Bonnie C. Wintle & Anna Lou Abatayo & Ernest Baskin & Henk Berkman & Erin M. Buchanan & Sara Capitán & Tabaré Capitán & Ginny Chan & Kent Jason, 2025. "Predicting the replicability of social and behavioural science claims in COVID-19 preprints," Nature Human Behaviour, Nature, vol. 9(2), pages 287-304, February.
    16. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    17. Simon Jäger & Christopher Roth & Nina Roussille & Benjamin Schoefer, 2024. "Worker Beliefs About Outside Options," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 139(3), pages 1505-1556.
    18. Katherine Casey & Rachel Glennerster & Edward Miguel & Maarten Voors, 2023. "Skill Versus Voice in Local Development," The Review of Economics and Statistics, MIT Press, vol. 105(2), pages 311-326, March.
    19. Tom Coupé & W. Robert Reed, 2021. "Do Negative Replications Affect Citations?," Working Papers in Economics 21/14, University of Canterbury, Department of Economics and Finance.
    20. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).

    More about this item

    JEL classification:

    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development
    • O17 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Formal and Informal Sectors; Shadow Economy; Institutional Arrangements

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26716. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.