IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26716.html
   My bibliography  Save this paper

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Author

Listed:
  • Stefano DellaVigna
  • Nicholas Otis
  • Eva Vivalt

Abstract

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values; (2) whether predictions are in raw units or standard deviations; (3) text-entry versus slider responses; and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered.

Suggested Citation

  • Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," NBER Working Papers 26716, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:26716
    Note: DEV LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26716.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    2. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    3. Olckers, Matthew, 2021. "On track for retirement?," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 76-88.
    4. Del Carmen,Giselle & Espinal Hernandez,Edgardo Enrique & De Gouvea Scot De Arruda,Thiago, 2022. "Targeting in Tax Compliance Interventions : Experimental Evidence from Honduras," Policy Research Working Paper Series 9967, The World Bank.
    5. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    6. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    7. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    8. Yang, Dean & Allen, James & Mahumane, Arlete & Riddell, James & Yu, Hang, 2023. "Knowledge, stigma, and HIV testing: An analysis of a widespread HIV/AIDS program," Journal of Development Economics, Elsevier, vol. 160(C).
    9. Henning Schaak & Jens Rommel & Julian Sagebiel & Jesus Barreiro-Hurlé & Douadia Bougherara & Luigi Cemablo & Marija Cerjak & Tajana Čop & Mikołaj Czajkowski & María Espinosa-Goded & Julia Höhler & Car, 2022. "How Well Can Experts Predict Farmers' Risk Preferences ?," Post-Print hal-03738351, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    3. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    6. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).
    7. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.
    8. Schaerer, Michael & du Plessis, Christilene & Nguyen, My Hoang Bao & van Aert, Robbie C.M. & Tiokhin, Leo & Lakens, Daniël & Giulia Clemente, Elena & Pfeiffer, Thomas & Dreber, Anna & Johannesson, Mag, 2023. "On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 179(C).
    9. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    10. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    11. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    12. Jäger, Simon & Roth, Christopher & Roussille, Nina & Schoefer, Benjamin, 2021. "Worker Beliefs about Outside Options," IZA Discussion Papers 14963, Institute of Labor Economics (IZA).
    13. Katherine Casey & Rachel Glennerster & Edward Miguel & Maarten Voors, 2023. "Skill Versus Voice in Local Development," The Review of Economics and Statistics, MIT Press, vol. 105(2), pages 311-326, March.
    14. Tom Coupé & W. Robert Reed, 2021. "Do Negative Replications Affect Citations?," Working Papers in Economics 21/14, University of Canterbury, Department of Economics and Finance.
    15. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    16. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    17. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    18. Da Ke, 2021. "Who Wears the Pants? Gender Identity Norms and Intrahousehold Financial Decision‐Making," Journal of Finance, American Finance Association, vol. 76(3), pages 1389-1425, June.
    19. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    20. Galanis, S. & Ioannou, C. & Kotronis, S., 2019. "Information Aggregation Under Ambiguity: Theory and Experimental Evidence," Working Papers 20/05, Department of Economics, City University London.

    More about this item

    JEL classification:

    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development
    • O17 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Formal and Informal Sectors; Shadow Economy; Institutional Arrangements

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26716. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.