IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/cahyq.html
   My bibliography  Save this paper

Bayesian Specification Curve Analysis

Author

Listed:
  • Semken, Christoph
  • Rossell, David

Abstract

Science suffers from a reproducibility crisis. Specification Curve Analysis (SCA) helps address this crisis by preventing the selective reporting of results and arbitrary data analysis choices. SCA plots the variability (or heterogeneity) of treatment effects against all ‘reasonable specifications’ (ways to conduct analysis). However, SCA has also been used for formal statistical inference on a type of global average (median) treatment effect (ATE), leading a study by Orben & Przybylski to conclude that ‘the association of [adolescent mental] well-being with regularly eating potatoes was nearly as negative as the association with technology use.’ In contrast, we find relevant associations between certain technologies and well-being, and sharp discrepancies between parent and teenager assessments. These heterogeneous effects are masked by taking medians. In layman’s terms, an ATE may appear practically irrelevant due to averaging over apples and oranges. In addition, the SCA median can have large bias and variance, due to over-weighting statistically implausible control variable specifications. With the Bayesian Specification Curve Analysis (BSCA) we extend SCA to estimate both individual and, if desired, average treatment effects, with controls weighted via Bayesian Model Averaging. The strategy allows to test individual effects, a missing feature in SCA, while improving statistical properties and protecting against false positives. We provide R code that implements BSCA and reproduces our analyses.

Suggested Citation

  • Semken, Christoph & Rossell, David, 2020. "Bayesian Specification Curve Analysis," OSF Preprints cahyq, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:cahyq
    DOI: 10.31219/osf.io/cahyq
    as

    Download full text from publisher

    File URL: https://osf.io/download/5fa57ec323568c0210d653f8/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/cahyq?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Michael Muthukrishna & Joseph Henrich, 2019. "A problem in theory," Nature Human Behaviour, Nature, vol. 3(3), pages 221-229, March.
    3. Uri Simonsohn & Joseph P. Simmons & Leif D. Nelson, 2020. "Specification curve analysis," Nature Human Behaviour, Nature, vol. 4(11), pages 1208-1214, November.
    4. Jean M. Twenge & Jonathan Haidt & Thomas E. Joiner & W. Keith Campbell, 2020. "Underestimating digital media harm," Nature Human Behaviour, Nature, vol. 4(4), pages 346-348, April.
    5. Hunt Allcott & Luca Braghieri & Sarah Eichmeyer & Matthew Gentzkow, 2020. "The Welfare Effects of Social Media," American Economic Review, American Economic Association, vol. 110(3), pages 629-676, March.
    6. Amy Orben & Andrew K. Przybylski, 2019. "The association between adolescent well-being and digital technology use," Nature Human Behaviour, Nature, vol. 3(2), pages 173-182, February.
    7. Adam Slez, 2019. "The Difference Between Instability and Uncertainty: Comment on Young and Holsteen (2017)," Sociological Methods & Research, , vol. 48(2), pages 400-430, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoph Semken & David Rossell, 2022. "Specification analysis for technology use and teenager well‐being: Statistical validity and a Bayesian proposal," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1330-1355, November.
    2. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    3. Linda K. Kaye & Amy Orben & David A. Ellis & Simon C. Hunter & Stephen Houghton, 2020. "The Conceptual and Methodological Mayhem of “Screen Time”," IJERPH, MDPI, vol. 17(10), pages 1-10, May.
    4. Hoong, Ruru, 2021. "Self control and smartphone use: An experimental study of soft commitment devices," European Economic Review, Elsevier, vol. 140(C).
    5. Jürges, Hendrik & Khanam, Rasheda, 2021. "Adolescents’ time allocation and skill production," Economics of Education Review, Elsevier, vol. 85(C).
    6. Ebersberger, Bernd & Kuckertz, Andreas, 2021. "Hop to it! The impact of organization type on innovation response time to the COVID-19 crisis," Journal of Business Research, Elsevier, vol. 124(C), pages 126-135.
    7. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    8. Omar Hegazi & Samer Alalalmeh & Ahmad Alfaresi & Soheil Dashtinezhad & Ahmed Bahada & Moyad Shahwan & Ammar Abdulrahman Jairoun & Tesleem K. Babalola & Haya Yasin, 2022. "Development, Validation, and Utilization of a Social Media Use and Mental Health Questionnaire among Middle Eastern and Western Adults: A Pilot Study from the UAE," IJERPH, MDPI, vol. 19(23), pages 1-18, November.
    9. Nicolás Ajzenman & Bruno Ferman & Pedro C. Sant’Anna, 2023. "Rooting for the Same Team: On the Interplay between Political and Social Identities in the Formation of Social Ties," Working Papers 231, Red Nacional de Investigadores en Economía (RedNIE).
    10. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    11. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    12. Larsen, Vegard H. & Thorsrud, Leif Anders & Zhulanova, Julia, 2021. "News-driven inflation expectations and information rigidities," Journal of Monetary Economics, Elsevier, vol. 117(C), pages 507-520.
    13. Eibich, Peter & Goldzahl, Léontine, 2021. "Does retirement affect secondary preventive care use? Evidence from breast cancer screening," Economics & Human Biology, Elsevier, vol. 43(C).
    14. Frederique Bordignon, 2020. "Self-correction of science: a comparative study of negative citations and post-publication peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1225-1239, August.
    15. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
    16. Aurelie Seguin & Wolfgang Forstmeier, 2012. "No Band Color Effects on Male Courtship Rate or Body Mass in the Zebra Finch: Four Experiments and a Meta-Analysis," PLOS ONE, Public Library of Science, vol. 7(6), pages 1-11, June.
    17. Dragana Radicic & Geoffrey Pugh & Hugo Hollanders & René Wintjes & Jon Fairburn, 2016. "The impact of innovation support programs on small and medium enterprises innovation in traditional manufacturing industries: An evaluation for seven European Union regions," Environment and Planning C, , vol. 34(8), pages 1425-1452, December.
    18. Christoph Kronenberg, 2021. "New(spaper) evidence of a reduction in suicide mentions during the 19th century US gold rush," Health Economics, John Wiley & Sons, Ltd., vol. 30(10), pages 2582-2594, September.
    19. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    20. Li, Lunzheng & Maniadis, Zacharias & Sedikides, Constantine, 2021. "Anchoring in Economics: A Meta-Analysis of Studies on Willingness-To-Pay and Willingness-To-Accept," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 90(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:cahyq. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.