IDEAS home Printed from https://ideas.repec.org/p/upf/upfgen/1807.html
   My bibliography  Save this paper

Non-standard errors

Author

Listed:
  • Albert J. Menkveld
  • Anna Dreber
  • Felix Holzmeister
  • Juergen Huber
  • Magnus Johannesson
  • Michael Kirchler
  • Sebastian Neussüs
  • Michael Razen
  • Utz Weitzel
  • Christian T. Brownlees
  • Javier Gil-Bazo
  • et al.

Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Suggested Citation

  • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neussüs & Michael Razen & Utz Weitzel & Christian T. Brownlees & Javier Gil-Baz, 2021. "Non-standard errors," Economics Working Papers 1807, Department of Economics and Business, Universitat Pompeu Fabra.
    • Handle: RePEc:upf:upfgen:1807
    as

    Download full text from publisher

    File URL: https://econ-papers.upf.edu/papers/1807.pdf
    File Function: Whole Paper
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Kirchler & Florian Lindner & Utz Weitzel, 2018. "Rankings and Risk‐Taking in the Finance Industry," Journal of Finance, American Finance Association, vol. 73(5), pages 2271-2302, October.
    2. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    3. Tran, Anh & Zeckhauser, Richard, 2012. "Rank as an inherent incentive: Evidence from a field experiment," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 645-650.
    4. Pastor, Lubos & Stambaugh, Robert F., 2003. "Liquidity Risk and Expected Stock Returns," Journal of Political Economy, University of Chicago Press, vol. 111(3), pages 642-685, June.
    5. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    6. Sophocles Mavroeidis & Mikkel Plagborg-Møller & James H. Stock, 2014. "Empirical Evidence on Inflation Expectations in the New Keynesian Phillips Curve," Journal of Economic Literature, American Economic Association, vol. 52(1), pages 124-188, March.
    7. Harvey, A C, 1976. "Estimating Regression Models with Multiplicative Heteroscedasticity," Econometrica, Econometric Society, vol. 44(3), pages 461-465, May.
    8. Geyer-Klingeberg, Jerome & Hang, Markus & Rathgeber, Andreas, 2020. "Meta-analysis in finance research: Opportunities, challenges, and contemporary applications," International Review of Financial Analysis, Elsevier, vol. 71(C).
    9. R. David Mclean & Jeffrey Pontiff, 2016. "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, American Finance Association, vol. 71(1), pages 5-32, February.
    10. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    11. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Papers 2006.04269, arXiv.org.
    12. Albert J. Menkveld, 2016. "The Economics of High-Frequency Trading: Taking Stock," Annual Review of Financial Economics, Annual Reviews, vol. 8(1), pages 1-24, October.
    13. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    14. Itzhak Ben-David & Francesco Franzoni & Byungwook Kim & Rabih Moussawi, 2021. "Competition for Attention in the ETF Space," NBER Working Papers 28369, National Bureau of Economic Research, Inc.
    15. Andrew C. Chang & Phillip Li, 2017. "A Preanalysis Plan to Replicate Sixty Economics Research Papers That Worked Half of the Time," American Economic Review, American Economic Association, vol. 107(5), pages 60-64, May.
    16. Tarun Chordia & Amit Goyal & Alessio Saretto & Andrew KarolyiEditor, 2020. "Anomalies and False Rejections," Review of Finance, European Finance Association, vol. 33(5), pages 2134-2179.
    17. Juhani T Linnainmaa & Michael R Roberts, 2018. "The History of the Cross-Section of Stock Returns," Review of Financial Studies, Society for Financial Studies, vol. 31(7), pages 2606-2649.
    18. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    19. Mas-Colell, Andreu & Whinston, Michael D. & Green, Jerry R., 1995. "Microeconomic Theory," OUP Catalogue, Oxford University Press, number 9780195102680.
    20. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    21. Lo, Andrew W & MacKinlay, A Craig, 1990. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," Review of Financial Studies, Society for Financial Studies, vol. 3(3), pages 431-467.
    22. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    23. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    24. Bao, Jack & O’Hara, Maureen & (Alex) Zhou, Xing, 2018. "The Volcker Rule and corporate bond market making in times of stress," Journal of Financial Economics, Elsevier, vol. 130(1), pages 95-113.
    25. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    26. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    27. Tarun Chordia & Amit Goyal & Alessio Saretto, 2020. "Anomalies and False Rejections," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2134-2179.
    28. James J. Angel & Lawrence E. Harris & Chester S. Spatt, 2015. "Equity Trading in the 21st Century: An Update," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 5(01), pages 1-39.
    29. John Adams & Darren Hayunga & Sattar Mansi & David Reeb & Vincenzo Verardi, 2019. "Identifying and treating outliers in finance," Financial Management, Financial Management Association International, vol. 48(2), pages 345-384, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Isabella Müller & Felix Noth & Lena Tonzer, 2022. "A Note on the Use of Syndicated Loan Data," Tinbergen Institute Discussion Papers 22-064/IV, Tinbergen Institute.
    2. Christoph Huber & Christian König-Kersting, 2022. "Experimenting with Financial Professionals," Working Papers 2022-07, Faculty of Economics and Statistics, Universität Innsbruck.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Todd Mitton, 2022. "Methodological Variation in Empirical Corporate Finance," Review of Financial Studies, Society for Financial Studies, vol. 35(2), pages 527-575.
    2. Chinco, Alex & Neuhierl, Andreas & Weber, Michael, 2021. "Estimating the anomaly base rate," Journal of Financial Economics, Elsevier, vol. 140(1), pages 101-126.
    3. Baltussen, Guido & Swinkels, Laurens & Van Vliet, Pim, 2021. "Global factor premiums," Journal of Financial Economics, Elsevier, vol. 142(3), pages 1128-1154.
    4. Harvey, Campbell R. & Liu, Yan, 2021. "Lucky factors," Journal of Financial Economics, Elsevier, vol. 141(2), pages 413-435.
    5. Cakici, Nusret & Zaremba, Adam & Bianchi, Robert J. & Pham, Nga, 2021. "False discoveries in the anomaly research: New insights from the Stock Exchange of Melbourne (1927–1987)," Pacific-Basin Finance Journal, Elsevier, vol. 70(C).
    6. Kewei Hou & Chen Xue & Lu Zhang, 2017. "Replicating Anomalies," NBER Working Papers 23394, National Bureau of Economic Research, Inc.
    7. Andrew Y. Chen, 2021. "The Limits of p‐Hacking: Some Thought Experiments," Journal of Finance, American Finance Association, vol. 76(5), pages 2447-2480, October.
    8. Smith, Simon C., 2022. "Time-variation, multiple testing, and the factor zoo," International Review of Financial Analysis, Elsevier, vol. 84(C).
    9. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    10. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    11. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    12. Andrew Y. Chen & Tom Zimmermann, 2022. "Open Source Cross-Sectional Asset Pricing," Critical Finance Review, now publishers, vol. 11(2), pages 207-264, May.
    13. Andrew Y. Chen, 2022. "Most claimed statistical findings in cross-sectional return predictability are likely true," Papers 2206.15365, arXiv.org, revised Aug 2022.
    14. Söhnke M. Bartram & Harald Lohre & Peter F. Pope & Ananthalakshmi Ranganathan, 2021. "Navigating the factor zoo around the world: an institutional investor perspective," Journal of Business Economics, Springer, vol. 91(5), pages 655-703, July.
    15. Hoang, Khoa & Cannavan, Damien & Gaunt, Clive & Huang, Ronghong, 2019. "Is that factor just lucky? Australian evidence," Pacific-Basin Finance Journal, Elsevier, vol. 57(C).
    16. Campbell R. Harvey & Yan Liu, 2022. "Luck versus Skill in the Cross Section of Mutual Fund Returns: Reexamining the Evidence," Journal of Finance, American Finance Association, vol. 77(3), pages 1921-1966, June.
    17. Cheng, Tingting & Yan, Cheng & Yan, Yayi, 2021. "Improved inference for fund alphas using high-dimensional cross-sectional tests," Journal of Empirical Finance, Elsevier, vol. 61(C), pages 57-81.
    18. Psaradellis, Ioannis & Laws, Jason & Pantelous, Athanasios A. & Sermpinis, Georgios, 2023. "Technical analysis, spread trading, and data snooping control," International Journal of Forecasting, Elsevier, vol. 39(1), pages 178-191.
    19. Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    20. Huber, Christoph & Kirchler, Michael, 2023. "Experiments in finance: A survey of historical trends," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).

    More about this item

    Keywords

    non-standard errors; multi-analyst approach; liquidity;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • G1 - Financial Economics - - General Financial Markets
    • G14 - Financial Economics - - General Financial Markets - - - Information and Market Efficiency; Event Studies; Insider Trading

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:upf:upfgen:1807. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.econ.upf.edu/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: http://www.econ.upf.edu/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.