IDEAS home Printed from https://ideas.repec.org/p/hal/journl/halshs-03500882.html
   My bibliography  Save this paper

Non-Standard Errors

Author

Listed:
  • Albert J Menkveld

    (VU - Vrije Universiteit Amsterdam [Amsterdam])

  • Anna Dreber

    (Stockholm School of Economics - Department of Economics)

  • Felix Holzmeister

    (Universität Innsbruck [Innsbruck])

  • Juergen Huber

    (Universität Innsbruck [Innsbruck])

  • Magnus Johannesson

    (Stockholm School of Economics - Department of Economics)

  • Michael Kirchler

    (Universität Innsbruck [Innsbruck])

  • Sebastian Neusüss

    (Aalto University)

  • Michael Razen

    (Universität Innsbruck [Innsbruck])

  • Utz Weitzel

    (VU - Vrije Universiteit Amsterdam [Amsterdam])

  • Gunther Capelle-Blancard

    (UP1 - Université Paris 1 Panthéon-Sorbonne, CES - Centre d'économie de la Sorbonne - UP1 - Université Paris 1 Panthéon-Sorbonne - CNRS - Centre National de la Recherche Scientifique)

  • David Abad-Díaz
  • Menachem Abudy
  • Tobias Adrian
  • Yacine Ait-Sahalia
  • Olivier Akmansoy
  • Jamie T Alcock
  • Vitali Alexeev
  • Arash Aloosh
  • Livia Amato
  • Diego Amaya
  • James J Angel
  • Alejandro T Avetikian
  • Amadeus Bach
  • Edwin Baidoo
  • Gaetan Bakalli
  • Bao Li
  • Andrea Barbon
  • Oksana Bashchenko
  • Parampreet C Bindra
  • Geir H Bjønnes
  • Jeffrey R Black
  • Bernard S Black
  • Dimitar Bogoev
  • Santiago Bohorquez Correa
  • Oleg Bondarenko
  • Charles S Bos
  • Ciril Bosch-Rosa
  • Elie Bouri
  • Christian Brownlees
  • Anna Calamia
  • Nga Cao
  • Laura M Capera Romero
  • Massimiliano Caporin
  • Allen Carrion
  • Tolga Caskurlu
  • Bidisha Chakrabarty
  • Jian Chen
  • Mikhail Chernov
  • William Cheung
  • Ludwig B Chincarini
  • Tarun Chordia
  • Sheung Chi Chow
  • Benjamin Clapham
  • Jean-Edouard Colliard
  • Carole Comerton-Forde
  • Edward Curran
  • Thong Dao
  • Wale Dare
  • Ryan J Davies
  • Riccardo de Blasis
  • Gianluca F de Nard
  • Fany Declerck
  • Oleg Deev
  • Hans Degryse
  • Solomon y Deku
  • Christophe Desagre
  • Mathijs A van Dijk
  • Chukwuma Dim
  • Thomas Dimpfl
  • Yun Jiang Dong
  • Philip A Drummond
  • Tom Dudda
  • Teodor Duevski
  • Ariadna Dumitrescu
  • Teodor Dyakov
  • Anne Haubo Dyhrberg
  • Michał Dzieliński
  • Asli Eksi
  • Izidin El Kalak
  • Saskia ter Ellen
  • Nicolas Eugster
  • Martin D D Evans
  • Michael Farrell
  • Ester Felez-Vinas
  • Gerardo Ferrara
  • El Mehdi Ferrouhi
  • Andrea Flori
  • Jonathan T Fluharty
  • Sean D V Foley
  • Kingsley y L Fong
  • Thierry Foucault
  • Tatiana Franus
  • Francesco Franzoni
  • Bart Frijns
  • Michael Frömmel
  • Servanna M Fu
  • Sascha C Füllbrunn
  • Baoqing Gan
  • Ge Gao
  • Thomas P Gehrig
  • Roland Gemayel
  • Dirk Gerritsen
  • Javier Gil-Bazo
  • Dudley Gilder
  • Lawrence R Glosten

Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Suggested Citation

  • Albert J Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & Gunther Capelle-Blancard & David Abad-Dí, 2021. "Non-Standard Errors," Post-Print halshs-03500882, HAL.
    • Handle: RePEc:hal:journl:halshs-03500882
    Note: View the original document on HAL open archive server: https://shs.hal.science/halshs-03500882
    as

    Download full text from publisher

    File URL: https://shs.hal.science/halshs-03500882/document
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Kirchler & Florian Lindner & Utz Weitzel, 2018. "Rankings and Risk‐Taking in the Finance Industry," Journal of Finance, American Finance Association, vol. 73(5), pages 2271-2302, October.
    2. Tran, Anh & Zeckhauser, Richard, 2012. "Rank as an inherent incentive: Evidence from a field experiment," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 645-650.
    3. Pastor, Lubos & Stambaugh, Robert F., 2003. "Liquidity Risk and Expected Stock Returns," Journal of Political Economy, University of Chicago Press, vol. 111(3), pages 642-685, June.
    4. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    5. Harvey, A C, 1976. "Estimating Regression Models with Multiplicative Heteroscedasticity," Econometrica, Econometric Society, vol. 44(3), pages 461-465, May.
    6. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    7. Albert J. Menkveld, 2016. "The Economics of High-Frequency Trading: Taking Stock," Annual Review of Financial Economics, Annual Reviews, vol. 8(1), pages 1-24, October.
    8. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    9. Andrew C. Chang & Phillip Li, 2017. "A Preanalysis Plan to Replicate Sixty Economics Research Papers That Worked Half of the Time," American Economic Review, American Economic Association, vol. 107(5), pages 60-64, May.
    10. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    11. Lo, Andrew W & MacKinlay, A Craig, 1990. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," Review of Financial Studies, Society for Financial Studies, vol. 3(3), pages 431-467.
    12. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    13. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    14. Bao, Jack & O’Hara, Maureen & (Alex) Zhou, Xing, 2018. "The Volcker Rule and corporate bond market making in times of stress," Journal of Financial Economics, Elsevier, vol. 130(1), pages 95-113.
    15. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    16. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    17. James J. Angel & Lawrence E. Harris & Chester S. Spatt, 2015. "Equity Trading in the 21st Century: An Update," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 5(01), pages 1-39.
    18. John Adams & Darren Hayunga & Sattar Mansi & David Reeb & Vincenzo Verardi, 2019. "Identifying and treating outliers in finance," Financial Management, Financial Management Association International, vol. 48(2), pages 345-384, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christoph Huber & Christian König-Kersting, 2022. "Experimenting with Financial Professionals," Working Papers 2022-07, Faculty of Economics and Statistics, Universität Innsbruck.
    2. Isabella Müller & Felix Noth & Lena Tonzer, 2022. "A Note on the Use of Syndicated Loan Data," Tinbergen Institute Discussion Papers 22-064/IV, Tinbergen Institute.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Huber, Christoph & Kirchler, Michael, 2023. "Experiments in finance: A survey of historical trends," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    4. Brinkerink, Jasper & De Massis, Alfredo & Kellermanns, Franz, 2022. "One finding is no finding: Toward a replication culture in family business research," Journal of Family Business Strategy, Elsevier, vol. 13(4).
    5. Hou, Kewei & Xue, Chen & Zhang, Lu, 2017. "Replicating Anomalies," Working Paper Series 2017-10, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    6. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    7. Bull, Charles & Courty, Pascal & Doyon, Maurice & Rondeau, Daniel, 2019. "Failure of the Becker–DeGroot–Marschak mechanism in inexperienced subjects: New tests of the game form misconception hypothesis," Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 235-253.
    8. Grabiszewski, Konrad & Horenstein, Alex, 2020. "Effort is not a monotonic function of skills: Results from a global mobile experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 634-652.
    9. Brade, Raphael & Himmler, Oliver & Jäckle, Robert, 2018. "Normatively Framed Relative Performance Feedback – Field Experiment and Replication," MPRA Paper 88830, University Library of Munich, Germany.
    10. So, Tony, 2020. "Classroom experiments as a replication device," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 86(C).
    11. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    12. Hoang, Khoa & Cannavan, Damien & Gaunt, Clive & Huang, Ronghong, 2019. "Is that factor just lucky? Australian evidence," Pacific-Basin Finance Journal, Elsevier, vol. 57(C).
    13. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    14. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    15. Robert Merl & Thomas Stöckl & Stefan Palan, 2021. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Working Paper Series, Social and Economic Sciences 2021-03, Faculty of Social and Economic Sciences, Karl-Franzens-University Graz.
    16. Oliver Braganza, 2020. "A simple model suggesting economically rational sample-size choice drives irreproducibility," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-19, March.
    17. Adam Altmejd & Anna Dreber & Eskil Forsell & Juergen Huber & Taisuke Imai & Magnus Johannesson & Michael Kirchler & Gideon Nave & Colin Camerer, 2019. "Predicting the replicability of social science lab experiments," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-18, December.
    18. Williams, Cole Randall, 2019. "How redefining statistical significance can worsen the replication crisis," Economics Letters, Elsevier, vol. 181(C), pages 65-69.
    19. Harvey, Campbell R. & Liu, Yan, 2021. "Lucky factors," Journal of Financial Economics, Elsevier, vol. 141(2), pages 413-435.
    20. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.

    More about this item

    Keywords

    non-standard errors; multi-analyst approach; liquidity;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • G1 - Financial Economics - - General Financial Markets
    • G14 - Financial Economics - - General Financial Markets - - - Information and Market Efficiency; Event Studies; Insider Trading

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:journl:halshs-03500882. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://hal.archives-ouvertes.fr/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.