IDEAS home Printed from https://ideas.repec.org/p/inn/wpaper/2021-31.html
   My bibliography  Save this paper

Non-Standard Errors

Author

Listed:
  • Albert J. Menkveld
  • Anna Dreber
  • Felix Holzmeister
  • Jürgen Huber
  • Magnus Johannesson
  • Michael Kirchler
  • Sebastian Neusüss
  • Michael Razen
  • Utz Weitzel
  • David Abad-Dí­az
  • Menachem Abudy
  • Tobias Adrian
  • Yacine Ait-Sahalia
  • Olivier Akmansoy
  • Jamie T. Alcock
  • Vitali Alexeev
  • Arash Aloosh
  • Livia Amato
  • Diego Amaya
  • James J. Angel
  • Alejandro T. Avetikian
  • Amadeus Bach
  • Edwin Baidoo
  • Gaetan Bakalli
  • Li Bao
  • Andrea Barbon
  • Oksana Bashchenko
  • Parampreet C. Bindra
  • Geir H. Bjønnes
  • Jeffrey R. Black
  • Bernard S. Black
  • Dimitar Bogoev
  • Santiago Bohorquez Correa
  • Oleg Bondarenko
  • Charles S. Bos
  • Ciril Bosch-Rosa
  • Elie Bouri
  • Christian Brownlees
  • Anna Calamia
  • Viet Nga Cao
  • Gunther Capelle-Blancard
  • Laura M. Capera Romero
  • Massimiliano Caporin
  • Allen Carrion
  • Tolga Caskurlu
  • Bidisha Chakrabarty
  • Jian Chen
  • Mikhail Chernov
  • William Cheung
  • Ludwig B. Chincarini
  • Tarun Chordia
  • Sheung Chi Chow
  • Benjamin Clapham
  • Jean-Edouard Colliard
  • Carole Comerton-Forde
  • Edward Curran
  • Thong Dao
  • Wale Dare
  • Ryan J. Davies
  • Riccardo De Blasis
  • Gianluca F. De Nard
  • Fany Declerck
  • Oleg Deev
  • Hans Degryse
  • Solomon Y. Deku
  • Christophe Desagre
  • Mathijs A. van Dijk
  • Chukwuma Dim
  • Thomas Dimpfl
  • Yun Jiang Dong
  • Philip A. Drummond
  • Tom Dudda
  • Teodor Duevski
  • Ariadna Dumitrescu
  • Teodor Dyakov
  • Anne Haubo Dyhrberg
  • Michal Dzielinski
  • Asli Eksi
  • Izidin El Kalak
  • Saskia ter Ellen
  • Nicolas Eugster
  • Martin D. D. Evans
  • Michael Farrell
  • Ester Felez-Vinas
  • Gerardo Ferrara
  • El Mehdi Ferrouhi
  • Andrea Flori
  • Jonathan T. Fluharty
  • Sean D. V. Foley
  • Kingsley Y. L. Fong
  • Thierry Foucault
  • Tatiana Franus
  • Francesco Franzoni
  • Bart Frijns
  • Michael Frömmel
  • Servanna M. Fu
  • Sascha C. Füllbrunn
  • Baoqing Gan
  • Ge Gao
  • Thomas P. Gehrig
  • Roland Gemayel
  • Dirk Gerritsen
  • Javier Gil-Bazo
  • Dudley Gilder
  • Lawrence R. Glosten
  • Thomas Gomez
  • Arseny Gorbenko
  • Joachim Grammig
  • Vincent Grégoire
  • Ufuk Güçbilmez
  • Björn Hagströmer
  • Julien Hambuckers
  • Erik Hapnes
  • Jeffrey H. Harris
  • Lawrence Harris
  • Simon Hartmann
  • Jean-Baptiste Hasse
  • Nikolaus Hautsch
  • Xue-Zhong (Tony) He
  • Davidson Heath
  • Simon Hediger
  • Terrence Hendershott
  • Ann Marie Hibbert
  • Erik Hjalmarsson
  • Seth Hoelscher
  • Peter Hoffmann
  • Craig W. Holden
  • Alex R. Horenstein
  • Wenqian Huang
  • Da Huang
  • Christophe Hurlin
  • Konrad Ilczuk
  • Alexey Ivashchenko
  • Subramanian R. Iyer
  • Hossein Jahanshahloo
  • Naji P. Jalkh
  • Charles M. Jones
  • Simon Jurkatis
  • Petri Jylhä
  • Andreas T. Kaeck
  • Gabriel Kaiser
  • Arzé Karam
  • Egle Karmaziene
  • Bernhard Kassner
  • Markku Kaustia
  • Ekaterina Kazak
  • Fearghal Kearney
  • Vincent van Kervel
  • Saad A. Khan
  • Marta K. Khomyn
  • Tony Klein
  • Olga Klein
  • Alexander Klos
  • Michael Koetter
  • Aleksey Kolokolov
  • Robert A. Korajczyk
  • Roman Kozhan
  • Jan P. Krahnen
  • Paul Kuhle
  • Amy Kwan
  • Quentin Lajaunie
  • F. Y. Eric C. Lam
  • Marie Lambert
  • Hugues Langlois
  • Jens Lausen
  • Tobias Lauter
  • Markus Leippold
  • Vladimir Levin
  • Yijie Li
  • Hui Li
  • Chee Yoong Liew
  • Thomas Lindner
  • Oliver Linton
  • Jiacheng Liu
  • Anqi Liu
  • Guillermo Llorente
  • Matthijs Lof
  • Ariel Lohr
  • Francis Longstaff
  • Alejandro Lopez-Lira
  • Shawn Mankad
  • Nicola Mano
  • Alexis Marchal
  • Charles Martineau
  • Francesco Mazzola
  • Debrah Meloso
  • Michael G. Mi
  • Roxana Mihet
  • Vijay Mohan
  • Sophie Moinas
  • David Moore
  • Liangyi Mu
  • Dmitriy Muravyev
  • Dermot Murphy
  • Gabor Neszveda
  • Christian Neumeier
  • Ulf Nielsson
  • Mahendrarajah Nimalendran
  • Sven Nolte
  • Lars L. Norden
  • Peter W. O'Neill
  • Khaled Obaid
  • Bernt A. Ødegaard
  • Per Östberg
  • Emiliano Pagnotta
  • Marcus Painter
  • Stefan Palan
  • Imon J. Palit
  • Andreas Park
  • Roberto Pascual
  • Paolo Pasquariello
  • Lubos Pastor
  • Vinay Patel
  • Andrew J. Patton
  • Neil D. Pearson
  • Loriana Pelizzon
  • Michele Pelli
  • Matthias Pelster
  • Christophe Pérignon
  • Cameron Pfiffer
  • Richard Philip
  • Tomás Plí­hal
  • Puneet Prakash
  • Oliver-Alexander Press
  • Tina Prodromou
  • Marcel Prokopczuk
  • Talis Putnins
  • Ya Qian
  • Gaurav Raizada
  • David Rakowski
  • Angelo Ranaldo
  • Luca Regis
  • Stefan Reitz
  • Thomas Renault
  • Rex W. Renjie
  • Roberto Reno
  • Steven J. Riddiough
  • Kalle Rinne
  • Paul J. Rintamäki
  • Ryan Riordan
  • Thomas Rittmannsberger
  • Iñaki Rodrí­guez Longarela
  • Dominik Roesch
  • Lavinia Rognone
  • Brian Roseman
  • Ioanid Rosu
  • Saurabh Roy
  • Nicolas Rudolf
  • Stephen R. Rush
  • Khaladdin Rzayev
  • Aleksandra A. Rzeznik
  • Anthony Sanford
  • Harikumar Sankaran
  • Asani Sarkar
  • Lucio Sarno
  • Olivier Scaillet
  • Stefan Scharnowski
  • Klaus R. Schenk-Hoppé
  • Andrea Schertler
  • Michael Schneider
  • Florian Schroeder
  • Norman Schürhoff
  • Philipp Schuster
  • Marco A. Schwarz
  • Mark S. Seasholes
  • Norman J. Seeger
  • Or Shachar
  • Andriy Shkilko
  • Jessica Shui
  • Mario Sikic
  • Giorgia Simion
  • Lee A. Smales
  • Paul Söderlind
  • Elvira Sojli
  • Konstantin Sokolov
  • Jantje Sönksen
  • Laima Spokeviciute
  • Denitsa Stefanova
  • Marti G. Subrahmanyam
  • Barnabas Szaszi
  • Oleksandr Talavera
  • Yuehua Tang
  • Nick Taylor
  • Wing Wah Tham
  • Erik Theissen
  • Julian Thimme
  • Ian Tonks
  • Hai Tran
  • Luca Trapin
  • Anders B. Trolle
  • M. Andreea Vaduva
  • Giorgio Valente
  • Robert A. Van Ness
  • Aurelio Vasquez
  • Thanos Verousis
  • Patrick Verwijmeren
  • Anders Vilhelmsson
  • Grigory Vilkov
  • Vladimir Vladimirov
  • Sebastian Vogel
  • Stefan Voigt
  • Wolf Wagner
  • Thomas Walther
  • Patrick Weiss
  • Michel van der Wel
  • Ingrid M. Werner
  • Joakim Westerholm
  • Christian Westheide
  • Hans C. Wika
  • Evert Wipplinger
  • Michael Wolf
  • Christian C. P. Wolff
  • Leonard Wolk
  • Wing-Keung Wong
  • Jan Wrampelmeyer
  • Zhen-Xing Wu
  • Shuo Xia
  • Dacheng Xiu
  • Ke Xu
  • Caihong Xu
  • Pradeep K. Yadav
  • José Yagüe
  • Cheng Yan
  • Antti Yang
  • Woongsun Yoo
  • Wenjia Yu
  • Yihe Yu
  • Shihao Yu
  • Bart Z. Yueshen
  • Darya Yuferova
  • Marcin Zamojski
  • Abalfazl Zareei
  • Stefan M. Zeisberger
  • Lu Zhang
  • S. Sarah Zhang
  • Xiaoyu Zhang
  • Lu Zhao
  • Zhuo Zhong
  • Zeyang (Ivy) Zhou
  • Chen Zhou
  • Xingyu S. Zhu
  • Marius Zoican
  • Remco Zwinkels
Registered:

Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Suggested Citation

  • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & David Abad-Dí­az & Menachem Abudy & Tobi, 2021. "Non-Standard Errors," Working Papers 2021-31, Faculty of Economics and Statistics, Universität Innsbruck.
  • Handle: RePEc:inn:wpaper:2021-31
    as

    Download full text from publisher

    File URL: https://www2.uibk.ac.at/downloads/c9821000/wpaper/2021-31.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Harvey, A C, 1976. "Estimating Regression Models with Multiplicative Heteroscedasticity," Econometrica, Econometric Society, vol. 44(3), pages 461-465, May.
    2. Andrew C. Chang & Phillip Li, 2017. "A Preanalysis Plan to Replicate Sixty Economics Research Papers That Worked Half of the Time," American Economic Review, American Economic Association, vol. 107(5), pages 60-64, May.
    3. Michael Kirchler & Florian Lindner & Utz Weitzel, 2018. "Rankings and Risk‐Taking in the Finance Industry," Journal of Finance, American Finance Association, vol. 73(5), pages 2271-2302, October.
    4. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    5. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    6. Lo, Andrew W & MacKinlay, A Craig, 1990. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," Review of Financial Studies, Society for Financial Studies, vol. 3(3), pages 431-467.
    7. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    8. Albert J. Menkveld, 2016. "The Economics of High-Frequency Trading: Taking Stock," Annual Review of Financial Economics, Annual Reviews, vol. 8(1), pages 1-24, October.
    9. Tran, Anh & Zeckhauser, Richard, 2012. "Rank as an inherent incentive: Evidence from a field experiment," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 645-650.
    10. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    11. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    12. Pastor, Lubos & Stambaugh, Robert F., 2003. "Liquidity Risk and Expected Stock Returns," Journal of Political Economy, University of Chicago Press, vol. 111(3), pages 642-685, June.
    13. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    14. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    15. James J. Angel & Lawrence E. Harris & Chester S. Spatt, 2015. "Equity Trading in the 21st Century: An Update," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 5(01), pages 1-39.
    16. John Adams & Darren Hayunga & Sattar Mansi & David Reeb & Vincenzo Verardi, 2019. "Identifying and treating outliers in finance," Financial Management, Financial Management Association International, vol. 48(2), pages 345-384, June.
    17. Bao, Jack & O’Hara, Maureen & (Alex) Zhou, Xing, 2018. "The Volcker Rule and corporate bond market making in times of stress," Journal of Financial Economics, Elsevier, vol. 130(1), pages 95-113.
    18. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christoph Huber & Christian König-Kersting, 2022. "Experimenting with Financial Professionals," Working Papers 2022-07, Faculty of Economics and Statistics, Universität Innsbruck.
    2. Isabella Müller & Felix Noth & Lena Tonzer, 2022. "A Note on the Use of Syndicated Loan Data," Tinbergen Institute Discussion Papers 22-064/IV, Tinbergen Institute.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Huber, Christoph & Kirchler, Michael, 2023. "Experiments in finance: A survey of historical trends," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    4. Christoph Huber & Michael Kirchler, 2022. "Experiments in Finance – A Survey of Historical Trends," Working Papers 2022-09, Faculty of Economics and Statistics, Universität Innsbruck.
    5. Kewei Hou & Chen Xue & Lu Zhang, 2017. "Replicating Anomalies," NBER Working Papers 23394, National Bureau of Economic Research, Inc.
    6. Robert Merl & Thomas Stöckl & Stefan Palan, 2021. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Working Paper Series, Social and Economic Sciences 2021-03, Faculty of Social and Economic Sciences, Karl-Franzens-University Graz.
    7. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    8. Bull, Charles & Courty, Pascal & Doyon, Maurice & Rondeau, Daniel, 2019. "Failure of the Becker–DeGroot–Marschak mechanism in inexperienced subjects: New tests of the game form misconception hypothesis," Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 235-253.
    9. Oliver Braganza, 2020. "A simple model suggesting economically rational sample-size choice drives irreproducibility," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-19, March.
    10. Adam Altmejd & Anna Dreber & Eskil Forsell & Juergen Huber & Taisuke Imai & Magnus Johannesson & Michael Kirchler & Gideon Nave & Colin Camerer, 2019. "Predicting the replicability of social science lab experiments," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-18, December.
    11. Williams, Cole Randall, 2019. "How redefining statistical significance can worsen the replication crisis," Economics Letters, Elsevier, vol. 181(C), pages 65-69.
    12. Harvey, Campbell R. & Liu, Yan, 2021. "Lucky factors," Journal of Financial Economics, Elsevier, vol. 141(2), pages 413-435.
    13. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    14. Grabiszewski, Konrad & Horenstein, Alex, 2020. "Effort is not a monotonic function of skills: Results from a global mobile experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 634-652.
    15. Brade, Raphael & Himmler, Oliver & Jäckle, Robert, 2018. "Normatively Framed Relative Performance Feedback – Field Experiment and Replication," MPRA Paper 88830, University Library of Munich, Germany.
    16. So, Tony, 2020. "Classroom experiments as a replication device," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 86(C).
    17. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    18. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.
    19. Hoang, Khoa & Cannavan, Damien & Gaunt, Clive & Huang, Ronghong, 2019. "Is that factor just lucky? Australian evidence," Pacific-Basin Finance Journal, Elsevier, vol. 57(C).
    20. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.

    More about this item

    Keywords

    non-standard errors; multi-analyst approach; liquidity;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • G1 - Financial Economics - - General Financial Markets
    • G14 - Financial Economics - - General Financial Markets - - - Information and Market Efficiency; Event Studies; Insider Trading

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inn:wpaper:2021-31. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/fuibkat.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Janette Walde (email available below). General contact details of provider: https://edirc.repec.org/data/fuibkat.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.