IDEAS home Printed from https://ideas.repec.org/a/spr/jbecon/v89y2019i4d10.1007_s11573-018-0919-3.html
   My bibliography  Save this article

Reaching for the stars: attention to multiple testing problems and method recommendations using simulation for business research

Author

Listed:
  • Christina C. Bartenschlager

    (University of Augsburg, University Center of Health Sciences at Klinikum Augsburg (UNIKA-T))

  • Jens O. Brunner

    (University of Augsburg, University Center of Health Sciences at Klinikum Augsburg (UNIKA-T))

Abstract

Multiple hypotheses tests decide on more than one hypothesis based upon the same data set. Despite the significant relevance for business research, we find that multiple testing methods are not systematically applied in our research area. As surprising this finding is, as crucial it is to the significance of research findings. False positive findings can inflate with the number of employed single tests. The focus on medicine and psychology related topics of current multiple testing literature might deter business researchers from employing the tests. Hence, we provide guidance to researchers by exposing the importance of multiple testing. Based on an application oriented systematization, Monte Carlo simulations, and a decision theoretic evaluation scheme, we highlight method recommendations for different conservatism definitions regarding errors. As implicit results we find, that the dependency structure in the data does not influence the method outcomes significantly. In addition, intuitive single step margin based approaches perform similar than sophisticated tests for typical multiple testing problems in a conservative setting. More liberal methods show a stable relation for all simulation designs.

Suggested Citation

  • Christina C. Bartenschlager & Jens O. Brunner, 2019. "Reaching for the stars: attention to multiple testing problems and method recommendations using simulation for business research," Journal of Business Economics, Springer, vol. 89(4), pages 447-479, June.
    • Handle: RePEc:spr:jbecon:v:89:y:2019:i:4:d:10.1007_s11573-018-0919-3
    DOI: 10.1007/s11573-018-0919-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11573-018-0919-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11573-018-0919-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yoav Benjamini & Abba M. Krieger & Daniel Yekutieli, 2006. "Adaptive linear step-up procedures that control the false discovery rate," Biometrika, Biometrika Trust, vol. 93(3), pages 491-507, September.
    2. Jens Stange & Taras Bodnar & Thorsten Dickhaus, 2015. "Uncertainty quantification for the family-wise error rate in multivariate copula models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 99(3), pages 281-310, July.
    3. Francisco Barillas & Jay Shanken, 2018. "Comparing Asset Pricing Models," Journal of Finance, American Finance Association, vol. 73(2), pages 715-754, April.
    4. Burkhard C. Schipper, 2015. "Sex Hormones and Competitive Bidding," Management Science, INFORMS, vol. 61(2), pages 249-266, February.
    5. Korhonen, Pekka J. & Malo, Pekka & Pajala, Tommi & Ravaja, Niklas & Somervuori, Outi & Wallenius, Jyrki, 2018. "Context matters: The impact of product type, emotional attachment and information overload on choice quality," European Journal of Operational Research, Elsevier, vol. 264(1), pages 270-279.
    6. Andreas Ostermaier, 2018. "Incentives for students: effects of certificates and deadlines on student performance," Journal of Business Economics, Springer, vol. 88(1), pages 65-96, January.
    7. Wesley R. Hartmann & Daniel Klapper, 2018. "Super Bowl Ads," Marketing Science, INFORMS, vol. 37(1), pages 78-96, January.
    8. Yoav Benjamini & Yosef Hochberg, 2000. "On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics," Journal of Educational and Behavioral Statistics, , vol. 25(1), pages 60-83, March.
    9. Tsai, Chen-An & Chen, James J., 2007. "Kernel estimation for adjusted p-values in multiple testing," Computational Statistics & Data Analysis, Elsevier, vol. 51(8), pages 3885-3897, May.
    10. Chen, Yuning & Hao, Jin-Kao, 2018. "Two phased hybrid local search for the periodic capacitated arc routing problem," European Journal of Operational Research, Elsevier, vol. 264(1), pages 55-65.
    11. Elmar Gerum & Sascha H. Mölls & Chunqian Shen, 2018. "Corporate governance, capital market orientation and firm performance: empirical evidence for large publicly traded German corporations," Journal of Business Economics, Springer, vol. 88(2), pages 203-252, February.
    12. Bessec, Marie & Fouquau, Julien, 2018. "Short-run electricity load forecasting with combinations of stationary wavelet transforms," European Journal of Operational Research, Elsevier, vol. 264(1), pages 149-164.
    13. Steffen Maas & Tassilo Schuster & Evi Hartmann, 2018. "Stakeholder Pressures, Environmental Practice Adoption and Economic Performance in the German Third-party Logistics Industry—A Contingency Perspective," Journal of Business Economics, Springer, vol. 88(2), pages 167-201, February.
    14. John D. Storey & Jonathan E. Taylor & David Siegmund, 2004. "Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 66(1), pages 187-205, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Steffen Heider & Jan Schoenfelder & Thomas Koperna & Jens O. Brunner, 2022. "Balancing control and autonomy in master surgery scheduling: Benefits of ICU quotas for recovery units," Health Care Management Science, Springer, vol. 25(2), pages 311-332, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guo Wenge & Peddada Shyamal, 2008. "Adaptive Choice of the Number of Bootstrap Samples in Large Scale Multiple Testing," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 7(1), pages 1-21, March.
    2. Yoav Benjamini, 2010. "Discovering the false discovery rate," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(4), pages 405-416, September.
    3. Döhler, Sebastian, 2018. "A discrete modification of the Benjamini–Yekutieli procedure," Econometrics and Statistics, Elsevier, vol. 5(C), pages 137-147.
    4. Shigeyuki Matsui & Hisashi Noma, 2011. "Estimating Effect Sizes of Differentially Expressed Genes for Power and Sample-Size Assessments in Microarray Experiments," Biometrics, The International Biometric Society, vol. 67(4), pages 1225-1235, December.
    5. Hanck, Christoph, 2011. "Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach," Economic Modelling, Elsevier, vol. 28(4), pages 1739-1746, July.
    6. Ghosh Debashis, 2012. "Incorporating the Empirical Null Hypothesis into the Benjamini-Hochberg Procedure," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 11(4), pages 1-21, July.
    7. Gharad Bryan & James J Choi & Dean Karlan, 2021. "Randomizing Religion: the Impact of Protestant Evangelism on Economic Outcomes," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 136(1), pages 293-380.
    8. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(3), pages 417-442, November.
    9. Ferreira José A. & Berkhof Johannes & Souverein Olga & Zwinderman Koos, 2009. "A Multiple Testing Approach to High-Dimensional Association Studies with an Application to the Detection of Associations between Risk Factors of Heart Disease and Genetic Polymorphisms," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 8(1), pages 1-56, January.
    10. Benoit Duchet & Filippo Ghezzi & Gihan Weerasinghe & Gerd Tinkhauser & Andrea A Kühn & Peter Brown & Christian Bick & Rafal Bogacz, 2021. "Average beta burst duration profiles provide a signature of dynamical changes between the ON and OFF medication states in Parkinson’s disease," PLOS Computational Biology, Public Library of Science, vol. 17(7), pages 1-42, July.
    11. Christina C. Bartenschlager & Michael Krapp, 2015. "Theorie und Methoden multipler statistischer Vergleiche," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 9(2), pages 107-129, November.
    12. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    13. Deckers, Thomas & Hanck, Christoph, 2009. "Multiple Testing Techniques in Growth Econometrics," MPRA Paper 17843, University Library of Munich, Germany.
    14. Nik Tuzov & Frederi Viens, 2011. "Mutual fund performance: false discoveries, bias, and power," Annals of Finance, Springer, vol. 7(2), pages 137-169, May.
    15. Sankaran, Kris & Holmes, Susan, 2014. "structSSI: Simultaneous and Selective Inference for Grouped or Hierarchically Structured Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 59(i13).
    16. Habiger, Joshua D. & Adekpedjou, Akim, 2014. "Optimal rejection curves for exact false discovery rate control," Statistics & Probability Letters, Elsevier, vol. 94(C), pages 21-28.
    17. Kim, Donggyu & Zhang, Chunming, 2014. "Adaptive linear step-up multiple testing procedure with the bias-reduced estimator," Statistics & Probability Letters, Elsevier, vol. 87(C), pages 31-39.
    18. Andrew Y. Chen, 2022. "Do t-Statistic Hurdles Need to be Raised?," Papers 2204.10275, arXiv.org, revised Apr 2024.
    19. Zhao, Haibing & Fung, Wing Kam, 2016. "A powerful FDR control procedure for multiple hypotheses," Computational Statistics & Data Analysis, Elsevier, vol. 98(C), pages 60-70.
    20. Cai, Qingyun, 2018. "A scoring criterion for rejection of clustered p-values," Computational Statistics & Data Analysis, Elsevier, vol. 121(C), pages 180-189.

    More about this item

    Keywords

    Multiple testing; Simultaneous inference; Simulation; Method selection;
    All these keywords.

    JEL classification:

    • M00 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - General - - - General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jbecon:v:89:y:2019:i:4:d:10.1007_s11573-018-0919-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.