IDEAS home Printed from https://ideas.repec.org/p/ecl/ohidic/2019-21.html
   My bibliography  Save this paper

Reusing Natural Experiments

Author

Listed:
  • Heath, Davidson

    (University of Utah David Eccles School of Business)

  • Ringgenberg, Matthew C.

    (University of Utah - Department of Finance)

  • Samadi, Mehrdad

    (Southern Methodist University (SMU) - Finance Department)

  • Werner, Ingrid M.

    (The Ohio State University - Fisher College of Business)

Abstract

Natural experiments are used in empirical research to make causal inferences. After a natural experiment is first used, other researchers often reuse the setting, examining different outcomes based on causal chain arguments. Using simulation evidence combined with two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we show that the repeated use of a natural experiment significantly increases the likelihood of false discoveries. To correct this, we propose multiple testing methods which account for dependence across tests and we show evidence of their efficacy.

Suggested Citation

  • Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    • Handle: RePEc:ecl:ohidic:2019-21
    as

    Download full text from publisher

    File URL: http://ssrn.com/abstract=3457525
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Julian Atanassov, 2013. "Do Hostile Takeovers Stifle Innovation? Evidence from Antitakeover Legislation and Corporate Patenting," Journal of Finance, American Finance Association, vol. 68(3), pages 1097-1131, June.
    2. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    5. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    6. Comment, Robert & Schwert, G. William, 1995. "Poison or placebo? Evidence on the deterrence and wealth effects of modern antitakeover measures," Journal of Financial Economics, Elsevier, vol. 39(1), pages 3-43, September.
    7. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    8. Vivian W. Fang & Allen H. Huang & Jonathan M. Karpoff, 2016. "Short Selling and Earnings Management: A Controlled Experiment," Journal of Finance, American Finance Association, vol. 71(3), pages 1251-1294, June.
    9. Marianne Bertrand & Sendhil Mullainathan, 1999. "Is There a Discretion in Wage Setting? A Test Using Takeover Legislation," RAND Journal of Economics, The RAND Corporation, vol. 30(3), pages 535-554, Autumn.
    10. Kenneth I. Wolpin & Mark R. Rosenzweig, 2000. "Natural "Natural Experiments" in Economics," Journal of Economic Literature, American Economic Association, vol. 38(4), pages 827-874, December.
    11. Leonid Kogan & Dimitris Papanikolaou & Amit Seru & Noah Stoffman, 2017. "Technological Innovation, Resource Allocation, and Growth," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(2), pages 665-712.
    12. Tarun Chordia & Amit Goyal & Alessio Saretto & Andrew KarolyiEditor, 2020. "Anomalies and False Rejections," Review of Finance, European Finance Association, vol. 33(5), pages 2134-2179.
    13. Marianne Bertrand & Sendhil Mullainathan, 2003. "Enjoying the Quiet Life? Corporate Governance and Managerial Preferences," Journal of Political Economy, University of Chicago Press, vol. 111(5), pages 1043-1075, October.
    14. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    15. Damian Clarke & Joseph P. Romano & Michael Wolf, 2020. "The Romano–Wolf multiple-hypothesis correction in Stata," Stata Journal, StataCorp LP, vol. 20(4), pages 812-843, December.
    16. Giroud, Xavier & Mueller, Holger M., 2010. "Does corporate governance matter in competitive industries?," Journal of Financial Economics, Elsevier, vol. 95(3), pages 312-331, March.
    17. Gormley, Todd A. & Matsa, David A., 2016. "Playing it safe? Managerial preferences, risk, and agency conflicts," Journal of Financial Economics, Elsevier, vol. 122(3), pages 431-455.
    18. Karl B. Diether & Kuan‐Hui Lee & Ingrid M. Werner, 2009. "It's SHO Time! Short‐Sale Price Tests and Market Quality," Journal of Finance, American Finance Association, vol. 64(1), pages 37-73, February.
    19. Alexander, Gordon J. & Peterson, Mark A., 2008. "The effect of price tests on trader behavior and market quality: An analysis of Reg SHO," Journal of Financial Markets, Elsevier, vol. 11(1), pages 84-111, February.
    20. Gustavo Grullon & Sébastien Michenaud & James P. Weston, 2015. "The Real Effects of Short-Selling Constraints," The Review of Financial Studies, Society for Financial Studies, vol. 28(6), pages 1737-1767.
    21. Blau, Benjamin M. & Griffith, Todd G., 2016. "Price clustering and the stability of stock prices," Journal of Business Research, Elsevier, vol. 69(10), pages 3933-3942.
    22. Wald, John K. & Long, Michael S., 2007. "The effect of state laws on capital structure," Journal of Financial Economics, Elsevier, vol. 83(2), pages 297-319, February.
    23. Liu, Lily Y. & Patton, Andrew J. & Sheppard, Kevin, 2015. "Does anything beat 5-minute RV? A comparison of realized measures across multiple asset classes," Journal of Econometrics, Elsevier, vol. 187(1), pages 293-311.
    24. Brown, Stephen & Hillegeist, Stephen A. & Lo, Kin, 2004. "Conference calls and information asymmetry," Journal of Accounting and Economics, Elsevier, vol. 37(3), pages 343-366, September.
    25. Florian S. Peters & Alexander F. Wagner, 2014. "The Executive Turnover Risk Premium," Journal of Finance, American Finance Association, vol. 69(4), pages 1529-1563, August.
    26. Morck, Randall & Yeung, Bernard, 2011. "Economics, History, and Causation," Business History Review, Cambridge University Press, vol. 85(1), pages 39-63, April.
    27. Dirk Jenter & Fadi Kanaan, 2015. "CEO Turnover and Relative Performance Evaluation," Journal of Finance, American Finance Association, vol. 70(5), pages 2155-2184, October.
    28. Jonathan M. Karpoff & Michael D. Wittry, 2018. "Institutional and Legal Context in Natural Experiments: The Case of State Antitakeover Laws," Journal of Finance, American Finance Association, vol. 73(2), pages 657-714, April.
    29. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    30. Armstrong, Christopher S. & Balakrishnan, Karthik & Cohen, Daniel, 2012. "Corporate governance and the information environment: Evidence from state antitakeover laws," Journal of Accounting and Economics, Elsevier, vol. 53(1), pages 185-204.
    31. Massa, Massimo & Qian, Wenlan & Xu, Weibiao & Zhang, Hong, 2015. "Competition of the informed: Does the presence of short sellers affect insider selling?," Journal of Financial Economics, Elsevier, vol. 118(2), pages 268-288.
    32. Romano, Joseph P. & Wolf, Michael, 2016. "Efficient computation of adjusted p-values for resampling-based stepdown multiple testing," Statistics & Probability Letters, Elsevier, vol. 113(C), pages 38-40.
    33. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    34. Joseph P. Romano & Michael Wolf, "undated". "Control of Generalized Error Rates in Multiple Testing," IEW - Working Papers 245, Institute for Empirical Research in Economics - University of Zurich.
    35. Bhargava, Rahul & Faircloth, Sheri & Zeng, Hongchao, 2017. "Takeover protection and stock price crash risk: Evidence from state antitakeover laws," Journal of Business Research, Elsevier, vol. 70(C), pages 177-184.
    36. Mulherin, J. Harold & Netter, Jeffry M. & Poulsen, Annette B., 2018. "Observations on research and publishing from nineteen years as editors of the Journal of Corporate Finance," Journal of Corporate Finance, Elsevier, vol. 49(C), pages 120-124.
    37. Dean P. Foster & Robert A. Stine, 2008. "α‐investing: a procedure for sequential control of expected false discoveries," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 70(2), pages 429-444, April.
    38. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    39. Tarun Chordia & Amit Goyal & Alessio Saretto, 2020. "Anomalies and False Rejections," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2134-2179.
    40. Meyer, Bruce D, 1995. "Natural and Quasi-experiments in Economics," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(2), pages 151-161, April.
    41. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    42. Jones, Jj, 1991. "Earnings Management During Import Relief Investigations," Journal of Accounting Research, Wiley Blackwell, vol. 29(2), pages 193-228.
    43. Baldwin Kate & Bhavnani Rikhil R., 2015. "Ancillary Studies of Experiments: Opportunities and Challenges," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 113-146, June.
    44. Gerald T. Garvey & Gordon Hanka, 1999. "Capital Structure and Corporate Control: The Effect of Antitakeover Statutes on Firm Leverage," Journal of Finance, American Finance Association, vol. 54(2), pages 519-546, April.
    45. Bill B. Francis & Iftekhar Hasan & Kose John & Liang Song, 2011. "Corporate Governance and Dividend Payout Policy: A Test Using Antitakeover Legislation," Financial Management, Financial Management Association International, vol. 40(1), pages 83-112, March.
    46. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    47. David De Angelis & Gustavo Grullon & Sébastien Michenaud, 2017. "The Effects of Short-Selling Threats on Incentive Contracts: Evidence from an Experiment," The Review of Financial Studies, Society for Financial Studies, vol. 30(5), pages 1627-1659.
    48. Easley, David & Kiefer, Nicholas M & O'Hara, Maureen, 1997. "One Day in the Life of a Very Common Stock," The Review of Financial Studies, Society for Financial Studies, vol. 10(3), pages 805-835.
    49. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    50. Cain, Matthew D. & McKeon, Stephen B. & Solomon, Steven Davidoff, 2017. "Do takeover laws matter? Evidence from five decades of hostile takeovers," Journal of Financial Economics, Elsevier, vol. 124(3), pages 464-485.
    51. Karpoff, Jonathan M. & Malatesta, Paul H., 1989. "The wealth effects of second-generation state takeover legislation," Journal of Financial Economics, Elsevier, vol. 25(2), pages 291-322, December.
    52. repec:fth:prinin:455 is not listed on IDEAS
    53. Angie Andrikogiannopoulou & Filippos Papakonstantinou, 2019. "Reassessing False Discoveries in Mutual Fund Performance: Skill, Luck, or Lack of Power?," Journal of Finance, American Finance Association, vol. 74(5), pages 2667-2688, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chad Brown & Paola Conconi & Aksel Erbahar & Lorenzo Trimarchi, 2020. "Trade Protection Along Supply Chains," Working Papers ECARES 2020-52, ULB -- Universite Libre de Bruxelles.
    2. Guernsey, Scott & Sepe, Simone M. & Serfling, Matthew, 2022. "Blood in the water: The value of antitakeover provisions during market shocks," Journal of Financial Economics, Elsevier, vol. 143(3), pages 1070-1096.
    3. Bui, Dien Giau & Hasan, Iftekhar & Lin, Chih-Yung & Nguyen, Hong Thoa, 2023. "Short-selling threats and bank risk-taking: Evidence from the financial crisis," Journal of Banking & Finance, Elsevier, vol. 150(C).
    4. Aepli, Manuel & Kuhn, Andreas & Schweri, Jürg, 2021. "Culture, norms, and the provision of training by employers: Evidence from the Swiss language border," Labour Economics, Elsevier, vol. 73(C).
    5. Leuz, Christian, 2022. "Towards a design-based approach to accounting research," Journal of Accounting and Economics, Elsevier, vol. 74(2).
    6. Chen, Xiaoqi & Chih-Chieh Chris, Hsieh & Tsang, Albert & Xiang, Yi, 2022. "Cross-border enforcement of securities laws and dividend payouts," The British Accounting Review, Elsevier, vol. 54(6).
    7. Yun Ke & Kin Lo & Jinfei Sheng & Jenny Li Zhang, 2023. "Do investors affect financial analysts’ behavior? Evidence from short sellers," Financial Management, Financial Management Association International, vol. 52(1), pages 199-224, March.
    8. Gong, Rong, 2020. "Short selling threat and corporate financing decisions," Journal of Banking & Finance, Elsevier, vol. 118(C).
    9. Tianyu Cai & Lixiong Guo & Yongxian Tan, 2024. "Short seller monitoring and real earnings management," The Financial Review, Eastern Finance Association, vol. 59(1), pages 203-225, February.
    10. Chen, Jie & Su, Xunhua & Tian, Xuan & Xu, Bin, 2022. "Does customer-base structure influence managerial risk-taking incentives?," Journal of Financial Economics, Elsevier, vol. 143(1), pages 462-483.
    11. Nilabhra Bhattacharya & Theodore E. Christensen & Qunfeng Liao & Bo Ouyang, 2022. "Can short sellers constrain aggressive non-GAAP reporting?," Review of Accounting Studies, Springer, vol. 27(2), pages 391-440, June.
    12. Merkley, Kenneth & Michaely, Roni & Pacelli, Joseph, 2020. "Cultural diversity on Wall Street: Evidence from consensus earnings forecasts," Journal of Accounting and Economics, Elsevier, vol. 70(1).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Todd Mitton, 2022. "Methodological Variation in Empirical Corporate Finance," The Review of Financial Studies, Society for Financial Studies, vol. 35(2), pages 527-575.
    2. Iskenderoglu, Cansu, 2021. "Managerial discretion and efficiency of internal capital markets," Journal of Corporate Finance, Elsevier, vol. 70(C).
    3. Tang, Yuehua, 2018. "When does competition mitigate agency problems?," Journal of Corporate Finance, Elsevier, vol. 51(C), pages 258-274.
    4. Guernsey, Scott & Sepe, Simone M. & Serfling, Matthew, 2022. "Blood in the water: The value of antitakeover provisions during market shocks," Journal of Financial Economics, Elsevier, vol. 143(3), pages 1070-1096.
    5. Andres Christian & Jacob Martin & Ulrich Lennart, 2019. "Takeover Protection and Firm Value," Review of Law & Economics, De Gruyter, vol. 15(1), pages 1-46, March.
    6. Malmendier, Ulrike M. & Borgschulte, Mark & Guenzel, Marius & Liu, Canyao, 2020. "CEO Stress, Aging, and Death," CEPR Discussion Papers 14933, C.E.P.R. Discussion Papers.
    7. Xavier Giroud & Holger M. Mueller, 2009. "Does Corporate Governance Matter in Competitive Industries?," NBER Working Papers 14877, National Bureau of Economic Research, Inc.
    8. Bui, Dien Giau & Hasan, Iftekhar & Lin, Chih-Yung & Nguyen, Hong Thoa, 2023. "Short-selling threats and bank risk-taking: Evidence from the financial crisis," Journal of Banking & Finance, Elsevier, vol. 150(C).
    9. Armstrong, Christopher S. & Balakrishnan, Karthik & Cohen, Daniel, 2012. "Corporate governance and the information environment: Evidence from state antitakeover laws," Journal of Accounting and Economics, Elsevier, vol. 53(1), pages 185-204.
    10. Giroud, Xavier & Mueller, Holger M., 2010. "Does corporate governance matter in competitive industries?," Journal of Financial Economics, Elsevier, vol. 95(3), pages 312-331, March.
    11. Gormley, Todd A. & Matsa, David A., 2016. "Playing it safe? Managerial preferences, risk, and agency conflicts," Journal of Financial Economics, Elsevier, vol. 122(3), pages 431-455.
    12. Ni, Xiaoran, 2020. "Does stakeholder orientation matter for earnings management: Evidence from non-shareholder constituency statutes," Journal of Corporate Finance, Elsevier, vol. 62(C).
    13. Cain, Matthew D. & McKeon, Stephen B. & Solomon, Steven Davidoff, 2017. "Do takeover laws matter? Evidence from five decades of hostile takeovers," Journal of Financial Economics, Elsevier, vol. 124(3), pages 464-485.
    14. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    15. Kim, Incheol & Lee, Suin & Sharma, Bina, 2023. "Competition law reform and firm performance: Evidence from developing countries," Emerging Markets Review, Elsevier, vol. 56(C).
    16. Bhargava, Rahul & Faircloth, Sheri & Zeng, Hongchao, 2017. "Takeover protection and stock price crash risk: Evidence from state antitakeover laws," Journal of Business Research, Elsevier, vol. 70(C), pages 177-184.
    17. Psaradellis, Ioannis & Laws, Jason & Pantelous, Athanasios A. & Sermpinis, Georgios, 2023. "Technical analysis, spread trading, and data snooping control," International Journal of Forecasting, Elsevier, vol. 39(1), pages 178-191.
    18. Omer Unsal & Blake Rayfield, 2020. "Correction to: Corporate governance and employee treatment: Evidence from takeover defenses," Journal of Economics and Finance, Springer;Academy of Economics and Finance, vol. 44(2), pages 392-416, April.
    19. Brockman, Paul & Luo, Juan & Xu, Limin, 2020. "The impact of short-selling pressure on corporate employee relations," Journal of Corporate Finance, Elsevier, vol. 64(C).
    20. Younge, Kenneth A. & Tong, Tony W., 2018. "Competitive pressure on the rate and scope of innovation," Journal of Economic Behavior & Organization, Elsevier, vol. 150(C), pages 162-181.

    More about this item

    JEL classification:

    • G1 - Financial Economics - - General Financial Markets
    • G10 - Financial Economics - - General Financial Markets - - - General (includes Measurement and Data)

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:ohidic:2019-21. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/cdohsus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.