IDEAS home Printed from https://ideas.repec.org/p/dkn/econwp/eco_2009_13.html
   My bibliography  Save this paper

Could It Be Better to Discard 90% of the Data? A Statistical Paradox

Author

Listed:
  • T.D. Stanley

    ()

  • Stephen B. Jarrell

    ()

  • Hristos Doucouliagos

    ()

Abstract

Conventional practice is to draw inferences from all available data and research results, even though there is ample evidence to suggest that empirical literatures suffer from publication selection bias. When a scientific literature is plagued by such bias, a simple discarding of the vast majority of empirical results can actually improve statistical inference and estimation. Simulations demonstrate that, if the majority of researchers, reviewers, and editors use statistical significance as a criterion for reporting or publishing an estimate, discarding 90% of the published findings greatly reduces publication selection bias and is often more efficient than conventional summary statistics. Improving statistical estimation and inference through removing so much data goes against statistical theory and practice; hence, it is paradoxical. We investigate a very simple method to reduce the effects of publication bias and to improve the efficiency of summary estimates of accumulated empirical research results that averages the most precise ten percent of the reported estimates (i.e., ‘Top10’). In the process, the critical importance of precision (the inverse of an estimate’s standard error) as a measure of a study’s quality is brought to light. Reviewers and journal editors should use precision as one objective measure of a study’s quality.

Suggested Citation

  • T.D. Stanley & Stephen B. Jarrell & Hristos Doucouliagos, 2009. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," Economics Series 2009_13, Deakin University, Faculty of Business and Law, School of Accounting, Economics and Finance.
  • Handle: RePEc:dkn:econwp:eco_2009_13
    as

    Download full text from publisher

    File URL: http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2009_13.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Trockel, Walter, 2011. "An axiomatization of the sequential Raiffa solution," Center for Mathematical Economics Working Papers 425, Center for Mathematical Economics, Bielefeld University.
    2. Kalai, Ehud & Smorodinsky, Meir, 1975. "Other Solutions to Nash's Bargaining Problem," Econometrica, Econometric Society, vol. 43(3), pages 513-518, May.
    3. Geoffroy Clippel, 2007. "An axiomatization of the Nash bargaining solution," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 29(2), pages 201-210, September.
    4. Kalai, Ehud, 1977. "Proportional Solutions to Bargaining Situations: Interpersonal Utility Comparisons," Econometrica, Econometric Society, vol. 45(7), pages 1623-1630, October.
    5. Hans Peters & Eric Van Damme, 1991. "Characterizing the Nash and Raiffa Bargaining Solutions by Disagreement Point Axioms," Mathematics of Operations Research, INFORMS, vol. 16(3), pages 447-461, August.
    6. Chun, Youngsub, 1988. "Nash solution and timing of bargaining," Economics Letters, Elsevier, vol. 28(1), pages 27-31.
    7. Diskin, A. & Koppel, M. & Samet, D., 2011. "Generalized Raiffa solutions," Games and Economic Behavior, Elsevier, vol. 73(2), pages 452-458.
    8. Nash, John, 1950. "The Bargaining Problem," Econometrica, Econometric Society, vol. 18(2), pages 155-162, April.
    9. Chun, Youngsub & Thomson, William, 1990. "Bargaining with Uncertain Disagreement Points," Econometrica, Econometric Society, vol. 58(4), pages 951-959, July.
    10. Sobel, Joel, 1981. "Distortion of Utilities and the Bargaining Problem," Econometrica, Econometric Society, vol. 49(3), pages 597-619, May.
    11. Roth, Alvin E, 1979. "Proportional Solutions to the Bargaining Problem," Econometrica, Econometric Society, vol. 47(3), pages 775-777, May.
    12. Edgeworth, Francis Ysidro, 1881. "Mathematical Psychics," History of Economic Thought Books, McMaster University Archive for the History of Economic Thought, number edgeworth1881, December.
    13. Tijs, S.H. & Jansen, M.J.M., 1982. "On the existence of values of arbitration games," Other publications TiSEM da09ab18-3256-4e0c-ad35-b, Tilburg University, School of Economics and Management.
    14. Thomson, William, 1987. "Monotonicity of bargaining solutions with respect to the disagreement point," Journal of Economic Theory, Elsevier, vol. 42(1), pages 50-58, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Havranek, Tomas & Kokes, Ondrej, 2015. "Income elasticity of gasoline demand: A meta-analysis," Energy Economics, Elsevier, vol. 47(C), pages 77-86.
    2. T.D. Stanley, 2013. "Does economics add up? An introduction to meta-regression analysis," European Journal of Economics and Economic Policies: Intervention, Edward Elgar Publishing, vol. 10(2), pages 207-220.
    3. Auspurg Katrin & Hinz Thomas, 2011. "What Fuels Publication Bias?: Theoretical and Empirical Analyses of Risk Factors Using the Caliper Test," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 231(5-6), pages 636-660, October.
    4. Reed, W. Robert, 2015. "A Monte Carlo analysis of alternative meta-analysis estimators in the presence of publication bias," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 9, pages 1-40.
    5. Dimos, Christos & Pugh, Geoff, 2016. "The effectiveness of R&D subsidies: A meta-regression analysis of the evaluation literature," Research Policy, Elsevier, vol. 45(4), pages 797-815.
    6. repec:eee:wdevel:v:96:y:2017:i:c:p:1-18 is not listed on IDEAS
    7. Doucouliagos, Chris & Stanley, T.D. & Giles, Margaret, 2012. "Are estimates of the value of a statistical life exaggerated?," Journal of Health Economics, Elsevier, vol. 31(1), pages 197-206.
    8. Doucouliagos, Hristos & Stanley, T.D. & Viscusi, W. Kip, 2014. "Publication selection and the income elasticity of the value of a statistical life," Journal of Health Economics, Elsevier, vol. 33(C), pages 67-75.
    9. Jon Nelson, 2013. "Meta-analysis of alcohol price and income elasticities – with corrections for publication bias," Health Economics Review, Springer, vol. 3(1), pages 1-10, December.
    10. Stern, David I. & Enflo, Kerstin, 2013. "Causality between energy and output in the long-run," Energy Economics, Elsevier, vol. 39(C), pages 135-146.
    11. Sefa K. Awaworyi, 2014. "The Impact of Microfinance Interventions: A Meta-analysis," Monash Economics Working Papers 03-14, Monash University, Department of Economics.
    12. T.D. Stanley & Hristos Doucouliagos, 2011. "Meta-Regression Approximations to Reduce Publication Selection Bias," Economics Series 2011_4, Deakin University, Faculty of Business and Law, School of Accounting, Economics and Finance.
    13. Nelson, Jon P., 2014. "Estimating the price elasticity of beer: Meta-analysis of data with heterogeneity, dependence, and publication bias," Journal of Health Economics, Elsevier, vol. 33(C), pages 180-187.
    14. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.

    More about this item

    Keywords

    Publication Selection; Meta-analysis; Precision; Simulations; Meta-Regression.;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:dkn:econwp:eco_2009_13. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dr Xueli Tang). General contact details of provider: http://edirc.repec.org/data/bsdeaau.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.