IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/215709.html
   My bibliography  Save this article

Can p-values be meaningfully interpreted without random sampling?

Author

Listed:
  • Hirschauer, Norbert
  • Grüner, Sven
  • Mußhoff, Oliver
  • Becker, Claudia
  • Jantsch, Antje

Abstract

Besides the inferential errors that abound in the interpretation of p-values, the probabilistic pre-conditions (i.e. random sampling or equivalent) for using them at all are not often met by observational studies in the social sciences. This paper systematizes different sampling designs and discusses the restrictive requirements of data collection that are the indispensable prerequisite for using p-values.

Suggested Citation

  • Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2020. "Can p-values be meaningfully interpreted without random sampling?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 14, pages 71-91.
    • Handle: RePEc:zbw:espost:215709
    DOI: 10.1214/20-SS129
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/215709/1/Hirschauer_2020_p_values.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.1214/20-SS129?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.
    3. David Trafimow, 2019. "Five Nonobvious Changes in Editorial Practice for Editors and Reviewers to Consider When Evaluating Submissions in a Post p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 340-345, March.
    4. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    5. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "Inference in economic experiments," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 14, pages 1-14.
    6. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    7. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    8. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    9. Lee, Byung-Joo & Marsh, Lawrence C, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    10. Andrew Gelman & John Carlin, 2017. "Some Natural Solutions to the -Value Communication Problem—and Why They Won’t Work," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 899-901, July.
    11. Joyce J Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(5), pages 1337-1352.
    12. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 106, University of California, Davis, Department of Economics.
    13. Donald Berry, 2017. "A -Value to Die For," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 895-897, July.
    14. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    15. Massenot, Baptiste & Pettinicchi, Yuri, 2018. "Can firms see into the future? Survey evidence from Germany," Journal of Economic Behavior & Organization, Elsevier, vol. 145(C), pages 66-79.
    16. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    17. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    18. Vella, Francis & Verbeek, Marno, 1999. "Two-step estimation of panel data models with censored endogenous variables and selection bias," Journal of Econometrics, Elsevier, vol. 90(2), pages 239-263, June.
    19. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 318, University of California, Davis, Department of Economics.
    20. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    21. Walter Krämer, 2011. "The Cult of Statistical Significance – What Economists Should and Should Not Do to Make their Data Talk," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 131(3), pages 455-468.
    22. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    23. Greene, William H, 1981. "Sample Selection Bias as a Specification Error: Comment," Econometrica, Econometric Society, vol. 49(3), pages 795-798, May.
    24. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    25. Joyce J. Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, John Wiley & Sons, vol. 101(5), pages 1337-1352, October.
    26. J. B. Copas & H. G. Li, 1997. "Inference for Non‐random Samples," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(1), pages 55-95.
    27. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    28. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    29. Byung‐Joo Lee & L. C. Marsh, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    30. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2019. "Economic experiments and inference," SocArXiv 67mws, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dufhues, Thomas & Möllers, Judith & Jantsch, Antje & Buchenrieder, Gertrud & Camfield, Laura, 2023. "Don’t look up! Individual income comparisons and subjective well-being of students in Thailand," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 24(2), pages 477-503.
    2. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "A primer on p-value thresholds and α-levels – two different kettles of fish," SocArXiv d46m2, Center for Open Science.
    3. Gruener, Sven, 2019. "An empirical study on Internet-based false news stories: experiences, problem awareness, and responsibilities," SocArXiv xbez9, Center for Open Science.
    4. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    5. , Hirschauer, 2022. "Some Thoughts About Statistical Inference In The 21st Century," SocArXiv exdfg, Center for Open Science.
    6. Peter Backus & Thien Nguyen, 2021. "The Effect of the Sex Buyer Law on the Market for Sex, Sexual Health and Sexual Violence," Economics Discussion Paper Series 2106, Economics, The University of Manchester.
    7. J. M. Santos & H. Horta & H. Luna, 2022. "The relationship between academics’ strategic research agendas and their preferences for basic research, applied research, or experimental development," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4191-4225, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miguel Santolino & Magnus Söderberg, 2014. "Modelling appellate courts’ responses in motor injury disputes," European Journal of Law and Economics, Springer, vol. 38(3), pages 393-407, December.
    2. Morrissey, Karyn & Kinderman, Peter & Pontin, Eleanor & Tai, Sara & Schwannauer, Mathias, 2016. "Web based health surveys: Using a Two Step Heckman model to examine their potential for population health analysis," Social Science & Medicine, Elsevier, vol. 163(C), pages 45-53.
    3. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    4. Embaye, Weldensie T. & Bergtold, Jason S. & Archer, David & Flora, Cornelia & Andrango, Graciela C. & Odening, Marting & Buysse, Jeroen, 2018. "Examining farmers' willingness to grow and allocate land for oilseed crops for biofuel production," Energy Economics, Elsevier, vol. 71(C), pages 311-320.
    5. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    6. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    7. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    8. Miguel Santolino & Magnus Söderberg, 2011. "The influence of decision-maker effort and case complexity on appealed rulings subject to multi-categorical selection," IREA Working Papers 201115, University of Barcelona, Research Institute of Applied Economics, revised Sep 2011.
    9. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    10. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    11. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    12. P.W. Miller & S. Rummery, 1989. "Gender Wage Discrimination in Australia: A reassessment," Economics Discussion / Working Papers 89-21, The University of Western Australia, Department of Economics.
    13. Nigel Driffield & Yong Yang, 2021. "Leveraging the benefits of location decisions into performance:A global view from matched MNEs," Working Papers 011, The Productivity Institute.
    14. Takashi Yamagata & Chris Orme, 2005. "On Testing Sample Selection Bias Under the Multicollinearity Problem," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 467-481.
    15. Spermann, Alexander & Strotmann, Harald, 2005. "The Targeted Negative Income Tax (TNIT) in Germany: Evidence from a Quasi Experiment," ZEW Discussion Papers 05-68, ZEW - Leibniz Centre for European Economic Research.
    16. Benjamin L. Collier & Andrew F. Haughwout & Howard C. Kunreuther & Erwann O. Michel‐Kerjan, 2020. "Firms’ Management of Infrequent Shocks," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(6), pages 1329-1359, September.
    17. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    18. Yang, Yong & Driffield, Nigel, 2022. "Leveraging the benefits of location decisions into performance: A global view from matched MNEs," Journal of Business Research, Elsevier, vol. 139(C), pages 468-483.
    19. Ian Gazeley & Rose Holmes & Andrew Newell & Kevin Reynolds & Hector Gutierrez Rufrancos, 2023. "Escaping from hunger before WW1: the nutritional transition and living standards in Western Europe and USA in the late nineteenth century," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(3), pages 533-565, September.
    20. Sumaryanto & Sri Hery Susilowati & Fitri Nurfatriani & Herlina Tarigan & Erwidodo & Tahlim Sudaryanto & Henri Wira Perkasa, 2022. "Determinants of Farmers’ Behavior towards Land Conservation Practices in the Upper Citarum Watershed in West Java, Indonesia," Land, MDPI, vol. 11(10), pages 1-21, October.

    More about this item

    Keywords

    sampling design; propensity scores; p-values; random sampling; selection bias; inference;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:215709. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.