IDEAS home Printed from https://ideas.repec.org/p/osf/socarx/yazr8.html
   My bibliography  Save this paper

Can p-values be meaningfully interpreted without random sampling?

Author

Listed:
  • Hirschauer, Norbert
  • Gruener, Sven
  • Mußhoff, Oliver
  • Becker, Claudia
  • Jantsch, Antje

Abstract

Besides the inferential errors that abound in the interpretation of p-values, the probabilistic pre-conditions (i.e. random sampling or equivalent) for using them at all are not often met by observa-tional studies in the social sciences. This paper systematizes different sampling designs and discusses the restrictive requirements of data collection that are the sine-qua-non for using p-values.

Suggested Citation

  • Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2019. "Can p-values be meaningfully interpreted without random sampling?," SocArXiv yazr8, Center for Open Science.
    • Handle: RePEc:osf:socarx:yazr8
    DOI: 10.31219/osf.io/yazr8
    as

    Download full text from publisher

    File URL: https://osf.io/download/5d56ae446e9c70001cfe3dd1/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/yazr8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    2. Lee, Byung-Joo & Marsh, Lawrence C, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    3. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "Inference in economic experiments," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 14, pages 1-14.
    4. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    5. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 318, University of California, Davis, Department of Economics.
    6. Walter Krämer, 2011. "The Cult of Statistical Significance – What Economists Should and Should Not Do to Make their Data Talk," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 131(3), pages 455-468.
    7. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    8. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.
    9. J. B. Copas & H. G. Li, 1997. "Inference for Non‐random Samples," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(1), pages 55-95.
    10. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    11. Byung‐Joo Lee & L. C. Marsh, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    12. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    13. Vella, Francis & Verbeek, Marno, 1999. "Two-step estimation of panel data models with censored endogenous variables and selection bias," Journal of Econometrics, Elsevier, vol. 90(2), pages 239-263, June.
    14. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    15. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    16. Greene, William H, 1981. "Sample Selection Bias as a Specification Error: Comment," Econometrica, Econometric Society, vol. 49(3), pages 795-798, May.
    17. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    18. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    19. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    20. Joyce J Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(5), pages 1337-1352.
    21. Massenot, Baptiste & Pettinicchi, Yuri, 2018. "Can firms see into the future? Survey evidence from Germany," Journal of Economic Behavior & Organization, Elsevier, vol. 145(C), pages 66-79.
    22. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 107, University of California, Davis, Department of Economics.
    23. Joyce J. Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, John Wiley & Sons, vol. 101(5), pages 1337-1352, October.
    24. David Trafimow, 2019. "Five Nonobvious Changes in Editorial Practice for Editors and Reviewers to Consider When Evaluating Submissions in a Post p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 340-345, March.
    25. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    26. Andrew Gelman & John Carlin, 2017. "Some Natural Solutions to the -Value Communication Problem—and Why They Won’t Work," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 899-901, July.
    27. Donald Berry, 2017. "A -Value to Die For," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 895-897, July.
    28. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    29. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    30. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2019. "Economic experiments and inference," SocArXiv 67mws, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    2. Thomas Dufhues & Judith Möllers & Antje Jantsch & Gertrud Buchenrieder & Laura Camfield, 2023. "Don’t Look Up! Individual Income Comparisons and Subjective Well-Being of Students in Thailand," Journal of Happiness Studies, Springer, vol. 24(2), pages 477-503, February.
    3. Gruener, Sven, 2019. "An empirical study on Internet-based false news stories: experiences, problem awareness, and responsibilities," SocArXiv xbez9, Center for Open Science.
    4. , Hirschauer, 2022. "Some Thoughts About Statistical Inference In The 21st Century," SocArXiv exdfg, Center for Open Science.
    5. Peter Backus & Thien Nguyen, 2021. "The Effect of the Sex Buyer Law on the Market for Sex, Sexual Health and Sexual Violence," Economics Discussion Paper Series 2106, Economics, The University of Manchester.
    6. J. M. Santos & H. Horta & H. Luna, 2022. "The relationship between academics’ strategic research agendas and their preferences for basic research, applied research, or experimental development," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4191-4225, July.
    7. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "A primer on p-value thresholds and α-levels – two different kettles of fish," SocArXiv d46m2, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miguel Santolino & Magnus Söderberg, 2014. "Modelling appellate courts’ responses in motor injury disputes," European Journal of Law and Economics, Springer, vol. 38(3), pages 393-407, December.
    2. Morrissey, Karyn & Kinderman, Peter & Pontin, Eleanor & Tai, Sara & Schwannauer, Mathias, 2016. "Web based health surveys: Using a Two Step Heckman model to examine their potential for population health analysis," Social Science & Medicine, Elsevier, vol. 163(C), pages 45-53.
    3. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    4. Embaye, Weldensie T. & Bergtold, Jason S. & Archer, David & Flora, Cornelia & Andrango, Graciela C. & Odening, Marting & Buysse, Jeroen, 2018. "Examining farmers' willingness to grow and allocate land for oilseed crops for biofuel production," Energy Economics, Elsevier, vol. 71(C), pages 311-320.
    5. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    6. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    7. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    8. Miguel Santolino & Magnus Söderberg, 2011. "The influence of decision-maker effort and case complexity on appealed rulings subject to multi-categorical selection," IREA Working Papers 201115, University of Barcelona, Research Institute of Applied Economics, revised Sep 2011.
    9. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    10. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    11. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    12. P.W. Miller & S. Rummery, 1989. "Gender Wage Discrimination in Australia: A reassessment," Economics Discussion / Working Papers 89-21, The University of Western Australia, Department of Economics.
    13. Nigel Driffield & Yong Yang, 2021. "Leveraging the benefits of location decisions into performance:A global view from matched MNEs," Working Papers 011, The Productivity Institute.
    14. Takashi Yamagata & Chris Orme, 2005. "On Testing Sample Selection Bias Under the Multicollinearity Problem," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 467-481.
    15. Spermann, Alexander & Strotmann, Harald, 2005. "The Targeted Negative Income Tax (TNIT) in Germany: Evidence from a Quasi Experiment," ZEW Discussion Papers 05-68, ZEW - Leibniz Centre for European Economic Research.
    16. Benjamin L. Collier & Andrew F. Haughwout & Howard C. Kunreuther & Erwann O. Michel‐Kerjan, 2020. "Firms’ Management of Infrequent Shocks," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(6), pages 1329-1359, September.
    17. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    18. Yang, Yong & Driffield, Nigel, 2022. "Leveraging the benefits of location decisions into performance: A global view from matched MNEs," Journal of Business Research, Elsevier, vol. 139(C), pages 468-483.
    19. Ian Gazeley & Rose Holmes & Andrew Newell & Kevin Reynolds & Hector Gutierrez Rufrancos, 2023. "Escaping from hunger before WW1: the nutritional transition and living standards in Western Europe and USA in the late nineteenth century," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(3), pages 533-565, September.
    20. Sumaryanto & Sri Hery Susilowati & Fitri Nurfatriani & Herlina Tarigan & Erwidodo & Tahlim Sudaryanto & Henri Wira Perkasa, 2022. "Determinants of Farmers’ Behavior towards Land Conservation Practices in the Upper Citarum Watershed in West Java, Indonesia," Land, MDPI, vol. 11(10), pages 1-21, October.

    More about this item

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:socarx:yazr8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://arabixiv.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.