IDEAS home Printed from https://ideas.repec.org/a/wly/emetrp/v88y2020i1p265-296.html
   My bibliography  Save this article

Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis

Author

Listed:
  • Alberto Abadie
  • Susan Athey
  • Guido W. Imbens
  • Jeffrey M. Wooldridge

Abstract

Consider a researcher estimating the parameters of a regression function based on data for all 50 states in the United States or on data for all visits to a website. What is the interpretation of the estimated parameters and the standard errors? In practice, researchers typically assume that the sample is randomly drawn from a large population of interest and report standard errors that are designed to capture sampling variation. This is common even in applications where it is difficult to articulate what that population of interest is, and how it differs from the sample. In this article, we explore an alternative approach to inference, which is partly design‐based. In a design‐based setting, the values of some of the regressors can be manipulated, perhaps through a policy intervention. Design‐based uncertainty emanates from lack of knowledge about the values that the regression outcome would have taken under alternative interventions. We derive standard errors that account for design‐based uncertainty instead of, or in addition to, sampling‐based uncertainty. We show that our standard errors in general are smaller than the usual infinite‐population sampling‐based standard errors and provide conditions under which they coincide.

Suggested Citation

  • Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
  • Handle: RePEc:wly:emetrp:v:88:y:2020:i:1:p:265-296
    DOI: 10.3982/ECTA12675
    as

    Download full text from publisher

    File URL: https://doi.org/10.3982/ECTA12675
    Download Restriction: no

    File URL: https://libkey.io/10.3982/ECTA12675?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Charles F. Manski, 2013. "Response to the Review of ‘Public Policy in an Uncertain World’," Economic Journal, Royal Economic Society, vol. 0, pages 412-415, August.
    2. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    3. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    4. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    5. Manski, Charles F., 2013. "Public Policy in an Uncertain World: Analysis and Decisions," Economics Books, Harvard University Press, number 9780674066892, Spring.
    6. Alberto Abadie & Susan Athey & Guido Imbens & Jeffrey Wooldridge, 2017. "When Should You Adjust Standard Errors for Clustering?," Papers 1710.02926, arXiv.org, revised Oct 2017.
    7. Stefano DellaVigna & Attila Lindner & Balázs Reizer & Johannes F. Schmieder, 2017. "Reference-Dependent Job Search: Evidence from Hungary," The Quarterly Journal of Economics, Oxford University Press, vol. 132(4), pages 1969-2018.
    8. MacKinnon, James G. & White, Halbert, 1985. "Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties," Journal of Econometrics, Elsevier, vol. 29(3), pages 305-325, September.
    9. Micere Keels & Greg Duncan & Stefanie Deluca & Ruby Mendenhall & James Rosenbaum, 2005. "Fifteen years later: Can residential mobility programs provide a long-term escape from neighborhood segregation, crime, and poverty," Demography, Springer;Population Association of America (PAA), vol. 42(1), pages 51-73, February.
    10. Munnell, Alicia H. & Geoffrey M. B. Tootell & Lynn E. Browne & James McEneaney, 1996. "Mortgage Lending in Boston: Interpreting HMDA Data," American Economic Review, American Economic Association, vol. 86(1), pages 25-53, March.
    11. Joshua D. Angrist, 1998. "Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants," Econometrica, Econometric Society, vol. 66(2), pages 249-288, March.
    12. Rema Hanna & Sendhil Mullainathan & Joshua Schwartzstein, 2014. "Learning Through Noticing: Theory and Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 129(3), pages 1311-1353.
    13. Liran Einav & Amy Finkelstein & Paul Schrimpf, 2015. "The Response of Drug Expenditure to Nonlinear Contract Design: Evidence from Medicare Part D," The Quarterly Journal of Economics, Oxford University Press, vol. 130(2), pages 841-899.
    14. Charles F. Manski & John V. Pepper, 2018. "How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions," The Review of Economics and Statistics, MIT Press, vol. 100(2), pages 232-244, May.
    15. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    16. Lu, Jiannan, 2016. "On randomization-based and regression-based inferences for 2K factorial designs," Statistics & Probability Letters, Elsevier, vol. 112(C), pages 72-78.
    17. Peter M. Aronow & Cyrus Samii, 2016. "Does Regression Produce Representative Estimates of Causal Effects?," American Journal of Political Science, John Wiley & Sons, vol. 60(1), pages 250-267, January.
    18. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, October.
    19. White, Halbert, 1980. "Using Least Squares to Approximate Unknown Regression Functions," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 21(1), pages 149-170, February.
    20. Alberto Abadie & Guido W. Imbens & Fanyin Zheng, 2014. "Inference for Misspecified Models With Fixed Regressors," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1601-1614, December.
    21. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ryan Rafaty & Geoffroy Dolphin & Felix Pretis, 2020. "Carbon Pricing and the Elasticity of CO2 Emissions," Working Papers Series inetwp140, Institute for New Economic Thinking.
    2. Peter Bergman & Elizabeth Kopko & Julio E. Rodriguez, 2021. "Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment," NBER Working Papers 28948, National Bureau of Economic Research, Inc.
    3. Jonathan Roth & Pedro H. C. Sant'Anna, 2021. "Efficient Estimation for Staggered Rollout Designs," Papers 2102.01291, arXiv.org, revised Jun 2021.
    4. Martinez, Isabel Z., 2016. "Beggar-Thy-Neighbour Tax Cuts: Mobility after a Local Income and Wealth Tax Reform in Switzerland," Economics Working Paper Series 1608, University of St. Gallen, School of Economics and Political Science.
    5. Ding, Peng, 2021. "The Frisch–Waugh–Lovell theorem for standard errors," Statistics & Probability Letters, Elsevier, vol. 168(C).
    6. Sloczynski, Tymon, 2020. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," IZA Discussion Papers 13283, Institute of Labor Economics (IZA).
    7. Ganesh Karapakula & James J. Heckman, 2020. "Using a Satisficing Model of Experimenter Decision-Making to Guide Finite-Sample Inference for Compromised Experiments," Working Papers 2020-063, Human Capital and Economic Opportunity Working Group.
    8. Peng Ding, 2020. "The Frisch--Waugh--Lovell Theorem for Standard Errors," Papers 2009.06621, arXiv.org.
    9. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Aug 2020.
    10. Lea Bottmer & Guido Imbens & Jann Spiess & Merrill Warnick, 2021. "A Design-Based Perspective on Synthetic Control Methods," Papers 2101.09398, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2017. "Sampling-based vs. Design-based Uncertainty in Regression Analysis," Papers 1706.01778, arXiv.org, revised Jun 2019.
    2. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    3. Susan Athey & Raj Chetty & Guido Imbens, 2020. "Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes," Papers 2006.09676, arXiv.org.
    4. Richard H. Spady & Sami Stouli, 2018. "Simultaneous Mean-Variance Regression," Bristol Economics Discussion Papers 18/697, School of Economics, University of Bristol, UK.
    5. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    6. Ding, Peng, 2021. "The Frisch–Waugh–Lovell theorem for standard errors," Statistics & Probability Letters, Elsevier, vol. 168(C).
    7. Bruce E. Hansen & Seojeong Lee, 2021. "Inference for Iterated GMM Under Misspecification," Econometrica, Econometric Society, vol. 89(3), pages 1419-1447, May.
    8. Peng Ding, 2020. "The Frisch--Waugh--Lovell Theorem for Standard Errors," Papers 2009.06621, arXiv.org.
    9. Sloczynski, Tymon, 2020. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," IZA Discussion Papers 13283, Institute of Labor Economics (IZA).
    10. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 106, University of California, Davis, Department of Economics.
    11. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    12. Bryan S. Graham & Cristine Campos de Xavier Pinto, 2018. "Semiparametrically efficient estimation of the average linear regression function," Papers 1810.12511, arXiv.org.
    13. Joshua D. Angrist & Jörn-Steffen Pischke, 2017. "Undergraduate Econometrics Instruction: Through Our Classes, Darkly," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 125-144, Spring.
    14. Miguel A. Delgado & Thomas J. Kniesner, 1997. "Count Data Models With Variance Of Unknown Form: An Application To A Hedonic Model Of Worker Absenteeism," The Review of Economics and Statistics, MIT Press, vol. 79(1), pages 41-49, February.
    15. Charles, Amelie & Darne, Olivier, 2006. "Large shocks and the September 11th terrorist attacks on international stock markets," Economic Modelling, Elsevier, vol. 23(4), pages 683-698, July.
    16. Packalen, Mikko & Wirjanto, Tony S., 2012. "Inference about clustering and parametric assumptions in covariance matrix estimation," Computational Statistics & Data Analysis, Elsevier, vol. 56(1), pages 1-14, January.
    17. Binswanger, Johannes & Oechslin, Manuel, 2020. "Better statistics, better economic policies?," European Economic Review, Elsevier, vol. 130(C).
    18. P. Dorian Owen, 2017. "Evaluating Ingenious Instruments for Fundamental Determinants of Long-Run Economic Growth and Development," Econometrics, MDPI, Open Access Journal, vol. 5(3), pages 1-33, September.
    19. Matias D. Cattaneo & Michael Jansson & Whitney K. Newey, 2018. "Inference in Linear Regression Models with Many Covariates and Heteroscedasticity," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1350-1361, July.
    20. Alfredo Di Tillio & Marco Ottaviani & Peter Norman Sørensen, 2017. "Persuasion Bias in Science: Can Economics Help?," Economic Journal, Royal Economic Society, vol. 127(605), pages 266-304, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:emetrp:v:88:y:2020:i:1:p:265-296. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/essssea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/essssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.