IDEAS home Printed from
   My bibliography  Save this article

Problems with Tests of the Missingness Mechanism in Quantitative Policy Studies


  • Rhoads Christopher H.

    (University of Connecticut)


Policy analysts involved in quantitative research have many options for handling missing data. The method chosen will often greatly influence the substantive policy conclusions that will be drawn from the data. The most frequent methods for handling missing data assume that the data are missing at random (MAR). The current paper notes that an omnibus, nonparametric test of the MAR assumption is impossible using the observed data alone. Nonetheless various purported tests of the missingness mechanism (including tests of MAR) appear in the literature. The current paper clarifies that all of these tests rely on some assumption that cannot be tested from the data. The paper notes that tests of the missingness mechanism are frequently misinterpreted and it clarifies the appropriate interpretation of such tests. Policy analysts are encouraged not to develop the false impression that modern procedures for handling missing data in conjunction with tests of the missingness mechanism provide protection against the ill effects of missing data. Any justification for a particular approach to handling missing data must be come from substantive knowledge of the missingness process, not from the data.

Suggested Citation

  • Rhoads Christopher H., 2012. "Problems with Tests of the Missingness Mechanism in Quantitative Policy Studies," Statistics, Politics and Policy, De Gruyter, vol. 3(1), pages 1-25, March.
  • Handle: RePEc:bpj:statpp:v:3:y:2012:i:1:p:25:n:1
    DOI: 10.1515/2151-7509.1012

    Download full text from publisher

    File URL:
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.

    File URL:
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Charles F. Manski, 1989. "Anatomy of the Selection Problem," Journal of Human Resources, University of Wisconsin Press, vol. 24(3), pages 343-360.
    2. William N. Evans & Matthew C. Farrelly, 1998. "The Compensating Behavior of Smokers: Taxes, Tar, and Nicotine," RAND Journal of Economics, The RAND Corporation, vol. 29(3), pages 578-595, Autumn.
    3. Libertad González, 2005. "Nonparametric bounds on the returns to language skills," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 20(6), pages 771-795.
    4. Christine E. Grella & Christy K. Scott & Mark A. Foss & Michael L. Dennis, 2008. "Gender Similarities and Differences in the Treatment, Relapse, and Recovery Cycle," Evaluation Review, , vol. 32(1), pages 113-137, February.
    5. Geert Molenberghs & Caroline Beunckens & Cristina Sotto & Michael G. Kenward, 2008. "Every missingness not at random model has a missingness at random counterpart with equal fit," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 70(2), pages 371-388, April.
    6. Buckler, Kevin & Unnever, James D., 2008. "Racial and ethnic perceptions of injustice: Testing the core hypotheses of comparative conflict theory," Journal of Criminal Justice, Elsevier, vol. 36(3), pages 270-278, July.
    7. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    8. Frieder R. Lang & Paul B. Baltes & Gert G. Wagner, 2007. "Desired Lifetime and End-of-Life Desires Across Adulthood From 20 to 90: A Dual-Source Information Model," Journals of Gerontology: Series B, Gerontological Society of America, vol. 62(5), pages 268-276.
    9. Daniel O. Scharfstein & Charles F. Manski & James C. Anthony, 2004. "On the Construction of Bounds in Prospective Studies with Missing Ordinal Outcomes: Application to the Good Behavior Game Trial," Biometrics, The International Biometric Society, vol. 60(1), pages 154-164, March.
    10. Hedeker, Donald, 1999. "MIXNO: a computer program for mixed-effects nominal logistic regression," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 4(i05).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antonio Di Paolo & Josep Lluís Raymond, 2012. "Language Knowledge and Earnings in Catalonia," Journal of Applied Economics, Taylor & Francis Journals, vol. 15(1), pages 89-118, May.
    2. van der Klaauw, Bas & Koning, Ruud H, 2003. "Testing the Normality Assumption in the Sample Selection Model with an Application to Travel Demand," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 31-42, January.
    3. Breunig, Christoph & Mammen, Enno & Simoni, Anna, 2018. "Nonparametric estimation in case of endogenous selection," Journal of Econometrics, Elsevier, vol. 202(2), pages 268-285.
    4. Guilhem Bascle, 2008. "Controlling for endogeneity with instrumental variables in strategic management research," Post-Print hal-00576795, HAL.
    5. Christelis, Dimitris & Messina, Julián, 2019. "Partial Identification of Population Average and Quantile Treatment Effects in Observational Data under Sample Selection," IDB Publications (Working Papers) 9520, Inter-American Development Bank.
    6. Ibáñez, Ana María & Muñoz, Juan Carlos & Verwimp, Philip, 2013. "Abandoning Coffee under the Threat of Violence and the Presence of Illicit Crops. Evidence from Colombia," Documentos CEDE Series 161356, Universidad de Los Andes, Economics Department.
    7. Claudia Olivetti & Barbara Petrongolo, 2008. "Unequal Pay or Unequal Employment? A Cross-Country Analysis of Gender Gaps," Journal of Labor Economics, University of Chicago Press, vol. 26(4), pages 621-654, October.
    8. Aizawa, T.;, 2019. "Reviewing the Existing Evidence of the Conditional Cash Transfer in India through the Partial Identification Approach," Health, Econometrics and Data Group (HEDG) Working Papers 19/24, HEDG, c/o Department of Economics, University of York.
    9. Panizza, Ugo & Qiang, Christine Zhen-Wei, 2005. "Public-private wage differential and gender gap in Latin America: Spoiled bureaucrats and exploited women?," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 34(6), pages 810-833, December.
    10. Harmon, Colm & Walker, Ian, 1995. "Estimates of the Economic Return to Schooling for the United Kingdom," American Economic Review, American Economic Association, vol. 85(5), pages 1278-1286, December.
    11. Rosalia Vazquez-Alvarez, 2003. "Anchoring Bias and Covariate Nonresponse," University of St. Gallen Department of Economics working paper series 2003 2003-19, Department of Economics, University of St. Gallen.
    12. Li, Phillip, 2011. "Estimation of sample selection models with two selection mechanisms," Computational Statistics & Data Analysis, Elsevier, vol. 55(2), pages 1099-1108, February.
    13. Marjan Petreski & Nikica Blazevski & Blagica Petreski, 2014. "Gender Wage Gap when Women are Highly Inactive: Evidence from Repeated Imputations with Macedonian Data," Journal of Labor Research, Springer, vol. 35(4), pages 393-411, December.
    14. Bo E. Honoré & Luojia Hu, 2020. "Selection Without Exclusion," Econometrica, Econometric Society, vol. 88(3), pages 1007-1029, May.
    15. Douglas L. Kruse, 1993. "Does Profit Sharing Affect Productivity?," NBER Working Papers 4542, National Bureau of Economic Research, Inc.
    16. Kaya, Ezgi, 2014. "Gender Wage Gap Trends in Europe: The Role of Occupational Allocation and Skill Prices," Cardiff Economics Working Papers E2014/23, Cardiff University, Cardiff Business School, Economics Section.
    17. Lindström, Hanna & Lundberg, Sofia & Marklund, Per-Olov, 2020. "How Green Public Procurement can drive conversion of farmland: An empirical analysis of an organic food policy," Ecological Economics, Elsevier, vol. 172(C).
    18. Adrián Rubli, 2012. "La importancia de corregir por el sesgo de selección en el análisis de las brechas salariales por género: un estudio para Argentina, Brasil y México," Ensayos Revista de Economia, Universidad Autonoma de Nuevo Leon, Facultad de Economia, vol. 0(2), pages 1-36, November.
    19. Verbeek, Marno & Nijman, Theo, 1992. "Testing for Selectivity Bias in Panel Data Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 33(3), pages 681-703, August.
    20. Fitzgerald John M, 2011. "Attrition in Models of Intergenerational Links Using the PSID with Extensions to Health and to Sibling Models," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 11(3), pages 1-63, September.

    More about this item


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:statpp:v:3:y:2012:i:1:p:25:n:1. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.