IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v42y2018i5-6p575-615.html

The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment

Author

Listed:
  • Reuben Ford
  • Douwêrê Grékou
  • Isaac Kwakye
  • Taylor Shek-wai Hui

Abstract

Background: This article reports on the Future to Discover Project—a Canadian randomized controlled trial of two high school interventions—where data on key postsecondary enrollment outcomes were collected for two phases. During the initial phase, outcomes were recorded from administrative data and follow-up surveys. During the later phase, data came from administrative records only. Objectives: The article provides analyses that are informative about the consequences of a change from administrative-only data to survey-only data (and vice versa) for the estimation of impacts. Results: The change from administrative-only to survey-only data tended to produce apparent drops in postsecondary enrollment rates that varied by subgroup and education outcome. Nonetheless, levels and significance of impact with respect to postsecondary enrollment remained relatively stable. Conclusions: The findings of the article provide evidence that estimating education program impacts in the context of a randomized experiment can be relatively robust to the data sources chosen. They suggest that internal validity and conclusions for policy need not be affected by changing data sources even when the change produces marked changes in levels of the outcome of interest observed.

Suggested Citation

  • Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    • Handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615
    DOI: 10.1177/0193841X18799093
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18799093
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18799093?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. W. Norton Grubb, 1993. "The Varied Economic Returns to Postsecondary Education: New Evidence from the Class of 1972," Journal of Human Resources, University of Wisconsin Press, vol. 28(2), pages 365-382.
    2. Frenette, Marc, 2007. "Why Are Youth from Lower-income Families Less Likely to Attend University? Evidence from Academic Abilities, Parental Influences, and Financial Constraints," Analytical Studies Branch Research Paper Series 2007295e, Statistics Canada, Analytical Studies Branch.
    3. repec:mpr:mprres:1795 is not listed on IDEAS
    4. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    5. Richard Dorsett & Richard Hendra & Philip K. Robins, 2018. "Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration," Evaluation Review, , vol. 42(5-6), pages 491-514, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    2. Edith Yang & Richard Hendra, 2018. "The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City," Evaluation Review, , vol. 42(5-6), pages 550-574, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    2. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    3. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.
    4. Zeman, Klarka & Frenette, Marc, 2007. "Why Are Most University Students Women? Evidence Based on Academic Performance, Study Habits and Parental Influences," Analytical Studies Branch Research Paper Series 2007303e, Statistics Canada, Analytical Studies Branch.
    5. Stephanie Riegg Cellini & Nicholas Turner, 2019. "Gainfully Employed?: Assessing the Employment and Earnings of For-Profit College Students Using Administrative Data," Journal of Human Resources, University of Wisconsin Press, vol. 54(2), pages 342-370.
    6. Cellini, Stephanie Riegg, 2012. "For-Profit Higher Education: An Assessment of Costs and Benefits," National Tax Journal, National Tax Association;National Tax Journal, vol. 65(1), pages 153-179, March.
    7. Christopher Dougherty, 2005. "Why Are the Returns to Schooling Higher for Women than for Men?," Journal of Human Resources, University of Wisconsin Press, vol. 40(4), pages 969-988.
    8. D. Flannery & J. Cullinan, 2014. "Where they go, what they do and why it matters: the importance of geographic accessibility and social class for decisions relating to higher education institution type, degree level and field of study," Applied Economics, Taylor & Francis Journals, vol. 46(24), pages 2952-2965, August.
    9. Paul Frijters & Luo Chuliang & Xin Meng, 2012. "Child Education and the Family Income Gradient in China," Discussion Papers Series 470, School of Economics, University of Queensland, Australia.
    10. McGuinness, Seamus & Kelly, Elish & Pham, Thi Thu Phuong & Ha, Thi Thu Thuy & Whelan, Adele, 2021. "Returns to education in Vietnam: A changing landscape," World Development, Elsevier, vol. 138(C).
    11. Emanuelle Bourbeau & Pierre Lefebvre & Philip Merrigan, 2011. "Provincial Returns to Education for 21 to 35 year-olds: Results from the 1991-2006 Canadian Analytic Censuses Files," Cahiers de recherche 1106, CIRPEE.
    12. Arkes, Jeremy, 1999. "What Do Educational Credentials Signal and Why Do Employers Value Credentials?," Economics of Education Review, Elsevier, vol. 18(1), pages 133-141, February.
    13. Nicholas Trachter, 2011. "Option Value and Transitions in a Model of Postsecondary Education," EIEF Working Papers Series 1103, Einaudi Institute for Economics and Finance (EIEF), revised Jan 2011.
    14. Sumon Kumar Bhaumik & Ira N. Gang & Myeong-Su Yun, 2008. "Gender and Ethnicity in Post-Conflict Kosovo," WIDER Working Paper Series RP2008-43, World Institute for Development Economic Research (UNU-WIDER).
    15. Colin Busby, 2010. "Manitoba’s Demographic Challenge: Why Improving Aboriginal Education Outcomes Is Vital for Economic Prosperity," e-briefs 99, C.D. Howe Institute.
    16. Darwin Miller, 2007. "Isolating the Causal Impact of Community College Enrollment on Educational Attainment and Labor Market Outcomes in Texas," Discussion Papers 06-033, Stanford Institute for Economic Policy Research.
    17. Judith Scott-Clayton & Qiao Wen, 2019. "Estimating Returns to College Attainment: Comparing Survey and State Administrative Data–Based Estimates," Evaluation Review, , vol. 43(5), pages 266-306, October.
    18. Iván Bornacelly, 2013. "Educación técnica y tecnológica para la reducción de la desigualdad salarial y la pobreza," Revista Desarrollo y Sociedad, Universidad de los Andes,Facultad de Economía, CEDE.
    19. Rebecca Walcott & Isabelle Cohen & Denise Ferris, 2024. "When Who Matters: Interviewer Effects and Survey Modality," Evaluation Review, , vol. 48(6), pages 1024-1049, December.
    20. Norton Grubb, W., 1997. "The returns to education in the sub-baccalaureate labor market, 1984-1990," Economics of Education Review, Elsevier, vol. 16(3), pages 231-245, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.