IDEAS home Printed from https://ideas.repec.org/a/eee/ecoedu/v90y2022ics0272775722000826.html
   My bibliography  Save this article

Can learning be measured by phone? Evidence from Kenya

Author

Listed:
  • Rodriguez-Segura, Daniel
  • Schueler, Beth E.

Abstract

School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but should be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power. Finally, in cases when collecting in-person data is possible, this study illustrates that, depending on the logistical capabilities of the data collecting agency, PBAs do not always have a cost-effectiveness advantage over in-person data collection.

Suggested Citation

  • Rodriguez-Segura, Daniel & Schueler, Beth E., 2022. "Can learning be measured by phone? Evidence from Kenya," Economics of Education Review, Elsevier, vol. 90(C).
  • Handle: RePEc:eee:ecoedu:v:90:y:2022:i:c:s0272775722000826
    DOI: 10.1016/j.econedurev.2022.102309
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0272775722000826
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.econedurev.2022.102309?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Noam Angrist & Peter Bergman & David K. Evans & Susannah Hares & Matthew C. H. Jukes & Thato Letsomo, 2020. "Practical Lessons for Phone-Based Assessments of Learning," Working Papers 534, Center for Global Development, revised 10 Jul 2020.
    2. Noam Angrist & Peter Bergman & Moitshepi Matsheng, 2022. "Experimental evidence on learning using low-tech when school is out," Nature Human Behaviour, Nature, vol. 6(7), pages 941-950, July.
    3. Gourlay, Sydney & Kilic, Talip & Martuscelli, Antonio & Wollburg, Philip & Zezza, Alberto, 2021. "Viewpoint: High-frequency phone surveys on COVID-19: Good practices, open questions," Food Policy, Elsevier, vol. 105(C).
    4. Azevedo,Joao Pedro Wagner De, 2020. "Learning Poverty : Measures and Simulations," Policy Research Working Paper Series 9446, The World Bank.
    5. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1627-1673.
    6. Azevedo,Joao Pedro Wagner De & Goldemberg,Diana & Montoya,Silvia & Nayar,Reema & Rogers,F. Halsey & Saavedra,Jaime & Stacy,Brian William, 2021. "Will Every Child Be Able to Read by 2030 ? Defining Learning Poverty and Mapping the Dimensions of the Challenge," Policy Research Working Paper Series 9588, The World Bank.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Crawfurd, Lee & Evans, David K. & Hares, Susannah & Sandefur, Justin, 2023. "Live tutoring calls did not improve learning during the COVID-19 pandemic in Sierra Leone," Journal of Development Economics, Elsevier, vol. 164(C).
    2. Rodriguez-Segura, Daniel & Schueler, Beth E., 2023. "Assessors influence results: Evidence on enumerator effects and educational impact evaluations," Journal of Development Economics, Elsevier, vol. 163(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Allen, James & Mahumane, Arlete & Riddell, James & Rosenblat, Tanya & Yang, Dean & Yu, Hang, 2022. "Teaching and incentives: Substitutes or complements?," Economics of Education Review, Elsevier, vol. 91(C).
    2. Rodriguez-Segura, Daniel, 2022. "A closer look at reading comprehension: Experimental evidence from Guatemala," International Journal of Educational Development, Elsevier, vol. 93(C).
    3. Crawfurd, Lee & Evans, David K. & Hares, Susannah & Sandefur, Justin, 2023. "Live tutoring calls did not improve learning during the COVID-19 pandemic in Sierra Leone," Journal of Development Economics, Elsevier, vol. 164(C).
    4. Diether W Beuermann & C Kirabo Jackson & Laia Navarro-Sola & Francisco Pardo, 2023. "What is a Good School, and Can Parents Tell? Evidence on the Multidimensionality of School Output," Review of Economic Studies, Oxford University Press, vol. 90(1), pages 65-101.
    5. Emily Beam & Priya Mukherjee & Laia Navarro-Sola, 2022. "Lowering Barriers to Remote Education: Experimental Impacts on Parental Responses and Learning," Working Papers 2022-030, Human Capital and Economic Opportunity Working Group.
    6. Blimpo, Moussa P. & Pugatch, Todd, 2021. "Entrepreneurship education and teacher training in Rwanda," Journal of Development Economics, Elsevier, vol. 149(C).
    7. Mo, Di & Bai, Yu & Shi, Yaojiang & Abbey, Cody & Zhang, Linxiu & Rozelle, Scott & Loyalka, Prashant, 2020. "Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China," Journal of Development Economics, Elsevier, vol. 146(C).
    8. Yue-Yi Hwa & Clare Leaver, 2021. "Management in education systems," Oxford Review of Economic Policy, Oxford University Press, vol. 37(2), pages 367-391.
    9. Rodriguez-Segura, Daniel & Campton, Cole & Crouch, Luis & Slade, Timothy S., 2021. "Looking beyond changes in averages in evaluating foundational learning: Some inequality measures," International Journal of Educational Development, Elsevier, vol. 84(C).
    10. Joshua Brubaker & Talip Kilic & Philip Wollburg, 2021. "Representativeness of individual-level data in COVID-19 phone surveys: Findings from Sub-Saharan Africa," PLOS ONE, Public Library of Science, vol. 16(11), pages 1-27, November.
    11. Al-Samarrai, Samer & Shrestha, Unika & Hasan, Amer & Nakajima, Nozomi & Santoso, Santoso & Wijoyo, Wisnu Harto Adi, 2018. "Introducing a performance-based component into Jakarta's school grants: What do we know about its impact after three years?," Economics of Education Review, Elsevier, vol. 67(C), pages 110-136.
    12. Cardim, Joana & Molina-Millán, Teresa & Vicente, Pedro C., 2023. "Can technology improve the classroom experience in primary education? An African experiment on a worldwide program," Journal of Development Economics, Elsevier, vol. 164(C).
    13. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1627-1673.
    14. Alban Conto, Carolina & Akseer, Spogmai & Dreesen, Thomas & Kamei, Akito & Mizunoya, Suguru & Rigole, Annika, 2021. "Potential effects of COVID-19 school closures on foundational skills and Country responses for mitigating learning loss," International Journal of Educational Development, Elsevier, vol. 87(C).
    15. Sara Nadel and Lant Pritchett, 2016. "Searching for the Devil in the Details: Learning about Development Program Design," Working Papers 434, Center for Global Development.
    16. Barrera-Osorio, Felipe & Cilliers, Jacobus & Cloutier, Marie-Hélène & Filmer, Deon, 2022. "Heterogenous teacher effects of two incentive schemes: Evidence from a low-income country," Journal of Development Economics, Elsevier, vol. 156(C).
    17. Dang,Hai-Anh H. & Pullinger,John James & Serajuddin,Umar & Stacy,Brian William, 2024. "Reviewing Assessment Tools for Measuring Country Statistical Capacity," Policy Research Working Paper Series 10717, The World Bank.
    18. Élisé Wendlassida Miningou & Medjy Pierre‐Louis & Jean‐Marc Bernard, 2022. "Improving learning outcomes in francophone Africa: More resources or improved efficiency?," African Development Review, African Development Bank, vol. 34(1), pages 127-141, March.
    19. Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
    20. Gouveia, Catarina & Kalakou, Sofia & Cardoso-Grilo, Teresa, 2023. "How to forecast mental healthcare needs? Distinguishing between perceived and unperceived needs and their impact on capacity requirements," Socio-Economic Planning Sciences, Elsevier, vol. 87(PA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecoedu:v:90:y:2022:i:c:s0272775722000826. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/econedurev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.