IDEAS home Printed from
   My bibliography  Save this paper

Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?


  • Adesina, Adedoyin
  • Akogun, Oladele
  • Dillon, Andrew
  • Friedman, Jed
  • Njobdi, Sani
  • Serneels, Pieter


The replication of studies is foundational to the scientific method. But replication can have different meanings varying from results verification, to reproduction, to reanalysis with an alternative specification. Few studies are repeated in the same setting. Yet this may be key when environmental or other mediating factors change over time, as it may help establish validity over time or shed light on why impact changes over time. One example is the varied impact a health intervention can have, particularly when targeting infectious diseases, as disease incidence may vary across years. This paper presents the findings of a study that repeats the same health intervention in the same site in three different years, estimating the effect of malaria testing and treatment on agricultural worker earnings, labor supply and productivity. We find a significant impact on worker earnings across the years, but the impact size varies over time. The treatment on the treated estimates are lower in a year when the malaria prevalence rate is low. The ‘treatment on the medically untreated’, which captures an information and behavioral effect identified in an earlier study, is smaller in years when prevalence is lower and the possibility of substituting into and out of lower effort, lower return tasks is absent. These results underline the importance of changes in the prevalence rate as well as the worker’s labor constraints. The results demonstrate that repetition, apart from providing a useful tool for validation, can also help shed light on reasons why effects may vary over time.

Suggested Citation

  • Adesina, Adedoyin & Akogun, Oladele & Dillon, Andrew & Friedman, Jed & Njobdi, Sani & Serneels, Pieter, 2017. "Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?," 2018 Allied Social Sciences Association (ASSA) Annual Meeting, January 5-7, 2018, Philadelphia, Pennsylvania 266292, Agricultural and Applied Economics Association.
  • Handle: RePEc:ags:assa18:266292
    DOI: 10.22004/ag.econ.266292

    Download full text from publisher

    File URL:
    Download Restriction: no

    File URL:
    Download Restriction: no

    File URL:
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item

    Other versions of this item:

    References listed on IDEAS

    1. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    2. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009 is not listed on IDEAS
    3. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    4. Andrew Dillon & Jed Friedman & Pieter Serneels, 0. "Health Information, Treatment, and Worker Productivity," Journal of the European Economic Association, European Economic Association, vol. 19(2), pages 1077-1115.
    5. Carlos A. Flores & Oscar A. Mitnik, 2013. "Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1691-1707, December.
    6. Tessa Bold & Mwangi Kimenyi & Germano Mwabu & Alice Ng'ang'a & Justin Sandefur, 2013. "Scaling-up What Works: Experimental Evidence on External Validity in Kenyan Education," CSAE Working Paper Series 2013-04, Centre for the Study of African Economies, University of Oxford.
    7. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    8. Grosh, M.E. & Baker, J.L., 1995. "Proxy Means Tests for Targetting Social Programs. Simulations and Speculation," Papers 118, World Bank - Living Standards Measurement.
    9. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    10. Andrew Dillon & Jed Friedman & Pieter Serneels, 2014. "Health information, treatment, and worker productivity: Experimental evidence from malaria testing and treatment among Nigerian sugarcane cutters," Working Paper series, University of East Anglia, Centre for Behavioural and Experimental Social Science (CBESS) 14-05, School of Economics, University of East Anglia, Norwich, UK..
    11. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009.pdf is not listed on IDEAS
    12. Chima, Reginald Ikechukwu & Goodman, Catherine A. & Mills, Anne, 2003. "The economic impact of malaria in Africa: a critical review of the evidence," Health Policy, Elsevier, vol. 63(1), pages 17-36, January.
    13. Ahmed, Akhter U. & Bouis, Howarth E., 2002. "Weighing what's practical: proxy means tests for targeting food subsidies in Egypt," Food Policy, Elsevier, vol. 27(5-6), pages 519-540.
    14. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    15. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    16. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    17. repec:pri:rpdevs:instruments_of_development.pdf is not listed on IDEAS
    18. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    2. Akogun, Oladele & Dillon, Andrew & Friedman, Jed & Prasann, Ashesh & Serneels, Pieter, 2017. "Productivity and Health: Alternative Productivity Measures using Physical Activity," 2017 Annual Meeting, July 30-August 1, Chicago, Illinois 258380, Agricultural and Applied Economics Association.
    3. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    4. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    5. Akogun,Oladele B & Dillon,Andrew S. & Friedman,Jed & Prasann,Ashesh & Serneels,Pieter Maria & Akogun,Oladele B & Dillon,Andrew S. & Friedman,Jed & Prasann,Ashesh & Serneels,Pieter Maria, 2017. "Productivity and health : alternative productivity estimates using physical activity," Policy Research Working Paper Series 8228, The World Bank.
    6. Gauri, Varun & Jamison, Julian C. & Mazar, Nina & Ozier, Owen, 2021. "Motivating bureaucrats through social recognition: External validity—A tale of two states," Organizational Behavior and Human Decision Processes, Elsevier, vol. 163(C), pages 117-131.
    7. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    8. Gauri, Varun & Jamison, Julian C. & Mazar, Nina & Ozier, Owen, 2019. "Motivating Bureaucrats through Social Recognition: External Validity — A Tale of Two States," IZA Discussion Papers 12251, Institute of Labor Economics (IZA).
    9. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    10. Sebastian Galiani & Patrick J. McEwan & Brian Quistorff, 2017. "External and Internal Validity of a Geographic Quasi-Experiment Embedded in a Cluster-Randomized Experiment," Advances in Econometrics, in: Regression Discontinuity Designs, volume 38, pages 195-236, Emerald Group Publishing Limited.
    11. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    12. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    13. Bernard, Tanguy & Hidrobo, Melissa & Le Port, Agnès & Rawat, Rahul, 2017. "Nutrition incentives in dairy contract farming in northern Senegal," IFPRI discussion papers 1629, International Food Policy Research Institute (IFPRI).
    14. Florent Bedecarrats & Isabelle Guérin & François Roubaud, 2017. "L'étalon-or des évaluations randomisées : du discours de la méthode à l'économie politique," Working Papers ird-01445209, HAL.
    15. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    16. Schurer, Stefanie, 2014. "Bouncing Back from Health Shocks: Locus of Control, Labor Supply, and Mortality," IZA Discussion Papers 8203, Institute of Labor Economics (IZA).
    17. Anett John & Kate Orkin, 2022. "Can Simple Psychological Interventions Increase Preventive Health Investment?," Journal of the European Economic Association, European Economic Association, vol. 20(3), pages 1001-1047.
    18. Seuring, Till & Serneels, Pieter & Suhrcke, Marc, 2019. "The impact of diabetes on labour market outcomes in Mexico: A panel data and biomarker analysis," Social Science & Medicine, Elsevier, vol. 233(C), pages 252-261.
    19. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    20. Samar Khairy Ghanem, 2018. "The relationship between population and the environment and its impact on sustainable development in Egypt using a multi-equation model," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 20(1), pages 305-342, February.

    More about this item


    Research Methods/ Statistical Methods;


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ags:assa18:266292. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: AgEcon Search (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.