IDEAS home Printed from
   My bibliography  Save this article

Evaluating Social Policy by Experimental and Nonexperimental Methods


  • Bratberg, Espen
  • Grasdal, Astrid
  • Risa, Alf Erling


Although it is important to establish causal relationships in social policy evaluation, the effects are difficult to observe due to sample selection. To evaluate the performance of estimators designed to handle sample selection bias, we analyze data from a Norwegian rehabilitation project with a randomized experimental design. The data permit us to compare the performance of different nonexperimental estimators with the experimental results. In our case study we find that nonexperimental evaluation based on sample selection estimators with selection terms that fail to meet conventional levels of statistical significance is highly unreliable. The difference in difference estimator and propensity score matching estimators perform better in our context. Copyright 2002 by The editors of the Scandinavian Journal of Economics.

Suggested Citation

  • Bratberg, Espen & Grasdal, Astrid & Risa, Alf Erling, 2002. " Evaluating Social Policy by Experimental and Nonexperimental Methods," Scandinavian Journal of Economics, Wiley Blackwell, vol. 104(1), pages 147-171.
  • Handle: RePEc:bla:scandj:v:104:y:2002:i:1:p:147-71

    Download full text from publisher

    File URL:
    File Function: link to full text
    Download Restriction: Access to full text is restricted to subscribers.

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    1. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    2. John F. Ermisch & Marco Francesconi, 2001. "Family structure and children's achievements," Journal of Population Economics, Springer;European Society for Population Economics, vol. 14(2), pages 249-270.
    3. Bronars, Stephen G & Grogger, Jeff, 1994. "The Economic Consequences of Unwed Motherhood: Using Twin Births as a Natural Experiment," American Economic Review, American Economic Association, vol. 84(5), pages 1141-1156, December.
    4. John V. Pepper & Michael J. Brien & Gregory E. Loya, 2002. "Teenage childbearing and cognitive development," Journal of Population Economics, Springer;European Society for Population Economics, vol. 15(3), pages 391-416.
    5. Manski, C.F., 1992. "Identification Problems in the Social Sciences," Working papers 9217, Wisconsin Madison - Social Systems.
    6. Manski, C.F., 1990. "The Selection Problem," Working papers 90-12, Wisconsin Madison - Social Systems.
    7. John Ermisch & Marco Francesconi & David J. Pevalin, 2004. "Parental partnership and joblessness in childhood and their influence on young people's outcomes," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 167(1), pages 69-101.
    8. Sandra E. Black & Paul J. Devereux & Kjell G. Salvanes, 2005. "Why the Apple Doesn't Fall Far: Understanding Intergenerational Transmission of Human Capital," American Economic Review, American Economic Association, vol. 95(1), pages 437-449, March.
    9. Ribar, David C, 1994. "Teenage Fertility and High School Completion," The Review of Economics and Statistics, MIT Press, vol. 76(3), pages 413-424, August.
    10. Helena Holmlund, 2005. "Estimating Long-Term Consequences of Teenage Childbearing: An Examination of the Siblings Approach," Journal of Human Resources, University of Wisconsin Press, vol. 40(3).
    11. Kaplan, Greg & Goodman, Alissa & Walker, Ian, 2004. "Understanding the Effects of Early Motherhood in Britain: The Effects on Mothers," IZA Discussion Papers 1131, Institute for the Study of Labor (IZA).
    12. An, Chong-Bum & Haveman, Robert & Wolfe, Barbara, 1993. "Teen Out-of-Wedlock Births and Welfare Receipt: The Role of Childhood Events and Economic Circumstances," The Review of Economics and Statistics, MIT Press, vol. 75(2), pages 195-208, May.
    13. Arnaud Chevalier & Tarja K. Viitanen & Tarja K. Viitanen, 2003. "The long-run labour market consequences of teenage motherhood in Britain," Journal of Population Economics, Springer;European Society for Population Economics, vol. 16(2), pages 323-343, May.
    14. Daniel Klepinger & Shelly Lundberg & Robert Plotnick, 1999. "How Does Adolescent Fertility Affect the Human Capital and Wages of Young Women?," Journal of Human Resources, University of Wisconsin Press, vol. 34(3), pages 421-448.
    15. Marco Francesconi, 2008. "Adult Outcomes for Children of Teenage Mothers," Scandinavian Journal of Economics, Wiley Blackwell, vol. 110(1), pages 93-117, March.
    16. V. Joseph Hotz & Charles H. Mullin & Seth G. Sanders, 1997. "Bounding Causal Effects Using Data from a Contaminated Natural Experiment: Analysing the Effects of Teenage Childbearing," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 575-603.
    17. Manski, C.F. & Sandefur, G.D. & Mclanahan, S. & Powers, D., 1990. "Alternative Estimates Of The Effect Of Family Stucture During Adolescence On Hight School Graduation," Working papers 90-31, Wisconsin Madison - Social Systems.
    18. Robert Haveman & Barbara Wolfe, 1995. "The Determinants of Children's Attainments: A Review of Methods and Findings," Journal of Economic Literature, American Economic Association, vol. 33(4), pages 1829-1878, December.
    19. Ashenfelter, Orley & Krueger, Alan B, 1994. "Estimates of the Economic Returns to Schooling from a New Sample of Twins," American Economic Review, American Economic Association, vol. 84(5), pages 1157-1173, December.
    20. Rosenzweig, Mark R & Wolpin, Kenneth I, 1980. "Testing the Quantity-Quality Fertility Model: The Use of Twins as a Natural Experiment," Econometrica, Econometric Society, vol. 48(1), pages 227-240, January.
    21. Currie, Janet & Cole, Nancy, 1993. "Welfare and Child Health: The Link between AFDC Participation and Birth Weight," American Economic Review, American Economic Association, vol. 83(4), pages 971-985, September.
    22. Ermisch, John & Pevalin, David J., 2003. "Does a 'teen-birth' have longer-term impacts on the mother? evidence from the 1970 British Cohort Study," ISER Working Paper Series 2003-28, Institute for Social and Economic Research.
    23. Strauss, John, 1990. "Households, Communities, and Preschool Children's Nutrition Outcomes: Evidence from Rural Cote d'Ivoire," Economic Development and Cultural Change, University of Chicago Press, vol. 38(2), pages 231-261, January.
    24. Arline T. Geronimus & Sanders Korenman, 1992. "The Socioeconomic Consequences of Teen Childbearing Reconsidered," The Quarterly Journal of Economics, Oxford University Press, vol. 107(4), pages 1187-1214.
    25. Rosenzweig, Mark R, 1986. "Birth Spacing and Sibling Inequality: Asymmetric Information within the Family," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 27(1), pages 55-76, February.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Astrid Grasdal, 2001. "The performance of sample selection estimators to control for attrition bias," Health Economics, John Wiley & Sons, Ltd., vol. 10(5), pages 385-398.
    2. Fredriksson, Per G. & Millimet, D.L.Daniel L., 2004. "Comparative politics and environmental taxation," Journal of Environmental Economics and Management, Elsevier, vol. 48(1), pages 705-722, July.
    3. Hoken, Hisatoshi & Su, Qun, 2015. "Measuring the effect of agricultural cooperatives on household income using PSM-DID : a case study of a rice-producing cooperative in China," IDE Discussion Papers 539, Institute of Developing Economies, Japan External Trade Organization(JETRO).
    4. repec:enr:rpaper:0004 is not listed on IDEAS
    5. List, John A. & McHone, W. Warren & Millimet, Daniel L., 2004. "Effects of environmental regulation on foreign and domestic plant births: is there a home field advantage?," Journal of Urban Economics, Elsevier, vol. 56(2), pages 303-326, September.
    6. Maasoumi, Esfandiar & Millimet, Daniel & Sarkar, Dipanwita, 2005. "The Distribution of Returns to Marriage," Departmental Working Papers 0503, Southern Methodist University, Department of Economics.
    7. Slottje, Daniel J. & Millimet, Daniel L. & Buchanan, Michael J., 2007. "Econometric analysis of copyrights," Journal of Econometrics, Elsevier, vol. 139(2), pages 303-317, August.
    8. Hoken, Hisatoshi, 2016. "Participation in farmer's cooperatives and its effects on agricultural incomes : evidence from vegetable-producing areas in China," IDE Discussion Papers 578, Institute of Developing Economies, Japan External Trade Organization(JETRO).
    9. List John A. & Millimet Daniel L & McHone Warren, 2004. "The Unintended Disincentive in the Clean Air Act," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 3(2), pages 1-28, February.
    10. repec:sgh:gosnar:y:2017:i:5:p:105-127 is not listed on IDEAS
    11. Steven Stern & John Pepper & David Dean & Robert Schmidt, 2011. "The Effects of Vocational Rehabilitation for People with Mental Illlness," Virginia Economics Online Papers 382, University of Virginia, Department of Economics.
    12. Bakhshi, Hasan & Edwards, John S. & Roper, Stephen & Scully, Judy & Shaw, Duncan & Morley, Lorraine & Rathbone, Nicola, 2015. "Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits," Research Policy, Elsevier, vol. 44(8), pages 1462-1472.

    More about this item

    JEL classification:

    • H30 - Public Economics - - Fiscal Policies and Behavior of Economic Agents - - - General
    • C44 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Operations Research; Statistical Decision Theory
    • C82 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Macroeconomic Data; Data Access


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:scandj:v:104:y:2002:i:1:p:147-71. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing) or (Christopher F. Baum). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.