Public Policy Evaluation: Introduction to Quantitative Methodologies
AbstractThis paper is a survey which describes and explains in non-technical terms the logic behind various methodologies used in conducting retrospective quantitative evaluations of public policy programs. The programs usually have as their main targets firms or individuals who benefit from direct subsidies and/or training. It is hypothesised that because of the technical nature of quantitative evaluations, some of the public officials to whom these evaluations are intended, may find them too complex to comprehend fully. Hence, those officials might disregard them up front, or form a biased opinion (positive or negative) or even accept the results on their face value. However, because all evaluations are subjective by definition, the public officials should have some basic knowledge on the logic behind the design and context of evaluations. Only then, can they judge themselves on their worth, and consequently decide to what degree they will take into account their findings and recommendations. The paper initially discusses the issues of accountability and causality and then introduces policy evaluation as a two phase process: First, estimations are made on the potential impact of the policy in question and then a judgement is passed on the worth of the impacts estimated, through a cost benefit analysis. The estimations in turn, comprise of two related areas: the design of the evaluation and the model specification. In designs, one has to consider whether counterfactual populations are included or not and whether the impact variables are in cross-sectional or longitudinal format. In model specifications the evaluator must decide which independent control variables he will include in the regression model so as to account for selection bias. In cost benefit analysis decisions have to be made as to whether the analysis will be made at partial equilibrium or general equilibrium level and whether the judgements formulated will be based purely on efficiency grounds or using just distributional criteria as well. The paper recommends among others, that (a) public policy evaluations should establish clear rules of causation between the public intervention and the potential impact measured, (b) limitations in the estimation and cost benefit analysis phase must be explicitly stated and (c) retrospective evaluations should be conducted at closer intervals after the end of the intervention so as to reduce the external heterogeneity generated due to the time lag between the results produced and the on-going programs.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Government Institute for Economic Research Finland (VATT) in its series Research Reports with number 90.
Date of creation: 31 Jul 2002
Date of revision:
Find related papers by JEL classification:
- B40 - Schools of Economic Thought and Methodology - - Economic Methodology - - - General
- D61 - Microeconomics - - Welfare Economics - - - Allocative Efficiency; Cost-Benefit Analysis
- C00 - Mathematical and Quantitative Methods - - General - - - General
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Jeffrey Smith, 2000.
"A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies,"
UWO Department of Economics Working Papers
20006, University of Western Ontario, Department of Economics.
- Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
- Devarajan, Shantayanan & Squire, Lyn & Suthiwart-Narueput, Sethaput, 1997. "Beyond Rate of Return: Reorienting Project Appraisal," World Bank Research Observer, World Bank Group, vol. 12(1), pages 35-46, February.
- James L. Heckman, 1999.
"Causal Parameters and Policy Analysis in Economcs: A Twentieth Century Retrospective,"
NBER Working Papers
7333, National Bureau of Economic Research, Inc.
- James J. Heckman, 2000. "Causal Parameters And Policy Analysis In Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, MIT Press, vol. 115(1), pages 45-97, February.
- Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M., 2001.
"Disentangling Treatment Effects of Polish Active Labor Market Policies: Evidence from Matched Samples,"
IZA Discussion Papers
355, Institute for the Study of Labor (IZA).
- Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M, 2002. "Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence from Matched Samples," CEPR Discussion Papers 3298, C.E.P.R. Discussion Papers.
- Jochen Kluve & Hartmut Lehmann & Christoph M. Schmidt, 2002. "Disentangling Treatment Effects of Polish Active Labor Market Policies: Evidence from Matched Samples," William Davidson Institute Working Papers Series 447, William Davidson Institute at the University of Michigan.
- Jochen Kluve & Hartmut Lehmann & Christoph M. Schmidt, 2000. "Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence from Matched Samples," CERT Discussion Papers 0007, Centre for Economic Reform and Transformation, Heriot Watt University.
- Mark Schreiner, 2001. "Evaluation and Microenterprise Programs," Development and Comp Systems 0108002, EconWPA, revised 27 Dec 2001.
- Takis Venetoklis, 2001. "Business Subsidies and Bureaucratic Behaviour," Research Reports 79, Government Institute for Economic Research Finland (VATT).
- Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
- A. Smith, Jeffrey & E. Todd, Petra, 2005.
"Does matching overcome LaLonde's critique of nonexperimental estimators?,"
Journal of Econometrics,
Elsevier, vol. 125(1-2), pages 305-353.
- Jeffrey Smith & Petra Todd, 2003. "Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20035, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
- Schmidt, Christoph M, 2001.
"Knowing What Works. The Case for Rigorous Programme Evaluation,"
CEPR Discussion Papers
2826, C.E.P.R. Discussion Papers.
- Schmidt, Christoph M., 1999. "Knowing What Works: The Case for Rigorous Program Evaluation," IZA Discussion Papers 77, Institute for the Study of Labor (IZA).
- Heckman, James J, 1979.
"Sample Selection Bias as a Specification Error,"
Econometric Society, vol. 47(1), pages 153-61, January.
- Rajeev H. Dehejia & Sadek Wahba, 2002.
"Propensity Score-Matching Methods For Nonexperimental Causal Studies,"
The Review of Economics and Statistics,
MIT Press, vol. 84(1), pages 151-161, February.
- Rajeev H. Dehejia & Sadek Wahba, 1998. "Propensity Score Matching Methods for Non-experimental Causal Studies," NBER Working Papers 6829, National Bureau of Economic Research, Inc.
- Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity score matching methods for non-experimental causal studies," Discussion Papers 0102-14, Columbia University, Department of Economics.
- Dehejia, R.H. & Wahba, S., 1998. "Propensity Score Matching Methods for Non-Experimental Causal Studies," Discussion Papers 1998_02, Columbia University, Department of Economics.
- Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
- McCloskey, Donald N, 1983. "The Rhetoric of Economics," Journal of Economic Literature, American Economic Association, vol. 21(2), pages 481-517, June.
- Heckman, James, 2001. "Accounting for Heterogeneity, Diversity and General Equilibrium in Evaluating Social Programmes," Economic Journal, Royal Economic Society, vol. 111(475), pages F654-99, November.
- LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
- repec:fth:coluec:9899-01 is not listed on IDEAS
- Richard Blundell & Monica Costa Dias, 2000. "Evaluation methods for non-experimental data," Fiscal Studies, Institute for Fiscal Studies, vol. 21(4), pages 427-468, January.
- Takis Venetoklis & Aki Kangasharju, 2002. "Business Subsidies and Employment of Firms: Overall Evaluation and Regional Extension," Discussion Papers 268, Government Institute for Economic Research Finland (VATT).
- Lechner, Michael, 1999. "Identification and Estimation of Causal Effects of Multiple Treatments Under the Conditional Independence Assumption," IZA Discussion Papers 91, Institute for the Study of Labor (IZA).
- James J. Heckman & Jeffrey A. Smith, 1998. "Evaluating the Welfare State," NBER Working Papers 6542, National Bureau of Economic Research, Inc.
- Anna Baranowska-Rataj & Anna Matysiak, 2012. "Czy znamy lekarstwo na nisk¹ dzietnoœæ? Wyniki miêdzynarodowych badañ ewaluacyjnych na temat polityki rodzinnej," Working Papers 47, Institute of Statistics and Demography, Warsaw School of Economics.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Anita Niskanen).
If references are entirely missing, you can add them using this form.