Public Policy Evaluation: Introduction to Quantitative Methodologies
This paper is a survey which describes and explains in non-technical terms the logic behind various methodologies used in conducting retrospective quantitative evaluations of public policy programs. The programs usually have as their main targets firms or individuals who benefit from direct subsidies and/or training. It is hypothesised that because of the technical nature of quantitative evaluations, some of the public officials to whom these evaluations are intended, may find them too complex to comprehend fully. Hence, those officials might disregard them up front, or form a biased opinion (positive or negative) or even accept the results on their face value. However, because all evaluations are subjective by definition, the public officials should have some basic knowledge on the logic behind the design and context of evaluations. Only then, can they judge themselves on their worth, and consequently decide to what degree they will take into account their findings and recommendations. The paper initially discusses the issues of accountability and causality and then introduces policy evaluation as a two phase process: First, estimations are made on the potential impact of the policy in question and then a judgement is passed on the worth of the impacts estimated, through a cost benefit analysis. The estimations in turn, comprise of two related areas: the design of the evaluation and the model specification. In designs, one has to consider whether counterfactual populations are included or not and whether the impact variables are in cross-sectional or longitudinal format. In model specifications the evaluator must decide which independent control variables he will include in the regression model so as to account for selection bias. In cost benefit analysis decisions have to be made as to whether the analysis will be made at partial equilibrium or general equilibrium level and whether the judgements formulated will be based purely on efficiency grounds or using just distributional criteria as well. The paper recommends among others, that (a) public policy evaluations should establish clear rules of causation between the public intervention and the potential impact measured, (b) limitations in the estimation and cost benefit analysis phase must be explicitly stated and (c) retrospective evaluations should be conducted at closer intervals after the end of the intervention so as to reduce the external heterogeneity generated due to the time lag between the results produced and the on-going programs.
|Date of creation:||31 Jul 2002|
|Date of revision:|
|Contact details of provider:|| Postal: |
Phone: +358 295 519 400
Fax: +358 295 519 599
Web page: http://www.vatt.fi/
More information through EDIRC
|Order Information:|| Email: |
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- A. Smith, Jeffrey & E. Todd, Petra, 2005.
"Does matching overcome LaLonde's critique of nonexperimental estimators?,"
Journal of Econometrics,
Elsevier, vol. 125(1-2), pages 305-353.
- Jeffrey Smith & Petra Todd, 2003. "Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20035, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
- Jochen Kluve & Hartmut Lehmann & Christoph M. Schmidt, 2000.
"Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence from Matched Samples,"
CERT Discussion Papers
0007, Centre for Economic Reform and Transformation, Heriot Watt University.
- Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M., 2001. "Disentangling Treatment Effects of Polish Active Labor Market Policies: Evidence from Matched Samples," IZA Discussion Papers 355, Institute for the Study of Labor (IZA).
- Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M, 2002. "Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence from Matched Samples," CEPR Discussion Papers 3298, C.E.P.R. Discussion Papers.
- Jochen Kluve & Hartmut Lehmann & Christoph M. Schmidt, 2002. "Disentangling Treatment Effects of Polish Active Labor Market Policies: Evidence from Matched Samples," William Davidson Institute Working Papers Series 447, William Davidson Institute at the University of Michigan.
- Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
- Rajeev H. Dehejia & Sadek Wahba, 1998.
"Propensity Score Matching Methods for Non-experimental Causal Studies,"
NBER Working Papers
6829, National Bureau of Economic Research, Inc.
- Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
- Jeffrey Smith, 2000.
"A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies,"
UWO Department of Economics Working Papers
20006, University of Western Ontario, Department of Economics.
- Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
- James J. Heckman & Jeffrey A. Smith, 1998. "Evaluating the Welfare State," NBER Working Papers 6542, National Bureau of Economic Research, Inc.
- James J. Heckman, 2000.
"Causal Parameters And Policy Analysis In Economics: A Twentieth Century Retrospective,"
The Quarterly Journal of Economics,
MIT Press, vol. 115(1), pages 45-97, February.
- James L. Heckman, 1999. "Causal Parameters and Policy Analysis in Economcs: A Twentieth Century Retrospective," NBER Working Papers 7333, National Bureau of Economic Research, Inc.
- Mark Schreiner, 2001. "Evaluation and Microenterprise Programs," Development and Comp Systems 0108002, EconWPA, revised 27 Dec 2001.
- Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
- Lechner, Michael, 1999. "Identification and Estimation of Causal Effects of Multiple Treatments Under the Conditional Independence Assumption," IZA Discussion Papers 91, Institute for the Study of Labor (IZA).
- LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
- Heckman, James J, 1979.
"Sample Selection Bias as a Specification Error,"
Econometric Society, vol. 47(1), pages 153-61, January.
- McCloskey, Donald N, 1983. "The Rhetoric of Economics," Journal of Economic Literature, American Economic Association, vol. 21(2), pages 481-517, June.
- Richard Blundell & Monica Costa Dias, 2000. "Evaluation methods for non-experimental data," Fiscal Studies, Institute for Fiscal Studies, vol. 21(4), pages 427-468, January.
- Devarajan, Shantayanan & Squire, Lyn & Suthiwart-Narueput, Sethaput, 1997. "Beyond Rate of Return: Reorienting Project Appraisal," World Bank Research Observer, World Bank Group, vol. 12(1), pages 35-46, February.
- Schmidt, Christoph M., 1999.
"Knowing What Works: The Case for Rigorous Program Evaluation,"
IZA Discussion Papers
77, Institute for the Study of Labor (IZA).
- Schmidt, Christoph M, 2001. "Knowing What Works. The Case for Rigorous Programme Evaluation," CEPR Discussion Papers 2826, C.E.P.R. Discussion Papers.
- Takis Venetoklis & Aki Kangasharju, 2002. "Business Subsidies and Employment of Firms: Overall Evaluation and Regional Extension," Discussion Papers 268, Government Institute for Economic Research Finland (VATT).
- Heckman, James, 2001. "Accounting for Heterogeneity, Diversity and General Equilibrium in Evaluating Social Programmes," Economic Journal, Royal Economic Society, vol. 111(475), pages F654-99, November.
- Takis Venetoklis, 2001. "Business Subsidies and Bureaucratic Behaviour," Research Reports 79, Government Institute for Economic Research Finland (VATT).
When requesting a correction, please mention this item's handle: RePEc:fer:resrep:90. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Anita Niskanen)
If references are entirely missing, you can add them using this form.