IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Public Policy Evaluation: Introduction to Quantitative Methodologies

  • Takis Venetoklis

This paper is a survey which describes and explains in non-technical terms the logic behind various methodologies used in conducting retrospective quantitative evaluations of public policy programs. The programs usually have as their main targets firms or individuals who benefit from direct subsidies and/or training. It is hypothesised that because of the technical nature of quantitative evaluations, some of the public officials to whom these evaluations are intended, may find them too complex to comprehend fully. Hence, those officials might disregard them up front, or form a biased opinion (positive or negative) or even accept the results on their face value. However, because all evaluations are subjective by definition, the public officials should have some basic knowledge on the logic behind the design and context of evaluations. Only then, can they judge themselves on their worth, and consequently decide to what degree they will take into account their findings and recommendations. The paper initially discusses the issues of accountability and causality and then introduces policy evaluation as a two phase process: First, estimations are made on the potential impact of the policy in question and then a judgement is passed on the worth of the impacts estimated, through a cost benefit analysis. The estimations in turn, comprise of two related areas: the design of the evaluation and the model specification. In designs, one has to consider whether counterfactual populations are included or not and whether the impact variables are in cross-sectional or longitudinal format. In model specifications the evaluator must decide which independent control variables he will include in the regression model so as to account for selection bias. In cost benefit analysis decisions have to be made as to whether the analysis will be made at partial equilibrium or general equilibrium level and whether the judgements formulated will be based purely on efficiency grounds or using just distributional criteria as well. The paper recommends among others, that (a) public policy evaluations should establish clear rules of causation between the public intervention and the potential impact measured, (b) limitations in the estimation and cost benefit analysis phase must be explicitly stated and (c) retrospective evaluations should be conducted at closer intervals after the end of the intervention so as to reduce the external heterogeneity generated due to the time lag between the results produced and the on-going programs.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.vatt.fi/file/vatt_publication_pdf/t90.pdf
Download Restriction: no

File URL: http://www.vatt.fi/publications/latestPublications/publication/Publication_1345_id/448
Download Restriction: no

Paper provided by Government Institute for Economic Research Finland (VATT) in its series Research Reports with number 90.

as
in new window

Length:
Date of creation: 31 Jul 2002
Date of revision:
Handle: RePEc:fer:resrep:90
Contact details of provider: Postal: Arkadiankatu 7, P.O. Box 1279, FI-00101 Helsinki
Phone: +358 295 519 400
Fax: +358 295 519 599
Web page: http://www.vatt.fi/
Email:


More information through EDIRC

Order Information: Email:


References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Dehejia, R.H. & Wahba, S., 1998. "Propensity Score Matching Methods for Non-Experimental Causal Studies," Discussion Papers 1998_02, Columbia University, Department of Economics.
  2. Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M, 2002. "Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence from Matched Samples," CEPR Discussion Papers 3298, C.E.P.R. Discussion Papers.
  3. McCloskey, Donald N, 1983. "The Rhetoric of Economics," Journal of Economic Literature, American Economic Association, vol. 21(2), pages 481-517, June.
  4. Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
  5. Schmidt, Christoph M., 1999. "Knowing What Works: The Case for Rigorous Program Evaluation," IZA Discussion Papers 77, Institute for the Study of Labor (IZA).
  6. Takis Venetoklis, 2001. "Business Subsidies and Bureaucratic Behaviour," Research Reports 79, Government Institute for Economic Research Finland (VATT).
  7. Mark Schreiner, 2001. "Evaluation and Microenterprise Programs," Development and Comp Systems 0108002, EconWPA, revised 27 Dec 2001.
  8. Jeffrey Smith & Petra Todd, 2003. "Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20035, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
  9. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," UWO Department of Economics Working Papers 20006, University of Western Ontario, Department of Economics.
  10. Devarajan, Shantayanan & Squire, Lyn & Suthiwart-Narueput, Sethaput, 1997. "Beyond Rate of Return: Reorienting Project Appraisal," World Bank Research Observer, World Bank Group, vol. 12(1), pages 35-46, February.
  11. Lechner, Michael, 1999. "Identification and Estimation of Causal Effects of Multiple Treatments Under the Conditional Independence Assumption," IZA Discussion Papers 91, Institute for the Study of Labor (IZA).
  12. Heckman, James, 2001. "Accounting for Heterogeneity, Diversity and General Equilibrium in Evaluating Social Programmes," Economic Journal, Royal Economic Society, vol. 111(475), pages F654-99, November.
  13. James J. Heckman & Jeffrey A. Smith, 1998. "Evaluating the Welfare State," NBER Working Papers 6542, National Bureau of Economic Research, Inc.
  14. Takis Venetoklis & Aki Kangasharju, 2002. "Business Subsidies and Employment of Firms: Overall Evaluation and Regional Extension," Discussion Papers 268, Government Institute for Economic Research Finland (VATT).
  15. James J. Heckman, 2000. "Causal Parameters And Policy Analysis In Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, MIT Press, vol. 115(1), pages 45-97, February.
  16. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 31(3), pages 129-137.
  17. Richard Blundell & Monica Costa Dias, 2000. "Evaluation methods for non-experimental data," Fiscal Studies, Institute for Fiscal Studies, vol. 21(4), pages 427-468, January.
  18. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  19. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:fer:resrep:90. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Anita Niskanen)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.