IDEAS home Printed from https://ideas.repec.org/a/cup/polals/v14y2006i01p37-62_00.html
   My bibliography  Save this article

Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment

Author

Listed:
  • Arceneaux, Kevin
  • Gerber, Alan S.
  • Green, Donald P.

Abstract

In the social sciences, randomized experimentation is the optimal research design for establishing causation. However, for a number of practical reasons, researchers are sometimes unable to conduct experiments and must rely on observational data. In an effort to develop estimators that can approximate experimental results using observational data, scholars have given increasing attention to matching. In this article, we test the performance of matching by gauging the success with which matching approximates experimental results. The voter mobilization experiment presented here comprises a large number of observations (60,000 randomly assigned to the treatment group and nearly two million assigned to the control group) and a rich set of covariates. This study is analyzed in two ways. The first method, instrumental variables estimation, takes advantage of random assignment in order to produce consistent estimates. The second method, matching estimation, ignores random assignment and analyzes the data as though they were nonexperimental. Matching is found to produce biased results in this application because even a rich set of covariates is insufficient to control for preexisting differences between the treatment and control group. Matching, in fact, produces estimates that are no more accurate than those generated by ordinary least squares regression. The experimental findings show that brief paid get-out-the-vote phone calls do not increase turnout, while matching and regression show a large and significant effect.

Suggested Citation

  • Arceneaux, Kevin & Gerber, Alan S. & Green, Donald P., 2006. "Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment," Political Analysis, Cambridge University Press, vol. 14(1), pages 37-62, January.
  • Handle: RePEc:cup:polals:v:14:y:2006:i:01:p:37-62_00
    as

    Download full text from publisher

    File URL: https://www.cambridge.org/core/product/identifier/S1047198700001285/type/journal_article
    File Function: link to article abstract page
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Maureen A. Pirog & Anne L. Buffardi & Colleen K. Chrisinger & Pradeep Singh & John Briney, 2009. "Are the alternatives to randomized assignment nearly as good? Statistical corrections to nonrandomized evaluations," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 28(1), pages 169-172.
    2. Mariella Gonzales & Gianmarco León-Ciliotta & Luis R. Martínez, 2022. "How Effective Are Monetary Incentives to Vote? Evidence from a Nationwide Policy," American Economic Journal: Applied Economics, American Economic Association, vol. 14(1), pages 293-326, January.
    3. Kevin Arceneaux & Alan S. Gerber & Donald P. Green, 2010. "A Cautionary Note on the Use of Matching to Estimate Causal Effects: An Empirical Example Comparing Matching Estimates to an Experimental Benchmark," Sociological Methods & Research, , vol. 39(2), pages 256-282, November.
    4. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    5. David A. Freedman & Richard A. Berk, 2008. "Weighting Regressions by Propensity Scores," Evaluation Review, , vol. 32(4), pages 392-409, August.
    6. Meredith, Marc & Malhotra, Neil, 2008. "Can October Surprise? A Natural Experiment Assessing Late Campaign Effects," Research Papers 2002, Stanford University, Graduate School of Business.
    7. Christoph F. Kurz & Martin Rehm & Rolf Holle & Christina Teuner & Michael Laxy & Larissa Schwarzkopf, 2019. "The effect of bariatric surgery on health care costs: A synthetic control approach using Bayesian structural time series," Health Economics, John Wiley & Sons, Ltd., vol. 28(11), pages 1293-1307, November.
    8. Elsayed, Ahmed & Roushdy, Rania, 2017. "Empowering Women under Social Constraints: Evidence from a Field Intervention in Rural Egypt," IZA Discussion Papers 11240, Institute of Labor Economics (IZA).
    9. Larru, Jose Maria, 2007. "La evaluación de impacto: qué es, cómo se mide y qué está aportando en la cooperación al desarrollo [Impact Assessment and Evaluation: What it is it, how can it be measured and what it is adding to," MPRA Paper 6928, University Library of Munich, Germany.
    10. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    11. David A. Freedman, 2006. "Statistical Models for Causation," Evaluation Review, , vol. 30(6), pages 691-713, December.
    12. David A. Freedman, 2009. "Limits of Econometrics," International Econometric Review (IER), Econometric Research Association, vol. 1(1), pages 5-17, April.
    13. Ying Jin & Dominik Rothenhäusler, 2024. "Tailored inference for finite populations: conditional validity and transfer across distributions," Biometrika, Biometrika Trust, vol. 111(1), pages 215-233.
    14. Casey A. Klofstad & Benjamin G. Bishin, 2014. "Do Social Ties Encourage Immigrant Voters to Participate in Other Campaign Activities?," Social Science Quarterly, Southwestern Social Science Association, vol. 95(2), pages 295-310, June.
    15. Fukui Hideki, 2023. "Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation," Statistics, Politics and Policy, De Gruyter, vol. 14(2), pages 205-326, June.
    16. Fengshi Niu & Harsha Nori & Brian Quistorff & Rich Caruana & Donald Ngwe & Aadharsh Kannan, 2022. "Differentially Private Estimation of Heterogeneous Causal Effects," Papers 2202.11043, arXiv.org.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cup:polals:v:14:y:2006:i:01:p:37-62_00. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kirk Stebbing (email available below). General contact details of provider: https://www.cambridge.org/pan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.