IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0144962.html
   My bibliography  Save this article

Evaluating Propensity Score Methods in a Quasi-Experimental Study of the Impact of Menu-Labeling

Author

Listed:
  • Stephanie L Mayne
  • Brian K Lee
  • Amy H Auchincloss

Abstract

Background: Quasi-experimental studies of menu labeling have found mixed results for improving diet. Differences between experimental groups can hinder interpretation. Propensity scores are an increasingly common method to improve covariate balance, but multiple methods exist and the improvements associated with each method have rarely been compared. In this re-analysis of the impact of menu labeling, we compare multiple propensity score methods to determine which methods optimize balance between experimental groups. Methods: Study participants included adult customers who visited full-service restaurants with menu labeling (treatment) and without (control). We compared the balance between treatment groups obtained by four propensity score methods: 1) 1:1 nearest neighbor matching (NN), 2) augmented 1:1 NN (using caliper of 0.2 and an exact match on an imbalanced covariate), 3) full matching, and 4) inverse probability weighting (IPW). We then evaluated the treatment effect on differences in nutrients purchased across the different methods. Results: 1:1 NN resulted in worse balance than the original unmatched sample (average standardized absolute mean distance [ASAM]: 0.185 compared to 0.171). Augmented 1:1 NN improved balance (ASAM: 0.038) but resulted in a large reduction in sample size. Full matching and IPW improved balance over the unmatched sample without a reduction in sample size (ASAM: 0.049 and 0.031, respectively). Menu labeling was associated with decreased calories, fat, sodium and carbohydrates in the unmatched analysis. Results were qualitatively similar in the propensity score matched/weighted models. Conclusions: While propensity scores offer an increasingly popular tool to improve causal inference, choosing the correct method can be challenging. Our results emphasize the benefit of examining multiple methods to ensure results are consistent, and considering approaches beyond the most popular method of 1:1 NN matching.

Suggested Citation

  • Stephanie L Mayne & Brian K Lee & Amy H Auchincloss, 2015. "Evaluating Propensity Score Methods in a Quasi-Experimental Study of the Impact of Menu-Labeling," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-12, December.
    • Handle: RePEc:plo:pone00:0144962
    DOI: 10.1371/journal.pone.0144962
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144962
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0144962&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0144962?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Denis Conniffe & Vanessa Gash & Philip J. O'Connell, 2000. "Evaluating State Programmes - “Natural Experiments” and Propensity Scores," The Economic and Social Review, Economic and Social Studies, vol. 31(4), pages 283-308.
    2. Ben B. Hansen, 2004. "Full Matching in an Observational Study of Coaching for the SAT," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 609-618, January.
    3. Peter C. Austin, 2009. "The Relative Ability of Different Propensity Score Methods to Balance Measured Covariates Between Treated and Untreated Subjects in Observational Studies," Medical Decision Making, , vol. 29(6), pages 661-677, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andre M. N. Renzaho & Stanley Chitekwe & Wen Chen & Sanjay Rijal & Thakur Dhakal & Pradiumna Dahal, 2017. "The Synergetic Effect of Cash Transfers for Families, Child Sensitive Social Protection Programs, and Capacity Building for Effective Social Protection on Children’s Nutritional Status in Nepal," IJERPH, MDPI, vol. 14(12), pages 1-22, December.
    2. Yahui Wang & Liangjie Xin & Xiubin Li & Jianzhong Yan, 2016. "Impact of Land Use Rights Transfer on Household Labor Productivity: A Study Applying Propensity Score Matching in Chongqing, China," Sustainability, MDPI, vol. 9(1), pages 1-18, December.
    3. Hebe N Gouda & Andrew Hodge & Raoul Bermejo III & Willibald Zeck & Eliana Jimenez-Soto, 2016. "The Impact of Healthcare Insurance on the Utilisation of Facility-Based Delivery for Childbirth in the Philippines," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-15, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gerry Boyle & Rory McElligott & Jim O'Leary, 2004. "Public-Private Wage Differentials in Ireland, 1994-2001," Economics Department Working Paper Series n1421004, Department of Economics, National University of Ireland - Maynooth.
    2. Anna Kovner & Chenyang Wei, 2012. "The private premium in public bonds," Staff Reports 553, Federal Reserve Bank of New York.
    3. Alexandre Chiavegatto Filho & Ichiro Kawachi, 2013. "Are sex-selective abortions a characteristic of every poor region? Evidence from Brazil," International Journal of Public Health, Springer;Swiss School of Public Health (SSPH+), vol. 58(3), pages 395-400, June.
    4. Zhenzhen Xu & John D. Kalbfleisch, 2013. "Repeated Randomization and Matching in Multi-Arm Trials," Biometrics, The International Biometric Society, vol. 69(4), pages 949-959, December.
    5. Raiden B. Hasegawa & Sameer K. Deshpande & Dylan S. Small & Paul R. Rosenbaum, 2020. "Causal Inference With Two Versions of Treatment," Journal of Educational and Behavioral Statistics, , vol. 45(4), pages 426-445, August.
    6. Reena Aggarwal & Isil Erel & René Stulz & Rohan Williamson, 2010. "Differences in Governance Practices between US and Foreign Firms: Measurement, Causes, and Consequences," NBER Chapters, in: Corporate Governance, National Bureau of Economic Research, Inc.
    7. Nolan, Anne, 2006. "Evaluating the Impact of Eligibility for Free Care on the Use of GP Services in Ireland: A Difference-in-Difference Matching Approach," Papers HRBWP25, Economic and Social Research Institute (ESRI).
    8. Orlando Jiménez, 2005. "Innovation-Oriented Environmental Regulations: Direct versus Indirect Regulations; an Empirical Analysis of Small and Medium-Sized Enterprises in Chile," Environment and Planning A, , vol. 37(4), pages 723-750, April.
    9. Kelly, Elish & McGuinness, Seamus & O'Connell, Philip J., 2011. "What Can Active Labour Market Policies Do?," Papers EC1, Economic and Social Research Institute (ESRI).
    10. Aggarwal, Reena & Erel, Isil & Stulz, Rene M. & Williamson, Rohan, 2006. "Do U.S. Firms Have the Best Corporate Governance? A Cross-Country Examination of the Relation between Corporate Governance and Shareholder Wealth," Working Paper Series 2006-25, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    11. Zhenzhen Xu & John D. Kalbfleisch, 2010. "Propensity Score Matching in Randomized Clinical Trials," Biometrics, The International Biometric Society, vol. 66(3), pages 813-823, September.
    12. Halpin, Brendan & Hill, John, 2008. "Active Labour Market Programmes and Poverty Dynamics in Ireland," MPRA Paper 10335, University Library of Munich, Germany.
    13. Lenis, David & Ackerman, Benjamin & Stuart, Elizabeth A., 2018. "Measuring model misspecification: Application to propensity score methods with complex survey data," Computational Statistics & Data Analysis, Elsevier, vol. 128(C), pages 48-57.
    14. McGuinness, Seamus & O'Connell, Philip J. & Kelly, Elish, 2011. "Carrots without Sticks: The Impacts of Job Search Assistance in a Regime with Minimal Monitoring and Sanctions," Papers WP409, Economic and Social Research Institute (ESRI).
    15. Alberto Abadie & Guido W. Imbens, 2012. "A Martingale Representation for Matching Estimators," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 833-843, June.
    16. Motoi Kusadokoro & Ai Hasegawa, 2017. "The Influence of Internal Migration on Migrant Children’s School Enrolment and Work in Turkey," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 29(2), pages 348-368, April.
    17. Jha, Anand & Cox, James, 2015. "Corporate social responsibility and social capital," Journal of Banking & Finance, Elsevier, vol. 60(C), pages 252-270.
    18. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    19. Loh, Wen Wei & Ren, Dongning, 2021. "Data-driven Covariate Selection for Confounding Adjustment by Focusing on the Stability of the Effect Estimator," OSF Preprints yve6u, Center for Open Science.
    20. Iacus, Stefano & Porro, Giuseppe, 2008. "Invariant and Metric Free Proximities for Data Matching: An R Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 25(i11).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0144962. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.