IDEAS home Printed from https://ideas.repec.org/a/wly/hlthec/v27y2018i6p984-1010.html
   My bibliography  Save this article

Alternative evaluation metrics for risk adjustment methods

Author

Listed:
  • Sungchul Park
  • Anirban Basu

Abstract

Risk adjustment is instituted to counter risk selection by accurately equating payments with expected expenditures. Traditional risk‐adjustment methods are designed to estimate accurate payments at the group level. However, this generates residual risks at the individual level, especially for high‐expenditure individuals, thereby inducing health plans to avoid those with high residual risks. To identify an optimal risk‐adjustment method, we perform a comprehensive comparison of prediction accuracies at the group level, at the tail distributions, and at the individual level across 19 estimators: 9 parametric regression, 7 machine learning, and 3 distributional estimators. Using the 2013–2014 MarketScan database, we find that no one estimator performs best in all prediction accuracies. Generally, machine learning and distribution‐based estimators achieve higher group‐level prediction accuracy than parametric regression estimators. However, parametric regression estimators show higher tail distribution prediction accuracy and individual‐level prediction accuracy, especially at the tails of the distribution. This suggests that there is a trade‐off in selecting an appropriate risk‐adjustment method between estimating accurate payments at the group level and lower residual risks at the individual level. Our results indicate that an optimal method cannot be determined solely on the basis of statistical metrics but rather needs to account for simulating plans' risk selective behaviors.

Suggested Citation

  • Sungchul Park & Anirban Basu, 2018. "Alternative evaluation metrics for risk adjustment methods," Health Economics, John Wiley & Sons, Ltd., vol. 27(6), pages 984-1010, June.
  • Handle: RePEc:wly:hlthec:v:27:y:2018:i:6:p:984-1010
    DOI: 10.1002/hec.3657
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/hec.3657
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gilleskie, Donna B. & Mroz, Thomas A., 2004. "A flexible approach for estimating the effects of covariates on health expenditures," Journal of Health Economics, Elsevier, vol. 23(2), pages 391-418, March.
    2. Jason Brown & Mark Duggan & Ilyana Kuziemko & William Woolston, 2014. "How Does Risk Selection Respond to Risk Adjustment? New Evidence from the Medicare Advantage Program," American Economic Review, American Economic Association, vol. 104(10), pages 3335-3364, October.
    3. Victor Chernozhukov & Iván Fernández‐Val & Blaise Melly, 2013. "Inference on Counterfactual Distributions," Econometrica, Econometric Society, vol. 81(6), pages 2205-2268, November.
    4. Andrew M. Jones & James Lomas & Nigel Rice, 2014. "Applying Beta‐Type Size Distributions To Healthcare Cost Regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 649-670, June.
    5. McDonald, James B, 1984. "Some Generalized Functions for the Size Distribution of Income," Econometrica, Econometric Society, vol. 52(3), pages 647-663, May.
    6. Randall P. Ellis & Denzil G. Fiebig & Meliyanni Johar & Glenn Jones & Elizabeth Savage, 2013. "Explaining Health Care Expenditure Variation: Large‐Sample Evidence Using Linked Survey And Health Administrative Data," Health Economics, John Wiley & Sons, Ltd., vol. 22(9), pages 1093-1110, September.
    7. Jones, A. M. & Lomas, J. & Moore, P. & Rice, N., 2013. "A quasi-Monte Carlo comparison of developments in parametric and semi-parametric regression methods for heavy tailed and non-normal data: with an application to healthcare costs," Health, Econometrics and Data Group (HEDG) Working Papers 13/30, HEDG, c/o Department of Economics, University of York.
    8. Thomas G. McGuire & Jacob Glazer, 2000. "Optimal Risk Adjustment in Markets with Adverse Selection: An Application to Managed Care," American Economic Review, American Economic Association, vol. 90(4), pages 1055-1071, September.
    9. Newhouse, Joseph P. & McWilliams, J. Michael & Price, Mary & Huang, Jie & Fireman, Bruce & Hsu, John, 2013. "Do Medicare Advantage plans select enrollees in higher margin clinical categories?," Journal of Health Economics, Elsevier, vol. 32(6), pages 1278-1288.
    10. Andrew M. Jones & James Lomas & Nigel Rice, 2015. "Healthcare Cost Regressions: Going Beyond the Mean to Estimate the Full Distribution," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1192-1212, September.
    11. Andrew M. Jones & James Lomas & Peter T. Moore & Nigel Rice, 2016. "A quasi-Monte-Carlo comparison of parametric and semiparametric regression methods for heavy-tailed and non-normal data: an application to healthcare costs," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 951-974, October.
    12. Van de ven, Wynand P.M.M. & Ellis, Randall P., 2000. "Risk adjustment in competitive health plan markets," Handbook of Health Economics, in: A. J. Culyer & J. P. Newhouse (ed.), Handbook of Health Economics, edition 1, volume 1, chapter 14, pages 755-845, Elsevier.
    13. Partha Deb & James F. Burgess, Jr., 2003. "A Quasi-experimental Comparison of Econometric Models for Health Care Expenditures," Economics Working Paper Archive at Hunter College 212, Hunter College Department of Economics.
    14. Han, Aaron & Hausman, Jerry A, 1990. "Flexible Parametric Estimation of Duration and Competing Risk Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(1), pages 1-28, January-M.
    15. McDonald, James B. & Xu, Yexiao J., 1995. "A generalization of the beta distribution with applications," Journal of Econometrics, Elsevier, vol. 69(2), pages 427-428, October.
    16. Joseph P. Newhouse & Mary Price & John Hsu & J. Michael McWilliams & Thomas G. McGuire, 2015. "How Much Favorable Selection Is Left in Medicare Advantage?," American Journal of Health Economics, University of Chicago Press, vol. 1(1), pages 1-26, Winter.
    17. Deb, Partha & Trivedi, Pravin K, 1997. "Demand for Medical Care by the Elderly: A Finite Mixture Approach," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 12(3), pages 313-336, May-June.
    18. McGuire, Thomas G. & Newhouse, Joseph P. & Normand, Sharon-Lise & Shi, Julie & Zuvekas, Samuel, 2014. "Assessing incentives for service-level selection in private health insurance exchanges," Journal of Health Economics, Elsevier, vol. 35(C), pages 47-63.
    19. Liran Einav & Amy Finkelstein & Raymond Kluender & Paul Schrimpf, 2016. "Beyond Statistics: The Economic Content of Risk Scores," American Economic Journal: Applied Economics, American Economic Association, vol. 8(2), pages 195-224, April.
    20. James J. Heckman, 2001. "Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture," Journal of Political Economy, University of Chicago Press, vol. 109(4), pages 673-748, August.
    21. Dimitris Bertsimas & Margrét V. Bjarnadóttir & Michael A. Kane & J. Christian Kryder & Rudra Pandey & Santosh Vempala & Grant Wang, 2008. "Algorithmic Prediction of Health-Care Costs," Operations Research, INFORMS, vol. 56(6), pages 1382-1392, December.
    22. de Meijer, Claudine & O’Donnell, Owen & Koopmanschap, Marc & van Doorslaer, Eddy, 2013. "Health expenditure growth: Looking beyond the average through decomposition of the full distribution," Journal of Health Economics, Elsevier, vol. 32(1), pages 88-105.
    23. Zhang, Guoqiang & Eddy Patuwo, B. & Y. Hu, Michael, 1998. "Forecasting with artificial neural networks:: The state of the art," International Journal of Forecasting, Elsevier, vol. 14(1), pages 35-62, March.
    24. Sungchul Park & Anirban Basu & Norma Coe & Fahad Khalil, 2017. "Service-level Selection: Strategic Risk Selection in Medicare Advantage in Response to Risk Adjustment," NBER Working Papers 24038, National Bureau of Economic Research, Inc.
    25. Steven C. Hill & G. Edward Miller, 2010. "Health expenditure estimation and functional form: applications of the generalized gamma and extended estimating equations models," Health Economics, John Wiley & Sons, Ltd., vol. 19(5), pages 608-627, May.
    26. Anirban Basu & Bhakti V. Arondekar & Paul J. Rathouz, 2006. "Scale of interest versus scale of estimation: comparing alternative estimators for the incremental costs of a comorbidity," Health Economics, John Wiley & Sons, Ltd., vol. 15(10), pages 1091-1107, October.
    27. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    28. Timothy J. Layton & Randall P. Ellis & Thomas G. McGuire, 2015. "Assessing Incentives for Adverse Selection in Health Plan Payment Systems," NBER Working Papers 21531, National Bureau of Economic Research, Inc.
    29. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anna Zink & Sherri Rose, 2020. "Fair regression for health care spending," Biometrics, The International Biometric Society, vol. 76(3), pages 973-982, September.
    2. Bergquist, Savannah L. & Layton, Timothy J. & McGuire, Thomas G. & Rose, Sherri, 2019. "Data transformations to improve the performance of health plan payment methods," Journal of Health Economics, Elsevier, vol. 66(C), pages 195-207.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrew M. Jones & James Lomas & Nigel Rice, 2015. "Healthcare Cost Regressions: Going Beyond the Mean to Estimate the Full Distribution," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1192-1212, September.
    2. Jones, A. & Lomas, J. & Rice, N., 2014. "Going Beyond the Mean in Healthcare Cost Regressions: a Comparison of Methods for Estimating the Full Conditional Distribution," Health, Econometrics and Data Group (HEDG) Working Papers 14/26, HEDG, c/o Department of Economics, University of York.
    3. Andrew M. Jones & James Lomas & Peter T. Moore & Nigel Rice, 2016. "A quasi-Monte-Carlo comparison of parametric and semiparametric regression methods for heavy-tailed and non-normal data: an application to healthcare costs," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 951-974, October.
    4. Sriubaite, I. & Harris, A. & Jones, A.M. & Gabbe, B., 2020. "Economic Consequences of Road Traffic Injuries. Application of the Super Learner algorithm," Health, Econometrics and Data Group (HEDG) Working Papers 20/20, HEDG, c/o Department of Economics, University of York.
    5. Jones, A.M, 2010. "Models For Health Care," Health, Econometrics and Data Group (HEDG) Working Papers 10/01, HEDG, c/o Department of Economics, University of York.
    6. Yi Yao & Joan Schmit & Julie Shi, 2019. "Promoting sustainability for micro health insurance: a risk-adjusted subsidy approach for maternal healthcare service," The Geneva Papers on Risk and Insurance - Issues and Practice, Palgrave Macmillan;The Geneva Association, vol. 44(3), pages 382-409, July.
    7. Timothy J. Layton & Randall P. Ellis & Thomas G. McGuire, 2015. "Assessing Incentives for Adverse Selection in Health Plan Payment Systems," Boston University - Department of Economics - Working Papers Series wp2015-024, Boston University - Department of Economics.
    8. Sungchul Park & Anirban Basu & Norma Coe & Fahad Khalil, 2017. "Service-level Selection: Strategic Risk Selection in Medicare Advantage in Response to Risk Adjustment," NBER Working Papers 24038, National Bureau of Economic Research, Inc.
    9. Borislava Mihaylova & Andrew Briggs & Anthony O'Hagan & Simon G. Thompson, 2011. "Review of statistical methods for analysing healthcare resources and costs," Health Economics, John Wiley & Sons, Ltd., vol. 20(8), pages 897-916, August.
    10. Andrew M. Jones & James Lomas & Nigel Rice, 2014. "Applying Beta‐Type Size Distributions To Healthcare Cost Regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 649-670, June.
    11. Julie Shi & Yi Yao & Gordon Liu, 2018. "Modeling individual health care expenditures in China: Evidence to assist payment reform in public insurance," Health Economics, John Wiley & Sons, Ltd., vol. 27(12), pages 1945-1962, December.
    12. Pilny, Adam & Wübker, Ansgar & Ziebarth, Nicolas R., 2017. "Introducing risk adjustment and free health plan choice in employer-based health insurance: Evidence from Germany," Journal of Health Economics, Elsevier, vol. 56(C), pages 330-351.
    13. Decarolis, Francesco & Guglielmo, Andrea, 2017. "Insurers’ response to selection risk: Evidence from Medicare enrollment reforms," Journal of Health Economics, Elsevier, vol. 56(C), pages 383-396.
    14. Besstremyannaya, Galina, 2017. "Measuring income equity in the demand for healthcare with finite mixture models," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 46, pages 5-29.
    15. Keith M. Marzilli Ericson & Kimberley H. Geissler & Benjamin Lubin, 2018. "The Impact of Partial-Year Enrollment on the Accuracy of Risk-Adjustment Systems: A Framework and Evidence," American Journal of Health Economics, MIT Press, vol. 4(4), pages 454-478, Fall.
    16. Dosis, Anastasios, 2019. "Optimal ex post risk adjustment in markets with adverse selection," Journal of Mathematical Economics, Elsevier, vol. 85(C), pages 52-59.
    17. Erengul Dodd & George Streftaris, 2017. "Prediction of settlement delay in critical illness insurance claims by using the generalized beta of the second kind distribution," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 66(2), pages 273-294, February.
    18. Timothy J. Layton & Thomas G. McGuire, 2017. "Marketplace Plan Payment Options for Dealing with High-Cost Enrollees," American Journal of Health Economics, University of Chicago Press, vol. 3(2), pages 165-191, Spring.
    19. Michael Geruso & Timothy J. Layton, 2017. "Selection in Health Insurance Markets and Its Policy Remedies," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 23-50, Fall.
    20. Joseph P. Newhouse & Mary Beth Landrum & Mary Price & J. Michael McWilliams & John Hsu & Thomas G. McGuire, 2019. "The Comparative Advantage of Medicare Advantage," American Journal of Health Economics, University of Chicago Press, vol. 5(2), pages 281-301, Spring.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:hlthec:v:27:y:2018:i:6:p:984-1010. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley Content Delivery). General contact details of provider: http://www3.interscience.wiley.com/cgi-bin/jhome/5749 .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.