IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v152y2020ics0167947320301341.html

A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics

Author

Listed:
  • Baak, M.
  • Koopman, R.
  • Snoek, H.
  • Klous, S.

Abstract

A prescription is presented for a new and practical correlation coefficient, ϕK, based on several refinements to Pearson’s hypothesis test of independence of two variables. The combined features of ϕK form an advantage over existing coefficients. Primarily, it works consistently between categorical, ordinal and interval variables, in essence by treating each variable as categorical, and can therefore be used to calculate correlations between variables of mixed type. Second, it captures nonlinear dependency. The strength of ϕK is similar to Pearson’s correlation coefficient, and is equivalent in case of a bivariate normal input distribution. These are useful properties when studying the correlations between variables with mixed types, where some are categorical. Two more innovations are presented: to the proper evaluation of statistical significance of correlations, and to the interpretation of variable relationships in a contingency table, in particular in case of sparse or low statistics samples and significant dependencies. Two practical applications are discussed. The presented algorithms are easy to use and available through a public Python library.11https://github.com/KaveIO/PhiK.

Suggested Citation

  • Baak, M. & Koopman, R. & Snoek, H. & Klous, S., 2020. "A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics," Computational Statistics & Data Analysis, Elsevier, vol. 152(C).
    • Handle: RePEc:eee:csdana:v:152:y:2020:i:c:s0167947320301341
    DOI: 10.1016/j.csda.2020.107043
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947320301341
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2020.107043?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. P. M. Kroonenberg & Albert Verbeek, 2018. "The Tale of Cochran's Rule: My Contingency Table has so Many Expected Values Smaller than 5, What Am I to Do?," The American Statistician, Taylor & Francis Journals, vol. 72(2), pages 175-183, April.
    2. Garrido, J. & Genest, C. & Schulz, J., 2016. "Generalized linear models for dependent frequency and severity of insurance claims," Insurance: Mathematics and Economics, Elsevier, vol. 70(C), pages 205-215.
    3. W. M. Patefield, 1981. "An Efficient Method of Generating Random R × C Tables with Given Row and Column Totals," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 30(1), pages 91-97, March.
    4. Yoo, Yeawon & Escobedo, Adolfo R. & Skolfield, J. Kyle, 2020. "A new correlation coefficient for comparing and aggregating non-strict and incomplete rankings," European Journal of Operational Research, Elsevier, vol. 285(3), pages 1025-1041.
    5. Kim, Donguk & Agresti, Alan, 1997. "Nearly exact tests of conditional independence and marginal homogeneity for sparse contingency tables," Computational Statistics & Data Analysis, Elsevier, vol. 24(1), pages 89-104, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chen, Zongqi & Gong, Feng & Song, Jiaming & Zhang, Kai, 2025. "Interpretable prediction of biomass-derived biochar characteristics: combining machine learning with shapley additive explanation," Energy, Elsevier, vol. 335(C).
    2. Mohd Fairuzz Rosli & Dr. Azlul Kalilah Zaghlol & Associate Professor Dr. Chuah Soo Cheng & Dr. Nurul Syifaa Mohd Shakil & Theng-Huey Goh, 2024. "A Study of Long-Run Equilibrium Relationship Between Foreign Direct Investment and Its Key Determinants in Malaysia," Information Management and Business Review, AMH International, vol. 16(3), pages 868-881.
    3. Cesar de Lima Nogueira, Silvio & Och, Stephan Hennings & Moura, Luis Mauro & Domingues, Eric & Coelho, Leandro dos Santos & Mariani, Viviana Cocco, 2023. "Prediction of the NOx and CO2 emissions from an experimental dual fuel engine using optimized random forest combined with feature engineering," Energy, Elsevier, vol. 280(C).
    4. Jialiang Cui & Vanessa Hoi Mei Cheung & Wenjie Huang & Wan Sang Kan, 2022. "Mental Distress during the COVID-19 Pandemic: A Cross-Sectional Study of Women Receiving the Comprehensive Social Security Allowance in Hong Kong," IJERPH, MDPI, vol. 19(16), pages 1-13, August.
    5. Yuan Liu & Chuyao Liao & Li Zhuo & Haiyan Tao, 2022. "Evaluating Effects of Dynamic Interventions to Control COVID-19 Pandemic: A Case Study of Guangdong, China," IJERPH, MDPI, vol. 19(16), pages 1-17, August.
    6. Chengcheng Zhang & Yujia Ding & Qidi Peng, 2023. "How do demand-side incentives relate to insurance transitioning behavior of public health insurance enrollees? A novel voting ensemble approach for ranking factors of mixed data types," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(2), pages 217-246, December.
    7. Leng, Lijian & Li, Tanghao & Zhan, Hao & Rizwan, Muhammad & Zhang, Weijin & Peng, Haoyi & Yang, Zequn & Li, Hailong, 2023. "Machine learning-aided prediction of nitrogen heterocycles in bio-oil from the pyrolysis of biomass," Energy, Elsevier, vol. 278(PB).
    8. Sefiw, Nibret & Alemayehu, Arragaw & Kebede, Belaynesh, 2025. "Conflict in land acquisition for cooperative housing development in Woldia, Ethiopia," Land Use Policy, Elsevier, vol. 154(C).
    9. Shi, Dehua & Xu, Han & Wang, Shaohua & Hu, Jia & Chen, Long & Yin, Chunfang, 2024. "Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network," Energy, Elsevier, vol. 305(C).
    10. Jose Ramon Saura & Rita Bužinskienė, 2025. "Behavioral economics, artificial intelligence and entrepreneurship: an updated framework for management," International Entrepreneurship and Management Journal, Springer, vol. 21(1), pages 1-33, December.
    11. Choi, Insu & Lee, Myounggu & Kim, Hyejin & Kim, Woo Chang, 2023. "Elucidating Directed Statistical Dependencies: Investigating Global Financial Market Indices' Influence on Korean Short Selling Activities," Pacific-Basin Finance Journal, Elsevier, vol. 79(C).
    12. Cimpoeru Smaranda & Roman Monica & Kobeissi Amira & Mohammad Heba, 2020. "How are European Migrants from the MENA Countries Affected by COVID-19? Insights from an Online Survey," Journal of Social and Economic Statistics, Sciendo, vol. 9(1), pages 128-143, August.
    13. Vargas-Pérez, Víctor A. & Cordón, Oscar & Chica, Manuel & Hernández, Juan M., 2025. "Social network of peer-to-peer accommodations for a visual decision support system in tourism: The case of the Canary Islands," Socio-Economic Planning Sciences, Elsevier, vol. 98(C).
    14. Cosimo Russo & Alberto Castro & Andrea Gioia & Vito Iacobellis & Angela Gorgoglione, 2023. "A Stormwater Management Framework for Predicting First Flush Intensity and Quantifying its Influential Factors," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 37(3), pages 1437-1459, February.
    15. Bas Bosma & Arjen Witteloostuijn, 2024. "Machine learning in international business," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 55(6), pages 676-702, August.
    16. Podani, János & Patonai, Katalin & Szabó, Péter & Szilágyi, András, 2022. "Coefficients of association between nominal and fully ranked ordinal variables with applications to ecological network analysis," Ecological Modelling, Elsevier, vol. 466(C).
    17. Zhou, Yu & Chen, Ben & Meng, Kai & Zhou, Haoran & Chen, Wenshang & Zhang, Ning & Deng, Qihao & Yang, Guanghua & Tu, Zhengkai, 2023. "Optimal design of a cathode flow field for performance enhancement of PEM fuel cell," Applied Energy, Elsevier, vol. 343(C).
    18. Tianqi Zhang & Yue Zhou & Ming Li & Haoran Zhang & Tong Wang & Yu Tian, 2022. "Impacts of Urbanization on Drainage System Health and Sustainable Drainage Recommendations for Future Scenarios—A Small City Case in China," Sustainability, MDPI, vol. 14(24), pages 1-24, December.
    19. Krzysztof Flasinski & Elzbieta Szaruga & Agnieszka Szlachta & Sylwia Kowalska, 2024. "Spatial Analysis of Well-Being in Language Learning from the Point of View of War Migrants from Ukraine," European Research Studies Journal, European Research Studies Journal, vol. 0(2), pages 849-867.
    20. Alla Yu. Vladova, 2022. "Remote Geotechnical Monitoring of a Buried Oil Pipeline," Mathematics, MDPI, vol. 10(11), pages 1-14, May.
    21. dos Santos Ferreira, Greicili & Martins dos Santos, Deilson & Luciano Avila, Sérgio & Viana Luiz Albani, Vinicius & Cardoso Orsi, Gustavo & Cesar Cordeiro Vieira, Pedro & Nilson Rodrigues, Rafael, 2023. "Short- and long-term forecasting for building energy consumption considering IPMVP recommendations, WEO and COP27 scenarios," Applied Energy, Elsevier, vol. 339(C).
    22. Qingyue Xiong & Qiujun Lan & Jiaqi Ma & Huiling Zhou & Gang Li & Zheng Yang, 2024. "DRAPE: optimizing private data release under adjustable privacy-utility equilibrium," Information Technology and Management, Springer, vol. 25(2), pages 199-217, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Agresti, Alan & Coull, Brent A., 1998. "Order-restricted inference for monotone trend alternatives in contingency tables," Computational Statistics & Data Analysis, Elsevier, vol. 28(2), pages 139-155, August.
    2. Yang Lu, 2019. "Flexible (panel) regression models for bivariate count–continuous data with an insurance application," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 182(4), pages 1503-1521, October.
    3. Omerašević Amela & Selimović Jasmina, 2020. "Classification Ratemaking Using Decision Tree in the Insurance Market of Bosnia and Herzegovina," South East European Journal of Economics and Business, Sciendo, vol. 15(2), pages 124-139, December.
    4. Dong-Young Lim, 2021. "A Neural Frequency-Severity Model and Its Application to Insurance Claims," Papers 2106.10770, arXiv.org, revised Mar 2025.
    5. Benchong Li & Liya Fu, 2018. "Exact test of goodness of fit for binomial distribution," Statistical Papers, Springer, vol. 59(3), pages 851-860, September.
    6. Peng Shi & Glenn M. Fung & Daniel Dickinson, 2022. "Assessing hail risk for property insurers with a dependent marked point process," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(1), pages 302-328, January.
    7. Borkowf, Craig B., 2004. "An efficient algorithm for generating two-way contingency tables with fixed marginal totals and arbitrary mean proportions, with applications to permutation tests," Computational Statistics & Data Analysis, Elsevier, vol. 44(3), pages 431-449, January.
    8. Roland R. Ramsahai, 2020. "Connecting actuarial judgment to probabilistic learning techniques with graph theory," Papers 2007.15475, arXiv.org.
    9. Ramon Alemany & Catalina Bolancé & Roberto Rodrigo & Raluca Vernic, 2020. "Bivariate Mixed Poisson and Normal Generalised Linear Models with Sarmanov Dependence—An Application to Model Claim Frequency and Optimal Transformed Average Severity," Mathematics, MDPI, vol. 9(1), pages 1-18, December.
    10. Yeawon Yoo & Adolfo R. Escobedo, 2021. "A New Binary Programming Formulation and Social Choice Property for Kemeny Rank Aggregation," Decision Analysis, INFORMS, vol. 18(4), pages 296-320, December.
    11. Kim, Donguk & Agresti, Alan, 1997. "Nearly exact tests of conditional independence and marginal homogeneity for sparse contingency tables," Computational Statistics & Data Analysis, Elsevier, vol. 24(1), pages 89-104, March.
    12. Shengkun Xie & Anna T. Lawniczak, 2018. "Estimating Major Risk Factor Relativities in Rate Filings Using Generalized Linear Models," IJFS, MDPI, vol. 6(4), pages 1-14, October.
    13. Pierre-Olivier Goffard & Patrick Laub, 2021. "Approximate Bayesian Computations to fit and compare insurance loss models," Working Papers hal-02891046, HAL.
    14. Wang, Yunyun & Oka, Tatsushi & Zhu, Dan, 2023. "Bivariate distribution regression with application to insurance data," Insurance: Mathematics and Economics, Elsevier, vol. 113(C), pages 215-232.
    15. Enrique Garcia Tejeda, 2022. "La concentracion espacial de los reportes de disparos al 911 en la Ciudad de Mexico: ¿Comportamiento racional en el uso de armas durante la pandemia Covid-19?," Sobre México. Revista de Economía, Sobre México. Temas en economía, vol. 3(5), pages 69-93.
    16. Salazar García, Juan Fernando & Guzmán Aguilar, Diana Sirley & Hoyos Nieto, Daniel Arturo, 2023. "Modelación de una prima de seguros mediante la aplicación de métodos actuariales, teoría de fallas y Black-Scholes en la salud en Colombia [Modelling of an insurance premium through the application," Revista de Métodos Cuantitativos para la Economía y la Empresa = Journal of Quantitative Methods for Economics and Business Administration, Universidad Pablo de Olavide, Department of Quantitative Methods for Economics and Business Administration, vol. 35(1), pages 330-359, June.
    17. Oh, Rosy & Jeong, Himchan & Ahn, Jae Youn & Valdez, Emiliano A., 2021. "A multi-year microlevel collective risk model," Insurance: Mathematics and Economics, Elsevier, vol. 100(C), pages 309-328.
    18. Goffard, Pierre-Olivier & Laub, Patrick J., 2021. "Approximate Bayesian Computations to fit and compare insurance loss models," Insurance: Mathematics and Economics, Elsevier, vol. 100(C), pages 350-371.
    19. Tatjana Miljkovic & Daniel Fernández, 2018. "On Two Mixture-Based Clustering Approaches Used in Modeling an Insurance Portfolio," Risks, MDPI, vol. 6(2), pages 1-18, May.
    20. Cheung, Eric C.K. & Ni, Weihong & Oh, Rosy & Woo, Jae-Kyung, 2021. "Bayesian credibility under a bivariate prior on the frequency and the severity of claims," Insurance: Mathematics and Economics, Elsevier, vol. 100(C), pages 274-295.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:152:y:2020:i:c:s0167947320301341. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.