IDEAS home Printed from https://ideas.repec.org/a/eee/jmvana/v97y2006i9p2034-2040.html
   My bibliography  Save this article

Interpreting Kullback-Leibler divergence with the Neyman-Pearson lemma

Author

Listed:
  • Eguchi, Shinto
  • Copas, John

Abstract

Kullback-Leibler divergence and the Neyman-Pearson lemma are two fundamental concepts in statistics. Both are about likelihood ratios: Kullback-Leibler divergence is the expected log-likelihood ratio, and the Neyman-Pearson lemma is about error rates of likelihood ratio tests. Exploring this connection gives another statistical interpretation of the Kullback-Leibler divergence in terms of the loss of power of the likelihood ratio test when the wrong distribution is used for one of the hypotheses. In this interpretation, the standard non-negativity property of the Kullback-Leibler divergence is essentially a restatement of the optimal property of likelihood ratios established by the Neyman-Pearson lemma. The asymmetry of Kullback-Leibler divergence is overviewed in information geometry.

Suggested Citation

  • Eguchi, Shinto & Copas, John, 2006. "Interpreting Kullback-Leibler divergence with the Neyman-Pearson lemma," Journal of Multivariate Analysis, Elsevier, vol. 97(9), pages 2034-2040, October.
    • Handle: RePEc:eee:jmvana:v:97:y:2006:i:9:p:2034-2040
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0047-259X(06)00086-8
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Shinto Eguchi, 2002. "A class of logistic-type discriminant functions," Biometrika, Biometrika Trust, vol. 89(1), pages 1-22, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Valdevino Félix de Lima, Wenia & David Costa do Nascimento, Abraão & José Amorim do Amaral, Getúlio, 2021. "Distance-based tests for planar shape," Journal of Multivariate Analysis, Elsevier, vol. 184(C).
    2. Ghosh, Indranil, 2023. "On the issue of convergence of certain divergence measures related to finding most nearly compatible probability distribution under the discrete set-up," Statistics & Probability Letters, Elsevier, vol. 203(C).
    3. Jinxin Liu & Guan Wang & Tong Zhao & Li Zhang, 2017. "Fault Diagnosis of On-Load Tap-Changer Based on Variational Mode Decomposition and Relevance Vector Machine," Energies, MDPI, vol. 10(7), pages 1-14, July.
    4. Gallindo, Lucas, 2007. "Computation of Power Loss in Likelihood Ratio Tests for Probability Densities Extended by Lehmann Alternatives," OSF Preprints pkcnb, Center for Open Science.
    5. Springborn, Michael & Sanchirico, James N., 2013. "A density projection approach for non-trivial information dynamics: Adaptive management of stochastic natural resources," Journal of Environmental Economics and Management, Elsevier, vol. 66(3), pages 609-624.
    6. Bhattacharya, Bhaskar & Hughes, Gareth, 2015. "On shape properties of the receiver operating characteristic curve," Statistics & Probability Letters, Elsevier, vol. 103(C), pages 73-79.
    7. Piyush Anand & Clarence Lee, 2023. "Using Deep Learning to Overcome Privacy and Scalability Issues in Customer Data Transfer," Marketing Science, INFORMS, vol. 42(1), pages 189-207, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Margaret Sullivan Pepe & Tianxi Cai & Gary Longton, 2006. "Combining Predictors for Classification Using the Area under the Receiver Operating Characteristic Curve," Biometrics, The International Biometric Society, vol. 62(1), pages 221-229, March.
    2. Osamu Komori & Shinto Eguchi & John B. Copas, 2015. "Generalized t-statistic for two-group classification," Biometrics, The International Biometric Society, vol. 71(2), pages 404-416, June.
    3. Osamu Komori, 2011. "A boosting method for maximization of the area under the ROC curve," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 63(5), pages 961-979, October.
    4. Kenichi Hayashi, 2018. "Asymptotic comparison of semi-supervised and supervised linear discriminant functions for heteroscedastic normal populations," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(2), pages 315-339, June.
    5. Yingye Zheng & Tianxi Cai & Ziding Feng, 2006. "Application of the Time-Dependent ROC Curves for Prognostic Accuracy with Multiple Biomarkers," Biometrics, The International Biometric Society, vol. 62(1), pages 279-287, March.
    6. Nishii, Ryuei & Eguchi, Shinto, 2006. "Image classification based on Markov random field models with Jeffreys divergence," Journal of Multivariate Analysis, Elsevier, vol. 97(9), pages 1997-2008, October.
    7. Wei Dai & Ming Yang & Chaolong Wang & Tianxi Cai, 2017. "Sequence robust association test for familial data," Biometrics, The International Biometric Society, vol. 73(3), pages 876-884, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jmvana:v:97:y:2006:i:9:p:2034-2040. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.