IDEAS home Printed from https://ideas.repec.org/a/eee/stapro/v68y2004i1p73-82.html

A note on margin-based loss functions in classification

Author

Listed:
  • Lin, Yi

Abstract

In many classification procedures, the classification function is obtained by minimizing a certain empirical risk on the training sample. The classification is then based on the sign of the classification function. In recent years, there have been a host of classification methods proposed that use different margin-based loss functions. The margin-based loss functions are often motivated as upper bounds of the misclassification loss, but this cannot explain the statistical properties of the classification procedures. We show that a large family of margin-based loss functions are Fisher consistent for classification. That is, the population minimizer of the loss function leads to the Bayes optimal rule of classification. Our result covers almost all margin-based loss functions that have been proposed in the literature. We give an inequality that links the Fisher consistency of margin-based loss functions with the consistency of methods based on these loss functions. We use this inequality to obtain the rate of convergence for the method of sieves based on a class of margin-based loss functions.

Suggested Citation

  • Lin, Yi, 2004. "A note on margin-based loss functions in classification," Statistics & Probability Letters, Elsevier, vol. 68(1), pages 73-82, June.
    • Handle: RePEc:eee:stapro:v:68:y:2004:i:1:p:73-82
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167-7152(04)00070-7
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Buhlmann P. & Yu B., 2003. "Boosting With the L2 Loss: Regression and Classification," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 324-339, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ling Peng & Xiaohui Liu & Xiangyong Tan & Yiweng Zhou & Shihua Luo, 2024. "The statistical rate for support matrix machines under low rankness and row (column) sparsity," Statistical Papers, Springer, vol. 65(7), pages 4567-4598, September.
    2. Alexandru V. Asimit & Ioannis Kyriakou & Simone Santoni & Salvatore Scognamiglio & Rui Zhu, 2022. "Robust Classification via Support Vector Machines," Risks, MDPI, vol. 10(8), pages 1-25, August.
    3. Adam N. Elmachtoub & Paul Grigas, 2022. "Smart “Predict, then Optimize”," Management Science, INFORMS, vol. 68(1), pages 9-26, January.
    4. Yang, Yi & Guo, Yuxuan & Chang, Xiangyu, 2021. "Angle-based cost-sensitive multicategory classification," Computational Statistics & Data Analysis, Elsevier, vol. 156(C).
    5. Chen, Zhongyuan & Xie, Jun, 2023. "Estimating heterogeneous treatment effects versus building individualized treatment rules: Connection and disconnection," Statistics & Probability Letters, Elsevier, vol. 199(C).
    6. Hayashi, Kenichi, 2012. "A simple extension of boosting for asymmetric mislabeled data," Statistics & Probability Letters, Elsevier, vol. 82(2), pages 348-356.
    7. Xiangyu Chang & Yinghui Huang & Mei Li & Xin Bo & Subodha Kumar, 2021. "Efficient Detection of Environmental Violators: A Big Data Approach," Production and Operations Management, Production and Operations Management Society, vol. 30(5), pages 1246-1270, May.
    8. Caiyi Li & Kaishuai Liu & Shuai Liu, 2025. "A Survey of Loss Functions in Deep Learning," Mathematics, MDPI, vol. 13(15), pages 1-50, July.
    9. Mun, Jongmin & Bang, Sungwan & Kim, Jaeoh, 2025. "Weighted support vector machine for extremely imbalanced data," Computational Statistics & Data Analysis, Elsevier, vol. 203(C).
    10. Artem Timoshenko & Caio Waisman, 2025. "Profit-Aligned CATE Estimation: Reconciling Policy Learning and Inference," Papers 2512.13400, arXiv.org, revised Apr 2026.
    11. Nam Ho-Nguyen & Fatma Kılınç-Karzan, 2022. "Risk Guarantees for End-to-End Prediction and Optimization Processes," Management Science, INFORMS, vol. 68(12), pages 8680-8698, December.
    12. Seokho Lee & Hyejin Shin & Sang Han Lee, 2016. "Label‐noise resistant logistic regression for functional data classification with an application to Alzheimer's disease study," Biometrics, The International Biometric Society, vol. 72(4), pages 1325-1335, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tutz, Gerhard & Pößnecker, Wolfgang & Uhlmann, Lorenz, 2015. "Variable selection in general multinomial logit models," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 207-222.
    2. Gerhard Tutz & Moritz Berger, 2018. "Tree-structured modelling of categorical predictors in generalized additive regression," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(3), pages 737-758, September.
    3. Mittnik, Stefan & Robinzonov, Nikolay & Spindler, Martin, 2015. "Stock market volatility: Identifying major drivers and the nature of their impact," Journal of Banking & Finance, Elsevier, vol. 58(C), pages 1-14.
    4. Wang Zhu & Wang C.Y., 2010. "Buckley-James Boosting for Survival Analysis with High-Dimensional Biomarker Data," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 9(1), pages 1-33, June.
    5. Martijn Kagie & Michiel Van Wezel, 2007. "Hedonic price models and indices based on boosting applied to the Dutch housing market," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 15(3‐4), pages 85-106, July.
    6. Hofner, Benjamin & Mayr, Andreas & Schmid, Matthias, 2016. "gamboostLSS: An R Package for Model Building and Variable Selection in the GAMLSS Framework," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 74(i01).
    7. Marra, Giampiero & Wood, Simon N., 2011. "Practical variable selection for generalized additive models," Computational Statistics & Data Analysis, Elsevier, vol. 55(7), pages 2372-2387, July.
    8. Ziwei Mei & Peter C. B. Phillips & Zhentao Shi, 2022. "The boosted HP filter is more general than you might think," Papers 2209.09810, arXiv.org, revised Apr 2024.
    9. R. Lehmann & K. Wohlrabe, 2016. "Looking into the black box of boosting: the case of Germany," Applied Economics Letters, Taylor & Francis Journals, vol. 23(17), pages 1229-1233, November.
    10. Kim, Hyun Hak & Swanson, Norman R., 2014. "Forecasting financial and macroeconomic variables using data reduction methods: New empirical evidence," Journal of Econometrics, Elsevier, vol. 178(P2), pages 352-367.
    11. Wolfgang Nierhaus & Timo Wollmershäuser, 2016. "ifo Konjunkturumfragen und Konjunkturanalyse: Band II," ifo Forschungsberichte, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 72.
    12. Fabio Trojani, 2007. "Accurate Short-Term Yield Curve Forecasting using Functional Gradient Descent," Journal of Financial Econometrics, Oxford University Press, vol. 5(4), pages 591-623, Fall.
    13. Stefanie Hieke & Axel Benner & Richard F Schlenk & Martin Schumacher & Lars Bullinger & Harald Binder, 2016. "Identifying Prognostic SNPs in Clinical Cohorts: Complementing Univariate Analyses by Resampling and Multivariable Modeling," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-18, May.
    14. Panagiotelis, Anastasios & Gamakumara, Puwasala & Athanasopoulos, George & Hyndman, Rob J., 2023. "Probabilistic forecast reconciliation: Properties, evaluation and score optimisation," European Journal of Operational Research, Elsevier, vol. 306(2), pages 693-706.
    15. Ben Taieb, Souhaib & Hyndman, Rob J., 2014. "A gradient boosting approach to the Kaggle load forecasting competition," International Journal of Forecasting, Elsevier, vol. 30(2), pages 382-394.
    16. Klaus Wohlrabe & Teresa Buchen, 2014. "Assessing the Macroeconomic Forecasting Performance of Boosting: Evidence for the United States, the Euro Area and Germany," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 33(4), pages 231-242, July.
    17. Faisal Zahid & Gerhard Tutz, 2013. "Multinomial logit models with implicit variable selection," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 7(4), pages 393-416, December.
    18. Imad Bou-Hamad & Abdel Latef Anouze & Denis Larocque, 2017. "An integrated approach of data envelopment analysis and boosted generalized linear mixed models for efficiency assessment," Annals of Operations Research, Springer, vol. 253(1), pages 77-95, June.
    19. Ju, Xiaomeng & Salibián-Barrera, Matías, 2021. "Robust boosting for regression problems," Computational Statistics & Data Analysis, Elsevier, vol. 153(C).
    20. Guilherme Schultz Lindenmeyer & Hudson Silva Torrent, 2024. "Boosting and Predictability of Macroeconomic Variables: Evidence from Brazil," Computational Economics, Springer;Society for Computational Economics, vol. 64(1), pages 377-409, July.

    More about this item

    Keywords

    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:stapro:v:68:y:2004:i:1:p:73-82. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.