IDEAS home Printed from
   My bibliography  Save this article

Cross-validated bagged learning


  • Petersen, Maya L.
  • Molinaro, Annette M.
  • Sinisi, Sandra E.
  • van der Laan, Mark J.


Many applications aim to learn a high dimensional parameter of a data generating distribution based on a sample of independent and identically distributed observations. For example, the goal might be to estimate the conditional mean of an outcome given a list of input variables. In this prediction context, bootstrap aggregating (bagging) has been introduced as a method to reduce the variance of a given estimator at little cost to bias. Bagging involves applying an estimator to multiple bootstrap samples and averaging the result across bootstrap samples. In order to address the curse of dimensionality, a common practice has been to apply bagging to estimators which themselves use cross-validation, thereby using cross-validation within a bootstrap sample to select fine-tuning parameters trading off bias and variance of the bootstrap sample-specific candidate estimators. In this article we point out that in order to achieve the correct bias variance trade-off for the parameter of interest, one should apply the cross-validation selector externally to candidate bagged estimators indexed by these fine-tuning parameters. We use three simulations to compare the new cross-validated bagging method with bagging of cross-validated estimators and bagging of non-cross-validated estimators.

Suggested Citation

  • Petersen, Maya L. & Molinaro, Annette M. & Sinisi, Sandra E. & van der Laan, Mark J., 2007. "Cross-validated bagged learning," Journal of Multivariate Analysis, Elsevier, vol. 98(9), pages 1693-1704, October.
  • Handle: RePEc:eee:jmvana:v:98:y:2007:i:9:p:1693-1704

    Download full text from publisher

    File URL:
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Sinisi Sandra E & van der Laan Mark J., 2004. "Deletion/Substitution/Addition Algorithm in Learning with Applications in Genomics," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-40, August.
    2. Hothorn, Torsten & Lausen, Berthold, 2005. "Bundling classifiers by bagging trees," Computational Statistics & Data Analysis, Elsevier, vol. 49(4), pages 1068-1078, June.
    3. Peter Hall & Richard J. Samworth, 2005. "Properties of bagged nearest neighbour classifiers," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(3), pages 363-379.
    4. Borra, Simone & Di Ciaccio, Agostino, 2002. "Improving nonparametric regression methods by bagging and boosting," Computational Statistics & Data Analysis, Elsevier, vol. 38(4), pages 407-420, February.
    5. Birkner Merrill D. & Sinisi Sandra E. & van der Laan Mark J., 2005. "Multiple Testing and Data Adaptive Regression: An Application to HIV-1 Sequence Data," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 4(1), pages 1-30, April.
    6. Mark van der Laan & Sandrine Dudoit & Aad van der Vaart, 2004. "The Cross-Validated Adaptive Epsilon-Net Estimator," U.C. Berkeley Division of Biostatistics Working Paper Series 1141, Berkeley Electronic Press.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. J. D’Haen & D. Van Den Poel & D. Thorleuchter, 2012. "Predicting Customer Profitability During Acquisition: Finding the Optimal Combination of Data Source and Data Mining Technique," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 12/818, Ghent University, Faculty of Economics and Business Administration.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jmvana:v:98:y:2007:i:9:p:1693-1704. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.