To Bag is to Prune

It is notoriously difficult to build a bad Random Forest (RF). Concurrently, RF blatantly overfits in-sample without apparent consequences out-of-sample. Arguments like the bias-variance trade-off or double descent cannot rationalize this paradox. I propose a new explanation: bootstrap aggregation and model perturbation as implemented by RF automatically prune a latent “true” tree. More generally, I document that randomized ensembles of greedily optimized learners implicitly perform optimal early stopping out-of-sample. So, letting RF overfit the training data is a dominant tuning strategy against nature’s undisclosed choice of noise level. Additionally, novel ensembles of Boosting and MARS are also eligible. I empirically demonstrate the property, with simulated and real data, by reporting that these new completely overfitting ensembles perform similarly to their tuned counterparts – or better.