A Theoretical Analysis of Bagging as a Linear Combination of Classifiers

TitleA Theoretical Analysis of Bagging as a Linear Combination of Classifiers
Publication TypeJournal Article
Year of Publication2008
AuthorsFumera, G, Roli, F, Serrau, A
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Date Published07/2008
Keywordscombining rules, ensemble construction, linear combiners, mcs01, mcs02, Multiple Classifier Systems

We apply an analytical framework for the analysis of linearly combined classifiers to ensembles generated by bagging. This provides an analytical model of bagging misclassification probability as a function of the ensemble size, which is a novel result in the literature. Experimental results on real data sets confirm the theoretical predictions. This allows us to derive a novel and theoretically grounded guideline for choosing bagging ensemble size. Furthermore, our results are consistent with explanations of bagging in terms of classifier instability and variance reduction, support the optimality of the simple average over the weighted average combining rule for ensembles generated by bagging, and apply to other randomization-based methods for constructing classifier ensembles. Although our results do not allow to compare bagging misclassification probability with the one of an individual classifier trained on the original training set, we discuss how the considered theoretical framework could be exploited to this aim.

Citation Key 193
fumera-TPAMI2008.pdf895.49 KB