A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

@inproceedings{citeulike:531106, abstract = {We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimental results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifiers (model selection), ten-fold cross-validation may be better than the more expensive leaveone -out cross-validation. We report on a largescale experiment---over half a million runs of C4.5 and a Naive-Bayes algorithm---to...}, author = {Kohavi, Ron }, booktitle = {IJCAI}, citeulike-article-id = {531106}, comment = {Proposition 1: Given a dataset and an inducer, if the inducer is stable under the perturbations caused by deleting the instances for the folds in k-fold cross-validation, the cross-validation estimation will be unbiased and the variance of the estimated accuracy will be approximately acc\_cv (1-acc\_cv)/n, where n is the number of instances in the dataset. Corollary 2: Given a dataset and an inducer, if the inducer is stable under the perturbations caused by deleting the test instances for the folds in k-fold cross-validation for various values of k, then the variance of the estimates will be the same.}, keywords = {cross\_validation, model\_selection, pattern\_recognition}, pages = {1137--1145}, posted-at = {2008-04-28 14:39:25}, priority = {0}, title = {A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection}, url = {http://citeseer.ist.psu.edu/kohavi95study.html}, year = {1995} }

TY - CONF ID - citeulike:531106 L3 - citeulike-article-id:531106 TI - A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection T2 - IJCAI SP - 1137 EP - 1145 N2 - We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimental results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifiers (model selection), ten-fold cross-validation may be better than the more expensive leaveone -out cross-validation. We report on a largescale experiment---over half a million runs of C4.5 and a Naive-Bayes algorithm---to... KW - cross_validation KW - model_selection KW - pattern_recognition N1 - Proposition 1: Given a dataset and an inducer, if the inducer is stable under the perturbations caused by deleting the instances for the folds in k-fold cross-validation, the cross-validation estimation will be unbiased and the variance of the estimated accuracy will be approximately acc_cv (1-acc_cv)/n, where n is the number of instances in the dataset. Corollary 2: Given a dataset and an inducer, if the inducer is stable under the perturbations caused by deleting the test instances for the folds in k-fold cross-validation for various values of k, then the variance of the estimates will be the same. AU - Kohavi, Ron PY - 1995/// UR - http://citeseer.ist.psu.edu/kohavi95study.html ER -