Using AUC and accuracy in evaluating learning algorithms

@article{citeulike:2775365, abstract = {The area under the ROC (receiver operating characteristics) curve, or simply AUC, has been traditionally used in medical diagnosis since the 1970s. It has recently been proposed as an alternative single-number measure for evaluating the predictive ability of learning algorithms. However, no formal arguments were given as to why AUC should be preferred over accuracy. We establish formal criteria for comparing two different measures for learning algorithms and we show theoretically and empirically that AUC is a better measure (defined precisely) than accuracy. We then reevaluate well-established claims in machine learning based on accuracy using AUC and obtain interesting and surprising new results. For example, it has been well-established and accepted that Naive Bayes and decision trees are very similar in predictive accuracy. We show, however, that Naive Bayes is significantly better than decision trees in AUC. The conclusions drawn in this paper may make a significant impact on machine learning and data mining applications.}, author = {Huang, Jin and Ling, C. X. }, booktitle = {Knowledge and Data Engineering, IEEE Transactions on}, citeulike-article-id = {2775365}, doi = {10.1109/TKDE.2005.50}, journal = {Knowledge and Data Engineering, IEEE Transactions on}, keywords = {algorithms, information, kdd, stochastic}, number = {3}, pages = {299--310}, posted-at = {2008-05-09 13:02:22}, priority = {2}, title = {Using AUC and accuracy in evaluating learning algorithms}, url = {http://dx.doi.org/10.1109/TKDE.2005.50}, volume = {17}, year = {2005} }

TY - JOUR ID - citeulike:2775365 TI - Using AUC and accuracy in evaluating learning algorithms T2 - Knowledge and Data Engineering, IEEE Transactions on JF - Knowledge and Data Engineering, IEEE Transactions on VL - 17 IS - 3 SP - 299 EP - 310 N2 - The area under the ROC (receiver operating characteristics) curve, or simply AUC, has been traditionally used in medical diagnosis since the 1970s. It has recently been proposed as an alternative single-number measure for evaluating the predictive ability of learning algorithms. However, no formal arguments were given as to why AUC should be preferred over accuracy. We establish formal criteria for comparing two different measures for learning algorithms and we show theoretically and empirically that AUC is a better measure (defined precisely) than accuracy. We then reevaluate well-established claims in machine learning based on accuracy using AUC and obtain interesting and surprising new results. For example, it has been well-established and accepted that Naive Bayes and decision trees are very similar in predictive accuracy. We show, however, that Naive Bayes is significantly better than decision trees in AUC. The conclusions drawn in this paper may make a significant impact on machine learning and data mining applications. KW - algorithms KW - information KW - kdd KW - stochastic AU - Huang, Jin AU - Ling, CX PY - 2005/// UR - http://dx.doi.org/10.1109/TKDE.2005.50 ER -