| Previous: Meta-Classifiers |
Calibration
Calibration transforms a classifier's output from [-¥, ¥] to [0,1]. In other words, calibration is one method to obtain probability estimates from a classifier. In the case where a classifier already produces probability estimates, calibration often improves these estimates. However, a calibration method requires a classifier that provides real-valued output.
The logistic correction was suggested by Friedman et al.'s view of AdaBoost as an additive logistic regression model minimizing E(exp(-yF(x))). Niculescu-Mizil and Caruana found that the logistic correction on AdaBoost on Decision Stumps perform reasonably well.
Features
- No model selection required
- Only constant time added on prediction
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. "Additive Logistic Regression: A Statistical View of Boosting." Department of Statistics Stanford University, 1998.
Niculescu-Mizil, Alexandru, and Rich Caruana. "An Empirical Comparison of Supervised Learning Algorithm" Paper presented at the 23rd International Conference on Machine Learning, Pittsburgh, Pennsylvania 2006.
The sigmoid correction (or Platt Calibration) is a parametric transformation and was first suggested by Platt to transform the predictions of SVM to fall between [0, 1]. Niculescu-Mizil and Caruana demonstrated that the sigmoid correction works well on other learning algorithms including AdaBoost.
Features
- Provides accurate probability estimates
- Requires 3-fold cross-validation to set parameters
Platt, John C. "Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods." In Advances in Large Margin Classifiers, edited by Peter J. Bartlett, Bernhard Schölkopf, Dale Schuurmans and Alex J. Smola, 61-74. Boston: MIT Press, 1999.
Niculescu-Mizil, Alexandru, and Rich Caruana. "An Empirical Comparison of Supervised Learning Algorithm" Paper presented at the 23rd International Conference on Machine Learning, Pittsburgh, Pennsylvania 2006.
A nonparametric alternative to the sigmoid correction is isotonic regression first posed by Robertson et al. Zadronzny and Elkan applied isotonic regression to Naive Bayes, boosted Naive Bayes and decision trees. This work was extended by Niculescu-Mizil and Caruana to a number of other methods.
Features
- Provides accurate probability estimates
- Non-parametric estimate
- Requires 3-fold cross-validation to set parameters
Zadrozny, Bianca, and Charles Elkan. "Transforming Classifier Scores into Accurate Multiclass Probability Estimates." Paper presented at the Eighth ACM SIGKDD, Edmonton, Alberta, Canada 2002.
Robertson, T., F. Wright, and R. Dykstra. Order Restricted Statistical Inference. New York: John Wiley and Sones, 1988.
Niculescu-Mizil, Alexandru, and Rich Caruana. "An Empirical Comparison of Supervised Learning Algorithm" Paper presented at the 23rd International Conference on Machine Learning, Pittsburgh, Pennsylvania 2006.
| Previous: Meta-Classifiers |