scikit-learn .predict() default threshold
Question:
I’m working on a classification problem with unbalanced classes (5% 1’s). I want to predict the class, not the probability.
In a binary classification problem, is scikit’s classifier.predict() using 0.5 by default?
If it doesn’t, what’s the default method? If it does, how do I change it?
In scikit some classifiers have the class_weight='auto' option, but not all do. With class_weight='auto', would .predict() use the actual population proportion as a threshold?
What would be the way to do this in a classifier like MultinomialNB that doesn’t support class_weight? Other than using predict_proba() and then calculation the classes myself.
Answers:
You seem to be confusing concepts here. Threshold is not a concept for a “generic classifier” – the most basic approaches are based on some tunable threshold, but most of the existing methods create complex rules for classification which cannot (or at least shouldn’t) be seen as a thresholding.
So first – one cannot answer your question for scikit’s classifier default threshold because there is no such thing.
Second – class weighting is not about threshold, is about classifier ability to deal with imbalanced classes, and it is something dependent on a particular classifier. For example – in SVM case it is the way of weighting the slack variables in the optimization problem, or if you prefer – the upper bounds for the lagrange multipliers values connected with particular classes. Setting this to ‘auto’ means using some default heuristic, but once again – it cannot be simply translated into some thresholding.
Naive Bayes on the other hand directly estimates the classes probability from the training set. It is called “class prior” and you can set it in the constructor with “class_prior” variable.
From the documentation:
Prior probabilities of the classes. If specified the priors are not adjusted according to the data.
is scikit’s classifier.predict() using 0.5 by default?
In probabilistic classifiers, yes. It’s the only sensible threshold from a mathematical viewpoint, as others have explained.
What would be the way to do this in a classifier like MultinomialNB that doesn’t support class_weight?
You can set the class_prior, which is the prior probability P(y) per class y. That effectively shifts the decision boundary. E.g.
# minimal dataset
>>> X = [[1, 0], [1, 0], [0, 1]]
>>> y = [0, 0, 1]
# use empirical prior, learned from y
>>> MultinomialNB().fit(X,y).predict([1,1])
array([0])
# use custom prior to make 1 more likely
>>> MultinomialNB(class_prior=[.1, .9]).fit(X,y).predict([1,1])
array([1])
The threshold in scikit learn is 0.5 for binary classification and whichever class has the greatest probability for multiclass classification. In many problems a much better result may be obtained by adjusting the threshold. However, this must be done with care and NOT on the holdout test data but by cross validation on the training data. If you do any adjustment of the threshold on your test data you are just overfitting the test data.
Most methods of adjusting the threshold is based on the receiver operating characteristics (ROC) and Youden’s J statistic but it can also be done by other methods such as a search with a genetic algorithm.
Here is a peer review journal article describing doing this in medicine:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2515362/
So far as I know there is no package for doing it in Python but it is relatively simple (but inefficient) to find it with a brute force search in Python.
This is some R code that does it.
## load data
DD73OP <- read.table("/my_probabilites.txt", header=T, quote=""")
library("pROC")
# No smoothing
roc_OP <- roc(DD73OP$tc, DD73OP$prob)
auc_OP <- auc(roc_OP)
auc_OP
Area under the curve: 0.8909
plot(roc_OP)
# Best threshold
# Method: Youden
#Youden's J statistic (Youden, 1950) is employed. The optimal cut-off is the threshold that maximizes the distance to the identity (diagonal) line. Can be shortened to "y".
#The optimality criterion is:
#max(sensitivities + specificities)
coords(roc_OP, "best", ret=c("threshold", "specificity", "sensitivity"), best.method="youden")
#threshold specificity sensitivity
#0.7276835 0.9092466 0.7559022
In case someone visits this thread hoping for ready-to-use function (python 2.7). In this example cutoff is designed to reflect ratio of events to non-events in original dataset df, while y_prob could be the result of .predict_proba method (assuming stratified train/test split).
def predict_with_cutoff(colname, y_prob, df):
n_events = df[colname].values
event_rate = sum(n_events) / float(df.shape[0]) * 100
threshold = np.percentile(y_prob[:, 1], 100 - event_rate)
print "Cutoff/threshold at: " + str(threshold)
y_pred = [1 if x >= threshold else 0 for x in y_prob[:, 1]]
return y_pred
Feel free to criticize/modify. Hope it helps in rare cases when class balancing is out of the question and the dataset itself is highly imbalanced.
The threshold can be set using clf.predict_proba()
for example:
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state = 2)
clf.fit(X_train,y_train)
# y_pred = clf.predict(X_test) # default threshold is 0.5
y_pred = (clf.predict_proba(X_test)[:,1] >= 0.3).astype(bool) # set threshold as 0.3
Scikit-learn classifiers generally choose the predicted class by taking the argmax of scores/probabilities (see LogisticRegression and DecisionTreeClassifier).
For binary classification problems, the argmax is equal to using a 0.5 threshold on probabilities. In this case, varying the threshold changes your confidence about the predicted classes.
You can tune/change the threshold according to your goals (i.e. maximize precision or recall). The concept is clearly explained in this post. It’s possible to automate the finding of the optimal threshold, of any classifier, by retrieving the predicted probabilities and optimizing a metric of interest on a validation set. This is done by ThresholdClassifier:
import numpy as np
from sklearn.metrics import fbeta_score
from sklearn.model_selection import train_test_split
from sklearn.base import clone, BaseEstimator, ClassifierMixin
class ThresholdClassifier(BaseEstimator, ClassifierMixin):
def __init__(self, estimator, refit=True, val_size=0.3):
self.estimator = estimator
self.refit = refit
self.val_size = val_size
def fit(self, X, y):
def scoring(th, y, prob):
pred = (prob > th).astype(int)
return 0 if not pred.any() else
-fbeta_score(y, pred, beta=0.1)
X_train, X_val, y_train, y_val = train_test_split(
X, y, stratify=y, test_size=self.val_size,
shuffle=True, random_state=1234
)
self.estimator_ = clone(self.estimator)
self.estimator_.fit(X_train, y_train)
prob_val = self.estimator_.predict_proba(X_val)[:,1]
thresholds = np.linspace(0,1, 200)[1:-1]
scores = [scoring(th, y_val, prob_val)
for th in thresholds]
self.score_ = np.min(scores)
self.th_ = thresholds[np.argmin(scores)]
if self.refit:
self.estimator_.fit(X, y)
if hasattr(self.estimator_, 'classes_'):
self.classes_ = self.estimator_.classes_
return self
def predict(self, X):
proba = self.estimator_.predict_proba(X)[:,1]
return (proba > self.th_).astype(int)
def predict_proba(self, X):
return self.estimator_.predict_proba(X)
When calling fit:
- a validation (
X_val and y_val) set is randomly generated from the received data;
- the
estimator is fitted on X_train and y_train;
- probabilities (
prob_val) are retrieved for the 1’s class on X_val;
- an optimal threshold value is found on
X_val by optimizing a metric of choice (fbeta_score in our case).
When calling predict: probabilities for the 1’s class are generated and cast into binary classes by the optimal threshold value found.
model = ThresholdClassifier(RandomForestClassifier()).fit(X_train, y_train)
pred_clas = model.predict(X_test)
ThresholdClassifier can be used with any sklearn classifier which produces probabilities. It can be easily customized according to different needs. It’s very useful in conjunction with GridSearchCV/RandomSearchCV to connect the search of parameters with the tune of a classification threshold.
model = RandomizedSearchCV(
ThresholdClassifier(RandomForestClassifier()),
dict(n_estimators=stats.randint(50,300)),
n_iter=20, random_state=1234,
cv=5, n_jobs=-1,
).fit(X_train, y_train)
I’m working on a classification problem with unbalanced classes (5% 1’s). I want to predict the class, not the probability.
In a binary classification problem, is scikit’s classifier.predict() using 0.5 by default?
If it doesn’t, what’s the default method? If it does, how do I change it?
In scikit some classifiers have the class_weight='auto' option, but not all do. With class_weight='auto', would .predict() use the actual population proportion as a threshold?
What would be the way to do this in a classifier like MultinomialNB that doesn’t support class_weight? Other than using predict_proba() and then calculation the classes myself.
You seem to be confusing concepts here. Threshold is not a concept for a “generic classifier” – the most basic approaches are based on some tunable threshold, but most of the existing methods create complex rules for classification which cannot (or at least shouldn’t) be seen as a thresholding.
So first – one cannot answer your question for scikit’s classifier default threshold because there is no such thing.
Second – class weighting is not about threshold, is about classifier ability to deal with imbalanced classes, and it is something dependent on a particular classifier. For example – in SVM case it is the way of weighting the slack variables in the optimization problem, or if you prefer – the upper bounds for the lagrange multipliers values connected with particular classes. Setting this to ‘auto’ means using some default heuristic, but once again – it cannot be simply translated into some thresholding.
Naive Bayes on the other hand directly estimates the classes probability from the training set. It is called “class prior” and you can set it in the constructor with “class_prior” variable.
From the documentation:
Prior probabilities of the classes. If specified the priors are not adjusted according to the data.
is scikit’s
classifier.predict()using 0.5 by default?
In probabilistic classifiers, yes. It’s the only sensible threshold from a mathematical viewpoint, as others have explained.
What would be the way to do this in a classifier like MultinomialNB that doesn’t support
class_weight?
You can set the class_prior, which is the prior probability P(y) per class y. That effectively shifts the decision boundary. E.g.
# minimal dataset
>>> X = [[1, 0], [1, 0], [0, 1]]
>>> y = [0, 0, 1]
# use empirical prior, learned from y
>>> MultinomialNB().fit(X,y).predict([1,1])
array([0])
# use custom prior to make 1 more likely
>>> MultinomialNB(class_prior=[.1, .9]).fit(X,y).predict([1,1])
array([1])
The threshold in scikit learn is 0.5 for binary classification and whichever class has the greatest probability for multiclass classification. In many problems a much better result may be obtained by adjusting the threshold. However, this must be done with care and NOT on the holdout test data but by cross validation on the training data. If you do any adjustment of the threshold on your test data you are just overfitting the test data.
Most methods of adjusting the threshold is based on the receiver operating characteristics (ROC) and Youden’s J statistic but it can also be done by other methods such as a search with a genetic algorithm.
Here is a peer review journal article describing doing this in medicine:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2515362/
So far as I know there is no package for doing it in Python but it is relatively simple (but inefficient) to find it with a brute force search in Python.
This is some R code that does it.
## load data
DD73OP <- read.table("/my_probabilites.txt", header=T, quote=""")
library("pROC")
# No smoothing
roc_OP <- roc(DD73OP$tc, DD73OP$prob)
auc_OP <- auc(roc_OP)
auc_OP
Area under the curve: 0.8909
plot(roc_OP)
# Best threshold
# Method: Youden
#Youden's J statistic (Youden, 1950) is employed. The optimal cut-off is the threshold that maximizes the distance to the identity (diagonal) line. Can be shortened to "y".
#The optimality criterion is:
#max(sensitivities + specificities)
coords(roc_OP, "best", ret=c("threshold", "specificity", "sensitivity"), best.method="youden")
#threshold specificity sensitivity
#0.7276835 0.9092466 0.7559022
In case someone visits this thread hoping for ready-to-use function (python 2.7). In this example cutoff is designed to reflect ratio of events to non-events in original dataset df, while y_prob could be the result of .predict_proba method (assuming stratified train/test split).
def predict_with_cutoff(colname, y_prob, df):
n_events = df[colname].values
event_rate = sum(n_events) / float(df.shape[0]) * 100
threshold = np.percentile(y_prob[:, 1], 100 - event_rate)
print "Cutoff/threshold at: " + str(threshold)
y_pred = [1 if x >= threshold else 0 for x in y_prob[:, 1]]
return y_pred
Feel free to criticize/modify. Hope it helps in rare cases when class balancing is out of the question and the dataset itself is highly imbalanced.
The threshold can be set using clf.predict_proba()
for example:
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state = 2)
clf.fit(X_train,y_train)
# y_pred = clf.predict(X_test) # default threshold is 0.5
y_pred = (clf.predict_proba(X_test)[:,1] >= 0.3).astype(bool) # set threshold as 0.3
Scikit-learn classifiers generally choose the predicted class by taking the argmax of scores/probabilities (see LogisticRegression and DecisionTreeClassifier).
For binary classification problems, the argmax is equal to using a 0.5 threshold on probabilities. In this case, varying the threshold changes your confidence about the predicted classes.
You can tune/change the threshold according to your goals (i.e. maximize precision or recall). The concept is clearly explained in this post. It’s possible to automate the finding of the optimal threshold, of any classifier, by retrieving the predicted probabilities and optimizing a metric of interest on a validation set. This is done by ThresholdClassifier:
import numpy as np
from sklearn.metrics import fbeta_score
from sklearn.model_selection import train_test_split
from sklearn.base import clone, BaseEstimator, ClassifierMixin
class ThresholdClassifier(BaseEstimator, ClassifierMixin):
def __init__(self, estimator, refit=True, val_size=0.3):
self.estimator = estimator
self.refit = refit
self.val_size = val_size
def fit(self, X, y):
def scoring(th, y, prob):
pred = (prob > th).astype(int)
return 0 if not pred.any() else
-fbeta_score(y, pred, beta=0.1)
X_train, X_val, y_train, y_val = train_test_split(
X, y, stratify=y, test_size=self.val_size,
shuffle=True, random_state=1234
)
self.estimator_ = clone(self.estimator)
self.estimator_.fit(X_train, y_train)
prob_val = self.estimator_.predict_proba(X_val)[:,1]
thresholds = np.linspace(0,1, 200)[1:-1]
scores = [scoring(th, y_val, prob_val)
for th in thresholds]
self.score_ = np.min(scores)
self.th_ = thresholds[np.argmin(scores)]
if self.refit:
self.estimator_.fit(X, y)
if hasattr(self.estimator_, 'classes_'):
self.classes_ = self.estimator_.classes_
return self
def predict(self, X):
proba = self.estimator_.predict_proba(X)[:,1]
return (proba > self.th_).astype(int)
def predict_proba(self, X):
return self.estimator_.predict_proba(X)
When calling fit:
- a validation (
X_valandy_val) set is randomly generated from the received data; - the
estimatoris fitted onX_trainandy_train; - probabilities (
prob_val) are retrieved for the 1’s class onX_val; - an optimal threshold value is found on
X_valby optimizing a metric of choice (fbeta_scorein our case).
When calling predict: probabilities for the 1’s class are generated and cast into binary classes by the optimal threshold value found.
model = ThresholdClassifier(RandomForestClassifier()).fit(X_train, y_train)
pred_clas = model.predict(X_test)
ThresholdClassifier can be used with any sklearn classifier which produces probabilities. It can be easily customized according to different needs. It’s very useful in conjunction with GridSearchCV/RandomSearchCV to connect the search of parameters with the tune of a classification threshold.
model = RandomizedSearchCV(
ThresholdClassifier(RandomForestClassifier()),
dict(n_estimators=stats.randint(50,300)),
n_iter=20, random_state=1234,
cv=5, n_jobs=-1,
).fit(X_train, y_train)