grid search over multiple classifiers
Question:
Is there a better inbuilt way to do grid search and test multiple models in a single pipeline? Of course the parameters of the models would be different, which made is complicated for me to figure this out. Here is what I did:
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.grid_search import GridSearchCV
def grid_search():
pipeline1 = Pipeline((
('clf', RandomForestClassifier()),
('vec2', TfidfTransformer())
))
pipeline2 = Pipeline((
('clf', KNeighborsClassifier()),
))
pipeline3 = Pipeline((
('clf', SVC()),
))
pipeline4 = Pipeline((
('clf', MultinomialNB()),
))
parameters1 = {
'clf__n_estimators': [10, 20, 30],
'clf__criterion': ['gini', 'entropy'],
'clf__max_features': [5, 10, 15],
'clf__max_depth': ['auto', 'log2', 'sqrt', None]
}
parameters2 = {
'clf__n_neighbors': [3, 7, 10],
'clf__weights': ['uniform', 'distance']
}
parameters3 = {
'clf__C': [0.01, 0.1, 1.0],
'clf__kernel': ['rbf', 'poly'],
'clf__gamma': [0.01, 0.1, 1.0],
}
parameters4 = {
'clf__alpha': [0.01, 0.1, 1.0]
}
pars = [parameters1, parameters2, parameters3, parameters4]
pips = [pipeline1, pipeline2, pipeline3, pipeline4]
print "starting Gridsearch"
for i in range(len(pars)):
gs = GridSearchCV(pips[i], pars[i], verbose=2, refit=False, n_jobs=-1)
gs = gs.fit(X_train, y_train)
print "finished Gridsearch"
print gs.best_score_
However, this approach is still giving the best model within each classifier, and not comparing between classifiers.
Answers:
Instead of using Grid Search for hyperparameter selection, you can use the ‘hyperopt’ library.
Please have a look at section 2.2 of this page. In the above case, you can use an hp.choice expression to select among the various pipelines and then define the parameter expressions for each one separately.
In your objective function, you need to have a check depending on the pipeline chosen and return the CV score for the selected pipeline and parameters (possibly via cross_val_score).
The trials object at the end of the execution, will indicate the best pipeline and parameters overall.
Although the solution from dubek is more straight forward, it does not help with interactions between parameters of pipeline elements that come before the classfier. Therefore, I have written a helper class to deal with it, and can be included in the default Pipeline setting of scikit. A minimal example:
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler, MaxAbsScaler
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
from pipelinehelper import PipelineHelper
iris = datasets.load_iris()
X_iris = iris.data
y_iris = iris.target
pipe = Pipeline([
('scaler', PipelineHelper([
('std', StandardScaler()),
('max', MaxAbsScaler()),
])),
('classifier', PipelineHelper([
('svm', LinearSVC()),
('rf', RandomForestClassifier()),
])),
])
params = {
'scaler__selected_model': pipe.named_steps['scaler'].generate({
'std__with_mean': [True, False],
'std__with_std': [True, False],
'max__copy': [True], # just for displaying
}),
'classifier__selected_model': pipe.named_steps['classifier'].generate({
'svm__C': [0.1, 1.0],
'rf__n_estimators': [100, 20],
})
}
grid = GridSearchCV(pipe, params, scoring='accuracy', verbose=1)
grid.fit(X_iris, y_iris)
print(grid.best_params_)
print(grid.best_score_)
It can also be used for other elements of the pipeline, not just the classifier.
Code is on github if anyone wants to check it out.
Edit: I have published this on PyPI if anyone is interested, just install ti using pip install pipelinehelper.
Another Simple solution to the problem.
First load all the estimators. Here I will be using classifiers mostly.
logi=LogisticRegression(penalty="elasticnet",l1_ratio=0.5,solver="saga", random_state=4, n_jobs=-1)
rf=RandomForestClassifier(random_state=4, n_jobs=-1, max_features="auto", warm_start=True)
gb=GradientBoostingClassifier(random_state=4, subsample=0.8, max_features="auto", warm_start=True)
svc=SVC(random_state=4, kernel='rbf')
ex=ExtraTreesClassifier(random_state=4, n_jobs=-1, max_features="auto", warm_start=True)
After that create a list of classifiers:
ensemble_clf=[rf, ex, gb, svc]
Now, Create all parameters for each classifier/estimator:-
params1={"max_depth": range(5,30,5), "min_samples_leaf": range(1,30,2),
"n_estimators":range(100,2000,200)}
params2={"criterion":["gini", "entropy"],"max_depth": range(5,30,5),
"min_samples_leaf": range(1,30,2), "n_estimators":range(100,2000,200)}
params3={"learning_rate":[0.001,0.01,0.1], "n_estimators":range(1000,3000,200)}
params4={"kernel":["rbf", "poly"], "gamma": ["auto", "scale"], "degree":range(1,6,1)}
Now create a list of them:
parameters_list=[params1, params2, params3, params4]
Now, comes the most important part:
Create a string names for all the models/classifiers or estimators:
This is used to create the Dataframes for comparison
model_log=["_rf", "_ex", "_gb", "_svc"]
Now run a for loop and use the Grid search:
for i in range(len(ensemble_clf)):
Grid=GridSearchCV(estimator=ensemble_clf[i], param_grid=parameters_list[i],
n_jobs=-1, cv=3, verbose=3).fit(TrainX_Std, TrainY)
globals()['Grid%s' % model_log[i]]=pd.DataFrame(Grid.cv_results_)
The “globals()[‘Grid%s’ % model_log[i]]=pd.DataFrame(Grid.cv_results_) “ will create dataframes individually for each of the estimators used and can be used to comparison by Sorting and picking up the best parameters of each and every estimator.
Hope this helps.
This is how I did it without a wrapper function.
You can evaluate any number of classifiers. Each one can have multiple parameters for hyperparameter optimization.
The one with best score will be saved to disk using pickle
from sklearn.svm import SVC
from operator import itemgetter
from sklearn.utils import shuffle
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.feature_extraction.text import TfidfVectorizer
import operator
#pipeline parameters
parameters =
[
{
'clf': [MultinomialNB()],
'tf-idf__stop_words': ['english', None],
'clf__alpha': [0.001, 0.1, 1, 10, 100]
},
{
'clf': [SVC()],
'tf-idf__stop_words': ['english', None],
'clf__C': [0.001, 0.1, 1, 10, 100, 10e5],
'clf__kernel': ['linear', 'rbf'],
'clf__class_weight': ['balanced'],
'clf__probability': [True]
},
{
'clf': [DecisionTreeClassifier()],
'tf-idf__stop_words': ['english', None],
'clf__criterion': ['gini','entropy'],
'clf__splitter': ['best','random'],
'clf__class_weight':['balanced', None]
}
]
#evaluating multiple classifiers
#based on pipeline parameters
#-------------------------------
result=[]
for params in parameters:
#classifier
clf = params['clf'][0]
#getting arguments by
#popping out classifier
params.pop('clf')
#pipeline
steps = [('tf-idf', TfidfVectorizer()), ('clf',clf)]
#cross validation using
#Grid Search
grid = GridSearchCV(Pipeline(steps), param_grid=params, cv=3)
grid.fit(features, labels)
#storing result
result.append
(
{
'grid': grid,
'classifier': grid.best_estimator_,
'best score': grid.best_score_,
'best params': grid.best_params_,
'cv': grid.cv
}
)
#sorting result by best score
result = sorted(result, key=operator.itemgetter('best score'),reverse=True)
#saving best classifier
grid = result[0]['grid']
joblib.dump(grid, 'classifier.pickle')
Another option is to use the HyperclassifierSearch (Github) package. It is close to the solution of bmurauer above.
However, you might
- find the DataFrame output for the best model helpful which skips timing info as the default
- find the three usage examples helpful
- like the shorter core code with around 100 lines
I developed the HyperclassifierSearch package (start with a pip install HyperclassifierSearch) based on the code from David Batista which I liked for the conciseness of the code.
Detail for 1., usage of the hyperclassifier evaluate_model function:
search = HyperclassifierSearch(models, params)
best_model = search.train_model(X, y)
search.evaluate_model(sort_by='mean_test_score', show_timing_info=False) # default parameters explicitly given
Is there a better inbuilt way to do grid search and test multiple models in a single pipeline? Of course the parameters of the models would be different, which made is complicated for me to figure this out. Here is what I did:
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.grid_search import GridSearchCV
def grid_search():
pipeline1 = Pipeline((
('clf', RandomForestClassifier()),
('vec2', TfidfTransformer())
))
pipeline2 = Pipeline((
('clf', KNeighborsClassifier()),
))
pipeline3 = Pipeline((
('clf', SVC()),
))
pipeline4 = Pipeline((
('clf', MultinomialNB()),
))
parameters1 = {
'clf__n_estimators': [10, 20, 30],
'clf__criterion': ['gini', 'entropy'],
'clf__max_features': [5, 10, 15],
'clf__max_depth': ['auto', 'log2', 'sqrt', None]
}
parameters2 = {
'clf__n_neighbors': [3, 7, 10],
'clf__weights': ['uniform', 'distance']
}
parameters3 = {
'clf__C': [0.01, 0.1, 1.0],
'clf__kernel': ['rbf', 'poly'],
'clf__gamma': [0.01, 0.1, 1.0],
}
parameters4 = {
'clf__alpha': [0.01, 0.1, 1.0]
}
pars = [parameters1, parameters2, parameters3, parameters4]
pips = [pipeline1, pipeline2, pipeline3, pipeline4]
print "starting Gridsearch"
for i in range(len(pars)):
gs = GridSearchCV(pips[i], pars[i], verbose=2, refit=False, n_jobs=-1)
gs = gs.fit(X_train, y_train)
print "finished Gridsearch"
print gs.best_score_
However, this approach is still giving the best model within each classifier, and not comparing between classifiers.
Instead of using Grid Search for hyperparameter selection, you can use the ‘hyperopt’ library.
Please have a look at section 2.2 of this page. In the above case, you can use an hp.choice expression to select among the various pipelines and then define the parameter expressions for each one separately.
In your objective function, you need to have a check depending on the pipeline chosen and return the CV score for the selected pipeline and parameters (possibly via cross_val_score).
The trials object at the end of the execution, will indicate the best pipeline and parameters overall.
Although the solution from dubek is more straight forward, it does not help with interactions between parameters of pipeline elements that come before the classfier. Therefore, I have written a helper class to deal with it, and can be included in the default Pipeline setting of scikit. A minimal example:
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler, MaxAbsScaler
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
from pipelinehelper import PipelineHelper
iris = datasets.load_iris()
X_iris = iris.data
y_iris = iris.target
pipe = Pipeline([
('scaler', PipelineHelper([
('std', StandardScaler()),
('max', MaxAbsScaler()),
])),
('classifier', PipelineHelper([
('svm', LinearSVC()),
('rf', RandomForestClassifier()),
])),
])
params = {
'scaler__selected_model': pipe.named_steps['scaler'].generate({
'std__with_mean': [True, False],
'std__with_std': [True, False],
'max__copy': [True], # just for displaying
}),
'classifier__selected_model': pipe.named_steps['classifier'].generate({
'svm__C': [0.1, 1.0],
'rf__n_estimators': [100, 20],
})
}
grid = GridSearchCV(pipe, params, scoring='accuracy', verbose=1)
grid.fit(X_iris, y_iris)
print(grid.best_params_)
print(grid.best_score_)
It can also be used for other elements of the pipeline, not just the classifier.
Code is on github if anyone wants to check it out.
Edit: I have published this on PyPI if anyone is interested, just install ti using pip install pipelinehelper.
Another Simple solution to the problem.
First load all the estimators. Here I will be using classifiers mostly.
logi=LogisticRegression(penalty="elasticnet",l1_ratio=0.5,solver="saga", random_state=4, n_jobs=-1)
rf=RandomForestClassifier(random_state=4, n_jobs=-1, max_features="auto", warm_start=True)
gb=GradientBoostingClassifier(random_state=4, subsample=0.8, max_features="auto", warm_start=True)
svc=SVC(random_state=4, kernel='rbf')
ex=ExtraTreesClassifier(random_state=4, n_jobs=-1, max_features="auto", warm_start=True)
After that create a list of classifiers:
ensemble_clf=[rf, ex, gb, svc]
Now, Create all parameters for each classifier/estimator:-
params1={"max_depth": range(5,30,5), "min_samples_leaf": range(1,30,2),
"n_estimators":range(100,2000,200)}
params2={"criterion":["gini", "entropy"],"max_depth": range(5,30,5),
"min_samples_leaf": range(1,30,2), "n_estimators":range(100,2000,200)}
params3={"learning_rate":[0.001,0.01,0.1], "n_estimators":range(1000,3000,200)}
params4={"kernel":["rbf", "poly"], "gamma": ["auto", "scale"], "degree":range(1,6,1)}
Now create a list of them:
parameters_list=[params1, params2, params3, params4]
Now, comes the most important part:
Create a string names for all the models/classifiers or estimators:
This is used to create the Dataframes for comparison
model_log=["_rf", "_ex", "_gb", "_svc"]
Now run a for loop and use the Grid search:
for i in range(len(ensemble_clf)):
Grid=GridSearchCV(estimator=ensemble_clf[i], param_grid=parameters_list[i],
n_jobs=-1, cv=3, verbose=3).fit(TrainX_Std, TrainY)
globals()['Grid%s' % model_log[i]]=pd.DataFrame(Grid.cv_results_)
The “globals()[‘Grid%s’ % model_log[i]]=pd.DataFrame(Grid.cv_results_) “ will create dataframes individually for each of the estimators used and can be used to comparison by Sorting and picking up the best parameters of each and every estimator.
Hope this helps.
This is how I did it without a wrapper function.
You can evaluate any number of classifiers. Each one can have multiple parameters for hyperparameter optimization.
The one with best score will be saved to disk using pickle
from sklearn.svm import SVC
from operator import itemgetter
from sklearn.utils import shuffle
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.feature_extraction.text import TfidfVectorizer
import operator
#pipeline parameters
parameters =
[
{
'clf': [MultinomialNB()],
'tf-idf__stop_words': ['english', None],
'clf__alpha': [0.001, 0.1, 1, 10, 100]
},
{
'clf': [SVC()],
'tf-idf__stop_words': ['english', None],
'clf__C': [0.001, 0.1, 1, 10, 100, 10e5],
'clf__kernel': ['linear', 'rbf'],
'clf__class_weight': ['balanced'],
'clf__probability': [True]
},
{
'clf': [DecisionTreeClassifier()],
'tf-idf__stop_words': ['english', None],
'clf__criterion': ['gini','entropy'],
'clf__splitter': ['best','random'],
'clf__class_weight':['balanced', None]
}
]
#evaluating multiple classifiers
#based on pipeline parameters
#-------------------------------
result=[]
for params in parameters:
#classifier
clf = params['clf'][0]
#getting arguments by
#popping out classifier
params.pop('clf')
#pipeline
steps = [('tf-idf', TfidfVectorizer()), ('clf',clf)]
#cross validation using
#Grid Search
grid = GridSearchCV(Pipeline(steps), param_grid=params, cv=3)
grid.fit(features, labels)
#storing result
result.append
(
{
'grid': grid,
'classifier': grid.best_estimator_,
'best score': grid.best_score_,
'best params': grid.best_params_,
'cv': grid.cv
}
)
#sorting result by best score
result = sorted(result, key=operator.itemgetter('best score'),reverse=True)
#saving best classifier
grid = result[0]['grid']
joblib.dump(grid, 'classifier.pickle')
Another option is to use the HyperclassifierSearch (Github) package. It is close to the solution of bmurauer above.
However, you might
- find the DataFrame output for the best model helpful which skips timing info as the default
- find the three usage examples helpful
- like the shorter core code with around 100 lines
I developed the HyperclassifierSearch package (start with a pip install HyperclassifierSearch) based on the code from David Batista which I liked for the conciseness of the code.
Detail for 1., usage of the hyperclassifier evaluate_model function:
search = HyperclassifierSearch(models, params)
best_model = search.train_model(X, y)
search.evaluate_model(sort_by='mean_test_score', show_timing_info=False) # default parameters explicitly given