How to tune parameters in Random Forest, using Scikit Learn?
Question:
class sklearn.ensemble.RandomForestClassifier(n_estimators=10,
criterion='gini',
max_depth=None,
min_samples_split=2,
min_samples_leaf=1,
min_weight_fraction_leaf=0.0,
max_features='auto',
max_leaf_nodes=None,
bootstrap=True,
oob_score=False,
n_jobs=1,
random_state=None,
verbose=0,
warm_start=False,
class_weight=None)
I’m using a random forest model with 9 samples and about 7000 attributes. Of these samples, there are 3 categories that my classifier recognizes.
I know this is far from ideal conditions but I’m trying to figure out which attributes are the most important in feature predictions. Which parameters would be the best to tweak for optimizing feature importance?
I tried different n_estimators and noticed that the amount of “significant features” (i.e. nonzero values in the feature_importances_ array) increased dramatically.
I’ve read through the documentation but if anyone has any experience in this, I would like to know which parameters are the best to tune and a brief explanation why.
Answers:
The crucial parts are usually three elements:
- number of estimators – usually bigger the forest the better, there is small chance of overfitting here
- max depth of each tree (default none, leading to full tree) – reduction of the maximum depth helps fighting with overfitting
- max features per split (default
sqrt(d)) – you might one to play around a bit as it significantly alters behaviour of the whole tree. sqrt heuristic is usually a good starting point but an actual sweet spot might be somewhere else
From my experience, there are three features worth exploring with the sklearn RandomForestClassifier, in order of importance:
-
n_estimators
-
max_features
-
criterion
n_estimators is not really worth optimizing. The more estimators you give it, the better it will do. 500 or 1000 is usually sufficient.
max_features is worth exploring for many different values. It may have a large impact on the behavior of the RF because it decides how many features each tree in the RF considers at each split.
criterion may have a small impact, but usually the default is fine. If you have the time, try it out.
Make sure to use sklearn’s GridSearch (preferably GridSearchCV, but your data set size is too small) when trying out these parameters.
If I understand your question correctly, though, you only have 9 samples and 3 classes? Presumably 3 samples per class? It’s very, very likely that your RF is going to overfit with that little amount of data, unless they are good, representative records.
This wonderful article has a detailed explanation of tunable parameters, how to track performance vs speed trade-off, some practical tips, and how to perform grid-search.
n_estimators is good one as others said. It is also good at dealing with the overfitting when increasing it.
But I think min_sample_split is also helpful when dealing with overfitting occurred in a small-sample but big-features dataset.
class sklearn.ensemble.RandomForestClassifier(n_estimators=10,
criterion='gini',
max_depth=None,
min_samples_split=2,
min_samples_leaf=1,
min_weight_fraction_leaf=0.0,
max_features='auto',
max_leaf_nodes=None,
bootstrap=True,
oob_score=False,
n_jobs=1,
random_state=None,
verbose=0,
warm_start=False,
class_weight=None)
I’m using a random forest model with 9 samples and about 7000 attributes. Of these samples, there are 3 categories that my classifier recognizes.
I know this is far from ideal conditions but I’m trying to figure out which attributes are the most important in feature predictions. Which parameters would be the best to tweak for optimizing feature importance?
I tried different n_estimators and noticed that the amount of “significant features” (i.e. nonzero values in the feature_importances_ array) increased dramatically.
I’ve read through the documentation but if anyone has any experience in this, I would like to know which parameters are the best to tune and a brief explanation why.
The crucial parts are usually three elements:
- number of estimators – usually bigger the forest the better, there is small chance of overfitting here
- max depth of each tree (default none, leading to full tree) – reduction of the maximum depth helps fighting with overfitting
- max features per split (default
sqrt(d)) – you might one to play around a bit as it significantly alters behaviour of the whole tree. sqrt heuristic is usually a good starting point but an actual sweet spot might be somewhere else
From my experience, there are three features worth exploring with the sklearn RandomForestClassifier, in order of importance:
-
n_estimators -
max_features -
criterion
n_estimators is not really worth optimizing. The more estimators you give it, the better it will do. 500 or 1000 is usually sufficient.
max_features is worth exploring for many different values. It may have a large impact on the behavior of the RF because it decides how many features each tree in the RF considers at each split.
criterion may have a small impact, but usually the default is fine. If you have the time, try it out.
Make sure to use sklearn’s GridSearch (preferably GridSearchCV, but your data set size is too small) when trying out these parameters.
If I understand your question correctly, though, you only have 9 samples and 3 classes? Presumably 3 samples per class? It’s very, very likely that your RF is going to overfit with that little amount of data, unless they are good, representative records.
This wonderful article has a detailed explanation of tunable parameters, how to track performance vs speed trade-off, some practical tips, and how to perform grid-search.
n_estimators is good one as others said. It is also good at dealing with the overfitting when increasing it.
But I think min_sample_split is also helpful when dealing with overfitting occurred in a small-sample but big-features dataset.