How to use SMOTENC inside pipeline (Error: Some of the categorical indices are out of range)?
Question:
I would greatly appreciate if you could let me know how to use SMOTENC. I wrote:
# Data
XX = pd.read_csv('Financial Distress.csv')
y = np.array(XX['Financial Distress'].values.tolist())
y = np.array([0 if i > -0.50 else 1 for i in y])
Na = np.array(pd.read_csv('Na.csv', header=None).values)
XX = XX.iloc[:, 3:127]
# Use get-dummies to convert categorical features into dummy ones
dis_features = ['x121']
X = pd.get_dummies(XX, columns=dis_features)
# # Divide Data into Train and Test
indices = np.arange(y.shape[0])
X_train, X_test, y_train, y_test, idx_train, idx_test = train_test_split(X, y, indices, stratify=y, test_size=0.3,
random_state=42)
num_indices=list(X)[:X.shape[1]-37]
cat_indices=list(X)[X.shape[1]-37:]
num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,123:160]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
pipeline=Pipeline(steps= [
# Categorical features
('feature_processing', FeatureUnion(transformer_list = [
('categorical', MultiColumn(cat_indices)),
#numeric
('numeric', Pipeline(steps = [
('select', MultiColumn(num_indices)),
('scale', StandardScaler())
]))
])),
('clf', rg)
]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices1), pipeline)
# # Grid Search to determine best params
cv=StratifiedKFold(n_splits=5,random_state=42)
rg_cv = GridSearchCV(pipeline_with_resampling, param_grid, cv=cv, scoring = 'f1')
rg_cv.fit(X_train, y_train)
Therefore, as it is indicated I have 5 categorical features. Really, indices 123 to 160 are related to one categorical feature with 37 possible values which is converted into 37 columns using get_dummies. Unfortunately, it throws the following error:
Traceback (most recent call last):
File "D:/mifs-master_2/MU/learning-from-imbalanced-classes-master/learning-from-imbalanced-classes-master/continuous/Final Logit/SMOTENC/logit-final - Copy.py", line 424, in <module>
rg_cv.fit(X_train, y_train)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 722, in fit
self._run_search(evaluate_candidates)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 1191, in _run_search
evaluate_candidates(ParameterGrid(self.param_grid))
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 711, in evaluate_candidates
cv.split(X, y, groups)))
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 917, in __call__
if self.dispatch_one_batch(iterator):
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 759, in dispatch_one_batch
self._dispatch(tasks)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblib_parallel_backends.py", line 182, in apply_async
result = ImmediateResult(func)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblib_parallel_backends.py", line 549, in __init__
self.results = batch()
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 225, in __call__
for func, args, kwargs in self.items]
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 225, in <listcomp>
for func, args, kwargs in self.items]
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_validation.py", line 528, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 237, in fit
Xt, yt, fit_params = self._fit(X, y, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 200, in _fit
cloned_transformer, Xt, yt, **fit_params_steps[name])
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibmemory.py", line 342, in __call__
return self.func(*args, **kwargs)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 576, in _fit_resample_one
X_res, y_res = sampler.fit_resample(X, y, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnbase.py", line 85, in fit_resample
output = self._fit_resample(X, y)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnover_sampling_smote.py", line 940, in _fit_resample
self._validate_estimator()
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnover_sampling_smote.py", line 933, in _validate_estimator
' should be between 0 and {}'.format(self.n_features_))
ValueError: Some of the categorical indices are out of range. Indices should be between 0 and 160
Thanks in advance.
Answers:
You can not dummies your categorical variables and use it later SMOTENC because it already implements in its algorithm get_dummies what will bias your model.
However, I recommend using SMOTE () instead of SMOTENC (), but in this case you must first apply get_demmies.
As it follows, two pipelines should be used:
num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:120,121:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,120]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
cat_indices = [94, 96, 98, 99, 120]
from imblearn.pipeline import make_pipeline
pipeline=Pipeline(steps= [
# Categorical features
('feature_processing', FeatureUnion(transformer_list = [
('categorical', MultiColumn(cat_indices1)),
#numeric
('numeric', Pipeline(steps = [
('select', MultiColumn(num_indices1)),
('scale', StandardScaler())
]))
])),
('clf', rg)
]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices), pipeline)
You cannot use scikit learn pipeline with imblearn pipeline. The imblearn pipeline implements fit_sample as well as fit_predict. Sklearn pipeline onle implements fit_predict. You cannot combine them.
First, don’t do the get_dummies. Then, change the way you do your categorical_features, and put a list of booleans for if it’s categorical or not.
Try this:
cat_cols = []
for col in x.columns:
if x[col].dtype == 'object': #or 'category' if that's the case
cat_cols.append(True)
else:
cat_cols.append(False)
Then pass cat_cols to your SMOTENC:
smote_nc = SMOTENC(categorical_features=cat_cols, random_state=0)
I would greatly appreciate if you could let me know how to use SMOTENC. I wrote:
# Data
XX = pd.read_csv('Financial Distress.csv')
y = np.array(XX['Financial Distress'].values.tolist())
y = np.array([0 if i > -0.50 else 1 for i in y])
Na = np.array(pd.read_csv('Na.csv', header=None).values)
XX = XX.iloc[:, 3:127]
# Use get-dummies to convert categorical features into dummy ones
dis_features = ['x121']
X = pd.get_dummies(XX, columns=dis_features)
# # Divide Data into Train and Test
indices = np.arange(y.shape[0])
X_train, X_test, y_train, y_test, idx_train, idx_test = train_test_split(X, y, indices, stratify=y, test_size=0.3,
random_state=42)
num_indices=list(X)[:X.shape[1]-37]
cat_indices=list(X)[X.shape[1]-37:]
num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,123:160]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
pipeline=Pipeline(steps= [
# Categorical features
('feature_processing', FeatureUnion(transformer_list = [
('categorical', MultiColumn(cat_indices)),
#numeric
('numeric', Pipeline(steps = [
('select', MultiColumn(num_indices)),
('scale', StandardScaler())
]))
])),
('clf', rg)
]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices1), pipeline)
# # Grid Search to determine best params
cv=StratifiedKFold(n_splits=5,random_state=42)
rg_cv = GridSearchCV(pipeline_with_resampling, param_grid, cv=cv, scoring = 'f1')
rg_cv.fit(X_train, y_train)
Therefore, as it is indicated I have 5 categorical features. Really, indices 123 to 160 are related to one categorical feature with 37 possible values which is converted into 37 columns using get_dummies. Unfortunately, it throws the following error:
Traceback (most recent call last):
File "D:/mifs-master_2/MU/learning-from-imbalanced-classes-master/learning-from-imbalanced-classes-master/continuous/Final Logit/SMOTENC/logit-final - Copy.py", line 424, in <module>
rg_cv.fit(X_train, y_train)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 722, in fit
self._run_search(evaluate_candidates)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 1191, in _run_search
evaluate_candidates(ParameterGrid(self.param_grid))
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_search.py", line 711, in evaluate_candidates
cv.split(X, y, groups)))
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 917, in __call__
if self.dispatch_one_batch(iterator):
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 759, in dispatch_one_batch
self._dispatch(tasks)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblib_parallel_backends.py", line 182, in apply_async
result = ImmediateResult(func)
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblib_parallel_backends.py", line 549, in __init__
self.results = batch()
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 225, in __call__
for func, args, kwargs in self.items]
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibparallel.py", line 225, in <listcomp>
for func, args, kwargs in self.items]
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnmodel_selection_validation.py", line 528, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 237, in fit
Xt, yt, fit_params = self._fit(X, y, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 200, in _fit
cloned_transformer, Xt, yt, **fit_params_steps[name])
File "C:UsersMarkazi.coAnaconda3libsite-packagessklearnexternalsjoblibmemory.py", line 342, in __call__
return self.func(*args, **kwargs)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnpipeline.py", line 576, in _fit_resample_one
X_res, y_res = sampler.fit_resample(X, y, **fit_params)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnbase.py", line 85, in fit_resample
output = self._fit_resample(X, y)
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnover_sampling_smote.py", line 940, in _fit_resample
self._validate_estimator()
File "C:UsersMarkazi.coAnaconda3libsite-packagesimblearnover_sampling_smote.py", line 933, in _validate_estimator
' should be between 0 and {}'.format(self.n_features_))
ValueError: Some of the categorical indices are out of range. Indices should be between 0 and 160
Thanks in advance.
You can not dummies your categorical variables and use it later SMOTENC because it already implements in its algorithm get_dummies what will bias your model.
However, I recommend using SMOTE () instead of SMOTENC (), but in this case you must first apply get_demmies.
As it follows, two pipelines should be used:
num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:120,121:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,120]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
cat_indices = [94, 96, 98, 99, 120]
from imblearn.pipeline import make_pipeline
pipeline=Pipeline(steps= [
# Categorical features
('feature_processing', FeatureUnion(transformer_list = [
('categorical', MultiColumn(cat_indices1)),
#numeric
('numeric', Pipeline(steps = [
('select', MultiColumn(num_indices1)),
('scale', StandardScaler())
]))
])),
('clf', rg)
]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices), pipeline)
You cannot use scikit learn pipeline with imblearn pipeline. The imblearn pipeline implements fit_sample as well as fit_predict. Sklearn pipeline onle implements fit_predict. You cannot combine them.
First, don’t do the get_dummies. Then, change the way you do your categorical_features, and put a list of booleans for if it’s categorical or not.
Try this:
cat_cols = []
for col in x.columns:
if x[col].dtype == 'object': #or 'category' if that's the case
cat_cols.append(True)
else:
cat_cols.append(False)
Then pass cat_cols to your SMOTENC:
smote_nc = SMOTENC(categorical_features=cat_cols, random_state=0)