Apply StandardScaler to parts of a data set
Question:
I want to use sklearn‘s StandardScaler. Is it possible to apply it to some feature columns but not others?
For instance, say my data is:
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
Age Name Weight
0 18 3 68
1 92 4 59
2 98 6 49
col_names = ['Name', 'Age', 'Weight']
features = data[col_names]
I fit and transform the data
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
scaled_features = pd.DataFrame(features, columns = col_names)
Name Age Weight
0 -1.069045 -1.411004 1.202703
1 -0.267261 0.623041 0.042954
2 1.336306 0.787964 -1.245657
But of course the names are not really integers but strings and I don’t want to standardize them. How can I apply the fit and transform methods only on the columns Age and Weight?
Answers:
Update:
Currently the best way to handle this is to use ColumnTransformer as explained here.
First create a copy of your dataframe:
scaled_features = data.copy()
Don’t include the Name column in the transformation:
col_names = ['Age', 'Weight']
features = scaled_features[col_names]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
Now, don’t create a new dataframe but assign the result to those two columns:
scaled_features[col_names] = features
print(scaled_features)
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
A more pythonic way to do this –
from sklearn.preprocessing import StandardScaler
data[['Age','Weight']] = data[['Age','Weight']].apply(
lambda x: StandardScaler().fit_transform(x))
data
Output –
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
Another option would be to drop Name column before scaling then merge it back together:
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
from sklearn.preprocessing import StandardScaler
# Save the variable you don't want to scale
name_var = data['Name']
# Fit scaler to your data
scaler.fit(data.drop('Name', axis = 1))
# Calculate scaled values and store them in a separate object
scaled_values = scaler.transform(data.drop('Name', axis = 1))
data = pd.DataFrame(scaled_values, index = data.index, columns = data.drop('ID', axis = 1).columns)
data['Name'] = name_var
print(data)
Introduced in v0.20 is ColumnTransformer which applies transformers to a specified set of columns of an array or pandas DataFrame.
import pandas as pd
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
col_names = ['Name', 'Age', 'Weight']
features = data[col_names]
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
ct = ColumnTransformer([
('somename', StandardScaler(), ['Age', 'Weight'])
], remainder='passthrough')
ct.fit_transform(features)
NB: Like Pipeline it also has a shorthand version make_column_transformer which doesn’t require naming the transformers
Output
-1.41100443, 1.20270298, 3.
0.62304092, 0.04295368, 4.
0.78796352, -1.24565666, 6.
The easiest way I find is:
from sklearn.preprocessing import StandardScaler
# I'm selecting only numericals to scale
numerical = temp.select_dtypes(include='float64').columns
# This will transform the selected columns and merge to the original data frame
temp.loc[:,numerical] = StandardScaler().fit_transform(temp.loc[:,numerical])
Output
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
Late to the party, but here’s my preferred solution:
#load data
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
#list for cols to scale
cols_to_scale = ['Age','Weight']
#create and fit scaler
scaler = StandardScaler()
scaler.fit(data[cols_to_scale])
#scale selected data
data[cols_to_scale] = scaler.transform(data[cols_to_scale])
I want to use sklearn‘s StandardScaler. Is it possible to apply it to some feature columns but not others?
For instance, say my data is:
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
Age Name Weight
0 18 3 68
1 92 4 59
2 98 6 49
col_names = ['Name', 'Age', 'Weight']
features = data[col_names]
I fit and transform the data
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
scaled_features = pd.DataFrame(features, columns = col_names)
Name Age Weight
0 -1.069045 -1.411004 1.202703
1 -0.267261 0.623041 0.042954
2 1.336306 0.787964 -1.245657
But of course the names are not really integers but strings and I don’t want to standardize them. How can I apply the fit and transform methods only on the columns Age and Weight?
Update:
Currently the best way to handle this is to use ColumnTransformer as explained here.
First create a copy of your dataframe:
scaled_features = data.copy()
Don’t include the Name column in the transformation:
col_names = ['Age', 'Weight']
features = scaled_features[col_names]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
Now, don’t create a new dataframe but assign the result to those two columns:
scaled_features[col_names] = features
print(scaled_features)
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
A more pythonic way to do this –
from sklearn.preprocessing import StandardScaler
data[['Age','Weight']] = data[['Age','Weight']].apply(
lambda x: StandardScaler().fit_transform(x))
data
Output –
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
Another option would be to drop Name column before scaling then merge it back together:
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
from sklearn.preprocessing import StandardScaler
# Save the variable you don't want to scale
name_var = data['Name']
# Fit scaler to your data
scaler.fit(data.drop('Name', axis = 1))
# Calculate scaled values and store them in a separate object
scaled_values = scaler.transform(data.drop('Name', axis = 1))
data = pd.DataFrame(scaled_values, index = data.index, columns = data.drop('ID', axis = 1).columns)
data['Name'] = name_var
print(data)
Introduced in v0.20 is ColumnTransformer which applies transformers to a specified set of columns of an array or pandas DataFrame.
import pandas as pd
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
col_names = ['Name', 'Age', 'Weight']
features = data[col_names]
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
ct = ColumnTransformer([
('somename', StandardScaler(), ['Age', 'Weight'])
], remainder='passthrough')
ct.fit_transform(features)
NB: Like Pipeline it also has a shorthand version make_column_transformer which doesn’t require naming the transformers
Output
-1.41100443, 1.20270298, 3.
0.62304092, 0.04295368, 4.
0.78796352, -1.24565666, 6.
The easiest way I find is:
from sklearn.preprocessing import StandardScaler
# I'm selecting only numericals to scale
numerical = temp.select_dtypes(include='float64').columns
# This will transform the selected columns and merge to the original data frame
temp.loc[:,numerical] = StandardScaler().fit_transform(temp.loc[:,numerical])
Output
Age Name Weight
0 -1.411004 3 1.202703
1 0.623041 4 0.042954
2 0.787964 6 -1.245657
Late to the party, but here’s my preferred solution:
#load data
data = pd.DataFrame({'Name' : [3, 4,6], 'Age' : [18, 92,98], 'Weight' : [68, 59,49]})
#list for cols to scale
cols_to_scale = ['Age','Weight']
#create and fit scaler
scaler = StandardScaler()
scaler.fit(data[cols_to_scale])
#scale selected data
data[cols_to_scale] = scaler.transform(data[cols_to_scale])