Keras: class weights (class_weight) for one-hot encoding
Question:
I’d like to use class_weight argument in keras model.fit to handle the imbalanced training data. By looking at some documents, I understood we can pass a dictionary like this:
class_weight = {0 : 1,
1: 1,
2: 5}
(In this example, class-2 will get higher penalty in the loss function.)
The problem is that my network’s output has one-hot encoding i.e. class-0 = (1, 0, 0), class-1 = (0, 1, 0), and class-3 = (0, 0, 1).
How can we use the class_weight for one-hot encoded output?
By looking at some codes in Keras, it looks like _feed_output_names contain a list of output classes, but in my case, model.output_names/model._feed_output_names returns ['dense_1']
Related: How to set class weights for imbalanced classes in Keras?
Answers:
I guess we can use sample_weights instead. Inside Keras, actually, class_weights are converted to sample_weights.
sample_weight: optional array of the same length as x, containing
weights to apply to the model’s loss for each sample. In the case of
temporal data, you can pass a 2D array with shape (samples,
sequence_length), to apply a different weight to every timestep of
every sample. In this case you should make sure to specify
sample_weight_mode="temporal" in compile().
sample_weights is used to provide a weight for each training sample. That means that you should pass a 1D array with the same number of elements as your training samples (indicating the weight for each of those samples)
A little bit of a convoluted answer, but the best I’ve found so far. This assumes your data is one-hot encoded, multi-class, and working only on the labels DataFrame df_y:
import pandas as pd
import numpy as np
# Create a pd.series that represents the categorical class of each one-hot encoded row
y_classes = df_y.idxmax(1, skipna=False)
from sklearn.preprocessing import LabelEncoder
# Instantiate the label encoder
le = LabelEncoder()
# Fit the label encoder to our label series
le.fit(list(y_classes))
# Create integer based labels Series
y_integers = le.transform(list(y_classes))
# Create dict of labels : integer representation
labels_and_integers = dict(zip(y_classes, y_integers))
from sklearn.utils.class_weight import compute_class_weight, compute_sample_weight
class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
sample_weights = compute_sample_weight('balanced', y_integers)
class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
This results in a sample_weights vector computed to balance an imbalanced dataset which can be passed to the Keras sample_weight property, and a class_weights_dict that can be fed to the Keras class_weight property in the .fit method. You don’t really want to use both, just choose one. I’m using class_weight right now because it’s complicated to get sample_weight working with fit_generator.
in _standardize_weights, keras does:
if y.shape[1] > 1:
y_classes = y.argmax(axis=1)
so basically, if you choose to use one-hot encoding, the classes are the column index.
You may also ask yourself how you can map the column index to the original classes of your data.
Well, if you use the LabelEncoder class of scikit learn to perform one-hot encoding, the column index maps the order of the unique labels computed by the .fit function.
The doc says
Extract an ordered array of unique labels
Example:
from sklearn.preprocessing import LabelBinarizer
y=[4,1,2,8]
l=LabelBinarizer()
y_transformed=l.fit_transorm(y)
y_transormed
> array([[0, 0, 1, 0],
[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 0, 1]])
l.classes_
> array([1, 2, 4, 8])
As a conclusion, the keys of the class_weights dictionary should reflect the order in the classes_ attribute of the encoder.
Here’s a solution that’s a bit shorter and faster. If your one-hot encoded y is a np.array:
import numpy as np
from sklearn.utils.class_weight import compute_class_weight
y_integers = np.argmax(y, axis=1)
class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
d_class_weights = dict(enumerate(class_weights))
d_class_weights can then be passed to class_weight in .fit.
I’d like to use class_weight argument in keras model.fit to handle the imbalanced training data. By looking at some documents, I understood we can pass a dictionary like this:
class_weight = {0 : 1,
1: 1,
2: 5}
(In this example, class-2 will get higher penalty in the loss function.)
The problem is that my network’s output has one-hot encoding i.e. class-0 = (1, 0, 0), class-1 = (0, 1, 0), and class-3 = (0, 0, 1).
How can we use the class_weight for one-hot encoded output?
By looking at some codes in Keras, it looks like _feed_output_names contain a list of output classes, but in my case, model.output_names/model._feed_output_names returns ['dense_1']
Related: How to set class weights for imbalanced classes in Keras?
I guess we can use sample_weights instead. Inside Keras, actually, class_weights are converted to sample_weights.
sample_weight: optional array of the same length as x, containing
weights to apply to the model’s loss for each sample. In the case of
temporal data, you can pass a 2D array with shape (samples,
sequence_length), to apply a different weight to every timestep of
every sample. In this case you should make sure to specify
sample_weight_mode="temporal" in compile().
sample_weights is used to provide a weight for each training sample. That means that you should pass a 1D array with the same number of elements as your training samples (indicating the weight for each of those samples)
A little bit of a convoluted answer, but the best I’ve found so far. This assumes your data is one-hot encoded, multi-class, and working only on the labels DataFrame df_y:
import pandas as pd
import numpy as np
# Create a pd.series that represents the categorical class of each one-hot encoded row
y_classes = df_y.idxmax(1, skipna=False)
from sklearn.preprocessing import LabelEncoder
# Instantiate the label encoder
le = LabelEncoder()
# Fit the label encoder to our label series
le.fit(list(y_classes))
# Create integer based labels Series
y_integers = le.transform(list(y_classes))
# Create dict of labels : integer representation
labels_and_integers = dict(zip(y_classes, y_integers))
from sklearn.utils.class_weight import compute_class_weight, compute_sample_weight
class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
sample_weights = compute_sample_weight('balanced', y_integers)
class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
This results in a sample_weights vector computed to balance an imbalanced dataset which can be passed to the Keras sample_weight property, and a class_weights_dict that can be fed to the Keras class_weight property in the .fit method. You don’t really want to use both, just choose one. I’m using class_weight right now because it’s complicated to get sample_weight working with fit_generator.
in _standardize_weights, keras does:
if y.shape[1] > 1:
y_classes = y.argmax(axis=1)
so basically, if you choose to use one-hot encoding, the classes are the column index.
You may also ask yourself how you can map the column index to the original classes of your data.
Well, if you use the LabelEncoder class of scikit learn to perform one-hot encoding, the column index maps the order of the unique labels computed by the .fit function.
The doc says
Extract an ordered array of unique labels
Example:
from sklearn.preprocessing import LabelBinarizer
y=[4,1,2,8]
l=LabelBinarizer()
y_transformed=l.fit_transorm(y)
y_transormed
> array([[0, 0, 1, 0],
[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 0, 1]])
l.classes_
> array([1, 2, 4, 8])
As a conclusion, the keys of the class_weights dictionary should reflect the order in the classes_ attribute of the encoder.
Here’s a solution that’s a bit shorter and faster. If your one-hot encoded y is a np.array:
import numpy as np
from sklearn.utils.class_weight import compute_class_weight
y_integers = np.argmax(y, axis=1)
class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
d_class_weights = dict(enumerate(class_weights))
d_class_weights can then be passed to class_weight in .fit.