How to return history of validation loss in Keras
Question:
Using Anaconda Python 2.7 Windows 10.
I am training a language model using the Keras exmaple:
print('Build model...')
model = Sequential()
model.add(GRU(512, return_sequences=True, input_shape=(maxlen, len(chars))))
model.add(Dropout(0.2))
model.add(GRU(512, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(len(chars)))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
def sample(a, temperature=1.0):
# helper function to sample an index from a probability array
a = np.log(a) / temperature
a = np.exp(a) / np.sum(np.exp(a))
return np.argmax(np.random.multinomial(1, a, 1))
# train the model, output generated text after each iteration
for iteration in range(1, 3):
print()
print('-' * 50)
print('Iteration', iteration)
model.fit(X, y, batch_size=128, nb_epoch=1)
start_index = random.randint(0, len(text) - maxlen - 1)
for diversity in [0.2, 0.5, 1.0, 1.2]:
print()
print('----- diversity:', diversity)
generated = ''
sentence = text[start_index: start_index + maxlen]
generated += sentence
print('----- Generating with seed: "' + sentence + '"')
sys.stdout.write(generated)
for i in range(400):
x = np.zeros((1, maxlen, len(chars)))
for t, char in enumerate(sentence):
x[0, t, char_indices[char]] = 1.
preds = model.predict(x, verbose=0)[0]
next_index = sample(preds, diversity)
next_char = indices_char[next_index]
generated += next_char
sentence = sentence[1:] + next_char
sys.stdout.write(next_char)
sys.stdout.flush()
print()
According to Keras documentation, the model.fit method returns a History callback, which has a history attribute containing the lists of successive losses and other metrics.
hist = model.fit(X, y, validation_split=0.2)
print(hist.history)
After training my model, if I run print(model.history) I get the error:
AttributeError: 'Sequential' object has no attribute 'history'
How do I return my model history after training my model with the above code?
UPDATE
The issue was that:
The following had to first be defined:
from keras.callbacks import History
history = History()
The callbacks option had to be called
model.fit(X_train, Y_train, nb_epoch=5, batch_size=16, callbacks=[history])
But now if I print
print(history.History)
it returns
{}
even though I ran an iteration.
Answers:
The dictionary with histories of “acc”, “loss”, etc. is available and saved in hist.history variable.
It’s been solved.
The losses only save to the History over the epochs. I was running iterations instead of using the Keras built in epochs option.
so instead of doing 4 iterations I now have
model.fit(......, nb_epoch = 4)
Now it returns the loss for each epoch run:
print(hist.history)
{'loss': [1.4358016599558268, 1.399221191623641, 1.381293383180471, 1.3758836857303727]}
Just an example started from
history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=0)
You can use
print(history.history.keys())
to list all data in history.
Then, you can print the history of validation loss like this:
print(history.history['val_loss'])
Another option is CSVLogger: https://keras.io/callbacks/#csvlogger.
It creates a csv file appending the result of each epoch. Even if you interrupt training, you get to see how it evolved.
I have also found that you can use verbose=2 to make keras print out the Losses:
history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=2)
And that would print nice lines like this:
Epoch 1/1
- 5s - loss: 0.6046 - acc: 0.9999 - val_loss: 0.4403 - val_acc: 0.9999
According to their documentation:
verbose: 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch.
The following simple code works great for me:
seqModel =model.fit(x_train, y_train,
batch_size = batch_size,
epochs = num_epochs,
validation_data = (x_test, y_test),
shuffle = True,
verbose=0, callbacks=[TQDMNotebookCallback()]) #for visualization
Make sure you assign the fit function to an output variable. Then you can access that variable very easily
# visualizing losses and accuracy
train_loss = seqModel.history['loss']
val_loss = seqModel.history['val_loss']
train_acc = seqModel.history['acc']
val_acc = seqModel.history['val_acc']
xc = range(num_epochs)
plt.figure()
plt.plot(xc, train_loss)
plt.plot(xc, val_loss)
Hope this helps.
source: https://keras.io/getting-started/faq/#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch
Actually, you can also do it with the iteration method. Because sometimes we might need to use the iteration method instead of the built-in epochs method to visualize the training results after each iteration.
history = [] #Creating a empty list for holding the loss later
for iteration in range(1, 3):
print()
print('-' * 50)
print('Iteration', iteration)
result = model.fit(X, y, batch_size=128, nb_epoch=1) #Obtaining the loss after each training
history.append(result.history['loss']) #Now append the loss after the training to the list.
start_index = random.randint(0, len(text) - maxlen - 1)
print(history)
This way allows you to get the loss you want while maintaining your iteration method.
For plotting the loss directly the following works:
import matplotlib.pyplot as plt
...
model_ = model.fit(X, Y, epochs= ..., verbose=1 )
plt.plot(list(model_.history.values())[0],'k-o')
Those who got still error like me:
Convert model.fit_generator() to model.fit()
Thanks to Alloush,
Following parameter must be included in model.fit():
validation_data = (x_test, y_test)
If it is not defined, val_acc and val_loss will not
be exist at output.
you can get loss and metrics like below:
returned history object is dictionary and you can access model loss( val_loss) or accuracy(val_accuracy) like below:
model_hist=model.fit(train_data,train_lbl,epochs=my_epoch,batch_size=sel_batch_size,validation_data=val_data)
acc=model_hist.history['accuracy']
val_acc=model_hist.history['val_accuracy']
loss=model_hist.history['loss']
val_loss=model_hist.history['val_loss']
dont forget that for getting val_loss or val_accuracy you should specify validation data in the "fit" function.
history = model.fit(partial_train_data, partial_train_targets,
validation_data=(val_data, val_targets),
epochs=num_epochs, batch_size=1, verbose=0)
mae_history = history.history['val_mean_absolute_error']
I had the same problem. The following code worked for me.
mae_history = history.history['val_mae']
Using Anaconda Python 2.7 Windows 10.
I am training a language model using the Keras exmaple:
print('Build model...')
model = Sequential()
model.add(GRU(512, return_sequences=True, input_shape=(maxlen, len(chars))))
model.add(Dropout(0.2))
model.add(GRU(512, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(len(chars)))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
def sample(a, temperature=1.0):
# helper function to sample an index from a probability array
a = np.log(a) / temperature
a = np.exp(a) / np.sum(np.exp(a))
return np.argmax(np.random.multinomial(1, a, 1))
# train the model, output generated text after each iteration
for iteration in range(1, 3):
print()
print('-' * 50)
print('Iteration', iteration)
model.fit(X, y, batch_size=128, nb_epoch=1)
start_index = random.randint(0, len(text) - maxlen - 1)
for diversity in [0.2, 0.5, 1.0, 1.2]:
print()
print('----- diversity:', diversity)
generated = ''
sentence = text[start_index: start_index + maxlen]
generated += sentence
print('----- Generating with seed: "' + sentence + '"')
sys.stdout.write(generated)
for i in range(400):
x = np.zeros((1, maxlen, len(chars)))
for t, char in enumerate(sentence):
x[0, t, char_indices[char]] = 1.
preds = model.predict(x, verbose=0)[0]
next_index = sample(preds, diversity)
next_char = indices_char[next_index]
generated += next_char
sentence = sentence[1:] + next_char
sys.stdout.write(next_char)
sys.stdout.flush()
print()
According to Keras documentation, the model.fit method returns a History callback, which has a history attribute containing the lists of successive losses and other metrics.
hist = model.fit(X, y, validation_split=0.2)
print(hist.history)
After training my model, if I run print(model.history) I get the error:
AttributeError: 'Sequential' object has no attribute 'history'
How do I return my model history after training my model with the above code?
UPDATE
The issue was that:
The following had to first be defined:
from keras.callbacks import History
history = History()
The callbacks option had to be called
model.fit(X_train, Y_train, nb_epoch=5, batch_size=16, callbacks=[history])
But now if I print
print(history.History)
it returns
{}
even though I ran an iteration.
The dictionary with histories of “acc”, “loss”, etc. is available and saved in hist.history variable.
It’s been solved.
The losses only save to the History over the epochs. I was running iterations instead of using the Keras built in epochs option.
so instead of doing 4 iterations I now have
model.fit(......, nb_epoch = 4)
Now it returns the loss for each epoch run:
print(hist.history)
{'loss': [1.4358016599558268, 1.399221191623641, 1.381293383180471, 1.3758836857303727]}
Just an example started from
history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=0)
You can use
print(history.history.keys())
to list all data in history.
Then, you can print the history of validation loss like this:
print(history.history['val_loss'])
Another option is CSVLogger: https://keras.io/callbacks/#csvlogger.
It creates a csv file appending the result of each epoch. Even if you interrupt training, you get to see how it evolved.
I have also found that you can use verbose=2 to make keras print out the Losses:
history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=2)
And that would print nice lines like this:
Epoch 1/1
- 5s - loss: 0.6046 - acc: 0.9999 - val_loss: 0.4403 - val_acc: 0.9999
According to their documentation:
verbose: 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch.
The following simple code works great for me:
seqModel =model.fit(x_train, y_train,
batch_size = batch_size,
epochs = num_epochs,
validation_data = (x_test, y_test),
shuffle = True,
verbose=0, callbacks=[TQDMNotebookCallback()]) #for visualization
Make sure you assign the fit function to an output variable. Then you can access that variable very easily
# visualizing losses and accuracy
train_loss = seqModel.history['loss']
val_loss = seqModel.history['val_loss']
train_acc = seqModel.history['acc']
val_acc = seqModel.history['val_acc']
xc = range(num_epochs)
plt.figure()
plt.plot(xc, train_loss)
plt.plot(xc, val_loss)
Hope this helps.
source: https://keras.io/getting-started/faq/#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch
Actually, you can also do it with the iteration method. Because sometimes we might need to use the iteration method instead of the built-in epochs method to visualize the training results after each iteration.
history = [] #Creating a empty list for holding the loss later
for iteration in range(1, 3):
print()
print('-' * 50)
print('Iteration', iteration)
result = model.fit(X, y, batch_size=128, nb_epoch=1) #Obtaining the loss after each training
history.append(result.history['loss']) #Now append the loss after the training to the list.
start_index = random.randint(0, len(text) - maxlen - 1)
print(history)
This way allows you to get the loss you want while maintaining your iteration method.
For plotting the loss directly the following works:
import matplotlib.pyplot as plt
...
model_ = model.fit(X, Y, epochs= ..., verbose=1 )
plt.plot(list(model_.history.values())[0],'k-o')
Those who got still error like me:
Convert model.fit_generator() to model.fit()
Thanks to Alloush,
Following parameter must be included in model.fit():
validation_data = (x_test, y_test)
If it is not defined, val_acc and val_loss will not
be exist at output.
you can get loss and metrics like below:
returned history object is dictionary and you can access model loss( val_loss) or accuracy(val_accuracy) like below:
model_hist=model.fit(train_data,train_lbl,epochs=my_epoch,batch_size=sel_batch_size,validation_data=val_data)
acc=model_hist.history['accuracy']
val_acc=model_hist.history['val_accuracy']
loss=model_hist.history['loss']
val_loss=model_hist.history['val_loss']
dont forget that for getting val_loss or val_accuracy you should specify validation data in the "fit" function.
history = model.fit(partial_train_data, partial_train_targets,
validation_data=(val_data, val_targets),
epochs=num_epochs, batch_size=1, verbose=0)
mae_history = history.history['val_mean_absolute_error']
I had the same problem. The following code worked for me.
mae_history = history.history['val_mae']