save and load keras.callbacks.History

Question

I’m training a deep neural net using Keras and looking for a way to save and later load the history object which is of keras.callbacks.History type. Here’s the setup:

history_model_1 = model_1.fit_generator(train_generator,
                          steps_per_epoch=100,
                          epochs=20,
                          validation_data=validation_generator,
                          validation_steps=50)

history_model_1 is the variable I want to be saved and loaded during another Python session.

Asked By: balkon16

||

Source

Answer 1

history_model_1 is a callback object. It contains all sorts of data and isn’t serializable.

However, it contains a dictionnary with all the values that you actually want to save (cf your comment) :

import json
# Get the dictionary containing each metric and the loss for each epoch
history_dict = history_model_1.history
# Save it under the form of a json file
json.dump(history_dict, open(your_history_path, 'w'))

You can now access the value of the loss at the 50th epoch like this :

print(history_dict['loss'][49])

Reload it with

history_dict = json.load(open(your_history_path, 'r'))

I hope this helps.

Answered By: Nassim Ben

Answer 2

You can use Pandas to save the history object as a CSV file.

import pandas as pd

pd.DataFrame.from_dict(history_model_1.history).to_csv('history.csv',index=False)

The JSON approach results in a TypeError: Object of type 'float32' is not JSON serializable. The reason for this is that the corresponding values in the history dictionary are NumPy arrays.

Answered By: Tobias Scheck

Answer 3

You can create a class so you will have the same structure and you can access in both cases with the same code.

import pickle
class History_trained_model(object):
    def __init__(self, history, epoch, params):
        self.history = history
        self.epoch = epoch
        self.params = params

with open(savemodel_path+'/history', 'wb') as file:
    model_history= History_trained_model(history.history, history.epoch, history.params)
    pickle.dump(model_history, file, pickle.HIGHEST_PROTOCOL)

then to access it:

with open(savemodel_path+'/history', 'rb') as file:
    history=pickle.load(file)

print(history.history)

Answered By: Dr. Fabien Tarrade

Answer 4

Taken from Tobias, use this updated version

import pandas as pd

pd.DataFrame.from_dict(history_model_1.history.history).to_csv('history.csv',index=False)

Answered By: ding dong

Answer 5

filename='log.csv'
history_logger=tf.keras.callbacks.CSVLogger(filename, separator=",", append=True)
history_model_1 = model_1.fit_generator(train_generator,
                          steps_per_epoch=100,
                          epochs=20,
                          callbacks=[history_logger],
                          validation_data=validation_generator,
                          validation_steps=50)

Answered By: David Dean

save and load keras.callbacks.History

Question:

Answers: