Retrieve keras model fit history afer execution interrupt
Question:
In a jupyter notebook, I’m doing a training in keras with a line similar to
history = model.fit(....., epoch=100)
When I see that var_loss converge I break manually the execution, and obviously history is not returned.
Is there a way to retrieve it? A member of model, or method, or a way to get the history object or its members?
Answers:
@Mquinteiro Try using a callback: https://keras.io/callbacks/. You could set the callback to terminate when the model is no longer improving (keras.callbacks.EarlyStopping) and then access the best model (save_best_only = True), or you could have the callback save checkpoints after each epoch (keras.callbacks.ModelCheckpoint) which you can access after manually stopping the execution.
I tested with Tensorflow v2.9.2 and history is an attribute on the model even if the training was interrupted (and without setting an explicit callback) when compiling the model.
For example, if interrupting a keras train like
history = model.fit(train_dataset, validation_data=val_dataset,
epochs=25, verbose=1)
, the history is accessible via model.history:
history = model.history
plt.plot(history.history["loss"])
plt.title("Training Loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.show()
In a jupyter notebook, I’m doing a training in keras with a line similar to
history = model.fit(....., epoch=100)
When I see that var_loss converge I break manually the execution, and obviously history is not returned.
Is there a way to retrieve it? A member of model, or method, or a way to get the history object or its members?
@Mquinteiro Try using a callback: https://keras.io/callbacks/. You could set the callback to terminate when the model is no longer improving (keras.callbacks.EarlyStopping) and then access the best model (save_best_only = True), or you could have the callback save checkpoints after each epoch (keras.callbacks.ModelCheckpoint) which you can access after manually stopping the execution.
I tested with Tensorflow v2.9.2 and history is an attribute on the model even if the training was interrupted (and without setting an explicit callback) when compiling the model.
For example, if interrupting a keras train like
history = model.fit(train_dataset, validation_data=val_dataset,
epochs=25, verbose=1)
, the history is accessible via model.history:
history = model.history
plt.plot(history.history["loss"])
plt.title("Training Loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.show()