How to save tensorflow neural networks at custom intervals

Question

I want to save my neural network such that I can make a video on how the neural network trains (plotting results throughout the training process). Since the most difference in results occur at the start, I would like to save the model more often at the start of training. For example, I might want to save my model after the following epochs:

1, 2, 4, 8, 16, 32, 64, 128, …

With tf.keras.callbacks.ModelCheckpoint I can only find methods for saving at a regular interval, for example after every epoch. However my computer does not have enough space to save the network that often. If I would take a longer interval, then I would lose all the large improvements the model makes at the start of training. Therefore I would like a different schedule (for example the one mentioned above). Preferably I would just use a list of the desired batches (or epochs) after which I want to save the network.

Is this possible with Tensorflow?

Asked By: FeddeHasQuestion

||

Source

Answer 1

You need to extend the Callback class. This gives you access to the on_epoch_end() function which is what you want. You can use my code below as a reference

Here’s a link to the documentation for more info: Writing your own callbacks

class CustomCallback(tf.keras.callbacks.Callback):
    def __init__(self, save_epoch_list):
        super(CustomCallback, self).__init__()
        self.save_epoch_list = save_epoch_list 


    def on_epoch_end(self, epoch, logs=None):
        if epoch in self.save_epoch_list:
            # insert save model code here
            pass

save_epoch_list = [2, 4, 8, 16, 32]

model.fit(
    x_train,
    y_train,
    batch_size=64,
    epochs=30,
    callbacks=[CustomCallback(save_epoch_list)],
)

Answered By: Rashaad Meyer

How to save tensorflow neural networks at custom intervals

Question:

Answers: