Custom keras callback to determine total training time

Question

I have to train my model ntimes using 3 different activation functions (relu, tanh, sigmoid) for statistical assessment of performance. One of the things i need to evaluate, is the total train time of each of these models. For that, i have created a custom keras callback like the following:

class TimingCallback(keras.callbacks.Callback):
    def __init__(self, iteracao, func, logs=None):
        self.iteracao = iteracao
        self.func = func
        self.starttime = None
        if logs is None:
            logs = {'relu': {}, 'tanh': {}, 'sigmoid': {}}
        self.logs = logs

    def on_train_begin(self, epoch, logs={}):
        self.starttime = time.time()

    def on_train_end(self, epoch, logs={}):
        self.logs[self.func].update({self.iteracao: (time.time() - self.starttime)/60})

So, what i am trying to do here, is running a forloop in all these activation functions, like this:

n = 10
dic_loss_train = {'relu': {}, 'tanh': {}, 'sigmoid': {}}
for funcao in dic_loss_train:
    for i in range(n):
        model = Sequential()
        model.add(...)
        model.compile(...)
        tempo_callback = TimingCallback(i, funcao)
        cp = ModelCheckpoint(...)
        earlystop = EarlyStopping(...)
        treinando = model.fit(x_train, y_train, batch_size=32,
                              epochs=5, verbose=2,
                              callbacks=[cp, earlystop, tempo_callback],
                              validation_split=val_size, shuffle=False)

I am having problem with the update method in on_train_end. What i was expecting to happen is getting a dictionary like the following:

{'relu': {0: 0.0583, 1: 0.0271, 2: 0.0266, 3: 0.0286, 4: 0.0289, 5: 0.0276, 6: 0.0271, 7: 0.0271, 8: 0.0279, 9: 0.0279}, 'tanh': {0: 0.0281, 1: 0.0302, 2: 0.0279, 3: 0.0274, 4: 0.0289, 5: 0.0273, 6: 0.0284, 7: 0.0273, 8: 0.0271, 9: 0.0276}, 'sigmoid': {0: 0.0227, 1: 0.0221, 2: 0.0229, 3: 0.0237, 4: 0.0242, 5: 0.0227, 6: 0.0224, 7: 0.0268, 8: 0.0229, 9: 0.0273}}

But instead, i am ending up with:

{'relu': {}, 'tanh': {}, 'sigmoid': {9: 0.0273}}

What i am doing wrong here? How can i fix this?

Asked By: Murilo

||

Source

Answer 1

You are creating a new TimingCallback – and with it a new, empty logs dictionary – in the for loop. You need to provide the dic_loss_train to the callback.

model.compile(...)
    tempo_callback = TimingCallback(i, funcao, logs=dic_loss_train)
    cp = ModelCheckpoint(...)

Answered By: AndrzejO

Custom keras callback to determine total training time

Question:

Answers: