Adding Dropout to testing/inference phase

Question

I’ve trained the following model for some timeseries in Keras:

    input_layer = Input(batch_shape=(56, 3864))
    first_layer = Dense(24, input_dim=28, activation='relu',
                        activity_regularizer=None,
                        kernel_regularizer=None)(input_layer)
    first_layer = Dropout(0.3)(first_layer)
    second_layer = Dense(12, activation='relu')(first_layer)
    second_layer = Dropout(0.3)(second_layer)
    out = Dense(56)(second_layer)
    model_1 = Model(input_layer, out)

Then I defined a new model with the trained layers of model_1 and added dropout layers with a different rate, drp, to it:

    input_2 = Input(batch_shape=(56, 3864))
    first_dense_layer = model_1.layers[1](input_2)
    first_dropout_layer = model_1.layers[2](first_dense_layer)
    new_dropout = Dropout(drp)(first_dropout_layer)
    snd_dense_layer = model_1.layers[3](new_dropout)
    snd_dropout_layer = model_1.layers[4](snd_dense_layer)
    new_dropout_2 = Dropout(drp)(snd_dropout_layer)
    output = model_1.layers[5](new_dropout_2)
    model_2 = Model(input_2, output)

Then I’m getting the prediction results of these two models as follow:

result_1 = model_1.predict(test_data, batch_size=56)
result_2 = model_2.predict(test_data, batch_size=56)

I was expecting to get completely different results because the second model has new dropout layers and theses two models are different (IMO), but that’s not the case. Both are generating the same result. Why is that happening?

Asked By: Birish

||

Source

Answer 1

As I mentioned in the comments, the Dropout layer is turned off in inference phase (i.e. test mode), so when you use model.predict() the Dropout layers are not active. However, if you would like to have a model that uses Dropout both in training and inference phase, you can pass training argument when calling it, as suggested by François Chollet:

# ...
new_dropout = Dropout(drp)(first_dropout_layer, training=True)
# ...

Alternatively, If you have already trained your model and now want to use it in inference mode and keep the Dropout layers (and possibly other layers which have different behavior in training/inference phase such as BatchNormalization) active, you can define a backend function that takes the model’s inputs as well as Keras learning phase:

from keras import backend as K

func = K.function(model.inputs + [K.learning_phase()], model.outputs)

# to use it pass 1 to set the learning phase to training mode
outputs = func([input_arrays] + [1.])

Answered By: today

Answer 2

your question has a simple solution in the latest version of Tensorflow. you can set the training argument of the call method to true.
you can run a code like the below code:

model(input,training=True)

by using training=True TensorFlow automatically applies the Dropout layer in inference mode.

Answered By: Ramin Bakhtiyari

Answer 3

As there are already some working code solutions above, I will simply add a few more details regarding dropout during inference to prevent confusion.

Based on the original paper, Dropout layers play the role of turning off (setting gradients to zero) the neuron nodes during training to reduce overfitting. However, once we finish off with training and start testing the model, we do not ‘touch’ any neurons, thus, all the units are considered to make the decision when inferencing. This causes previously ‘dead’ neuron weights to be large than expected due to the usage of Dropout. To prevent this, a scaling factor is applied to balance the network node. To be more precise, if a unit is retained with probability p during training, the outgoing weights of that unit are multiplied by p during the prediction stage.

Answered By: Elvin Aghammadzada

Adding Dropout to testing/inference phase

Question:

Answers: