Activity regularizer in Keras: before or after activation?

Question:

Suppose I have:

output = Dense(units=12, activation='sigmoid', activity_regularizer=L1(1e-2))(input)

Keras documentation says activity regularizer "apply a penalty on the layer’s output", but it does not specify whether "output" means the output of the dense operation only, or that of the entire layer including activation.

For my problem I need the activity regularizer to apply after activation. In case Keras implements it the other way around, how can I fix it?

Asked By: Felix Fourcolor

||

Answers:

Keras applies the activity regularization after the entire layer including activation.

If you scroll to the end of the Dense layer call method you will see that, if defined, the activation is applied on the output before returning it.

The activity regularization is applied after this call function in the Layer base class. See here

Answered By: thmslmr

As already mentioned by thmslmr: the activity regularizer is applied on the output of the layer, therefore after the activation function is applied.

If you want to apply regularization before the softmax activation, you can move the activation function into a separate layer :

model.add(layers.Dense(units=12, activity_regularizer=L1(1e-2) )
model.add(layers.Activation('softmax'))
Answered By: user3224103
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.