Keras: change learning rate

Question

I’m trying to change the learning rate of my model after it has been trained with a different learning rate.

I read here, here, here and some other places i can’t even find anymore.

I tried:

model.optimizer.learning_rate.set_value(0.1)
model.optimizer.lr = 0.1
model.optimizer.learning_rate = 0.1
K.set_value(model.optimizer.learning_rate, 0.1)
K.set_value(model.optimizer.lr, 0.1)
model.optimizer.lr.assign(0.1)

… but none of them worked!
I don’t understand how there could be such confusion around such a simple thing. Am I missing something?

EDIT: Working example

Here is a working example of what I’d like to do:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer)

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50)

# Change learning rate to 0.001 and train for 50 more epochs

model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50)

Asked By: Luca

||

Source

Answer 1

You should define it in the compile function :

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer,
              metrics=['categorical_accuracy'])

Looking at your comment, if you want to change the learning rate after the beginning you need to use a scheduler : link

Edit with your code and scheduler:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np

def lr_scheduler(epoch, lr):
    if epoch > 50:
        lr = 0.001
        return lr
    return lr

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer)

callbacks = [keras.callbacks.LearningRateScheduler(lr_scheduler, verbose=1)]

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=100, callbacks=callbacks)

Answered By: Orphee Faucoz

Answer 2

You can change lr during training with

from keras.callbacks import LearningRateScheduler

# This is a sample of a scheduler I used in the past
def lr_scheduler(epoch, lr):
    decay_rate = 0.85
    decay_step = 1
    if epoch % decay_step == 0 and epoch:
        return lr * pow(decay_rate, np.floor(epoch / decay_step))
    return lr

Apply scheduler to your model

callbacks = [LearningRateScheduler(lr_scheduler, verbose=1)]

model = build_model(pretrained_model=ka.InceptionV3, input_shape=(224, 224, 3))
history = model.fit(train, callbacks=callbacks, epochs=EPOCHS, verbose=1)

Answered By: Kenan

Answer 3

Suppose that you use Adam optimizer in keras, you’d want to define your optimizer before you compile your model with it.

For example, you can define

myadam = keras.optimizers.Adam(learning_rate=0.1)

Then, you compile your model with this optimizer.

I case you want to change your optimizer (with different type of optimizer or with different learning rate), you can define a new optimizer and compile your existing model with the new optimizer.

Hope this helps!

Answered By: Loc Tran

Answer 4

There is another way, you have to find the variable that holds the learning rate and assign it another value.

optimizer = tf.keras.optimizers.Adam(0.001)
optimizer.learning_rate.assign(0.01)
print(optimizer.learning_rate)

output:

<tf.Variable 'learning_rate:0' shape=() dtype=float32, numpy=0.01>

Answered By: Tomé Silva

Answer 5

You can change the learning rate as follows:

from keras import backend as K
K.set_value(model.optimizer.learning_rate, 0.001)

Included into your complete example it looks as follows:

from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import keras
import numpy as np

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse', optimizer=optimizer)

print("Learning rate before first fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50, verbose=0)

# Change learning rate to 0.001 and train for 50 more epochs
K.set_value(model.optimizer.learning_rate, 0.001)
print("Learning rate before second fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), 
          np.random.randn(50), 
          initial_epoch=50, 
          epochs=50,
          verbose=0)

I’ve just tested this with keras 2.3.1. Not sure why the approach didn’t seem to work for you.

Answered By: Timo.S

Answer 6

Some time ago I had a project for which I needed something similar. My idea to change the learning rate was to compile a new model with the new rate, then load the parameter weights from de old model to the new one.

For your example:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np


# Initial model

model = Sequential()
model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse', optimizer=optimizer)

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50)


# Change learning rate to 0.001 and train for 50 more epochs

new_model = Sequential()
new_model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.001)
new_model.compile(loss='mse', optimizer=optimizer)

new_model.set_weights(model.get_weights())
model = new_model

model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50)

With this you could see a worse fit of your model in the first epochs because ADAM uses previous steps to optimize and you will lose them.

Hope it helps someone!

Answered By: Juan Pablo Cortés

Keras: change learning rate

Question:

Answers: