How to get reproducible weights initializaiton in Keras?

Question

I set both numpy and tensorflow random seeds as suggested
Generate some data – this part is reproducible, gives same results always
Create a simple network and make a prediction (without training, just with random weights) – prediction is different every time

import numpy as np
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras import Sequential, optimizers
import tensorflow as tf

np.random.seed(32)
tf.set_random_seed(33)

random_data = np.random.rand(10, 2048)
print(random_data[:,0])

def make_classifier():
    model = Sequential()
    model.add(Dense(1024, activation='relu', input_dim=2048))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
              loss='binary_crossentropy')
    return model
model = make_classifier()
# model.summary()
model.predict(random_data)

When I re-run the whole cell, the print statement always outputs [0.85888927 0.23846818 0.17757634 0.07244977 0.71119893 0.09223853 0.86074647 0.31838194 0.7568638 0.38197083]. However the prediction is different every time:

array([[0.5825965 ],
       [0.8677979 ],
       [0.70151913],
       [0.64572096],
       [0.78101623],
       [0.76483005],
       [0.7946336 ],
       [0.6281612 ],
       [0.8208673 ],
       [0.8273002 ]], dtype=float32)

array([[0.51012236],
       [0.6562015 ],
       [0.5593666 ],
       [0.686155  ],
       [0.6488372 ],
       [0.5966359 ],
       [0.6236731 ],
       [0.58099884],
       [0.68447435],
       [0.58886844]], dtype=float32)

And so on.

How do I get reproducible prediction for a just-initialized network?
When does the weights initialization happen exactly? when I compile the model or?..

Asked By: Dmitry Toda

||

Source

Answer 1

tf.keras.initializers objects have a seed argument for reproducible initialization.

import tensorflow as tf
import numpy as np

initializer = tf.keras.initializers.GlorotUniform(seed=42)

for _ in range(10):
    print(np.round(initializer((4,)), 3))

[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]

In a Keras layer, you can use it like this:

tf.keras.layers.Dense(1024, 
                      activation='relu', 
                      input_dim=2048,
                      kernel_initializer=tf.keras.initializers.GlorotUniform(seed=42))

Answered By: Nicolas Gervais

Answer 2

I’ve been struggling a lot with this and turns out there are quite a few points that have to be set in order to achieve complete consistency for every case:

First of all, make sure the data (and the order of the data) that you feed to your model is consistent. Then, for the model weights initialization:

1)numpy random seed

import numpy as np
np.seed(1)

2)tensor flow random seed

import tensorflow as tf
tf.set_random_seed(2)

3)python random seed

import random
random.seed(3)

Additionally to that, you have to set two (if you have multiprocessing capabilities) arguments to model.fit. These ones are not often mentioned on the answers I’ve seen around:

model.fit(..., shuffle=False, use_multiprocessing=False)

Only then I have achieved complete consistency in training runs.

Hope that helps people!

Answered By: Lucas Azevedo

Answer 3

I think a better practice is

tf.keras.utils.set_random_seed(
    seed
)

ref: https://www.tensorflow.org/api_docs/python/tf/keras/utils/set_random_seed

Answered By: Litchy

How to get reproducible weights initializaiton in Keras?

Question:

Answers: