tensorflow evalutaion and earlystopping gives infinity overflow error
Question:
I a model as seen in the code below, but when trying to evaluate it or using earlystopping on it it gives me the following error:
numdigits = int(np.log10(self.target)) + 1
OverflowError: cannot convert float infinity to integer
I must state that without using .EarlyStopping
or model.evaluate
everything works well.
I know that np.log10(0)
gives -inf
so that could be a potential cause, but why is there a 0
there in the first place and how can it be prevented? How can this problem be fixed?
NOTES
this is the code I use:
import tensorflow as tf
from tensorflow import keras
TRAIN_PERCENT = 0.9
model = keras.Sequential([
keras.layers.Dense(128, input_shape=(100,), activation='relu'),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(100)
])
earlystop_callback = keras.callbacks.EarlyStopping(min_delta=0.0001, patience=1
, monitor='accuracy'
)
optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(optimizer=optimizer, loss="mse", metrics=['accuracy'])
X_set, Y_set = some_get_data_function()
sep = int(len(X_set)/TRAIN_PERCENT)
X_train, Y_train = X_set[:sep], Y_set[:sep]
X_test, Y_test = X_set[sep:], Y_set[sep:]
model.fit(X_train, Y_train, batch_size=16, epochs=5, callbacks=[earlystop_callback])
ev = model.evaluate(X_test, Y_test)
print(ev)
X,Y sets are np
arrays. X is an array of arrays of 100 integers between 0
and 10
. Y is an array of arrays of 100 integers, all of them are either 0
or 1
.
Answers:
Well it’s hard to tell exactly as I can’t run code without some_get_data_function()
realization but recently I’ve got same error when mistakenly passed EMPTY array to model.evaluate
. Taking into account that @meTchaikovsky comment solved your issue it’s certainly due to messed up input arrays.
in my case, the data was small – so doing something like
cached_train = train.shuffle(100_000).batch(8192).cache()
cached_test = test.batch(4096).cache()
would have resulted in an empty cached_test
! this does not affect model.fit
so always check the length of the data using cached_test.__len__()
to make sure you are not passing an empty data to the function.
Setting batch size to 1 fixed this error for me. Play around with smaller batch sizes.
I a model as seen in the code below, but when trying to evaluate it or using earlystopping on it it gives me the following error:
numdigits = int(np.log10(self.target)) + 1
OverflowError: cannot convert float infinity to integer
I must state that without using .EarlyStopping
or model.evaluate
everything works well.
I know that np.log10(0)
gives -inf
so that could be a potential cause, but why is there a 0
there in the first place and how can it be prevented? How can this problem be fixed?
NOTES
this is the code I use:
import tensorflow as tf
from tensorflow import keras
TRAIN_PERCENT = 0.9
model = keras.Sequential([
keras.layers.Dense(128, input_shape=(100,), activation='relu'),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(100)
])
earlystop_callback = keras.callbacks.EarlyStopping(min_delta=0.0001, patience=1
, monitor='accuracy'
)
optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(optimizer=optimizer, loss="mse", metrics=['accuracy'])
X_set, Y_set = some_get_data_function()
sep = int(len(X_set)/TRAIN_PERCENT)
X_train, Y_train = X_set[:sep], Y_set[:sep]
X_test, Y_test = X_set[sep:], Y_set[sep:]
model.fit(X_train, Y_train, batch_size=16, epochs=5, callbacks=[earlystop_callback])
ev = model.evaluate(X_test, Y_test)
print(ev)
X,Y sets are np
arrays. X is an array of arrays of 100 integers between 0
and 10
. Y is an array of arrays of 100 integers, all of them are either 0
or 1
.
Well it’s hard to tell exactly as I can’t run code without some_get_data_function()
realization but recently I’ve got same error when mistakenly passed EMPTY array to model.evaluate
. Taking into account that @meTchaikovsky comment solved your issue it’s certainly due to messed up input arrays.
in my case, the data was small – so doing something like
cached_train = train.shuffle(100_000).batch(8192).cache()
cached_test = test.batch(4096).cache()
would have resulted in an empty cached_test
! this does not affect model.fit
so always check the length of the data using cached_test.__len__()
to make sure you are not passing an empty data to the function.
Setting batch size to 1 fixed this error for me. Play around with smaller batch sizes.