model.evaluate() not giving the same loss value as model.predict()

[goncrosa]

I can't seem to understand why is model.predict giving me the same value as the last iteration of my training but model.evaluate is giving me a different value

My code looks like this

random_seed = random.randint(0, 1000) # Random seed between 0 and 1000
print("Random Seed:", random_seed)
tf.random.set_seed(random_seed)

initializer = tf.keras.initializers.GlorotUniform(seed=42)

def build_mlpdo(hidden_layer1, hidden_layer2, dropout_rate):
model = Sequential([
Input(shape=(5,)),
Dense(hidden_layer1, activation='sigmoid', kernel_initializer=initializer),
Dropout(dropout_rate),
Dense(hidden_layer2, activation='sigmoid', kernel_initializer=initializer),
Dropout(dropout_rate),
Dense(1, activation='linear', kernel_initializer=initializer) # Output layer
])

model.compile(optimizer=Adam(learning_rate=0.002), loss='mse')
return model

modeldo = build_mlpdo(hidden_layer1=2, hidden_layer2=9, dropout_rate=0.5)

early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=pa, restore_best_weights=True)
historydo = modeldo.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=ep, batch_size=b_s, callbacks=[early_stop])

modeldo.evaluate(X_val, y_val)
test_predictions = modeldo.predict(X_val)
test_predictions = test_predictions.ravel()
mse = np.mean((test_predictions - y_val)**2)
print("Val MSE:", mse)

giving me the outputs
Epoch 10/10
25/25 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4850 - val_loss: 0.0973
7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1068
7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
Val MSE: 0.09726350368324282

[harshaljanjani]

Thanks for posting the issue @goncrosa!

modeldo.evaluate(X_val, y_val)
test_predictions = modeldo.predict(X_val)
test_predictions = test_predictions.ravel()
mse = np.mean((test_predictions - y_val)**2)
print("Val MSE:", mse)

When using model.evaluate(), the network processes data batch-wise, while your manual calculation using model.predict() processes all predictions simultaneously. To get consistent results, you should explicitly set training=False when making predictions or stick to using model.evaluate() consistently when evaluating the model, either case should get you the consistent results you're searching for!

Relevant Documentation:

1. https://keras.io/guides/training_with_built_in_methods
2. https://www.tensorflow.org/api_docs/python/tf/keras/Model#predict

I've also attached a gist demonstrating the fix.

[goncrosa]

what i don't understand is that predict() is different from evaluate() and in your example, it gives the same value. Moreover I now changed "modeldo.evaluate(X_val, y_val)" to "eval_mse = model.evaluate(X_val, y_val, verbose=0)
print(f"MSE from model.evaluate(): {eval_mse:.6f}")" and it gives the same value as predict(), why?

[dhantule]

Hi @goncrosa, thanks for reporting this. similar issue is #20788.

I have tested this and values are identical upto 1e-7 in this gist. Possible cause could be the progress bar callback being delayed by a batch. We'll dig into this issue and update you.