RuntimeError: There was no new checkpoint after the training. Eval status: missing checkpoint

[harirajeev]

While running this model on the CONLL2003 data ( TF 1.11) , i am getting this error

global_step/sec: 3.00248
loss = 5.6152534, step = 101 (33.300 sec)
global_step/sec: 2.88663
loss = 4.2953496, step = 201 (34.641 sec)
global_step/sec: 3.08292
loss = 3.5392659, step = 301 (32.438 sec)
Saving checkpoints for 344 into results\model\model.ckpt.
Estimator is not trained yet. Will start an evaluation when a checkpoint is ready.

RuntimeError: There was no new checkpoint after the training. Eval status: missing checkpoint

Could you please paste a picture of your model weights directory ?.

thanks

[guillaumegenthial]

Hi @harirajeev ,
What's probably happening is that by default the evaluation will be triggered every 120 seconds, if a new checkpoint is available. If your computer is too slow, and it does not save a new checkpoint in this lapse of time, it will try to run evaluation and skip as no new checkpoint is available.

[errolPereira]

How to delay the eval trigger? @guillaumegenthial