Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The loss got NAN #9

Open
zaynmi opened this issue May 26, 2020 · 3 comments
Open

The loss got NAN #9

zaynmi opened this issue May 26, 2020 · 3 comments

Comments

@zaynmi
Copy link

zaynmi commented May 26, 2020

Hello Ranjay, thanks for your work.
I have run the train_iq.py using the default lr 0.001, but after 4 epochs, the loss and other metrics got Nan. Is there something wrong with the training script?
Looking forward to your reply. Thanks again.
@duchengyao
Copy link

the same issue

@ozancaglayan
Copy link

Got the same thing in the middle of epoch 2 as well.

@mahmudhasankhan
Copy link

Same, after weight decaying starts all the loss got Nan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ozancaglayan @duchengyao @zaynmi @mahmudhasankhan