Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NaN on RTX 3090 GPU training #6

Open
PengJihua opened this issue May 16, 2024 · 1 comment
Open

NaN on RTX 3090 GPU training #6

PengJihua opened this issue May 16, 2024 · 1 comment

Comments

@PengJihua
Copy link

I have been attempting to train the StarGAN v2 model using an RTX 3090 GPU, but I encounter a persistent issue where the loss values turn to NaN, and the generated images are solely composed of noise. This is puzzling, especially since the same code and data work flawlessly on a Tesla P100 GPU.

Could you please shed some light on what might be causing these discrepancies between the two GPUs? Any insights or suggestions you could provide would be immensely appreciated, as I am quite perplexed by this behavior.

@eps696 eps696 changed the title I have been attempting to train the StarGAN v2 model using an RTX 3090 GPU, but I encounter a persistent issue where the loss values turn to NaN, and the generated images are solely composed of noise. This is puzzling, especially since the same code and data work flawlessly on a Tesla P100 GPU. Could you please shed some light on what might be causing these discrepancies between the two GPUs? Any insights or suggestions you could provide would be immensely appreciated, as I am quite perplexed by this behavior. NaN on RTX 3090 GPU training May 17, 2024
@eps696
Copy link
Owner

eps696 commented May 17, 2024

alas, i'm not an expert in GPU details, so can't guess out of the blue what may be the difference.
and that's too little info to guess anything. is it on colab or local? what python/torch versions are used? did you try other data on 3090?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants