How can the training be real deterministic in nnunet?

[sywangsha]

Hi,
Thanks for your great work.
I'm wondering how to make the training be real deterministic.
When I train with the same data and same code in nnUNet, and with --ndet=False, I got different models and there is more than 0.01 difference in their DICE score. This can make me confused when I try to compare some difference training designs.

I noticed the following code in the network_trainer, but it seems not worked.

    np.random.seed(12345)
    torch.manual_seed(12345)
    torch.cuda.manual_seed_all(12345)

Does anyone have any idea? Thanks.

[FabianIsensee]

Good point. I have not tried to get my trainings deterministic in a long time. I don't know what the solution is. If you figure it out I would be happy if you could share it :-)

[FabianIsensee]

(i do not like deterministic training because it gives you a false sense of security and hides the noise - the noise is important because it prevents you from overfitting to your random seed)