Running backward tracing, using ParticleSet.from_parcels_file(... restart=True)

[subniten]

Hi,

I have a large dataset, so I need to divide my particle set execution into time chunks. When I try to run it in backward tracing mode, I can't make it work.

I have made a notebook, and uploaded a subset (the lower left corner) of my domain reproducing the error.

[erikvansebille]

Thanks for reporting this, @subniten. I can not reproduce this error easily on a simple test case, though. But may I ask, why do you need to manually split your execute? Parcels should be able to handle extremely long-time datasets, because it uses deferred_loading. What happens if you use all your files to create the FieldSet?

[subniten]

You mean you can’t reproducera my error? I provided data as well in the linked github repo.

If I run the whole tracing in one go, it never finishes (180 days and a mesh grid 237 x 503 x 593). I tried splitting it up in chunks of 5000 parcels and time chunks of 15 or 30 days, in forward mode I can run 110’000 in a couple of hours.

If I don’t split it up in time chunks it has so far crashed.

In the real repo I’m doing this automatically, the linked case is just to reproduce the error.

[subniten]

Asked a friend to run it on his linux host, and he gets a lot of warning messages, but it runs, see
attached .pdf. I run on a M1-processor.

[subniten]

Thanks for reporting this, @subniten. I can not reproduce this error easily on a simple test case, though. But may I ask, why do you need to manually split your execute? Parcels should be able to handle extremely long-time datasets, because it uses deferred_loading. What happens if you use all your files to create the FieldSet?

It seems I had a bug somewhere else in my code loading the fields, now it seems I can run a backward tracing w/o exhausting the memory as before. I don't where the bug really was, but it doesn't matter now. :D

[subniten]

*I don't know

[erikvansebille]

OK, good to know that it works now!