Multiple network scenario simulation - EPANET files remain open

[r-haider]

Hi,

I'm working with your toolkit for analyzing different configurations within an existing hydraulic network. I generate different scenarios, model them in sequence, and then perform some analysis around the results from EPANET. However, I'm running into an issue with all the temporary files that must be created if a change is made to the network.

I am using the function ENsaveinpfile to save the changes I've made to an open network (I make all changes on this same open network), and then have to open the new file to run the simulation. - I follow the network change example you provide (thank you for the detail!)

However, since I run multiple scenarios, I end up generating many temporary files, and cannot continue beyond ~50 simulations as there are too many files open (I know I can increase the space allocated to temporary files, but this isn't a reasonable solution as I'll need to run 1000+ simulations).

I've taken a look at the source code, epanettools.py at the _close() function and see that you've noted there is a bug. I've also tried to manually force a file close or remove (see list below), but get Windows 32 error (file cannot be accessed because it's being used by another process).

1.    new_solver._close()
   

2.    new_solver.ENclose()
   

3.    os.remove(new_solver.rptfile)
   

Is this a bug that you will be working to address soon (I'm nearing the end of my project timeline and I would need this bug fixed), or have you been able to find a workaround that doesn't involve excessive memory allocation?

[asselapathirana]

Hi, Thank you for your mail. Yes, this is a problem that I know about. But fixing it is not easy by only using python - the underlying problem(s) are with the C code. The best work around for that is the following: When doing many simulations, use the multiprocessing library of python ( https://docs.python.org/3/library/multiprocessing.html). It has the added advantage that you can harness the power of multiple processor cores to speed up your calculations! This is the 'trick' I do in the software SWMM5EA (https://github.com/asselapathirana/swmm5-ea/tree/master/swmm5ea) as well (I need to run thousands of swmm - a model very similar to epanet -- simulations there). Hope this helps. Assela Assela Pathirana, IHE Delft Institute for Water Education T: +31(0)15 215 1854 Westvest 7, 2611 AX Delft, The Netherlands

…

On 15 May 2017 at 17:26, r-haider ***@***.***> wrote: Hi, I'm working with your toolkit for analyzing different configurations within an existing hydraulic network. I generate different scenarios, model them in sequence, and then perform some analysis around the results from EPANET. However, I'm running into an issue with all the temporary files that must be created if a change is made to the network. I am using the function ENsaveinpfile to save the changes I've made to an open network (I make all changes on this same open network), and then have to open the new file to run the simulation. - I follow the network change example you provide (thank you for the detail!) However, since I run multiple scenarios, I end up generating many temporary files, and cannot continue beyond ~50 simulations as there are too many files open (I know I can increase the space allocated to temporary files, but this isn't a reasonable solution as I'll need to run 1000+ simulations). I've taken a look at the source code, epanettools.py at the _close() function and see that you've noted there is a bug. I've also tried to manually force a file close or remove (see list below), but get Windows 32 error (file cannot be accessed because it's being used by another process). 1. new_solver._close() 2. new_solver.ENclose() 3. os.remove(new_solver.rptfile) Is this a bug that you will be working to address soon (I'm nearing the end of my project timeline and I would need this bug fixed), or have you been able to find a workaround that doesn't involve excessive memory allocation? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#22>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAT7VxY5mAiScUVzPV6NiKDt5hNJWfIVks5r6G6LgaJpZM4NbS4_> .

[AMahajna]

r-haider, where you employing coupled simulation-optimization? Where you able to increase the number of simulations by employing multiprocessing successfully?

[asselapathirana]

One of the additional advantages of the 'workaround' I explained is - that makes it very easy to implement multiprocessing. Hint: use the multiprocessing package. Regards, Assela Assela Pathirana, IHE Delft Institute for Water Education T: +31(0)15 215 1854 Westvest 7, 2611 AX Delft, The Netherlands

…

On Sun, 9 Jun 2019 at 22:34, AMahajna ***@***.***> wrote: r-haider, where you employing coupled simulation-optimization? Where you able to increase the number of simulations by employing multiprocessing successfully? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=AACPWV4DWQ7XXY3DWOQ5VWDPZVSOXA5CNFSM4DLNFY72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXIRT7Q#issuecomment-500242942>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACPWV6YLFCFSHIBTVZO2OTPZVSOXANCNFSM4DLNFY7Q> .

[AMahajna]

Dear Sir,
Thank you so much for your valuable contributions!
Indeed, multiprocessing solves the issue. :)