Reading a BP5 series (groupBased) using 120+ GB memory, any fix around it? #1724
-
|
Hello, However, I have ran a large simulation which in return produced a huge data file. The ".bp" directory that is created is around: 122GB size large. While, I was accessing it using: import openpmd_api as io
series = io.Series(file_path, io.Access.read_only)The RAM usage spikes to 120GB, which kind of makes sense due to the size of the file. Without performing any additional statistics on the data, the RAM used reaches 120GB. Basically when the python code hits the series in the script it stays there to read the series and it reaches 120GB RAM without progessing to next lines of code. Is there a way to not use this much memory, because my machine is maxed at 134GB of RAM. Is there a method for lazy loading and accessing the corresponding data as needed? It seems just calling the series as I did, loads up everything even though lets say I just want to process every 50 or 100 turns data but it is written as per turn data if that makes sense. Any suggestions will be appreciated! |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 19 replies
-
|
Hello,
You can verify 2. and 3. using bpls: In this combination, the BP5 engine of ADIOS2 has been previously observed to have quadratic metadata output size depending on the number of steps created, due to the assumptions that it makes for its serialization. This means that if you write lots of Iterations, this exact situation may occur. At the moment, the recommended alternative is to use file-based encoding which creates a new file for each output step. This can be done by including an expansion pattern in the filename: For converting your existing dataset to file-based encoding, we will have to use an open mode in ADIOS2 that does not try to consume all metadata at once. There is one slight problem in that: Due to the difficulty in correctly associating attributes to steps in that read mode, openPMD-api 0.16 no longer supports that read mode on group-based files, meaning that you would have to temporarily downgrade to openPMD-api 0.15 for that. I'm sorry for the troubles that this may cause, I will add a fix for the upcoming patch release so that we no longer create files in default configurations that are unreadable in certain read modes. |
Beta Was this translation helpful? Give feedback.
-
|
When I try the following: bpls openPMD/monitor.bp/ -alt /iterationEncoding -P 'SelectSteps=0'The output is: Step 0:
Error: None of the variables/attributes matched any name/regexp you providedI am guessing this due to:
I am sorry, I am not an expert at this. |
Beta Was this translation helpful? Give feedback.
-
|
No worries, then let's try the full attribute output of the first step: |
Beta Was this translation helpful? Give feedback.
-
|
The command: bpls monitor.bp/ -alt -P 'SelectSteps=0'Gives: Step 0:This output is similar to what I tried with python script importing adios2 library. Showing that there is no step available? |
Beta Was this translation helpful? Give feedback.
-
|
Hm, I think I'll have to test this for myself in order to figure out which ways would even still work for accessing an output like that. I assumed that step-by-step reading would still work at least, but maybe not. To confirm that this really is the metadata's fault (it very likely is), can you try listing the sizes of the single files in the .bp directory, please? E.g.: |
Beta Was this translation helpful? Give feedback.
-
|
First of all I want to apologize, because for some reason the monitor.bp directory had such a small file sizes. Whereas, earlier it was filled with data.. I am not sure if this because of some action I did earlier with python script. However, I did have a backup of that directory. So trying things once again: 103849503 68G -rw-rw-r-- 1 ogilanli ogilanli 68G Mar 2 14:56 data.0
103849502 24G -rw-rw-r-- 1 ogilanli ogilanli 24G Mar 2 14:56 md.0
103849498 1.2M -rw-rw-r-- 1 ogilanli ogilanli 1.2M Mar 2 14:56 md.idx
103849504 31G -rw-rw-r-- 1 ogilanli ogilanli 31G Mar 2 14:56 mmd.0
103809042 4.0K -rw-rw-r-- 1 ogilanli ogilanli 836 Mar 2 14:56 profiling.jsonSecond: 122G monitor.bp/Now trying previous commands once again:
Running, another simulation with smaller parameters: 1K turns, with 3K particles:
total 766M
234752668 208M -rw-rw-r-- 1 ogilanli ogilanli 208M Mar 3 11:35 data.0
234750098 249M -rw-rw-r-- 1 ogilanli ogilanli 249M Mar 3 11:35 md.0
234750097 76K -rw-rw-r-- 1 ogilanli ogilanli 72K Mar 3 11:35 md.idx
234752669 310M -rw-rw-r-- 1 ogilanli ogilanli 310M Mar 3 11:35 mmd.0
234752672 4.0K -rw-rw-r-- 1 ogilanli ogilanli 793 Mar 3 11:35 profiling.jsonWith: Repeating the previous commands: bpls monitor.bp/ -al /iterationEncoding Gives: string /iterationEncoding attr = "groupBased"Secondly: Step 0:
uint64_t /data/1/particles/beam/id {3000}
double /data/1/particles/beam/momentum/t {3000}
double /data/1/particles/beam/momentum/x {3000}
double /data/1/particles/beam/momentum/y {3000}
double /data/1/particles/beam/position/t {3000}
double /data/1/particles/beam/position/x {3000}
double /data/1/particles/beam/position/y {3000}
double /data/1/particles/beam/qm {3000}
double /data/1/particles/beam/weighting {3000}
Step 1:
uint64_t /data/939/particles/beam/id {3000}
double /data/939/particles/beam/momentum/t {3000}
double /data/939/particles/beam/momentum/x {3000}
double /data/939/particles/beam/momentum/y {3000}
double /data/939/particles/beam/position/t {3000}
double /data/939/particles/beam/position/x {3000}
double /data/939/particles/beam/position/y {3000}
double /data/939/particles/beam/qm {3000}
double /data/939/particles/beam/weighting {3000}I am sorry about earlier and the confusion it created! I did not know it was empy inside! Last time I checked it was all good with 123 GB of sizes. I guess I did something that messed it up... Hopefully now it cleared off somethings! Thanks for replies! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @OLuckyG, Thank you for your report! I am co-developing both ImpactX and openPMD-api and will try to add a few more details to see if we can get this figured out, together with the details already shared by @franzpoeschel. I am tracking this as an issue report in BLAST-ImpactX/impactx#868 as well now. First of all, to fully understand your problem: Do you mind sharing your full analysis routines? Is the memory already spiking just at If you don't mind, can you share a reproducer, e.g., the ImpactX input (or simplified version) and a demonstrator in analysis? How many turns (outputs) and particles are in your >120GB simulation? My guess currently is that your memory is not spiking on Otherwise, I can guide you in the meantime to change the output mode in your ImpactX file to use another format that does not have this issue. Also, to give you quick relief, are you aware of the new
@franzpoeschel FYI: we default to group based encoding, so that might be part of the problem with BP5. |
Beta Was this translation helpful? Give feedback.
-
|
@ax3l I am using ImpactX via Python! Followed the steps through conda installer via mamba. Only reason following this path was, could not install adios2 through different channels... I just realized there is a switch for encoding!! I am not that expert in this so I just left it as default which seems to be group-based. I did not know it would have caused a problem like this, since it was all good with the h5 file! |
Beta Was this translation helpful? Give feedback.
-
|
@OLuckyG perfect, then the options you can use for now are: ADIOS2 BP4monitor = BeamMonitor("monitor", backend="bp4", encoding="g")or monitor = BeamMonitor("monitor", backend="bp4", encoding="f")ADIOS2 BP5monitor = BeamMonitor("monitor", backend="bp5", encoding="f")or experimental monitor = BeamMonitor("monitor", backend="bp5", encoding="v")HDF5monitor = BeamMonitor("monitor", backend="h5", encoding="g")or monitor = BeamMonitor("monitor", backend="h5", encoding="f") |
Beta Was this translation helpful? Give feedback.
-
Yep, that default is definitely a bug and we will fix this now thanks to your report. Unfortunately, monitor = BeamMonitor("monitor", backend="bp5", encoding="g")was the default since I updated ADIOS2 from 2.8 to 2.10, which I change and throw an error for in ImpactX |
Beta Was this translation helpful? Give feedback.
-
|
@ax3l Honestly, I am not sure what difference will it make in terms of the output between group based and file based encodings. Will that imply to do some changes the way files are being read for analysis or the openpmd_api will take care of it the backend? |
Beta Was this translation helpful? Give feedback.
-
|
What we call "file based" encoding creates a new openPMD file per turn of your beam monitor, e.g., import openpmd_api as io
series = io.Series("openPMD_%T.bp", io.Access.read_only)The "group based" and "variable based" encoding uses a single logical file, e.g., import openpmd_api as io
series = io.Series("openPMD.bp", io.Access.read_only)The file name "encoding" is all that changes for you. |
Beta Was this translation helpful? Give feedback.
-
|
I have replicated a similar workflow as yours and I think that I can verify this is a metadata issue in BP5. Using PIConGPU to create 10000 output steps, once using variable-based encoding, once using group-based encoding: The more than 60GB size difference is entirely in metadata. For your upcoming runs, either:
For your existing data, either:
|
Beta Was this translation helpful? Give feedback.
-
|
Got it thanks!! I already set up a simulation in the H5 format! I was using the BP format for the sake of accessing the data while it was being written because I do not want to wait a long time to see that the results did not go as it was planned so it gives an early access to the data so I do not waste time in waiting for the results and I can change things as needed! Thanks for all the replies! I will go ahead and mark it as answered! |
Beta Was this translation helpful? Give feedback.
-
|
Fix to ImpactX: BLAST-ImpactX/impactx#870 @OLuckyG you can still use this (i.e., |
Beta Was this translation helpful? Give feedback.
-
|
Should we limit the number of steps written in BP5 + group-based encoding to a number of ~1000? |
Beta Was this translation helpful? Give feedback.
-
|
Hm, it sounds to me like with the current (memory) read-back scaling we want to abort on any write attempt using BP5 + group-based, no? |
Beta Was this translation helpful? Give feedback.
I have replicated a similar workflow as yours and I think that I can verify this is a metadata issue in BP5. Using PIConGPU to create 10000 output steps, once using variable-based encoding, once using group-based encoding: