Serial VTK and binary outputs crash when ghost zone output is enabled

[jmstone]

In GitLab by @jfields7 on Feb 28, 2023, 13:42

When trying to write .vtk output without MPI enabled, adding the flag ghost_zones = true to the parameter file causes a segmentation fault during output. This is true on both CPU and GPU. It seems to be related to the size of the output data array; in serial, its size is calculated using nout1*nout2*nout3, but these variables only include the physical grid cells. This also seems to be an issue with serial binary output, but not ASCII tables. I haven't tested it, but I don't think the MPI output is affected by this issue, as it appears to write each MeshBlock separately.

I haven't looked at it in much detail, but if this is nontrivial to fix, then it would be worthwhile to spit out an error or a warning at startup if ghost_zones = true and file_type = vtk or file_type = bin when MPI_PARALLEL_ENABLED is 0.

[jmstone]

In GitLab by @felker on Apr 5, 2023, 16:08

When I played around with this problem a month ago, I think I was able to reproduce the segfault, but I am unable to reproduce this now. Maybe there is another condition?

On my M2 MacBook:

mkdir build; cd build
cmake -D CMAKE_BUILD_TYPE=Debug ../
make -j20
cd src
./athena -i ../../inputs/tests/linear_wave_mhd.athinput time/nlim=10 output3/ghost_zones=true output4/ghost_zones=true output4/file_type=bin

Root grid = 1 x 1 x 1 MeshBlocks
Total number of MeshBlocks = 1
Number of logical  levels of refinement = 0 (1 levels total)
Number of physical levels of refinement = 0 (1 levels total)
Number of parallel ranks = 1

Setup complete, executing task list...

cycle=0 time=0.000000e+00 dt=7.028600e-03
cycle=1 time=7.028600e-03 dt=7.028606e-03
cycle=2 time=1.405721e-02 dt=7.028617e-03
cycle=3 time=2.108582e-02 dt=7.028626e-03
cycle=4 time=2.811445e-02 dt=7.028633e-03
cycle=5 time=3.514308e-02 dt=7.028639e-03
cycle=6 time=4.217172e-02 dt=7.028644e-03
cycle=7 time=4.920037e-02 dt=7.028650e-03
cycle=8 time=5.622902e-02 dt=7.028655e-03
cycle=9 time=6.325767e-02 dt=7.028660e-03
cycle=10 time=7.028633e-02 dt=7.028664e-03

Terminating on cycle limit
time=7.028633e-02 cycle=10
tlim=2.500000e+00 nlim=10

MeshBlock-cycles = 10
cpu time used  = 2.036897e+01
zone-cycles/cpu_second = 3.217443e+04

And the vtk files at least are differently sized depending on the ghost zone setting:

❯ ll vtk*
vtk:
total 3.8M
-rw-r--r-- 1 felker staff 1.3M Apr  5 15:07 LinWave.mhd_w.00000.vtk
-rw-r--r-- 1 felker staff 1.3M Apr  5 15:07 LinWave.mhd_w.00001.vtk
-rw-r--r-- 1 felker staff 1.3M Apr  5 15:07 LinWave.mhd_w.00002.vtk

vtk-gzs:
total 5.1M
-rw-r--r-- 1 felker staff 1.7M Apr  5 15:06 LinWave.mhd_w.00000.vtk
-rw-r--r-- 1 felker staff 1.7M Apr  5 15:06 LinWave.mhd_w.00001.vtk
-rw-r--r-- 1 felker staff 1.7M Apr  5 15:06 LinWave.mhd_w.00002.vtk

I also tried with multiple MeshBlocks without issue in serial CPU operation. Is this still a problem for you / can you share a complete reproducer?

[jmstone]

In GitLab by @felker on May 15, 2023, 14:50

Ping on this @jfields7

[jmstone]

In GitLab by @jmstone216 on Jun 29, 2023, 11:57

It is probably too late to comment on this now, but this is a feature not a bug. The VTK format does not support multiple MeshBlocks in a single file (everything has to written as one large array). Thus if you try to write ghost zones in VTK files that contain more than one MeshBlock that introduces extra data in the array interior that cannot be represented, and I guess causes the code to crash in some cases. I need to add a trap that does not allow users to write ghost zones in VTK format with multiple MBs. Note that you can write just one MeshBlock in a VTK file (specified using the grid ID of that block), and in that case it is possible to write the ghost zones.