How to avoid the cloudvolume.exceptions.EmptyVolumeException

[liuyx599]

Hi, I recently generated mesh and skeleton for the segmentation results of a larger electron microscopy dataset, and I used the following commands:

Generate mesh:

igneous mesh forge  $segdir --dir mesh --mip 2 --queue $meshforge_queue --sharded
igneous execute -x $meshforge_queue
igneous mesh merge-sharded $segdir --queue $meshmerge_queue --nlod 0
igneous execute -x $meshmerge_queue

ptq status $meshforge_queue
ptq status $meshmerge_queue

Generate skeleton:

igneous skeleton forge  $segdir --mip 2 --queue $skelforge_queue --scale 4 --const 100 --sharded
igneous execute -x $skelforge_queue
igneous skeleton merge-sharded $segdir --queue $skelmerge_queue
igneous execute -x $skelmerge_queue

ptq status $skelforge_queue
ptq status $skelmerge_queue

When running the command igneous execute, the same error seems to occur in both scripts:

INFO Running SkeletonTask(cloudpath='file:///EM/autoseg',shape=[513, 513, 513],offset=[47044, 34201, 25090],mip=2,teasar_params={'scale': 4.0, 'const': 100.0, 'pdrf_exponent': 4, 'pdrf_scale': 100000, 'soma_detection_threshold': 1100.0, 'soma_acceptance_threshold': 3500.0, 'soma_invalidation_scale': 1.0, 'soma_invalidation_const': 300.0, 'max_paths': None},will_postprocess=True,info=None,object_ids=None,mask_ids=None,fix_branching=True,fix_borders=True,fix_avocados=False,fill_holes=False,dust_threshold=1000,progress=False,parallel=1,fill_missing=False,sharded=True,frag_path=None,spatial_index=True,spatial_grid_shape=[512, 512, 512],synapses=None,dust_global=False)  (id: 963ec416-ff67-4856-94e6-85644f45e71b)
ERROR SkeletonTask(cloudpath='file:///EM/autoseg',shape=[513, 513, 513],offset=[15812, 39833, 18946],mip=2,teasar_params={'scale': 4.0, 'const': 100.0, 'pdrf_exponent': 4, 'pdrf_scale': 100000, 'soma_detection_threshold': 1100.0, 'soma_acceptance_threshold': 3500.0, 'soma_invalidation_scale': 1.0, 'soma_invalidation_const': 300.0, 'max_paths': None},will_postprocess=True,info=None,object_ids=None,mask_ids=None,fix_branching=True,fix_borders=True,fix_avocados=False,fill_holes=False,dust_threshold=1000,progress=False,parallel=1,fill_missing=False,sharded=True,frag_path=None,spatial_index=True,spatial_grid_shape=[512, 512, 512],synapses=None,dust_global=False) raised Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)
 Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/taskqueue/taskqueue.py", line 375, in poll
    task.execute(*execute_args, **execute_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/igneous/tasks/skeleton.py", line 103, in execute
    all_labels = vol[ bbox.to_slices() ]
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 551, in __getitem__
    img = self.download(requested_bbox, self.mip)
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 731, in download
    tup = self.image.download(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 190, in download
    return rx.download_sharded(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 99, in download_sharded
    img3d = decode_fn(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 623, in decode
    return _decode_helper(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 684, in _decode_helper
    raise EmptyVolumeException(input_bbox)
cloudvolume.exceptions.EmptyVolumeException: Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)

The staus of $skelforge_queue

Inserted: 312832
Enqueued: 312828 (100.0% left)
Completed: 4 (0.0%)
Leased: 0 (0.0% of queue)

Then, I try to access the autoseg data by CloudVolume

import cloudvolume

cv = cloudvolume.CloudVolume("file:///EM/autoseg",mip=2,cache=True")
cv.key         # '32x32x40'
cv.bounds  # Bbox([6596, 7577, 14850],[54702, 40340, 27858], dtype=int32)

cv[15940, 40089, 18946]

The same error seems to have occurred

  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 551, in __getitem__
    img = self.download(requested_bbox, self.mip)
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 731, in download
    tup = self.image.download(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 190, in download
    return rx.download_sharded(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 99, in download_sharded
    img3d = decode_fn(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 664, in decode_single_voxel
    return _decode_helper(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 684, in _decode_helper
    raise EmptyVolumeException(input_bbox)
cloudvolume.exceptions.EmptyVolumeException: Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)

The cv.meta.info of the autoseg(mip=2) is

{'chunk_sizes': [[64, 64, 64]],
   'compressed_segmentation_block_size': [8, 8, 8],
   'encoding': 'compressed_segmentation',
   'key': '32x32x40',
   'resolution': [32, 32, 40],
   'sharding': {'@type': 'neuroglancer_uint64_sharded_v1',
    'data_encoding': 'gzip',
    'hash': 'identity',
    'minishard_bits': 6,
    'minishard_index_encoding': 'gzip',
    'preshift_bits': 9,
    'shard_bits': 12},
   'size': [48106, 32763, 13008],
   'voxel_offset': [6596, 7577, 14850]},

I was observing the region Bbox([15940, 40089, 18946],[16004, 40153, 19010] on Neuroglancer and it seems to be at the edge of the entire electron microscopy dataset now. It is indeed a dark patch with no segmentation results.

In large-scale electron microscopy datasets, such as FAFB, MiCrONS, H01, etc., the vast majority of neuron segmentation results are in the middle region of the dataset, and it seems to be not uncommon to have no segmentation results in the edge region.

I'm wondering if there is another way to avoid this error, and I'm wondering if cloudvlomue, or igenous, has a relevant api parameter that defaults empty regions to 0 when accessed, which might avoid the above EmptyVolumeException.

[william-silversmith]

Hi! Yes, this is a common problem with a fix. There's a reliability tradeoff since to fix it, we have to ignore missing file errors and autofill them with black. In practice, it's been fine for a few years, but we have seen some missing tiles years ago in earlier versions of the software when used at huge scale.

To fix this, generate your tasks with the --fill-missing flag. e.g.

igneous mesh forge  $segdir --dir mesh --mip 2 --queue $meshforge_queue --sharded --fill-missing
igneous skeleton forge  $segdir --mip 2 --queue $skelforge_queue --scale 4 --const 100 --sharded --fill-missing

Happy meshing and skeletonizing!