Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak in bounding_box_query? #850

Open
grst opened this issue Jan 29, 2025 · 0 comments
Open

Memory leak in bounding_box_query? #850

grst opened this issue Jan 29, 2025 · 0 comments

Comments

@grst
Copy link
Contributor

grst commented Jan 29, 2025

I am working with a spatialdata object with 49 visium samples:

sdata 
SpatialData object, with associated Zarr store: /cfs/<redacted>/02_nfcore_spatialvi/output/aggregation/data/merged_sdata.zarr
├── Images
│     ├── 'study1_sample1_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample1_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample2_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample2_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample3_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample3_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample4_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample4_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample5_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample5_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample6_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample6_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample7_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample7_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample8_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample8_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample9_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample9_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample10_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample10_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample11_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample11_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample12_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample12_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample13_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample13_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample14_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample14_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample15_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample15_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study1_sample16_hires_image': DataArray[cyx] (3, 2000, 2000)
│     ├── 'study1_sample16_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study2_sampleBI33_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study2_sampleBI33_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study2_sampleBI39_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study2_sampleBI39_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study2_sampleBI64_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study2_sampleBI64_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study2_sampleBI66_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study2_sampleBI66_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB1_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB1_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB2_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB2_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB3_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB3_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB4_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB4_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB5_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB5_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB6_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB6_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB7_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB7_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB8_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB8_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB9_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB9_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB10_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB10_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB11_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB11_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB12_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB12_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB13_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB13_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB14_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB14_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB15_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB15_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study3_sampleB16_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study3_sampleB16_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB1_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB1_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB2_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB2_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB5_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB5_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB6_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB6_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB7_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB7_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB9_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB9_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB10_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB10_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB11_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB11_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB12_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB12_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB13_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB13_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB14_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB14_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB15_hires_image': DataArray[cyx] (3, 3000, 3000)
│     ├── 'study4_sampleB15_lowres_image': DataArray[cyx] (3, 600, 600)
│     ├── 'study4_sampleB16_hires_image': DataArray[cyx] (3, 3000, 3000)
│     └── 'study4_sampleB16_lowres_image': DataArray[cyx] (3, 600, 600)
├── Shapes
│     ├── 'study1_sample1': GeoDataFrame shape: (245, 2) (2D shapes)
│     ├── 'study1_sample2': GeoDataFrame shape: (522, 2) (2D shapes)
│     ├── 'study1_sample3': GeoDataFrame shape: (320, 2) (2D shapes)
│     ├── 'study1_sample4': GeoDataFrame shape: (472, 2) (2D shapes)
│     ├── 'study1_sample5': GeoDataFrame shape: (823, 2) (2D shapes)
│     ├── 'study1_sample6': GeoDataFrame shape: (206, 2) (2D shapes)
│     ├── 'study1_sample7': GeoDataFrame shape: (192, 2) (2D shapes)
│     ├── 'study1_sample8': GeoDataFrame shape: (397, 2) (2D shapes)
│     ├── 'study1_sample9': GeoDataFrame shape: (122, 2) (2D shapes)
│     ├── 'study1_sample10': GeoDataFrame shape: (722, 2) (2D shapes)
│     ├── 'study1_sample11': GeoDataFrame shape: (271, 2) (2D shapes)
│     ├── 'study1_sample12': GeoDataFrame shape: (332, 2) (2D shapes)
│     ├── 'study1_sample13': GeoDataFrame shape: (472, 2) (2D shapes)
│     ├── 'study1_sample14': GeoDataFrame shape: (1272, 2) (2D shapes)
│     ├── 'study1_sample15': GeoDataFrame shape: (222, 2) (2D shapes)
│     ├── 'study1_sample16': GeoDataFrame shape: (274, 2) (2D shapes)
│     ├── 'study2_sampleBI33': GeoDataFrame shape: (6485, 2) (2D shapes)
│     ├── 'study2_sampleBI39': GeoDataFrame shape: (10255, 2) (2D shapes)
│     ├── 'study2_sampleBI64': GeoDataFrame shape: (9504, 2) (2D shapes)
│     ├── 'study2_sampleBI66': GeoDataFrame shape: (5764, 2) (2D shapes)
│     ├── 'study3_sampleB1': GeoDataFrame shape: (3202, 2) (2D shapes)
│     ├── 'study3_sampleB2': GeoDataFrame shape: (6127, 2) (2D shapes)
│     ├── 'study3_sampleB3': GeoDataFrame shape: (2859, 2) (2D shapes)
│     ├── 'study3_sampleB4': GeoDataFrame shape: (7092, 2) (2D shapes)
│     ├── 'study3_sampleB5': GeoDataFrame shape: (8514, 2) (2D shapes)
│     ├── 'study3_sampleB6': GeoDataFrame shape: (6704, 2) (2D shapes)
│     ├── 'study3_sampleB7': GeoDataFrame shape: (8184, 2) (2D shapes)
│     ├── 'study3_sampleB8': GeoDataFrame shape: (7764, 2) (2D shapes)
│     ├── 'study3_sampleB9': GeoDataFrame shape: (5941, 2) (2D shapes)
│     ├── 'study3_sampleB10': GeoDataFrame shape: (6814, 2) (2D shapes)
│     ├── 'study3_sampleB11': GeoDataFrame shape: (4438, 2) (2D shapes)
│     ├── 'study3_sampleB12': GeoDataFrame shape: (4798, 2) (2D shapes)
│     ├── 'study3_sampleB13': GeoDataFrame shape: (8419, 2) (2D shapes)
│     ├── 'study3_sampleB14': GeoDataFrame shape: (7236, 2) (2D shapes)
│     ├── 'study3_sampleB15': GeoDataFrame shape: (9298, 2) (2D shapes)
│     ├── 'study3_sampleB16': GeoDataFrame shape: (4689, 2) (2D shapes)
│     ├── 'study4_sampleB1': GeoDataFrame shape: (5475, 2) (2D shapes)
│     ├── 'study4_sampleB2': GeoDataFrame shape: (5765, 2) (2D shapes)
│     ├── 'study4_sampleB5': GeoDataFrame shape: (3165, 2) (2D shapes)
│     ├── 'study4_sampleB6': GeoDataFrame shape: (2332, 2) (2D shapes)
│     ├── 'study4_sampleB7': GeoDataFrame shape: (3461, 2) (2D shapes)
│     ├── 'study4_sampleB9': GeoDataFrame shape: (5700, 2) (2D shapes)
│     ├── 'study4_sampleB10': GeoDataFrame shape: (6566, 2) (2D shapes)
│     ├── 'study4_sampleB11': GeoDataFrame shape: (3704, 2) (2D shapes)
│     ├── 'study4_sampleB12': GeoDataFrame shape: (4223, 2) (2D shapes)
│     ├── 'study4_sampleB13': GeoDataFrame shape: (2828, 2) (2D shapes)
│     ├── 'study4_sampleB14': GeoDataFrame shape: (2050, 2) (2D shapes)
│     ├── 'study4_sampleB15': GeoDataFrame shape: (6022, 2) (2D shapes)
│     └── 'study4_sampleB16': GeoDataFrame shape: (4254, 2) (2D shapes)
└── Tables
      └── 'all': AnnData (196496, 11693)
with coordinate systems:
    ▸ 'downscaled_hires', with elements:
        study1_sample1_hires_image (Images), study1_sample2_hires_image (Images), study1_sample3_hires_image (Images), study1_sample4_hires_image (Images), study1_sample5_hires_image (Images), study1_sample6_hires_image (Images), study1_sample7_hires_image (Images), study1_sample8_hires_image (Images), study1_sample9_hires_image (Images), study1_sample10_hires_image (Images), study1_sample11_hires_image (Images), study1_sample12_hires_image (Images), study1_sample13_hires_image (Images), study1_sample14_hires_image (Images), study1_sample15_hires_image (Images), study1_sample16_hires_image (Images), study2_sampleBI33_hires_image (Images), study2_sampleBI39_hires_image (Images), study2_sampleBI64_hires_image (Images), study2_sampleBI66_hires_image (Images), study3_sampleB1_hires_image (Images), study3_sampleB2_hires_image (Images), study3_sampleB3_hires_image (Images), study3_sampleB4_hires_image (Images), study3_sampleB5_hires_image (Images), study3_sampleB6_hires_image (Images), study3_sampleB7_hires_image (Images), study3_sampleB8_hires_image (Images), study3_sampleB9_hires_image (Images), study3_sampleB10_hires_image (Images), study3_sampleB11_hires_image (Images), study3_sampleB12_hires_image (Images), study3_sampleB13_hires_image (Images), study3_sampleB14_hires_image (Images), study3_sampleB15_hires_image (Images), study3_sampleB16_hires_image (Images), study4_sampleB1_hires_image (Images), study4_sampleB2_hires_image (Images), study4_sampleB5_hires_image (Images), study4_sampleB6_hires_image (Images), study4_sampleB7_hires_image (Images), study4_sampleB9_hires_image (Images), study4_sampleB10_hires_image (Images), study4_sampleB11_hires_image (Images), study4_sampleB12_hires_image (Images), study4_sampleB13_hires_image (Images), study4_sampleB14_hires_image (Images), study4_sampleB15_hires_image (Images), study4_sampleB16_hires_image (Images), study1_sample1 (Shapes), study1_sample2 (Shapes), study1_sample3 (Shapes), study1_sample4 (Shapes), study1_sample5 (Shapes), study1_sample6 (Shapes), study1_sample7 (Shapes), study1_sample8 (Shapes), study1_sample9 (Shapes), study1_sample10 (Shapes), study1_sample11 (Shapes), study1_sample12 (Shapes), study1_sample13 (Shapes), study1_sample14 (Shapes), study1_sample15 (Shapes), study1_sample16 (Shapes), study2_sampleBI33 (Shapes), study2_sampleBI39 (Shapes), study2_sampleBI64 (Shapes), study2_sampleBI66 (Shapes), study3_sampleB1 (Shapes), study3_sampleB2 (Shapes), study3_sampleB3 (Shapes), study3_sampleB4 (Shapes), study3_sampleB5 (Shapes), study3_sampleB6 (Shapes), study3_sampleB7 (Shapes), study3_sampleB8 (Shapes), study3_sampleB9 (Shapes), study3_sampleB10 (Shapes), study3_sampleB11 (Shapes), study3_sampleB12 (Shapes), study3_sampleB13 (Shapes), study3_sampleB14 (Shapes), study3_sampleB15 (Shapes), study3_sampleB16 (Shapes), study4_sampleB1 (Shapes), study4_sampleB2 (Shapes), study4_sampleB5 (Shapes), study4_sampleB6 (Shapes), study4_sampleB7 (Shapes), study4_sampleB9 (Shapes), study4_sampleB10 (Shapes), study4_sampleB11 (Shapes), study4_sampleB12 (Shapes), study4_sampleB13 (Shapes), study4_sampleB14 (Shapes), study4_sampleB15 (Shapes), study4_sampleB16 (Shapes)
    ▸ 'downscaled_lowres', with elements:
        study1_sample1_lowres_image (Images), study1_sample2_lowres_image (Images), study1_sample3_lowres_image (Images), study1_sample4_lowres_image (Images), study1_sample5_lowres_image (Images), study1_sample6_lowres_image (Images), study1_sample7_lowres_image (Images), study1_sample8_lowres_image (Images), study1_sample9_lowres_image (Images), study1_sample10_lowres_image (Images), study1_sample11_lowres_image (Images), study1_sample12_lowres_image (Images), study1_sample13_lowres_image (Images), study1_sample14_lowres_image (Images), study1_sample15_lowres_image (Images), study1_sample16_lowres_image (Images), study2_sampleBI33_lowres_image (Images), study2_sampleBI39_lowres_image (Images), study2_sampleBI64_lowres_image (Images), study2_sampleBI66_lowres_image (Images), study3_sampleB1_lowres_image (Images), study3_sampleB2_lowres_image (Images), study3_sampleB3_lowres_image (Images), study3_sampleB4_lowres_image (Images), study3_sampleB5_lowres_image (Images), study3_sampleB6_lowres_image (Images), study3_sampleB7_lowres_image (Images), study3_sampleB8_lowres_image (Images), study3_sampleB9_lowres_image (Images), study3_sampleB10_lowres_image (Images), study3_sampleB11_lowres_image (Images), study3_sampleB12_lowres_image (Images), study3_sampleB13_lowres_image (Images), study3_sampleB14_lowres_image (Images), study3_sampleB15_lowres_image (Images), study3_sampleB16_lowres_image (Images), study4_sampleB1_lowres_image (Images), study4_sampleB2_lowres_image (Images), study4_sampleB5_lowres_image (Images), study4_sampleB6_lowres_image (Images), study4_sampleB7_lowres_image (Images), study4_sampleB9_lowres_image (Images), study4_sampleB10_lowres_image (Images), study4_sampleB11_lowres_image (Images), study4_sampleB12_lowres_image (Images), study4_sampleB13_lowres_image (Images), study4_sampleB14_lowres_image (Images), study4_sampleB15_lowres_image (Images), study4_sampleB16_lowres_image (Images), study1_sample1 (Shapes), study1_sample2 (Shapes), study1_sample3 (Shapes), study1_sample4 (Shapes), study1_sample5 (Shapes), study1_sample6 (Shapes), study1_sample7 (Shapes), study1_sample8 (Shapes), study1_sample9 (Shapes), study1_sample10 (Shapes), study1_sample11 (Shapes), study1_sample12 (Shapes), study1_sample13 (Shapes), study1_sample14 (Shapes), study1_sample15 (Shapes), study1_sample16 (Shapes), study2_sampleBI33 (Shapes), study2_sampleBI39 (Shapes), study2_sampleBI64 (Shapes), study2_sampleBI66 (Shapes), study3_sampleB1 (Shapes), study3_sampleB2 (Shapes), study3_sampleB3 (Shapes), study3_sampleB4 (Shapes), study3_sampleB5 (Shapes), study3_sampleB6 (Shapes), study3_sampleB7 (Shapes), study3_sampleB8 (Shapes), study3_sampleB9 (Shapes), study3_sampleB10 (Shapes), study3_sampleB11 (Shapes), study3_sampleB12 (Shapes), study3_sampleB13 (Shapes), study3_sampleB14 (Shapes), study3_sampleB15 (Shapes), study3_sampleB16 (Shapes), study4_sampleB1 (Shapes), study4_sampleB2 (Shapes), study4_sampleB5 (Shapes), study4_sampleB6 (Shapes), study4_sampleB7 (Shapes), study4_sampleB9 (Shapes), study4_sampleB10 (Shapes), study4_sampleB11 (Shapes), study4_sampleB12 (Shapes), study4_sampleB13 (Shapes), study4_sampleB14 (Shapes), study4_sampleB15 (Shapes), study4_sampleB16 (Shapes)
    ▸ 'global', with elements:
        study1_sample1 (Shapes), study1_sample2 (Shapes), study1_sample3 (Shapes), study1_sample4 (Shapes), study1_sample5 (Shapes), study1_sample6 (Shapes), study1_sample7 (Shapes), study1_sample8 (Shapes), study1_sample9 (Shapes), study1_sample10 (Shapes), study1_sample11 (Shapes), study1_sample12 (Shapes), study1_sample13 (Shapes), study1_sample14 (Shapes), study1_sample15 (Shapes), study1_sample16 (Shapes), study2_sampleBI33 (Shapes), study2_sampleBI39 (Shapes), study2_sampleBI64 (Shapes), study2_sampleBI66 (Shapes), study3_sampleB1 (Shapes), study3_sampleB2 (Shapes), study3_sampleB3 (Shapes), study3_sampleB4 (Shapes), study3_sampleB5 (Shapes), study3_sampleB6 (Shapes), study3_sampleB7 (Shapes), study3_sampleB8 (Shapes), study3_sampleB9 (Shapes), study3_sampleB10 (Shapes), study3_sampleB11 (Shapes), study3_sampleB12 (Shapes), study3_sampleB13 (Shapes), study3_sampleB14 (Shapes), study3_sampleB15 (Shapes), study3_sampleB16 (Shapes), study4_sampleB1 (Shapes), study4_sampleB2 (Shapes), study4_sampleB5 (Shapes), study4_sampleB6 (Shapes), study4_sampleB7 (Shapes), study4_sampleB9 (Shapes), study4_sampleB10 (Shapes), study4_sampleB11 (Shapes), study4_sampleB12 (Shapes), study4_sampleB13 (Shapes), study4_sampleB14 (Shapes), study4_sampleB15 (Shapes), study4_sampleB16 (Shapes)
with the following elements not in the Zarr store:
    ▸ all (Tables)
with the following elements in the Zarr store but not in the SpatialData object:
    ▸ study1_sample5_table (Tables)
    ▸ study1_sample6_table (Tables)
    ▸ study3_sampleB14_table (Tables)
    ▸ study1_sample1_table (Tables)
    ▸ study1_sample7_table (Tables)
    ▸ study1_sample2_table (Tables)
    ▸ study4_sampleB2_table (Tables)
    ▸ study1_sample4_table (Tables)
    ▸ study3_sampleB11_table (Tables)
    ▸ study3_sampleB15_table (Tables)
    ▸ study1_sample9_table (Tables)
    ▸ study3_sampleB6_table (Tables)
    ▸ study4_sampleB13_table (Tables)
    ▸ study4_sampleB12_table (Tables)
    ▸ study3_sampleB2_table (Tables)
    ▸ study1_sample10_table (Tables)
    ▸ study3_sampleB8_table (Tables)
    ▸ study1_sample13_table (Tables)
    ▸ study4_sampleB1_table (Tables)
    ▸ study4_sampleB11_table (Tables)
    ▸ study4_sampleB5_table (Tables)
    ▸ study1_sample8_table (Tables)
    ▸ study3_sampleB16_table (Tables)
    ▸ study4_sampleB16_table (Tables)
    ▸ study4_sampleB14_table (Tables)
    ▸ study3_sampleB1_table (Tables)
    ▸ study4_sampleB15_table (Tables)
    ▸ study2_sampleBI39_table (Tables)
    ▸ study1_sample14_table (Tables)
    ▸ study4_sampleB10_table (Tables)
    ▸ study2_sampleBI64_table (Tables)
    ▸ study1_sample15_table (Tables)
    ▸ study3_sampleB12_table (Tables)
    ▸ study3_sampleB4_table (Tables)
    ▸ study1_sample11_table (Tables)
    ▸ study1_sample3_table (Tables)
    ▸ study3_sampleB13_table (Tables)
    ▸ study3_sampleB7_table (Tables)
    ▸ study3_sampleB5_table (Tables)
    ▸ study4_sampleB6_table (Tables)
    ▸ study2_sampleBI33_table (Tables)
    ▸ study1_sample12_table (Tables)
    ▸ study3_sampleB9_table (Tables)
    ▸ study3_sampleB3_table (Tables)
    ▸ study4_sampleB7_table (Tables)
    ▸ study3_sampleB10_table (Tables)
    ▸ study2_sampleBI66_table (Tables)
    ▸ study1_sample16_table (Tables)
    ▸ study4_sampleB9_table (Tables)

I noticed that when running a bounding box query in a loop over all samples, it gets slower and slower and consumes a lot of memory.

from datetime import datetime
import numpy as np
import spatialdata as sd
import psutil

for _ in range(300):
    tic = datetime.now()
    bb = np.array(list(sd.get_extent(sdata, elements=["study2_sampleBI33"], coordinate_system="downscaled_hires").values()))
    sdata_bb = sd.bounding_box_query(sdata, ("x", "y"), bb[:, 0], bb[:, 1], "downscaled_hires")
    toc = datetime.now()
    print(f"time elapsed: {(toc - tic).total_seconds()} s; memory used: {psutil.Process().memory_info().rss / 1e9} GB")
time elapsed: 9.881882 s; memory used: 46.823497728 GB
time elapsed: 9.644644 s; memory used: 60.7720448 GB
time elapsed: 10.161626 s; memory used: 74.720657408 GB
time elapsed: 31.029099 s; memory used: 88.669437952 GB
time elapsed: 52.600431 s; memory used: 102.618013696 GB
time elapsed: 177.883717 s; memory used: 116.706766848 GB
time elapsed: 146.74199 s; memory used: 131.057217536 GB
time elapsed: 160.282223 s; memory used: 145.419689984 GB
time elapsed: 99.229229 s; memory used: 159.781974016 GB
time elapsed: 54.737399 s; memory used: 174.199156736 GB
time elapsed: 39.372628 s; memory used: 188.563365888 GB
time elapsed: 83.340401 s; memory used: 202.927603712 GB
time elapsed: 80.725091 s; memory used: 217.291235328 GB
time elapsed: 54.610381 s; memory used: 231.655145472 GB
time elapsed: 65.644117 s; memory used: 246.018277376 GB
time elapsed: 59.632265 s; memory used: 260.435939328 GB
time elapsed: 92.770736 s; memory used: 274.79996416 GB
time elapsed: 68.518154 s; memory used: 289.163894784 GB
time elapsed: 89.953097 s; memory used: 303.529623552 GB
time elapsed: 72.767521 s; memory used: 317.894258688 GB
time elapsed: 60.128739 s; memory used: 332.256210944 GB
time elapsed: 97.550062 s; memory used: 346.6724352 GB
time elapsed: 16.891613 s; memory used: 53.195804672 GB
time elapsed: 10.562216 s; memory used: 67.1446016 GB
time elapsed: 10.575739 s; memory used: 81.093279744 GB
time elapsed: 10.60881 s; memory used: 95.042097152 GB
time elapsed: 10.071131 s; memory used: 108.990795776 GB
time elapsed: 10.788739 s; memory used: 122.939232256 GB
time elapsed: 10.600906 s; memory used: 136.887844864 GB
time elapsed: 10.857134 s; memory used: 150.836383744 GB
time elapsed: 11.364936 s; memory used: 164.785098752 GB
time elapsed: 11.049787 s; memory used: 178.733674496 GB
time elapsed: 11.108218 s; memory used: 192.682438656 GB
time elapsed: 31.675223 s; memory used: 206.630981632 GB
time elapsed: 28.117611 s; memory used: 220.579495936 GB
time elapsed: 19.649157 s; memory used: 234.528223232 GB
time elapsed: 13.645791 s; memory used: 248.476798976 GB
time elapsed: 10.456291 s; memory used: 262.42537472 GB

It seems eventually garbage collection kicks in, but it does get a lot slower before that.

When forcing a gc.collect() after each iteration, the problem is gone - but that's not something I would typically do.

time elapsed: 11.078948 s; memory used: 48.512786432 GB
time elapsed: 10.736634 s; memory used: 48.512892928 GB
time elapsed: 11.165556 s; memory used: 48.512897024 GB
time elapsed: 10.7712 s; memory used: 48.512897024 GB
time elapsed: 10.836686 s; memory used: 48.512897024 GB
time elapsed: 10.36456 s; memory used: 48.512897024 GB
time elapsed: 10.898954 s; memory used: 48.512897024 GB
time elapsed: 10.771655 s; memory used: 48.512909312 GB
time elapsed: 10.383164 s; memory used: 48.512909312 GB
time elapsed: 10.23622 s; memory used: 48.512909312 GB
time elapsed: 10.185976 s; memory used: 48.512909312 GB
time elapsed: 9.935837 s; memory used: 48.512909312 GB
time elapsed: 10.32161 s; memory used: 48.512909312 GB
time elapsed: 10.238209 s; memory used: 48.512909312 GB
time elapsed: 10.780759 s; memory used: 48.512909312 GB
time elapsed: 10.409318 s; memory used: 48.512909312 GB
time elapsed: 10.442059 s; memory used: 48.513101824 GB
time elapsed: 10.311367 s; memory used: 48.513101824 GB
time elapsed: 10.396318 s; memory used: 48.513101824 GB
time elapsed: 10.189232 s; memory used: 48.513101824 GB
time elapsed: 9.997263 s; memory used: 48.513101824 GB
time elapsed: 9.915348 s; memory used: 48.513101824 GB
time elapsed: 10.007159 s; memory used: 48.51310592 GB
time elapsed: 10.333178 s; memory used: 48.51308544 GB
time elapsed: 10.172472 s; memory used: 48.51308544 GB
time elapsed: 9.798267 s; memory used: 48.51308544 GB
time elapsed: 9.770812 s; memory used: 48.51308544 GB
time elapsed: 10.800851 s; memory used: 48.51308544 GB
time elapsed: 10.98336 s; memory used: 48.51308544 GB

Environment:

uv.lock.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@grst