MONAI transforms extremely slow

[sharonlee12]

I am using Monai for training, and my code is as follows：
transforms = monai.transforms.Compose([ monai.transforms.LoadImaged(keys=["image", "label_path"]), monai.transforms.AddChanneld(keys=["image", "label_path"]), monai.transforms.Orientationd(keys=["image", "label_path"], axcodes="RAS"), monai.transforms.Spacingd(keys=["image", "label_path"], pixdim=(args.space_x, args.space_y, args.space_z), mode=("bilinear", "nearest")), monai.transforms.ScaleIntensityRanged(keys=["image"], a_min=args.a_min, a_max=args.a_max, b_min=args.b_min, b_max=args.b_max, clip=True), monai.transforms.CropForegroundd(keys=["image", "label_path"], source_key="image"), monai.transforms.SpatialPadd(keys=["image", "label_path"], spatial_size=(args.roi_x, args.roi_y, args.roi_z), mode='constant'), monai.transforms.RandCropByPosNegLabeld(keys=["image", "label_path"], label_key="label_path", spatial_size=(args.roi_x, args.roi_y, args.roi_z), pos=1, neg=1, num_samples=args.num_samples, image_key="image", image_threshold=0), monai.transforms.RandShiftIntensityd(keys=["image"], offsets=0.10, prob=0.20), monai.transforms.ToTensord(keys=["image", "label_path"]), ])
The dataset I am using is PersistentDataset
train_ds = PersistentDataset(data=train_dicts, transform=preprocess_sample, cache_dir="./data/cache") train_ds = PersistentDataset(data=train_dicts, transform=preprocess_sample, cache_dir="./data/cache") trainloader = DataLoader(train_ds, batch_size=1, num_workers=32, pin_memory=torch.cuda.is_available())
But I found that processing an image during the training process still takes a long time：

Does anyone know the possible reason

[KumoLiu]

Hi @sharonlee12,
While the PersistentDataset helps with the I/O operations, it unfortunately cannot eliminate the transfer time between the GPU and CPU. If your goal is to further enhance the speed of transformations, you might want to consider using the CacheDataset. For additional insights and performance comparisons, you can refer to the following tutorials:

1. https://github.com/Project-MONAI/tutorials/blob/main/acceleration/dataset_type_performance.ipynb
2. https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb

Hope it helps, thanks.

[sharonlee12]

Hi @sharonlee12, While the PersistentDataset helps with the I/O operations, it unfortunately cannot eliminate the transfer time between the GPU and CPU. If your goal is to further enhance the speed of transformations, you might want to consider using the CacheDataset. For additional insights and performance comparisons, you can refer to the following tutorials:

1. https://github.com/Project-MONAI/tutorials/blob/main/acceleration/dataset_type_performance.ipynb
2. https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb

Hope it helps, thanks.

Hello! I am glad to receive your reply. I am a beginner on Monai. I have previously seen someone summarize that PersistentDataset can be suitable for situations with large amounts of data. I have also tried using cachedataset before, but due to my large amount of data, the CPU seemed to burst during the process of reading data.

Secondly, I would like to ask how long it usually takes for you to process a single image (batch) using Monai for 3D medical image segmentation?

Hope to receive your reply, thank you!

[KumoLiu]

Hi @sharonlee12, determining a specific timeframe can be challenging without knowing the details of your setup. However, you could run the tutorial and assess the results to see if they align with your expectations. The tutorial also includes some benchmark results which might provide further insights.

Thanks.