[Q&A] OSError: [Errno 5] Input/output error when using Pytorch dataloader

[gonzaq94]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.3.8

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 18.04.3 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I am training a model with one server and two sites. When using the Pytorch dataloader, I encounter the "OSError: [Errno 5] Input/output error". The data used for both sites are actually saved in the same location, could this be the cause of the error?

2023-12-08 11:39:38,124 - ClientRunner - ERROR - [identity=site-2, run=88fb217e-099b-471c-91cf-a7c95bec7f5c, peer=mammo-classifier-two_sites-server_ama10-nvflare_2.3, peer_run=88fb217e-099b-471c-91cf-a7c95bec7f5c, task_name=train, task_id=feb03ace-849e-4d0a-8fab-10b4bf3bb85a]: Processing error from executor LearnerExecutor: OSError: [Errno 5] Input/output error
2023-12-08 11:39:38,124 - ClientRunner - ERROR - Traceback (most recent call last):
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/nvflare/private/fed/client/client_runner.py", line 259, in _process_task
reply = executor.execute(task.name, task.data, fl_ctx, self.task_abort_signal)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/nvflare/app_common/executors/learner_executor.py", line 85, in execute
return self.train(shareable, fl_ctx, abort_signal)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/nvflare/app_common/executors/learner_executor.py", line 100, in train
train_result = self.learner.train(shareable, fl_ctx, abort_signal)
File "/home/user/gitlab/classifier/mmar/custom/learners/mammo_learner.py", line 2230, in train
self.local_train(
File "/home/user/gitlab/classifier/mmar/custom/learners/mammo_learner.py", line 2051, in local_train
_state_output = self.trainer.run()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/monai/engines/trainer.py", line 53, in run
super().run()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/monai/engines/workflow.py", line 278, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 704, in run
return self._internal_run()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 783, in _internal_run
self._handle_exception(e)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 464, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/monai/handlers/stats_handler.py", line 179, in exception_raised
raise e
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 751, in _internal_run
self._setup_engine()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 732, in _setup_engine
self._setup_dataloader_iter()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/ignite/engine/engine.py", line 729, in _setup_dataloader_iter
self._dataloader_iter = iter(self.state.dataloader)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in iter
return self._get_iterator()
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 381, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/home/nvflare-2.3.8-env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1034, in init
w.start()
File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/lib/python3.8/multiprocessing/context.py", line 276, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 16, in init
util._flush_std_streams()
File "/usr/lib/python3.8/multiprocessing/util.py", line 412, in _flush_std_streams
sys.stdout.flush()
OSError: [Errno 5] Input/output error

[gonzaq94]

I have noticed that when the number of workers of the dataloader is 0, this issue does not arise

[YuanTingHsieh]

@gonzaq94 if two sites are downloading and saving dataset into the same location at the same time, it could cause racing issues.

If you want to avoid that, you can download the dataset BEFORE you run any jobs.
And then when running the jobs, they just read from the location.

Have you tried running with number of workers >0 again?

[gonzaq94]

The datasets are already downloaded to that location. All the dataloader does is reading in that location (nothing is written or saved in that location).
The issue still appears when running with workers > 0

[YuanTingHsieh]

@gonzaq94 I look it up, looks like this is a PyTorch issue: pytorch/pytorch#28820 https://pytorch.org/docs/stable/data.html#platform-specific-behaviors pytorch/pytorch#12831 https://stackoverflow.com/questions/77174223/simplest-dataloader-fails-when-num-workers0