You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Taking the serve-imagenet-shards as an example, I implemented for my own WebDataset. There are a few points I would like to highlight:
using multiple workers in the dataloader and providing an address range as zpub://0.0.0.0:788{0..4} results in daemonic processes are not allowed to have children error. To circumvent this, I used from concurrent.futures import ProcessPoolExecutor as Pool instead of multiprocessing.Pool. This worked fine, though there is risk of zombie processes on exit of the main script if the grandchildren processes are still running.
On the client side, I am able to get the data from the address range, however, I am trying to run a multiprocess based webdataset.WebLoader as below:
Taking the
serve-imagenet-shards
as an example, I implemented for my own WebDataset. There are a few points I would like to highlight:zpub://0.0.0.0:788{0..4}
results indaemonic processes are not allowed to have children
error. To circumvent this, I usedfrom concurrent.futures import ProcessPoolExecutor as Pool
instead ofmultiprocessing.Pool
. This worked fine, though there is risk of zombie processes on exit of the main script if the grandchildren processes are still running.webdataset.WebLoader
as below:The above code hangs and is unable to get any data if I use num_workers > 0.
Is there a way to do this?
The text was updated successfully, but these errors were encountered: