Lots of features are broken with the ParallelRunner, what do we do?

[astrojuanlu]

A collection of recent mentions of the ParallelRunner:

• [Investigate] ParallelRunner does not work with S3-linked data catalog #2162
• Investigate why Spaceflights project failing with ParallelRunner  #3674
• Partitioned dataset does not work with parallel runner because of caching in exists method kedro-plugins#623
• Viz hook is broken with ParallelRunner [Blocked by Framework]  kedro-viz#1801

This isn't necessarily the fault of the ParallelRunner, but it's how users experience it.

I'm always hesitant to recommend the ParallelRunner when people have performance issues because of problems like these. There might be different root causes, but the end result is that the runner appears broken to users.

Should we

• Try to fix them?
• Recommend alternative approaches, like Dask https://docs.kedro.org/en/stable/deployment/dask.html or Ray Document how users can use Ray with Kedro  #479?

[astrojuanlu]

In particular, there seems to be ways to check whether an object can be pickled, with some useful debugging info https://stackoverflow.com/a/32206955