Add support for multi-node for %load_node or %load_pipeline

[noklam]

Description

%load_node assume error comes from the node but this is not always the case.

This happens more commonly in Data Engineering pipeline, where you apply a series of transformation, aggregation on a set of table and pass it to next node. For example, you may get a "Column not found" error. loading the node that thrown out an error is only the first step to inspect the data, but there are still couple of manual steps to figure out where is the source of error.

The process roughly work as a binary search of the upstream nodes.

Context

This augment the existing debugging feature of Kedro and making this much easier for DS & DE

Runner is an abstraction that is powerful but not beginner friendly, bring the execution explicitly into a notebook cell is helpful

It's not a trivial task to figure out the correct execution order from a Kedro pipeline to a imperative manner (i.e. cells run sequentially in a notebook). The abstraction is a distraction mostly during debugging.

Possible Implementation

Limitation: Creating multiple cells is not easy, I tried in %load_node the first time but settle with the current solution because IPython do have limitations. We may be able to do this in Jupyter Notebook (not VSCode notebook) because there are better support

Possible Alternatives

[dundermain]

Hey @merelcht and @noklam , I would like to work on this issue. Let me know if that is okay with you.

[noklam]

@dundermain awesome, we haven't started with this ticket, but if this is something you would like to take a stab, go for it.

[dundermain]

Thanks a lot @noklam

[dundermain]

@noklam I am thinking of some approaches to solve this issue. Let me know if I am making any mistakes.

Approach: Since pipelines are made of nodes, I am planning to write a magic function "load_pipeline" that will take the pipeline name as input and return the contents of all the nodes in that pipeline. The only problem that I can see, and which you have already mentioned, is creating multiple cells. Since the pipeline can have multiple nodes, it might complicate it further.

Another approach that I can think of is instead of giving the pipeline name, the user can pass the node name to the load_pipeline function just like they do for the load_node function. Then, a function like _find_pipeline will search for the pipeline to which the given node belongs and then execute the load_pipeline function.

Let me know your thoughts on this. I am still experimenting with handling that multiple-cell part and will give an update soon.