How to get intermediate node results before execution ends?

[maaft]

I have a usecase where I chain multiple models into one onnx "supergraph" (dashed section). I do this for increased throughput due to less context switches (gpu->cpu->gpu-> ...).

The problem is, that my external process (in blue) needs to work on the bounding-boxes coming from my ObjectDetection as early as possible so it can finish its post-processing in parallel to the OCR model.

Is it somehow possible to get intermediate results of an ONNX graph before the session ends its execution?

"bounding_boxes" output is already part of the "outputs" of the supergraph. Therefore, I guess that the data will be written into the output tensor as soon as its computed. If this is true, all I'm asking for really is to get a callback or whatever as soon as my data is ready for read-out. Can be unsafe, I don't care really.

In this case, I could solve this by creating a custom-op that I attach at every output I want to track. This custom hook would forward data to my external process.

Though obviously, I'd prefer a more ready-to-use solution, if possible.

Any ideas appreciated! :)