ORT backend always returns tensor on CPU

[aklife97]

Description
The ORT backend always returns output tensors on CPU even when the instance is on GPU - when run using BLS through the python backend.

Expected behavior
The output tensor should be on the GPU when the instance kind is GPU for the ONNX model.

[CoderHam]

@Tabrizian @tanmayv25 can you look into the same

[askhade]

This issue explains the current limitation and why output is always on CPU : triton-inference-server/server#3364

[askhade]

Once triton-inference-server/server#3364 is merged we will enable output binding to gpus in ort backend.

[Slyne]

Any update ?
@askhade
@deadeyegoodwin

[askhade]

@Slyne This will be available in triton 22.02 release.

[Slyne]

@askhade Thank you for informing!

[vu0607]

@askhade
I serving encoder-decoder model (TrOCR) on Triton onnx backend. I meet a problem:
First, I call and get output from encoder model in server. Afterthat, because output on GPU, I need to transfer to CPU for converting to numpy before call output from decoder model on server. It make botteneck. Hope you can help me with issue. Thanks a lot.