gRPC response is cropped, where REST /generate fully sent

[mkhludnev]

Hello,
I use triton with vLLM model. For some reason vLLM backend prepends response with prompt. Don't know why, asked here.
As a result of RAG approach I've got really long responses. The problem is that REST /generate return long response fully, and gRPC crops long response at certain length. I'm using python gRPC client, but I witness cropping in verbose log.
More details, excepts from configs and logs are here.

[oandreeva-nv]

Hi @mkhludnev, could you please give us a rough estimation of how long your prompt is, or provide an example of a PROMPT for a reproducer, we'd highly appreciate it.

cc @itskevinwang

[mkhludnev]

Sure.
curl got 200 words rest-triton.txt
gRPC takes only 20 grpc-triton.txt
I use langchain module as a client.
Here's the model for vllm_bacnkend
model.json
config.pbtxt.txt

I'm trying to find a brief reproducer but it's tough to a noob like myself.

[oandreeva-nv]

Thank you! I think this should be enough for now.

[mkhludnev]

You know what.. I feel terribly sorry.
It might be it's a question of "parameters":{"max_tokens":4000}.
I'm trying to figure it out how to send it with https://github.com/triton-inference-server/vllm_backend/blob/main/samples/client.py

[mkhludnev]

Pardon triton-inference-server/vllm_backend#34

[oandreeva-nv]

Thanks for providing solution!