How to control request size within a batch?

[cringelord000222]

Hi all,

I've tried --max-batch-size command however it doesn't work as I've expected, I thought it was supposed to limit tgi_batch_current_size. I'd like to control the queue size & how many requests per inference batch. Can someone please clarify on this?

These are my launch commands and version:

# from docker compose file

image: ghcr.io/huggingface/text-generation-inference:2.0.4
command: --model-id /models/meta-llama/Meta-Llama-3-70B-Instruct --quantize bitsandbytes-nf4 --num-shard 2 --max-input-tokens 5000 --max-total-tokens 6000 --max-batch-size 10

My situation:

1. Concurrently sent 40 requests to TGI server, --max-batch-size 10
tgi_batch_current_size goes over 10 after a short while

# from /metrics

tgi_request_count 40
tgi_request_success 15
tgi_batch_inference_count{method="prefill"} 15
tgi_batch_inference_count{method="decode"} 559
tgi_batch_concat{reason="backpressure"} 7
tgi_batch_concat{reason="wait_exceeded"} 7
tgi_batch_inference_success{method="prefill"} 15
tgi_batch_inference_success{method="decode"} 558
tgi_batch_current_max_tokens 65664
tgi_queue_size 13
tgi_batch_current_size 12

Expected situation:

1. Concurrently sent 40 requests to TGI server, --max-batch-size 10
tgi_batch_current_size stays at 10, and the other 30 request will stay in tgi_queue_size until its done.

TLDR: tgi_batch_current_size doesn't align with --max-batch-size.

Thanks in advance.

[haroldship]

I'm also curious about this and also the relationship between max-batch-size and max-concurrent-requests

[fawern]

The --max-batch-size limits how many requests are grouped together, but TGi adjusts dynamically to handle more requests under heavy load (e.g due to long tokens or backpressure). This can cause the batch size to exceed your set limit.

How to fix it:

Reduce --max-total-tokens to limit the total tokens in a batch.
Limit the queue size using --max-queued-requests.
Make sure your input token sizes are not too large.
Monitor server metrics and adjust these values to better control batching.