QVQ Prefill stage slow

[WuNein]

Using int4 QVQ 72b model. https://huggingface.co/kosbu/QVQ-72B-Preview-AWQ
basic config: 4 2080ti 22G tp=4

python3 -m sglang.launch_server --model-path /root/model/QVQ-72B-Preview-AWQ --host 0.0.0.0 --port 30000 --tp 4 --mem-fraction-static 0.7

As you may see, the prefilling stage take 20s.

What i can do to optimize the speed?
Or do i have option to turn off prefilling, when performing only one request?

[zhaochenyang20]

Well. This is quite strange. What GPUs and prompts are you using?

[zhaochenyang20]

Okay. For 2080 that's normal. 😂 We may do not have better plan for this. Sorry.

[WuNein]

Okay. For 2080 that's normal. 😂 We may do not have better plan for this. Sorry.

One more thing, it is not quick on duo a6000s (w/ nvlink)
Prefilling Batch takes a lot of time.

[zhaochenyang20]

@WuNein This could also make sense. A6000 is not a well-performed GPU indeed. And I think there is something related:

#2488

The bad thing is that we do not have these general devices.