(fish-speech v1.5) bigger real time factor on short texts

[twocode]

Self Checks

• This template is only for bug reports. For questions, please visit Discussions.
• I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. English 中文 日本語 Portuguese (Brazil)
• I have searched for existing issues, including closed ones. Search issues
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template and fill in all required fields.

Cloud or Self Hosted

Self Hosted (Docker)

Environment Details

Tesla T4

Steps to Reproduce

1. Server starts in Docker as

    "python", "-m", "tools.api_server", \
    "--listen", "0.0.0.0:8080", \
    "--llama-checkpoint-path", "checkpoints/fish-speech-1.5", \
    "--decoder-checkpoint-path", "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth", \
    "--decoder-config-name", "firefly_gan_vq", \
    "--compile", \
    "--half" \

2. Upload reference audios

3. Client makes request specifying reference_id.

✔️ Expected Behavior

I hope to see a tts latency similar to fish-speech v1.4 at around 500ms for a non-referenced audio generation from a short text with only a few characters.

❌ Actual Behavior

The real time factor for short text chunks is bigger than longer texts.

{"level":"info","timestamp":"2024-12-12T17:59:15.231Z","caller":"mando/engine.go:366","msg":"TTS performance","pid":1,"audio_duration_ms":1646,"latency_ms":1913,"text":"好的，"}
{"level":"info","timestamp":"2024-12-12T17:59:20.715Z","caller":"mando/engine.go:366","msg":"TTS performance","pid":1,"audio_duration_ms":6009,"latency_ms":2822,"text":"让我们开始另一个故事！\n\n在一个神秘的王国里，住着一位勇敢的小骑士，"}
{"level":"info","timestamp":"2024-12-12T17:59:25.770Z","caller":"mando/engine.go:366","msg":"TTS performance","pid":1,"audio_duration_ms":13428,"latency_ms":4964,"text":"名叫亚瑟。亚瑟非常渴望成为一名伟大的骑士，保护他的村庄和朋友们。有一天，村庄里传来了一个坏消息：一条凶猛的龙来到了附近的山上，"}

In my application log above, audio_duration_ms is the length of the audio and latency_ms is the tts duration.
The shortest text here had a real-time-factor < 1.

[twocode]

I tested without inference audio and the issue remains.

[johnwick123f]

Yep same exact thing, 500ms latency with very short text on a T4 gpu with api. Slower then realtime(4sec to gen 3sec of audio). Only on longer text, it is much faster then realtime.

[leng-yue]

T4 is a very old GPU, it may not work well when doing the prefilling.

[twocode]

Thanks. What made me wonder was 1.4 was running quite well on T4. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Leng Yue ***@***.***> Sent: Friday, December 20, 2024 7:45:11 PM To: fishaudio/fish-speech ***@***.***> Cc: Xiangyu Hu ***@***.***>; Author ***@***.***> Subject: Re: [fishaudio/fish-speech] (fish-speech v1.5) bigger real time factor on short texts (Issue #744) T4 is a very old GPU, it may not work well when doing the prefilling. — Reply to this email directly, view it on GitHub<#744 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA4WKREAXVQUMSBEI6F7AD32GP7MPAVCNFSM6AAAAABTQ3H7EOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWHA2DCNJZGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[leng-yue]

This is quite interesting, as you see, we didn't change model architecture a lot during the update. One thing I am consider is that the embedding table is much larger now, it may then cause some memory bound?

[twocode]

Interesting. Thanks for pointing out the variance in embedding table. I'll find a time to validate. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Leng Yue ***@***.***> Sent: Saturday, December 21, 2024 11:35:34 PM To: fishaudio/fish-speech ***@***.***> Cc: Xiangyu Hu ***@***.***>; Author ***@***.***> Subject: Re: [fishaudio/fish-speech] (fish-speech v1.5) bigger real time factor on short texts (Issue #744) This is quite interesting, as you see, we didn't change model architecture a lot during the update. One thing I am consider is that the embedding table is much larger now, it may then cause some memory bound? — Reply to this email directly, view it on GitHub<#744 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA4WKRBZ3B3W4QQGEAHGQJT2GWDENAVCNFSM6AAAAABTQ3H7EOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGE2TGNJQHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[twocode]

Hi @leng-yue can you let know what are the sizes of embedding table for both 1.4 and 1.5? Would a g5.xlarge (with 24GiB GPU memory) be sufficient?
Thank you.

[leng-yue]

Even 6G is enough, vocab size 32k -> 100k.