We must catchup 11.54 seconds! even on thicc server

[bombelman]

Hi all,
I'm using LiquidSoap on AzuraCast 0.20.4 (Nov 23, 2024) and have been suggested to post my liquidsoap related question here.

I have a 4 core server on a pretty good host and max usage is well below 100%.
However I still get glitches/disconnects when streaming from 1 or more stations.

The log is filled with:
2025/01/08 02:37:55 [clock.main:2] We must catchup 0.04 seconds!
2025/01/08 02:38:30 [clock.main:2] We must catchup 0.18 seconds!
2025/01/08 02:38:36 [clock.main:2] We must catchup 0.06 seconds!
2025/01/08 02:39:48 [clock.main:2] We must catchup 0.20 seconds!
2025/01/08 02:40:04 [clock.main:2] We must catchup 0.03 seconds!
2025/01/08 02:40:09 [clock.main:2] We must catchup 0.13 seconds!
2025/01/08 02:40:32 [clock.main:2] We must catchup 0.16 seconds!
2025/01/08 02:40:53 [clock.main:2] We must catchup 0.19 seconds!
2025/01/08 02:41:04 [clock.main:2] We must catchup 0.32 seconds!
2025/01/08 02:41:08 [clock.main:2] We must catchup 0.30 seconds!
2025/01/08 02:42:12 [clock.main:2] We must catchup 0.14 seconds!
2025/01/08 02:43:49 [clock.main:2] We must catchup 0.20 seconds!

I'm not using any custom liquidsoap configs, only the default AzuraCast settings.
I have 3350 tracks and about 10 jingles.

The server load is well below the max, so I'm guessing that is not the issue,
even when it would look that way (?).

Who can help me troubleshoot this further?

Original post here: AzuraCast/AzuraCast#7534

Much appreciated!

[gAlleb]

For 4 stations you have a HUGE CPU consumption, it shouldn't be more than 0.5-0.7 average out of 4 cores or 10-20% total on a fair VPS. Probably a lode is increased with autocue

Do you use external audio files or from the host itself?

Also do you use HLS? It's enough to have an HLS with one bitrate per station cause it actually doesn't make sense to have 3 very slightly different bitrates.

Do you have any steal/wait on the machine?

[bombelman]

all files are local, mp3.
not using HLS.
this server has no wait/steal.
even with autocue load increase, it's around 75% of all cores, nowhere near it's max...

[gAlleb]

4 stations of AzuraCast just can't consume 75% of 4 cores. What exactly consumes this cpu? Can you check with top or htop?

[bombelman]

For clarification:
In this Azuracast instance there are 4 stations running with autodj in liquidsoap.
Another 3 have the autodj disabled and are pure live-only.
(While this can be done with DNAS only, I understand it is still going through a liquidsoap process.
not sure if this affects the overall load that much)

Attached are some usage screenshots from the AzuraCast interface and htop.

I own and operate this server, @gAlleb let me know if you need any other info to troubleshoot this.
IF I need more server resources, then that will have to be done.
However if it is software related issue, I would like to see if and how it can be fixed.

[vitoyucepi]

Are you experiencing the problem on all stations or just two with stereotool?
May I ask you to disable stereotool on all stations and see if the problem persists?

[bombelman]

Are you experiencing the problem on all stations or just two with stereotool? May I ask you to disable stereotool on all stations and see if the problem persists?

okay I'll try

[bombelman]

I disabled stereotool on the 2 autodj stations using it.
Overall load decreased around 10% and issues seemed to have stopped from happening,
however this was also the case after restarting the station,
so I did re-enable stereotool on 1 station (main) and so far all stations are running fine.
I did see the error pop up a few times, and am still monitoring it.
The other station is running on master_me.

As it stands, my conclusion is, stereotool is adding enough load to be causing some glitches, at lease when used with 2 or even 1 station.

I do want to keep using stereotool or any (premium) processing.

My questions:

• is there any optimising possible with stereotool, resulting in less cpu load?
• is there any optimising possible with master_me as an alternative to stereotool?
• Any alternatives to stereotool with less cpu load?

Thanks all.

[vitoyucepi]

Are you experiencing the problem on all stations or just two with stereotool?

[bombelman]

Are you experiencing the problem on all stations or just two with stereotool?

Good Question,
not sure if the other stations were affected, those were running on the server but were not live/production stations yet.

The other 2 are.

[bombelman]

I'm on Hetzner CX32, 4 VCPU (cores) AMD.
Live graph is barely above 300% (out of 400%)