Refreshed token doesn't contain capabilities for all channels

[sacOO7]

• Customer has reported an issue where refreshed token doesn't contain all channels, causing existing channels to reconnect again.

The current issue is that when Ably hits the token expiry time, it makes multiple token refresh calls. Although the current token has all the necessary capabilities, the server sometimes responds with a token containing only the channel_name provided in the payload. This causes the token to be marked as invalid, which then triggers another refresh cycle to obtain a token with proper capabilities.

• Link to internal conversation => https://ably-real-time.slack.com/archives/CURL4U2FP/p1733135454672739

┆Issue is synchronized with this Jira Bug by Unito

[sacOO7]

A temporary solution for now would be to set maximum ABLY_TOKEN_EXPIRY in .env file.

ABLY_TOKEN_EXPIRY= 86400  // 1 day ( 24 * 60 * 60 ) seconds

[sacOO7]

• Generally, ably requests the client to renew the token 30 seconds before expiry as per RTN22
• Link to internal conversation -> https://ably-real-time.slack.com/archives/C8SPU4589/p1724236742837859

[graphem]

Yeah seems I am experiecing this and this is killing my server, as all the clients seems to make tons of request again.

[graphem]

@sacOO7 Any news on merging this and doing a release?

[sacOO7]

Currently, this is kept on hold and will be picked up soon, can you try increasing TOKEN_EXPIRY for the time being to say 6 hrs, wdyt

[graphem]

Ok thanks! Seems the token expiry is not making a difference so maybe my issues is something different.

[sacOO7]

@graphem okay, make sure to report whatever issue you are facing 👍
Thanks !

[graphem]

@sacOO7 Essentially I am using Laravel Echo with Ably and I am using the listen method on channels, but lately I have observed a spike in traffic on the website and after investiguation it comes for tons of call to /broadcasting/auth which is the endpoint to get the Ably token. But it is getting out of control, like I am getting 100s of calls per seconds from single client, like there are stuck in a loop, so I was curious if this issue was related with token expiration and having client on a loop getting the token over and over. This right now puts a lot of pressure on our system, it looks almost like a DDOS but with just normal user traffic since all our route are auth protected so only login user can access the web app.

[sacOO7]

@graphem can you log userId or ip address of users requesting tokens?
Also, if you dont mind, how many users use your app at a time? and how many private/presence channels they use on average?

[graphem]

I am not using the presence channels. I am using very few channels basically the app might have up to 30 different private channel and might have 2-3 public channels. And we are running like 5 apps per server. I am going to try to log the entries to get more details but I was simply observing the access logs and it was just insane traffic. We have around 2000 users concurrent per servers right now. We use decent baremetal servers which can take the traffic but this is not sustainable right now as we grow.

[sacOO7]

Okay, it will be great if you can analyse logs and check which requests users are sending consistently?
I am confident you must have battle tested app on the local. It will also be great to analyse if requests are coming from mobile browsers that are causing any such issues? I think you can also use available AI tools to analyse logs.

[graphem]

Yes it is a tough one cause we are not able to reproduce locally our in our staging environment. I am going to analyse the log further today and see if I can find more info

[sacOO7]

@graphem thanks. We will be eagerly waiting for your analysis 👍

[graphem]

Is there a way to send private files as I don't want to share sensitive info here

[sacOO7]

Sure, you can open ticket at https://ably.com/support and share information there 👍

[graphem]

Ok cool I already have a ticket, I will gather more info

[sacOO7]

Cool 👍

[graphem]

Actually seems the token expiry seems to make a difference today, so it might be that issue

[sacOO7]

Okay ... I genuinely think it should.
Just make sure you set it correctly ABLY_TOKEN_EXPIRY in .env file as per README file

[graphem]

Ok I changed the settings on all my app that seems to do the trick, will have confirmation tomorrow when the app is more busy.

[sacOO7]

Great 👍
you can also try increasing the timeout value to say 12 or 24 hrs. Let us know about your findings.

[graphem]

@sacOO7 What is the setting for the timeout?

[sacOO7]

I didn't get your question? You mean token expiry or something else

[graphem]

Great 👍 you can also try increasing the timeout value to say 12 or 24 hrs. Let us know about your findings.

You mentioned a timeout value here

[sacOO7]

Ohh I meant this -> #55 (comment)
I meant, client token will expire/timeout after given expiry : )

[sacOO7]

Btw, were you able to achieve desired behaviour?

[graphem]

Oh yeah sorry I increased it, our server load is 200% better this morning, this seems to have done the trick

[sacOO7]

Good to hear that 👍
Feel free to raise issue if needed : )

[graphem]

Ok thanks! Yeah it is much better looking forward for the merge here. Thanks!