hyper fails on certain URLs?

[ldgabbay]

I have been trying to retrieve some files with hyper but for some reason they seem to hang and fail, but the same request with curl or actix-web work just fine. This seems to be a failure at the hyper level, but I am not sure how to debug the inner workings of hyper, but maybe someone can enlighten me?

As a demonstration, one of the URLs that fail only on hyper is:
https://api2.bybit.com/linear/funding-rate/list?symbol=MATICUSDT&export=true
This URL takes ~6 seconds to generate the response, but it does work reliably. hyper waits longer than this and fail rather than succeed.

The code I'm using is as simple as:

use hyper_tls::HttpsConnector;
use hyper::Client;
use hyper::Uri;

...
let https = HttpsConnector::new();
let client = Client::builder().build::<_, hyper::Body>(https);    
let response = client.get(Uri::from_str(url.as_str())?).await?;

and the program fails with:

Error: hyper::Error(IncompleteMessage)

Again, curl and actix-web work just fine. For now, I've switched to actix-web, but I would much prefer to be based on hyper. Please let me know if you have any suggestions, and if you can point me to ways to debug the issue. Thanks!

[seanmonstar]

Huh, when I try this URL out with hyper/reqwest, it never finishes for me, but curl works fine... I wonder, is hyper somehow missing a "read" notification from epoll? I'll have to add more logging and investigate.

[seanmonstar]

Ah, so this is interesting. I filled up a bunch of logging, and it didn't seem like we were ever getting a read notification. Then I compared closely the headers that curl was sending, and the only difference was a user-agent header. So, I added that to the hyper example, and suddenly the server sends a response!

So, I'm going to suggest that it's a server issue that is perhaps checking for that header, and it doesn't properly handle the lack of that header.

[ldgabbay]

thank you so much for debugging this... i lost hours over it. you are very much appreciated!!!