Support http over unix domain sockets #392

ducaale · 2024-12-29T16:57:56Z

Adds a --unix-socket option similar to cURL for binding to a Unix domain socket. When used, all requests, including redirects, will go through the specified Unix socket.

Resources used:

Resolves #230

ducaale · 2024-12-29T17:01:51Z

src/unix_socket.rs

+
+    pub fn execute(&self, request: Request) -> Result<Response> {
+        self.rt.block_on(async {
+            // TODO: Support named pipes by replacing tokio::net::UnixStream::connect(..) with:


Besides Docker for Window, I couldn't find any other examples of HTTP over named pipes, so I'm not sure if it's worth supporting at this time.

ducaale · 2024-12-29T17:09:43Z

src/main.rs

+            #[cfg(target_family = "unix")]
+            if let Some(unix_socket) = args.unix_socket {


There is a fork of hyperlocal that adds support for windows but it might be better to wait until it is merged upstream

ducaale · 2024-12-30T09:27:16Z

src/unix_socket.rs

+                }
+            });
+
+            // TODO: don't ignore value from --timeout option


Regardless of how this is handled, I need to ensure the correct exit code is used for timeout errors

Note to self: refer to impl<B> hyper::body::Body for TotalTimeoutBody<B> for how to implement timeout for body.

Also, we should consider moving to a combination of connect_timeout and read_timeout which is closer to how it is handled in python requests/HTTPie. See https://requests.readthedocs.io/en/stable/user/advanced/#timeouts

Hmm, it seems read_timeout is not yet implemented for reqwest::blocking::Client

blyxxyz

Nice! I haven't looked at everything yet

src/main.rs

blyxxyz · 2024-12-29T17:18:31Z

src/cli.rs

@@ -347,6 +347,12 @@ Example: --print=Hb"
    #[clap(short = '6', long)]
    pub ipv6: bool,

+    /// Connect using a Unix domain socket.
+    ///
+    /// Example: --unix_socket=/var/run/temp.sock


With this API you still have to make up a hostname for the URL, like curl, and unlike got and the HTTPie plugin.

AFAICT the typical use cases are for local services and not for proxies, meaning that the hostname is superfluous. But it also doesn't look like we're going to get a URL-based convention for this any time soon. And splitting it off into an option prevents redirect attacks. (Not sure how it should interact with sessions though... we need to look carefully at other areas of interference.)

A full command with the : shorthand might help users, e.g.

xh :/index.html --unix-socket=/var/run/temp.sock

There is a good chance that unix domain sockets will be treated as a proxy in reqwest. This has the advantage that anything relying on hostname like cookies and Host header will keep working. Also, see denoland/deno#8821 (comment)

However, I still think we can support xh http://unix:/var/run/docker.sock:/containers/json by treating it as a syntactic sugar for xh :/containers/json --unix-socket=/var/run/docker.sock.

We can always add syntactic sugar later. Right now there doesn't seem to be any consensus about the URL format.

If it ever gets to the point where browsers start supporting UDS URLs (as a non-experimental feature) then we should follow suit.

blyxxyz · 2024-12-31T13:52:35Z

src/unix_socket.rs

+        let starting_time = Instant::now();
+        let mut response = self.execute(request)?;
+        response.extensions_mut().insert(ResponseMeta {
+            request_duration: starting_time.elapsed(),
+            content_download_duration: None,
+        });


Don't we skip any subsequent middleware this way? I don't understand this yet but it might be the wrong layer to do it.

It is the last layer, but returning early and short-circuiting middlewares in this way might be confusing, not to mention the duplicated logic. Let me see if I can switch between UnixSocket and reqwest client inside ClientWithMiddleware.

the former is not yet supported in blocking client

blyxxyz

Looks good so far! (I haven't understood everything yet.)

Do you have tips for manually testing this?

blyxxyz · 2025-01-11T18:15:12Z

src/cli.rs

+    #[clap(
+        long,
+        value_name = "FILE",
+        conflicts_with_all=["proxy", "verify", "cert", "cert_key", "ssl", "resolve", "interface", "ipv4", "ipv6", "https", "http_version"]


Many of these might plausibly appear in an alias or config file, like --https, --ssl.

If someone passes --proxy they probably made a mistake but I think the list could be shorter. Maybe a guideline could be whether the option is compatible with a normal request to http://127.0.0.1?

Were you thinking of silently ignoring some of the flags or maybe printing a warning?

--https for example doesn't force HTTPS, it just changes the default scheme of the URL. So maybe the user gets an error if they pass :/index.html because that gets converted to https://localhost/index.html and the socket doesn't support TLS, but if they use --https with an explicit http://localhost/index.html then that should be perfectly fine, just like in xh --https http://127.0.0.1 the --https flag is useless but harmless.

blyxxyz · 2025-01-21T19:10:31Z

src/unix_socket.rs

+        tokio::task::spawn(async move {
+            if let Err(err) = conn.await {
+                log::error!("Connection failed: {:?}", err);
+            }
+        });


Is it OK to merely log this? I assume it leads to a harder error somewhere down the line and it's just hard to get it out of the task?

blyxxyz · 2025-01-21T19:23:39Z

src/unix_socket.rs

+            let mut sender = with_timeout(self.connect(), self.timeout).await??;
+            let response = with_timeout(sender.send_request(http_request), self.timeout).await??;


Does this mean we have a separate timeout for each stage?

Will there be a timeout for e.g. a large file upload?

blyxxyz · 2025-01-21T19:30:43Z

tests/server/mod.rs

+    let file_name: String = rand::thread_rng()
+        .sample_iter(&rand::distributions::Alphanumeric)
+        .take(10)
+        .map(char::from)
+        .collect();


Looks like tempfile can be used for this: https://docs.rs/tempfile/latest/tempfile/struct.Builder.html#examples-11

That way we also get cleanup

ducaale · 2025-01-21T22:48:34Z

Do you have tips for manually testing this?

I have been using Docker API but you can also use Bun which natively supports unix domain sockets

// bun index.js

async function* hello(times) {
  for (let i = 0; i < times; i++) {
    yield `data: hello ${i}\n\n`;
    await Bun.sleep(1000);
  }
}

const server = Bun.serve({
  unix: "/tmp/my-socket.sock",
  fetch(req){
    return new Response(hello(10), {
      headers: {
        "Content-Type": "text/event-stream",
        Connection: "keep-alive",
      }
    });
  }
});

console.log(`Listening on unix:///tmp/my-socket.sock`);

I'm noticing some decoding errors so I will have to investigate that

$ cargo r -- -v : --unix-socket=/tmp/my-socket.sock

...lines omitted

data: hello 7

data: hello 8

xh: error: error decoding response body

Caused by:
    0: request or response body error
    1: error reading a body from connection
    2: unexpected EOF during chunk size line

ducaale added 4 commits December 29, 2024 17:44

reduce lifetime of printer closure passed to client middleware

8ba7774

support http over unix domain sockets

aa8fbf5

implement test server for http_unix

fe56317

add initial tests for http_unix

531dba9

ducaale commented Dec 29, 2024

View reviewed changes

fix clippy warnings

bf4b1b8

ducaale commented Dec 29, 2024

View reviewed changes

disable http_unix tests in windows

80242c2

ducaale marked this pull request as ready for review December 29, 2024 22:18

ducaale marked this pull request as draft December 29, 2024 22:19

ducaale added 3 commits December 30, 2024 10:17

add middleware for managing cookies

8cee2fe

add test for cookies

652e3df

avoid mocking host header

b6d59d9

ducaale commented Dec 30, 2024

View reviewed changes

blyxxyz reviewed Dec 30, 2024

View reviewed changes

ducaale added 6 commits December 30, 2024 14:31

use shortened version of cfg unix check

cb8d396

throw an error if unix-socket used in unsupported os

c7fb645

provide complete example for unix-socket usage

e0791b3

fix missing import

f09a191

check that host header is passed

67730e1

Merge branch 'master' into http-over-unix-socket

de52108

blyxxyz reviewed Dec 31, 2024

View reviewed changes

ducaale added 5 commits December 31, 2024 20:38

store unix_client in ClientWithMiddleware

0a80407

Merge branch 'master' into http-over-unix-socket

3d887f0

Merge branch 'master' into http-over-unix-socket

2eb6367

warn or error if unix-socket used with unsupported option

31c4420

disable failing badssl.com tests

8bcbb0d

ducaale marked this pull request as ready for review January 7, 2025 10:46

ducaale added 2 commits January 7, 2025 23:46

implement read timeout for unix_socket requests

d601946

implement connect timeout for unix_socket requests

c84e8ef

ducaale added 3 commits January 11, 2025 10:25

switch from read timeout to total timeout

e362e7f

the former is not yet supported in blocking client

revert disabling badssl tests

92df488

Merge branch 'master' into http-over-unix-socket

bd8dbdd

ducaale requested a review from blyxxyz January 11, 2025 18:11

Merge branch 'master' into http-over-unix-socket

7db9b38

blyxxyz reviewed Jan 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support http over unix domain sockets #392

Support http over unix domain sockets #392

ducaale commented Dec 29, 2024 •

edited

Loading

ducaale Dec 29, 2024

ducaale Dec 29, 2024

ducaale Dec 30, 2024

ducaale Jan 7, 2025 •

edited

Loading

ducaale Jan 8, 2025

blyxxyz left a comment

blyxxyz Dec 29, 2024

ducaale Dec 31, 2024

blyxxyz Dec 31, 2024

blyxxyz Dec 31, 2024

ducaale Dec 31, 2024

blyxxyz left a comment

blyxxyz Jan 11, 2025

ducaale Jan 21, 2025

blyxxyz Jan 22, 2025

blyxxyz Jan 21, 2025

blyxxyz Jan 21, 2025

blyxxyz Jan 21, 2025

ducaale commented Jan 21, 2025

		#[cfg(target_family = "unix")]
		if let Some(unix_socket) = args.unix_socket {

		let mut sender = with_timeout(self.connect(), self.timeout).await??;
		let response = with_timeout(sender.send_request(http_request), self.timeout).await??;

Support http over unix domain sockets #392

Are you sure you want to change the base?

Support http over unix domain sockets #392

Conversation

ducaale commented Dec 29, 2024 • edited Loading

Resources used:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ducaale Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blyxxyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blyxxyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ducaale commented Jan 21, 2025

ducaale commented Dec 29, 2024 •

edited

Loading

ducaale Jan 7, 2025 •

edited

Loading