Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support http over unix domain sockets #392

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8ba7774
reduce lifetime of printer closure passed to client middleware
ducaale Dec 29, 2024
aa8fbf5
support http over unix domain sockets
ducaale Dec 29, 2024
fe56317
implement test server for http_unix
ducaale Dec 29, 2024
531dba9
add initial tests for http_unix
ducaale Dec 29, 2024
bf4b1b8
fix clippy warnings
ducaale Dec 29, 2024
80242c2
disable http_unix tests in windows
ducaale Dec 29, 2024
8cee2fe
add middleware for managing cookies
ducaale Dec 30, 2024
652e3df
add test for cookies
ducaale Dec 30, 2024
b6d59d9
avoid mocking host header
ducaale Dec 30, 2024
cb8d396
use shortened version of cfg unix check
ducaale Dec 30, 2024
c7fb645
throw an error if unix-socket used in unsupported os
ducaale Dec 30, 2024
e0791b3
provide complete example for unix-socket usage
ducaale Dec 30, 2024
f09a191
fix missing import
ducaale Dec 30, 2024
67730e1
check that host header is passed
ducaale Dec 30, 2024
de52108
Merge branch 'master' into http-over-unix-socket
ducaale Dec 30, 2024
0a80407
store unix_client in ClientWithMiddleware
ducaale Dec 31, 2024
3d887f0
Merge branch 'master' into http-over-unix-socket
ducaale Jan 2, 2025
2eb6367
Merge branch 'master' into http-over-unix-socket
ducaale Jan 7, 2025
31c4420
warn or error if unix-socket used with unsupported option
ducaale Jan 7, 2025
8bcbb0d
disable failing badssl.com tests
ducaale Jan 7, 2025
d601946
implement read timeout for unix_socket requests
ducaale Jan 7, 2025
c84e8ef
implement connect timeout for unix_socket requests
ducaale Jan 8, 2025
e362e7f
switch from read timeout to total timeout
ducaale Jan 11, 2025
92df488
revert disabling badssl tests
ducaale Jan 11, 2025
bd8dbdd
Merge branch 'master' into http-over-unix-socket
ducaale Jan 11, 2025
7db9b38
Merge branch 'master' into http-over-unix-socket
ducaale Jan 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,11 @@ dirs = "5.0"
encoding_rs = "0.8.28"
encoding_rs_io = "0.1.7"
flate2 = "1.0.22"
futures-core = { version = "0.3.28", default-features = false }
http = "1"
# Add "tracing" feature to hyper once it stabilizes
hyper = { version = "1.2", default-features = false }
hyper-util = { version = "0.1", features = ["tokio"] }
indicatif = "0.17"
jsonxf = "1.1.0"
memchr = "2.4.1"
Expand All @@ -38,6 +41,7 @@ mime_guess = "2.0"
once_cell = "1.8.0"
os_display = "0.1.3"
pem = "3.0"
pin-project-lite = "0.2"
regex-lite = "0.1.5"
roff = "0.2.1"
rpassword = "7.2.0"
Expand All @@ -48,6 +52,7 @@ serde_urlencoded = "0.7.0"
supports-hyperlinks = "3.0.0"
termcolor = "1.1.2"
time = "0.3.16"
tokio = { version = "1", features = ["rt-multi-thread"] }
unicode-width = "0.1.9"
url = "2.2.2"
ruzstd = { version = "0.7", default-features = false, features = ["std"]}
Expand Down
12 changes: 11 additions & 1 deletion src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,16 @@ Example: --print=Hb"
#[clap(short = '6', long)]
pub ipv6: bool,

/// Connect using a Unix domain socket.
///
/// Example: xh :/index.html --unix-socket=/var/run/temp.sock
#[clap(
long,
value_name = "FILE",
conflicts_with_all=["proxy", "verify", "cert", "cert_key", "ssl", "resolve", "interface", "ipv4", "ipv6", "https", "http_version"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many of these might plausibly appear in an alias or config file, like --https, --ssl.

If someone passes --proxy they probably made a mistake but I think the list could be shorter. Maybe a guideline could be whether the option is compatible with a normal request to http://127.0.0.1?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you thinking of silently ignoring some of the flags or maybe printing a warning?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--https for example doesn't force HTTPS, it just changes the default scheme of the URL. So maybe the user gets an error if they pass :/index.html because that gets converted to https://localhost/index.html and the socket doesn't support TLS, but if they use --https with an explicit http://localhost/index.html then that should be perfectly fine, just like in xh --https http://127.0.0.1 the --https flag is useless but harmless.

)]
pub unix_socket: Option<PathBuf>,

/// Do not attempt to read stdin.
///
/// This disables the default behaviour of reading the request body from stdin
Expand Down Expand Up @@ -1011,7 +1021,7 @@ impl FromStr for Print {
}
}

#[derive(Debug, Clone)]
#[derive(Debug, Clone, Copy)]
pub struct Timeout(Duration);

impl Timeout {
Expand Down
44 changes: 44 additions & 0 deletions src/cookie.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
use std::sync::Arc;

use anyhow::Result;
use reqwest::{
blocking::{Request, Response},
cookie::CookieStore,
header,
};

use crate::middleware::{Context, Middleware};

pub struct CookieMiddleware<T>(Arc<T>);

impl<T> CookieMiddleware<T> {
pub fn new(cookie_jar: Arc<T>) -> Self {
CookieMiddleware(cookie_jar)
}
}

impl<T: CookieStore> Middleware for CookieMiddleware<T> {
fn handle(&mut self, mut ctx: Context, mut request: Request) -> Result<Response> {
let url = request.url().clone();

if let Some(header) = self.0.cookies(&url) {
request
.headers_mut()
.entry(header::COOKIE)
.or_insert(header);
}

let response = self.next(&mut ctx, request)?;

let mut cookies = response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.peekable();
if cookies.peek().is_some() {
self.0.set_cookies(&mut cookies, &url);
}

Ok(response)
}
}
88 changes: 56 additions & 32 deletions src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
mod auth;
mod buffer;
mod cli;
mod cookie;
mod decoder;
mod download;
mod formatting;
Expand All @@ -15,6 +16,8 @@ mod redirect;
mod request_items;
mod session;
mod to_curl;
#[cfg(unix)]
mod unix_socket;
mod utils;

use std::env;
Expand All @@ -27,6 +30,7 @@ use std::str::FromStr;
use std::sync::Arc;

use anyhow::{anyhow, Context, Result};
use cookie::CookieMiddleware;
use cookie_store::{CookieStore, RawCookie};
use redirect::RedirectFollower;
use reqwest::blocking::Client;
Expand All @@ -35,7 +39,6 @@ use reqwest::header::{
};
use reqwest::tls;
use url::Host;
use utils::reason_phrase;

use crate::auth::{Auth, DigestAuthMiddleware};
use crate::buffer::Buffer;
Expand All @@ -45,7 +48,7 @@ use crate::middleware::ClientWithMiddleware;
use crate::printer::Printer;
use crate::request_items::{Body, FORM_CONTENT_TYPE, JSON_ACCEPT, JSON_CONTENT_TYPE};
use crate::session::Session;
use crate::utils::{test_mode, test_pretend_term, url_with_query};
use crate::utils::{reason_phrase, test_mode, test_pretend_term, url_with_query};

#[cfg(not(any(feature = "native-tls", feature = "rustls")))]
compile_error!("Either native-tls or rustls feature must be enabled!");
Expand Down Expand Up @@ -86,6 +89,15 @@ fn main() {
eprintln!();
eprintln!("Try running without the --native-tls flag.");
}
if msg.starts_with("deadline has elapsed") {
process::exit(2);
}
#[cfg(unix)]
{
if err.downcast_ref::<unix_socket::TimeoutError>().is_some() {
process::exit(2);
}
}
if let Some(err) = err.downcast_ref::<reqwest::Error>() {
if err.is_timeout() {
process::exit(2);
Expand Down Expand Up @@ -286,9 +298,6 @@ fn run(args: Cli) -> Result<i32> {
None => client,
};

let cookie_jar = Arc::new(reqwest_cookie_store::CookieStoreMutex::default());
client = client.cookie_provider(cookie_jar.clone());

client = match (args.ipv4, args.ipv6) {
(true, false) => client.local_address(IpAddr::from(Ipv4Addr::UNSPECIFIED)),
(false, true) => client.local_address(IpAddr::from(Ipv6Addr::UNSPECIFIED)),
Expand Down Expand Up @@ -340,6 +349,8 @@ fn run(args: Cli) -> Result<i32> {
log::trace!("{client:#?}");
let client = client.build()?;

let cookie_jar = Arc::new(reqwest_cookie_store::CookieStoreMutex::default());

let mut session = match &args.session {
Some(name_or_path) => Some(
Session::load_session(url.clone(), name_or_path.clone(), args.is_session_read_only)
Expand Down Expand Up @@ -561,42 +572,55 @@ fn run(args: Cli) -> Result<i32> {
printer.print_request_body(&mut request)?;
}

let mut client = ClientWithMiddleware::new(client);

if !args.offline {
let mut response = {
let history_print = args.history_print.unwrap_or(print);
let mut client = ClientWithMiddleware::new(&client);
if args.all {
client = client.with_printer(|prev_response, next_request| {
if history_print.response_headers {
printer.print_response_headers(prev_response)?;
}
if history_print.response_body {
printer.print_response_body(
prev_response,
response_charset,
response_mime,
)?;
printer.print_separator()?;
}
if history_print.response_meta {
printer.print_response_meta(prev_response)?;
}
if history_print.request_headers {
printer.print_request_headers(next_request, &*cookie_jar)?;
}
if history_print.request_body {
printer.print_request_body(next_request)?;
}
Ok(())
});
}
if args.follow {
client = client.with(RedirectFollower::new(args.max_redirects.unwrap_or(10)));
}
if let Some(Auth::Digest(username, password)) = &auth {
client = client.with(DigestAuthMiddleware::new(username, password));
}
client.execute(request)?
client = client.with(CookieMiddleware::new(cookie_jar.clone()));
if let Some(socket_path) = args.unix_socket {
#[cfg(not(unix))]
{
return Err(anyhow::anyhow!(
"HTTP over Unix domain sockets is not supported on this platform"
));
}
#[cfg(unix)]
{
client = client.with_unix_socket(
socket_path,
args.timeout.and_then(|t| t.as_duration()),
)?;
}
}
client.execute(request, |prev_response, next_request| {
if !args.all {
return Ok(());
}
if history_print.response_headers {
printer.print_response_headers(prev_response)?;
}
if history_print.response_body {
printer.print_response_body(prev_response, response_charset, response_mime)?;
printer.print_separator()?;
}
if history_print.response_meta {
printer.print_response_meta(prev_response)?;
}
if history_print.request_headers {
printer.print_request_headers(next_request, &*cookie_jar)?;
}
if history_print.request_body {
printer.print_request_body(next_request)?;
}
Ok(())
})?
};

let status = response.status();
Expand Down
Loading
Loading