Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce new SSLContext API & escalate deprecations. #3319

Merged
merged 23 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
a18f796
Drop unneccessary 'URL.raw' property (#3317)
tomchristie Sep 24, 2024
34e1110
Remove deprecated 'app' parameter. (#3315)
tomchristie Sep 24, 2024
dece72d
Remove deprecated 'proxies' parameter. (#3314)
tomchristie Sep 26, 2024
5766bf5
Merge branch 'master' into version-1.0
tomchristie Sep 26, 2024
7a04661
Drop `sniffio` requirement. (#3323)
tomchristie Sep 27, 2024
823ab04
Merge branch 'master' into version-1.0
tomchristie Sep 27, 2024
acf9987
Merge branch 'master' into version-1.0
tomchristie Sep 27, 2024
4ec069b
Merge branch 'master' into version-1.0
karpetrosyan Oct 2, 2024
2763690
Add `httpx.SSLContext` configuration. (#3022)
karpetrosyan Oct 8, 2024
1aa0a14
Drop overloaded usage of 'verify' and 'cert'. (#3335)
tomchristie Oct 10, 2024
69b0f55
Drop `_compat.py` module. (#3343)
tomchristie Oct 12, 2024
cf042b6
Cleanup `response.elapsed` implementation. (#3345)
tomchristie Oct 17, 2024
8bceb60
Warn with `stacklevel=2` to show caller of deprecation (#3136)
hugovk Oct 17, 2024
9a0561e
Merge branch 'master' into version-1.0
tomchristie Oct 24, 2024
ce270b9
Allow deprecated `verify=<...>` and `cert=<...>`. (#3366)
tomchristie Oct 28, 2024
f3dafec
Update .github/CONTRIBUTING.md
tomchristie Oct 28, 2024
140f61d
Update CHANGELOG.md
tomchristie Oct 28, 2024
f0b7c0d
Update CHANGELOG.md
tomchristie Oct 28, 2024
ffbddc1
Merge branch 'master' into version-1.0
tomchristie Oct 28, 2024
1d93ad8
Version 0.28. (#3370)
tomchristie Oct 28, 2024
d43a10e
Update CHANGELOG.md
tomchristie Oct 28, 2024
1cb3fda
Update docs/advanced/ssl.md
tomchristie Oct 28, 2024
e830698
Apply suggestions from code review
tomchristie Oct 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,9 +211,10 @@ this is where our previously generated `client.pem` comes in:
```
import httpx

proxies = {"all": "http://127.0.0.1:8080/"}
ssl_context = httpx.SSLContext()
ssl_context.load_verify_locations("/path/to/client.pem")

with httpx.Client(proxies=proxies, verify="/path/to/client.pem") as client:
with httpx.Client(proxy="http://127.0.0.1:8080/", ssl_context=ssl_context) as client:
response = client.get("https://example.org")
print(response.status_code) # should print 200
```
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
push:
branches: ["master"]
pull_request:
branches: ["master", 'version*']
branches: ["master", "version-*"]

jobs:
tests:
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Version 0.28.0

Version 0.28.0 introduces an `httpx.SSLContext()` class and `ssl_context` parameter.

* Added `httpx.SSLContext` class and `ssl_context` parameter. (#3022, #3335)
* The `verify` and `cert` arguments have been deprecated and will now raise warnings. (#3022, #3335)
* The deprecated `proxies` argument has now been removed.
* The deprecated `app` argument has now been removed.
* The `URL.raw` property has now been removed.

## 0.27.2 (27th August, 2024)

### Fixed
Expand Down
195 changes: 147 additions & 48 deletions docs/advanced/ssl.md
Original file line number Diff line number Diff line change
@@ -1,100 +1,199 @@
When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA).

## Changing the verification defaults
### Enabling and disabling verification

By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). This is what you want in most cases, even though some advanced situations may require you to use a different set of certificates.
By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases...

If you'd like to use a custom CA bundle, you can use the `verify` parameter.
```pycon
>>> httpx.get("https://expired.badssl.com/")
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)
```

```python
import httpx
Verification is configured through [the SSL Context API](https://docs.python.org/3/library/ssl.html#ssl-contexts).

r = httpx.get("https://example.org", verify="path/to/client.pem")
```pycon
>>> context = httpx.SSLContext()
>>> context
<SSLContext(verify=True)>
>>> httpx.get("https://www.example.com", ssl_context=context)
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)
```

Alternatively, you can pass a standard library `ssl.SSLContext`.
You can use this to disable verification completely and allow insecure requests...

```pycon
>>> import ssl
>>> import httpx
>>> context = ssl.create_default_context()
>>> context.load_verify_locations(cafile="/tmp/client.pem")
>>> httpx.get('https://example.org', verify=context)
>>> context = httpx.SSLContext(verify=False)
>>> context
<SSLContext(verify=False)>
>>> httpx.get("https://expired.badssl.com/", ssl_context=context)
<Response [200 OK]>
```

We also include a helper function for creating properly configured `SSLContext` instances.
### Configuring client instances

If you're using a `Client()` instance you should pass any SSL context when instantiating the client.

```pycon
>>> context = httpx.create_ssl_context()
>>> context = httpx.SSLContext()
>>> client = httpx.Client(ssl_context=context)
```

The `create_ssl_context` function accepts the same set of SSL configuration arguments
(`trust_env`, `verify`, `cert` and `http2` arguments)
as `httpx.Client` or `httpx.AsyncClient`
The `client.get(...)` method and other request methods on a `Client` instance *do not* support changing the SSL settings on a per-request basis.

If you need different SSL settings in different cases you should use more than one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool.

### Configuring certificate stores

By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/).

You can load additional certificate verification using the [`.load_verify_locations()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations) API:

```pycon
>>> import httpx
>>> context = httpx.create_ssl_context(verify="/tmp/client.pem")
>>> httpx.get('https://example.org', verify=context)
>>> context = httpx.SSLContext()
>>> context.load_verify_locations(cafile="path/to/certs.pem")
>>> client = httpx.Client(ssl_context=context)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

Or you can also disable the SSL verification entirely, which is _not_ recommended.
Or by providing an certificate directory:

```python
import httpx
```pycon
>>> context = httpx.SSLContext()
>>> context.load_verify_locations(capath="path/to/certs")
>>> client = httpx.Client(ssl_context=context)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

### Client side certificates

You can also specify a local cert to use as a client-side certificate, using the [`.load_cert_chain()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain) API:

r = httpx.get("https://example.org", verify=False)
```pycon
>>> context = httpx.SSLContext()
>>> context.load_cert_chain(certfile="path/to/client.pem")
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>
```

## SSL configuration on client instances
Or including a keyfile...

If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client.
```pycon
>>> context = httpx.SSLContext()
>>> context.load_cert_chain(
certfile="path/to/client.pem",
keyfile="path/to/client.key"
)
>>> httpx.get("https://example.org", ssl_context=context)
<Response [200 OK]>
```

```python
client = httpx.Client(verify=False)
Or including a keyfile and password...

```pycon
>>> context = httpx.SSLContext(cert=cert)
>>> context = httpx.SSLContext()
>>> context.load_cert_chain(
certfile="path/to/client.pem",
keyfile="path/to/client.key"
password="password"
)
>>> httpx.get("https://example.org", ssl_context=context)
<Response [200 OK]>
```

The `client.get(...)` method and other request methods *do not* support changing the SSL settings on a per-request basis. If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool.
### Using alternate SSL contexts

## Client Side Certificates
You can also use an alternate `ssl.SSLContext` instances.

You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file, or two-tuple of (certificate file, key file), or a three-tuple of (certificate file, key file, password)
For example, [using the `truststore` package](https://truststore.readthedocs.io/)...

```python
cert = "path/to/client.pem"
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
import ssl
import truststore
import httpx

ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client = httpx.Client(ssl_context=ssl_context)
```

Alternatively...
Or working [directly with Python's standard library](https://docs.python.org/3/library/ssl.html)...

```python
cert = ("path/to/client.pem", "path/to/client.key")
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
import ssl
import httpx

ssl_context = ssl.create_default_context()
client = httpx.Client(ssl_context=ssl_context)
```

Or...
### Working with `SSL_CERT_FILE` and `SSL_CERT_DIR`

Unlike `requests`, the `httpx` package does not automatically pull in [the environment variables `SSL_CERT_FILE` or `SSL_CERT_DIR`](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_default_verify_paths.html). If you want to use these they need to be enabled explicitly.

For example...

```python
cert = ("path/to/client.pem", "path/to/client.key", "password")
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
context = httpx.SSLContext()

# Use `SSL_CERT_FILE` or `SSL_CERT_DIR` if configured.
if os.environ.get("SSL_CERT_FILE") or os.environ.get("SSL_CERT_DIR"):
context.load_verify_locations(
cafile=os.environ.get("SSL_CERT_FILE"),
capath=os.environ.get("SSL_CERT_DIR"),
)
```

## Making HTTPS requests to a local server
## `SSLKEYLOGFILE`

Valid values: a filename

If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only.

Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer.

Example:

```python
# test_script.py
import httpx

with httpx.Client() as client:
r = client.get("https://google.com")
```

```console
SSLKEYLOGFILE=test.log python test_script.py
cat test.log
# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
```

### Making HTTPS requests to a local server

When making requests to local servers, such as a development server running on `localhost`, you will typically be using unencrypted HTTP connections.

If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it:

1. Use [trustme](https://github.com/python-trio/trustme) to generate a pair of server key/cert files, and a client cert file.
1. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.)
1. Tell HTTPX to use the certificates stored in `client.pem`:
2. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.)
3. Tell HTTPX to use the certificates stored in `client.pem`:

```python
client = httpx.Client(verify="/tmp/client.pem")
response = client.get("https://localhost:8000")
```pycon
>>> import httpx
>>> context = httpx.SSLContext()
>>> context.load_verify_locations(cafile="/tmp/client.pem")
>>> r = httpx.get("https://localhost:8000", ssl_context=context)
>>> r
Response <200 OK>
```
4 changes: 1 addition & 3 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,12 +171,10 @@ Also note that `requests.Session.request(...)` allows a `proxies=...` parameter,

## SSL configuration

When using a `Client` instance, the `trust_env`, `verify`, and `cert` arguments should always be passed on client instantiation, rather than passed to the request method.
When using a `Client` instance, the ssl configurations should always be passed on client instantiation, rather than passed to the request method.

If you need more than one different SSL configuration, you should use different client instances for each SSL configuration.

Requests supports `REQUESTS_CA_BUNDLE` which points to either a file or a directory. HTTPX supports the `SSL_CERT_FILE` (for a file) and `SSL_CERT_DIR` (for a directory) OpenSSL variables instead.

## Request body on HTTP methods

The HTTP `GET`, `DELETE`, `HEAD`, and `OPTIONS` methods are specified as not supporting a request body. To stay in line with this, the `.get`, `.delete`, `.head` and `.options` functions do not support `content`, `files`, `data`, or `json` arguments.
Expand Down
60 changes: 0 additions & 60 deletions docs/environment_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,66 +8,6 @@ Environment variables are used by default. To ignore environment variables, `tru

Here is a list of environment variables that HTTPX recognizes and what function they serve:

## `SSLKEYLOGFILE`

Valid values: a filename

If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only.

Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer.

Example:

```python
# test_script.py
import httpx

with httpx.AsyncClient() as client:
r = client.get("https://google.com")
```

```console
SSLKEYLOGFILE=test.log python test_script.py
cat test.log
# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
```

## `SSL_CERT_FILE`

Valid values: a filename

If this environment variable is set then HTTPX will load
CA certificate from the specified file instead of the default
location.

Example:

```console
SSL_CERT_FILE=/path/to/ca-certs/ca-bundle.crt python -c "import httpx; httpx.get('https://example.com')"
```

## `SSL_CERT_DIR`

Valid values: a directory following an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html).

If this environment variable is set and the directory follows an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html) (ie. you ran `c_rehash`) then HTTPX will load CA certificates from this directory instead of the default location.

Example:

```console
SSL_CERT_DIR=/path/to/ca-certs/ python -c "import httpx; httpx.get('https://example.com')"
```

## Proxies

The environment variables documented below are used as a convention by various HTTP tooling, including:
Expand Down
Loading