show full HTTP response on failure (Feat/1605)

[willi-mueller]

Description

This PR implements error logging of the full HTTP response on error when the dlt.sources.helpers.requests.Session is used.

Unless a custom session with different configuration is used, all HTTP responses with an error code 4xx will log an error.

This includes:

1. the RESTClient
2. each call made by the dlt.sources.rest_api

Related Issues

• Resolves show full response when printing failed api calls #1605

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	9e066d8
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6734c5996d93be0008e82824
😎 Deploy Preview	https://deploy-preview-1895--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[burnash]

I propose an alternative solution to logging the full HTTP response on error: we can use a context manager catch_http_error that captures HTTP errors and allows for flexible and controlled error handling / debugging.

Example:

import dlt
from dlt.sources.helpers.rest_client import paginate, catch_http_error


@dlt.resource(
    table_name="issues",
    write_disposition="replace",
    primary_key="id",
)
def get_issues():
    for page in paginate(
        "https://api.github.com/repos/dlt-hub/dlt/issues",
        headers={"Authorization": "Bearer invalid_token_here"},
    ):
        yield page


pipeline = dlt.pipeline(
    pipeline_name="github_issues_merge",
    destination="duckdb",
)

with catch_http_error() as http_error:
    load_info = pipeline.run(get_issues)

if http_error:
    print(f"HTTP {http_error.response.status_code}: {http_error.response.content}")

Pros:

• improved flexibility: the use can interact with captured HTTPError
• cleaner and safer logs
• selective usage: no need to alter the session

The rationale is that in my opinion accessing the HTTPError should be done on the higher level of abstraction than Requests's session as when the user is using the request/response directly they already have easy access to HTTP errors.
So the idea of this change is to provide some way to extract HTTPError from the dlt exceptions.