Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RESTClient: extend paginators to handle the HTTP POST method #1359

Open
4 tasks
burnash opened this issue May 14, 2024 · 2 comments
Open
4 tasks

RESTClient: extend paginators to handle the HTTP POST method #1359

burnash opened this issue May 14, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@burnash
Copy link
Collaborator

burnash commented May 14, 2024

Background

The current implementation of paginators in the RESTClient is designed to manage pagination through the HTTP GET method.

Problem

Certain APIs require the use of the HTTP POST method to retrieve paginated data (typically used when the request involves sending a significant amount of data). For example, Notion's Query a database uses JSON in the request body to filter and paginate data.

Task

  • Identify the paginator classes in the RESTClient that need to be extended to handle the HTTP POST method. The suspects are likely JSONResponsePaginator (aka JSONLinkPaginator) and JSONCursorPaginator.
  • Modify the paginator classes to support the HTTP POST method.
  • Add tests
  • Update the RESTClient documentation
@burnash burnash self-assigned this May 14, 2024
@burnash burnash added the enhancement New feature or request label May 14, 2024
@burnash burnash changed the title RESTClient: extend paginators to handle POST method RESTClient: extend paginators to handle the POST method May 14, 2024
@rudolfix rudolfix moved this from Todo to In Progress in dlt core library May 16, 2024
@rudolfix rudolfix moved this from In Progress to Todo in dlt core library May 16, 2024
@rudolfix
Copy link
Collaborator

@burnash should we close it?

@burnash
Copy link
Collaborator Author

burnash commented May 28, 2024

@rudolfix no, it's not implemented yet.

@rudolfix rudolfix moved this from Todo to Planned in dlt core library Jun 11, 2024
@burnash burnash changed the title RESTClient: extend paginators to handle the POST method RESTClient: extend paginators to handle the HTTP POST method Jul 9, 2024
@burnash burnash moved this from Planned to Todo in dlt core library Mar 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

3 participants