discover missing openAPI info

[rudolfix]

Info Discovery

To fully convert endpoint to resource we need to figure out as follows (assuming that this is REST Api that has any semblance to what REST Api should be: https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design/)

all of those could be LLM prompts but some of them are better solved heuristically

1. figure out that is the endpoint entity (ie. issue) and locate it the response data type

• used to name resource/table

2. figure out if the endpoint returns a single entity or a list of entities

• well constructed endpoint path will contain path param at the end for "details" type of endpoint returning single entity
• has a list as a response or a list under a well known key (then returns a list)

3. figure out unique id/primary key of the entity

if a list type
4. figure out pagination

• one of 4 most common paginators
• identify pagination params to remove them from resource signature
• we are able to figure out header and json/cursor based pagination from the response structure and header information
• in worst case we can autodetect paginators from the data at runtime

5. figure out if endpoint supports incremental loading

• figure out cursor field (ie. updated_at)
• before and since parameters

6. figure out if we should generate a transformer

details type
6a. a "details" type may be a transformer.

• identify parent resource (possibly by matching paths)
• match path param and response field (possibly primary key) to bind transformer to incoming data from parent

6b. list type
they may be standalone resources (ie. issues in a repo) or a transformers (when connected to a parent ie. to a list of repo in organization)

• see 5a.
• also code generation should allow to create a resource and a transformer

Representation

Discovery results should be passed to the code generation level. We have an idea to save it as https://github.com/dlt-hub/verified-sources/blob/enh/api_helper/sources/rest_api/typing.py#L107 and let people tweak it so it is easy to re-generate the code (ie. by chaning the paginator type or fixing transformer settings)