Simplify schema management - Specifying preferred_types without full schema definitions

[khoadaniel]

Feature description

dlt should let users specify preferred_types or other concerned configurations in the import-schema yaml without requiring users to provide a full schema description.

Are you a dlt user?

Yes, I run dlt in production.

Use case

I need to specify preferred_types within my schema settings to ensure all ID-related fields are treated as text. Here's a snippet (from the import-yaml file) for context:

settings:
  preferred_types:
    re:.*id.*: text

To apply this settings, I need to keep the rest of the schema (which I do not want to maintain) in the import-yaml.
If I delete any of the following from the import-schema yaml file, dlt would fail.

• tables
• normalizers
  or even
• version_hash

Example error:

<class 'dlt.common.exceptions.DictValidationException'>
In .: following required fields are missing {'tables'}

So given a normal user experience, I care only about a few config lines to govern my schema, but I have to declare the rest of the yaml file. I think this is not a good practice.

Proposed solution

dlt should let users specify preferred_types or other concerned configurations in the import-schema yaml without requiring users to provide a full schema description.

Related issues

No response