Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove the OS filename length restriction impacting the identifiers length #191

Open
rudolfix opened this issue Mar 19, 2023 · 0 comments
Open
Labels
bug Something isn't working

Comments

@rudolfix
Copy link
Collaborator

Background
extract and normalize stage use files to pass intermediary data (extracted and normalized files). the metadata on that files is stored in file name. it contains schema name, table name and other information
the os file name limit is ~255 characters (but may be much less on some NFSses etc.)
that impact our ability to create long table names on bigquery/duckdb even with #157

Tasks
We need to remove metadata information from file name or find a way to shorten the file name ie. by using hashes. File names should be human readable though.

see test: test_run_with_table_name_exceeding_path_length which must start passing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Todo
Development

No branches or pull requests

1 participant