Save compressed load files with .gz extension

[steinitzu]

Problem description

Load files are always saved with the extension of the file format, regardless of whether compression is enabled.
I.e. s3://path/to/load/file.jsonl may or may not be compressed.
This causes issues with e.g. databricks loader which can't parse gzip files without .gz extension so compression must be disabled. Possibly affects snowflake and redshift too, I think we assume json files are always compressed.

Solution

The data writer should add .gz extension when compression is enabled.
There are few places where we parse filenames to detect format, usually relying on filename.endswith('.<file_format>') or os.path.splitext. These need to be refactored, we need a clean way to get format/compression from filename.

[elviskahoro]

+1 this!

You can disable compression by setting the env variable:
os.environ["DATA_WRITER__DISABLE_COMPRESSION"] = str(True)

[rudolfix]

@elviskahoro there are many ways to disable compression in code. all of them change the configuration though ie.
os.environ["DATA_WRITER__DISABLE_COMPRESSION"] = "True"
dlt.config["data_writer.disable_compression"] = True

[kiernan]

I've also noticed this when attempting to writing jsonl to cloud storage and then read from it in another pipeline.

Despite using the built-in filesystem destination module to write it, the filesystem source module and read_jsonl() transformer will fail to parse it.

It does work when adding another transformer like this in-between:

    @dlt.transformer()
    def set_file_type(
        items: Iterator[FileItemDict],
        encoding: Optional[str] = None,
        mime_type: Optional[str] = None,
    ) -> Iterator[FileItemDict]:
        for file_obj in items:
            if encoding is not None:
                file_obj["encoding"] = encoding
            if mime_type is not None:
                file_obj["mime_type"] = mime_type
        return items