Follow the XDG Base Directory spec for local pipelines' data

[goosethedev]

Feature description

When running a dlt pipeline on Linux, a directory gets created at ~/.dlt. According to the XDG Base Directory specification, application specific data should be stored at $XDG_DATA_HOME/dlt if the environment variable is set.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

No response

Proposed solution

Check if the $XDG_DATA_HOME env var is set. If so, use that location to store the pipelines' data.

An additional condition in the run_context.py file could be enough.

Related issues

No response

[sh-rp]

Hey @goosethedev, would you like to provide a PR for this? It should be only a couple of lines of codes and I can point you in the right direction.

[goosethedev]

Yes, I'll prepare a PR. However, how would be best to handle cases where the $XDG_DATA_HOME env var is already set, but ~/.dlt is used currently? Some options I can think of:

• Moving the directory if present.
• Recreating the data and displaying a warning to remove manually the previous directory.

[sh-rp]

I would do option 2

[rudolfix]

implementation tip: there's already an env variable that will place data_dir of dlt wherever you want:

DLT_DATA_DIR = "DLT_DATA_DIR"
"""Sets default directory where pipelines' data (working directories) will be stored"""

my take would be to find how this is used and add a fallback with XDG_DATA_HOME. it should be just one place in the code (excl. tests) somewhere in run_context,py

btw. what about other directories? https://specifications.freedesktop.org/basedir-spec/latest/
@sh-rp maybe we should implement dlt RunContext that stores dlt files per those settings instead of modifying the default? otherwise I see backward compatibility problems as mentioned above?

[goosethedev]

@sh-rp @rudolfix I ended up using the XDG_DATA_HOME env var if it is set and ~/.dlt doesn't exist. Otherwise ~/.dlt is used and no compat breakage.

As for other XDG dirs, I'm not familiar with what other data dlt stores globally, but I think for now the data dir is a good start.