Problem: using "shared_directory" config setting causes path errors

[djjuhasz]

Describe the bug
When the a3m "shared_directory" setting is explicitly set, a3m fails with an error in the "Download package" job:

To Reproduce
Steps to reproduce the behavior:

1. Set up a3m to run with Docker: https://a3m.readthedocs.io/en/latest/docker.html
2. Start a transfer using the A3M_SHARED_DIRECTORY environment variable (Note: the example below specifies the default shared_directory)
3. See error below

Call:

docker run \
  --interactive --tty --rm \
  --volume="/tmp/demo/transfers:/tmp/demo/transfers" \
  --volume="/tmp/demo/completed:/home/a3m/.local/share/a3m/share/completed" \
  --entrypoint=python \
  -e A3M_SHARED_DIRECTORY="/home/a3m/.local/share/a3m/share" \
  ghcr.io/artefactual-labs/a3m:latest \
  -m a3m.cli.client --name=small \
  file:///tmp/demo/transfers/small.zip

Error

[Errno 13] Permission denied: 'a3m.transfer.1m4hapvu.f313dae4-e7f3-4302-813b-bc24f4a30693'Traceback (most recent     │
│ call last):                                                                                                          │
│   File "/a3m/a3m/client/job.py", line 121, in JobContext                                                             │
│     yield                                                                                                            │
│   File "/a3m/a3m/client/clientScripts/a3m_download_transfer.py", line 167, in call                                   │
│     job.set_status(main(job, transfer_id, transfer_path, url))                                                       │
│                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                        │
│   File "/a3m/a3m/client/clientScripts/a3m_download_transfer.py", line 148, in main                                   │
│     _process_file_url(job, Path(parsed.path), Path(transfer_path), transfer_id)                                      │
│   File "/a3m/a3m/client/clientScripts/a3m_download_transfer.py", line 129, in _process_file_url                      │
│     _transfer_file(job, path, transfer_path, transfer_id, copy=True)                                                 │
│   File "/a3m/a3m/client/clientScripts/a3m_download_transfer.py", line 77, in _transfer_file                          │
│     with _create_tmpdir(transfer_id, purpose="transfer") as tmp_dir:                                                 │
│   File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/contextlib.py", line 137, in __enter__                       │
│     return next(self.gen)                                                                                            │
│            ^^^^^^^^^^^^^^                                                                                            │
│   File "/a3m/a3m/client/clientScripts/a3m_download_transfer.py", line 38, in _create_tmpdir                          │
│     with TemporaryDirectory(                                                                                         │
│          ^^^^^^^^^^^^^^^^^^^                                                                                         │
│   File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/tempfile.py", line 882, in __init__                          │
│     self.name = mkdtemp(suffix, prefix, dir)                                                                         │
│                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                         │
│   File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/tempfile.py", line 384, in mkdtemp                           │
│     _os.mkdir(file, 0o700)                                                                                           │
│ PermissionError: [Errno 13] Permission denied: 'a3m.transfer.1m4hapvu.f313dae4-e7f3-4302-813b-bc24f4a30693'          │

Expected behaviour

If the directory specified by A3M_SHARED_DIRECTORY exists and is writable, then a3m should process the transfer using the specified directory.

Screenshots

Additional context

1. When the "shared_directory" config value is not specified then https://github.com/artefactual-labs/a3m/blob/main/a3m/settings/common.py#L168-L176 sets the config paths "shared_directory", "temp_dir", "processing_directory", and "rejected_directory" to the default values. If "shared_directory" is set explicitly, then the other three paths are not set.
2. If all of the "shared" directory paths are explicitly set, then processing completes successfully (see "example 1" below)
3. If "shared_directory" is set to path other than the default "/home/a3m/.local/share/a3m/share" then the directory must already exist and be writable even when all of the share "subdirectory" paths are set as well (See "example 2" below)

Example 1

❯ docker run \
--interactive --tty --rm \
--volume="/tmp/demo/transfers:/tmp/demo/transfers" \
--volume="/tmp/demo/completed:/home/a3m/.local/share/a3m/share/completed" \
--entrypoint=python \
-e A3M_SHARED_DIRECTORY="/home/a3m/.local/share/a3m/share" \
-e A3M_TEMP_DIR="/home/a3m/.local/share/a3m/share/tmp" \
-e A3M_PROCESSING_DIRECTORY="/home/a3m/.local/share/a3m/share/currentlyProcessing" \
-e A3M_REJECTED_DIRECTORY="/home/a3m/.local/share/a3m/share/rejected" \
ghcr.io/artefactual-labs/a3m:latest \
-m a3m.cli.client --name=small \
file:///tmp/demo/transfers/small.zip

AIP 8f3e42ec-cc38-4e5a-8a8c-ddf5a2eaf3c4 is being generated...
Processing completed successfully!

Example 2

❯ docker run \
--interactive --tty --rm \
--volume="/tmp/demo/transfers:/tmp/demo/transfers" \
--volume="/tmp/demo/completed:/home/a3m/share/completed" \
--entrypoint=python \
-e A3M_SHARED_DIRECTORY="/home/a3m/share" \
-e A3M_TEMP_DIR="/home/a3m/share/tmp" \
-e A3M_PROCESSING_DIRECTORY="/home/a3m/share/currentlyProcessing" \
-e A3M_REJECTED_DIRECTORY="/home/a3m/share/rejected" \
ghcr.io/artefactual-labs/a3m:latest \
-m a3m.cli.client --name=small \
file:///tmp/demo/transfers/small.zip

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/a3m/a3m/cli/client/__main__.py", line 179, in <module>
    main()
  File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a3m/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/a3m/a3m/cli/client/__main__.py", line 71, in main
    with ClientWrapper(address, wait_for_ready) as cw:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/a3m/a3m/cli/client/wrapper.py", line 23, in __init__
    self._create_server()
  File "/a3m/a3m/cli/client/wrapper.py", line 48, in _create_server
    self.server = create_server(
                  ^^^^^^^^^^^^^^
  File "/a3m/a3m/server/runner.py", line 185, in create_server
    shared_dirs.create()
  File "/a3m/a3m/server/shared_dirs.py", line 26, in create
    os.makedirs(dirname, mode=0o770)
  File "<frozen os>", line 215, in makedirs
  File "<frozen os>", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/home/a3m/share/currentlyProcessing'