Add functionality to easily create a container backup (#161)

The `disk_objectstore.backup_utils` module is added that contains various
utilities that make it easy to create a backup of a container. It was
designed with the following criteria in mind:

* Backup should use rsync to make incremental backup as efficient as
  possible.
* The mechanism should support keeping a certain number of backups in the
  target location, removing the oldest copies if the number exceeds the
  target number. 
* The mechanism should support backing up to a target that is reachable
  over an SSH connection.
* The mechanism should be easily reusable by other storage solutions that
  use the disk objectstore as part of the storage, e.g., `aiida-core`.
* The mechanism should work safely while the storage is being used.

The `BackupManager` class implements all functionality that relates to
copying files (over SSH or locally) using rsync and keeping the target
number of backup copies. It takes a callable that should contain the
actual backup logic to create a backup of a containers contents. The
callable should take an instance of the `BackupManager` as a first
argument as it should use its `call_rsync` method to copy any files to
the target directory.

This design makes for a slightly awkward cyclic dependency where the
manager calls a callable which takes the manager as an argument, but this
is done to make the actual backup logic configurable by other packages.

The actual container backup logic is implemented in `backup_container`
which is the callable passed to `BackupManager.backup_auto_folders`.
This function carefully copies the content of a container's folder to the
target directory in a specific order which guarantees that the backup is
coherent even while the container is actively being used.

.github/workflows/ci.yml

@@ -6,22 +6,19 @@ name: Continuous integration
 on: [push, pull_request]
 
 jobs:
-
   pre-commit:
-
     runs-on: ubuntu-latest
     timeout-minutes: 5
 
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v2
-      with:
-        python-version: "3.8"
-    - uses: pre-commit/[email protected]
+      - uses: actions/checkout@v2
+      - name: Set up Python 3.8
+        uses: actions/setup-python@v2
+        with:
+          python-version: "3.8"
+      - uses: pre-commit/[email protected]
 
   tests:
-
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -30,30 +27,37 @@ jobs:
         python-version: ["3.8", "3.9", "3.10", "3.11"]
 
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install dependencies (including dev dependencies at frozen version)
-      # I'm using pip install -e to make sure that the coverage properly traces the runs
-      # also of the concurrent tests (maybe we can achieve this differently)
-      run: |
-        python -m pip install --upgrade pip
-        pip install -e .[progressbar,optionaltests]
-        pip install -r requirements.lock
-    - name: Test with pytest
-      # No need to run the benchmarks, they will run in a different workflow
-      # Also, run in very verbose mode so if there is an error we get a complete diff
-      run: pytest -vv --cov=disk_objectstore --benchmark-skip
-      env:
-        SQLALCHEMY_WARN_20: 1
-    - name: Create xml coverage
-      run: coverage xml
-    - name: Upload coverage to Codecov
-      uses: codecov/codecov-action@v3
-      with:
-        files: ./coverage.xml
-        name: disk-objectstore
-        ## Commenting the following lines - if often fails, and if at least one manages to push, it should be enough
-        # fail_ci_if_error: true
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v1
+        with:
+          python-version: ${{ matrix.python-version }}
+        # Set up 'ssh localhost' that is used in testing the backup command
+        # skipped for windows, as it doesn't support this setup or the backup command
+      - name: set up 'ssh localhost'
+        if: matrix.os != 'windows-latest'
+        run: |
+          .github/workflows/setup-ssh-localhost.sh
+          ssh -v localhost
+      - name: Install dependencies (including dev dependencies at frozen version)
+        # I'm using pip install -e to make sure that the coverage properly traces the runs
+        # also of the concurrent tests (maybe we can achieve this differently)
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .[progressbar,optionaltests]
+          pip install -r requirements.lock
+      - name: Test with pytest
+        # No need to run the benchmarks, they will run in a different workflow
+        # Also, run in very verbose mode so if there is an error we get a complete diff
+        run: pytest -vv --cov=disk_objectstore --benchmark-skip
+        env:
+          SQLALCHEMY_WARN_20: 1
+      - name: Create xml coverage
+        run: coverage xml
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          files: ./coverage.xml
+          name: disk-objectstore
+          ## Commenting the following lines - if often fails, and if at least one manages to push, it should be enough
+          # fail_ci_if_error: true

.github/workflows/setup-ssh-localhost.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -ev
+
+ssh-keygen -q -t rsa -b 4096 -N "" -f "${HOME}/.ssh/id_rsa"
+ssh-keygen -y -f "${HOME}/.ssh/id_rsa" >> "${HOME}/.ssh/authorized_keys"
+ssh-keyscan -H localhost >> "${HOME}/.ssh/known_hosts"
+
+chmod 700 "${HOME}/.ssh"
+chmod 600 "${HOME}/.ssh"/*