refactor backend (#73)

* refactor backend usage * remove unneeded Downcast * error handling; add opendal backend * deactivate opendal:sftp for windows * fix most clippy lints * fix windows builds * fix clippy lint * update MSRV (as required by ordered-multimap, needed by opendal->rsa) * opendal - more services, add logging and retry * some code clean-ups * clean-up code and comments * clippy hints and s3 convenience backend * correct path * temporarily ignore rsa advisory until there is a workaround * fix: import and typo Signed-off-by: simonsan <[email protected]> * fix: unused import on win Signed-off-by: simonsan <[email protected]> * chore: deactivate security warning for rsa for cargo-audit and cargo-deny and annotate with fixme Signed-off-by: simonsan <[email protected]> * style: dprint fmt Signed-off-by: simonsan <[email protected]> * ci: test also examples Signed-off-by: simonsan <[email protected]> * chore: run cargo-msrv and update msrv to 1.71.1 Signed-off-by: simonsan <[email protected]> * chore: check and merge imports Signed-off-by: simonsan <[email protected]> * refactor: move RestErrorKind to error.rs Signed-off-by: simonsan <[email protected]> * refactor: move RcloneErrorKind to error.rs Signed-off-by: simonsan <[email protected]> * style: fmt Signed-off-by: simonsan <[email protected]> * chore: apply clippy fixes Signed-off-by: simonsan <[email protected]> * chore: borrowing for tree streamer Signed-off-by: simonsan <[email protected]> * chore: borrowing for path list Signed-off-by: simonsan <[email protected]> * chore: close name Signed-off-by: simonsan <[email protected]> * chore: clippy Signed-off-by: simonsan <[email protected]> * chore: clippy values_mut iter Signed-off-by: simonsan <[email protected]> * style: cargo fmt Signed-off-by: simonsan <[email protected]> * fix: readd error to docs that was removed Signed-off-by: simonsan <[email protected]> * fix: use `new` -> Self for HotColdBackend Signed-off-by: simonsan <[email protected]> * fix: lifetimes in public api Signed-off-by: simonsan <[email protected]> * chore: add docs and remove dbg! Signed-off-by: simonsan <[email protected]> * fix: log error in warm up Signed-off-by: simonsan <[email protected]> * docs: add docs Signed-off-by: simonsan <[email protected]> * feat(choose): implement trait for choosing backends Signed-off-by: simonsan <[email protected]> * refactor: move backend crate out of rustic_core, first try Signed-off-by: simonsan <[email protected]> * refactor: move tests into workspace crate Signed-off-by: simonsan <[email protected]> * refactor: choose Signed-off-by: simonsan <[email protected]> * refactor: move backup example and update to new api Signed-off-by: simonsan <[email protected]> * WIP: stash Signed-off-by: simonsan <[email protected]> * refactor: init backends Signed-off-by: simonsan <[email protected]> * chore: cleanup Signed-off-by: simonsan <[email protected]> * refactor(backends): use RepositoryBackends struct Signed-off-by: simonsan <[email protected]> * chore: cleanup Signed-off-by: simonsan <[email protected]> * doc: tiny doc for RepositoryBackends Signed-off-by: simonsan <[email protected]> * refactor: implement test cases Signed-off-by: simonsan <[email protected]> * refactor: rework examples and move them to workspace examples Signed-off-by: simonsan <[email protected]> * fix: remove examples from manifest.include and thus packaging as they are now outside of the package in workspace Signed-off-by: simonsan <[email protected]> * chore: remove opendal dep from core crate Signed-off-by: simonsan <[email protected]> * chore: cleanup errors Signed-off-by: simonsan <[email protected]> * fix(api): fix export of overwrite for mergable struct fields Signed-off-by: simonsan <[email protected]> * docs: document FindInBackend Signed-off-by: simonsan <[email protected]> * refactor: move errors to error.rs Signed-off-by: simonsan <[email protected]> * chore: ignore lint for unused imports as overwrite is not being recognized as being used Signed-off-by: simonsan <[email protected]> * chore: update package details Signed-off-by: simonsan <[email protected]> * chore: cleanup imports Signed-off-by: simonsan <[email protected]> * style: dprint fmt Signed-off-by: simonsan <[email protected]> * ci: Check for workspace crates's MSRV Signed-off-by: simonsan <[email protected]> * ci: update actions to be used on workspaces or especially on both workspace crates Signed-off-by: simonsan <[email protected]> * feat(s3): implement extra s3 backend, delegated read- and writebackend impl of opendal Signed-off-by: simonsan <[email protected]> * chore: remove glob import Signed-off-by: simonsan <[email protected]> * chore: using features and fixing stuff so tests run through on them with `cargo +stable hack test --feature-powerset --workspace` Signed-off-by: simonsan <[email protected]> * chore: remove cyclic dependency Signed-off-by: simonsan <[email protected]> * style: dprint fmt Signed-off-by: simonsan <[email protected]> * chore: set default features to include all backends Signed-off-by: simonsan <[email protected]> * chore: change back delimiter to `:` Signed-off-by: simonsan <[email protected]> * refactor: impl AsRef<str> in Backend::new() * chore: implement to_inner() on S3Backend, but keep impl Write/Read backend for now * chore(deps): update dependencies Signed-off-by: simonsan <[email protected]> * chore: rename backends to backend to account for crate name Signed-off-by: simonsan <[email protected]> * fix: add back cfg in backend/util.rs for parsing the provider string Signed-off-by: simonsan <[email protected]> * fix: add missing attribute Signed-off-by: simonsan <[email protected]> * chore: use clearer naming Signed-off-by: simonsan <[email protected]> * fix: use locahost Signed-off-by: simonsan <[email protected]> * fix: add match case for unc only Signed-off-by: simonsan <[email protected]> * fix: test with respecting features as some things don't need to work and won't work on *nix Signed-off-by: simonsan <[email protected]> * fix: allow unused imports in tests Signed-off-by: simonsan <[email protected]> * fix: test with respecting platforms Signed-off-by: simonsan <[email protected]> * use options_hot and options_cold * simplify s3 backend impl * add evaluate_password to RepositoryOptions * don't panic when repository is missing * rclone backend: Remove Arc * expose less in rustic_core crate * clean-up dependencies for rustic_core * clean-up dependencies for rustic_backend * don't validate repo path (-> init command) * fix clippy lint * fix tokio dependency * style: fmt Signed-off-by: simonsan <[email protected]> --------- Signed-off-by: simonsan <[email protected]> Co-authored-by: simonsan <[email protected]>
rustic-rs · Jan 17, 2024 · 310f5a1 · 310f5a1
1 parent 5d78cf8
commit 310f5a1
Show file tree

Hide file tree

Showing 123 changed files with 3,741 additions and 2,661 deletions.
diff --git a/.cargo/audit.toml b/.cargo/audit.toml
@@ -0,0 +1,6 @@
+[advisories]
+ignore = [
+  # FIXME!: See https://github.com/RustCrypto/RSA/issues/19#issuecomment-1822995643.
+  # There is no workaround available yet.
+  "RUSTSEC-2023-0071",
+]
diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml
@@ -28,6 +28,7 @@ jobs:
       - uses: rustsec/audit-check@dd51754d4e59da7395a4cd9b593f0ff2d61a9b95 # v1.4.1
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
+          ignore: RUSTSEC-2023-0071  # rsa thingy, ignored for now
 
   cargo-deny:
     if: ${{ github.repository_owner == 'rustic-rs' }}

diff --git a/.github/workflows/careful.yml b/.github/workflows/careful.yml
@@ -41,59 +41,61 @@ jobs:
       - name: Run Cargo Careful
         run: cargo +${{ matrix.rust }} careful test
 
-  miri:
-    name: Miri Test
-    runs-on: ${{ matrix.job.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        rust: [nightly] # runs on nightly only
-        job:
-          - os: macos-latest
-          - os: ubuntu-latest
-          - os: windows-latest
-    steps:
-      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-        if: github.event_name != 'pull_request'
-        with:
-          fetch-depth: 0
+  # TODO: don't run miri for now, due to addition of workspace
+  #1      crates and we'll need to figure out if we want to run miri
+  # miri:
+  #   name: Miri Test
+  #   runs-on: ${{ matrix.job.os }}
+  #   strategy:
+  #     fail-fast: false
+  #     matrix:
+  #       rust: [nightly] # runs on nightly only
+  #       job:
+  #         - os: macos-latest
+  #         - os: ubuntu-latest
+  #         - os: windows-latest
+  #   steps:
+  #     - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
+  #       if: github.event_name != 'pull_request'
+  #       with:
+  #         fetch-depth: 0
 
-      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-        if: github.event_name == 'pull_request'
-        with:
-          ref: ${{ github.event.pull_request.head.sha }}
-          fetch-depth: 0
+  #     - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
+  #       if: github.event_name == 'pull_request'
+  #       with:
+  #         ref: ${{ github.event.pull_request.head.sha }}
+  #         fetch-depth: 0
 
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@1482605bfc5719782e1267fd0c0cc350fe7646b8 # v1
-        with:
-          toolchain: ${{ matrix.rust }}
-          components: miri
-      - uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2
+  #     - name: Install Rust toolchain
+  #       uses: dtolnay/rust-toolchain@1482605bfc5719782e1267fd0c0cc350fe7646b8 # v1
+  #       with:
+  #         toolchain: ${{ matrix.rust }}
+  #         components: miri
+  #     - uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2
 
-      - name: Run Cargo Clean
-        run: cargo +${{ matrix.rust }} clean # miri needs clean builds
+  #     - name: Run Cargo Clean
+  #       run: cargo +${{ matrix.rust }} clean # miri needs clean builds
 
-      - name: Patch Cargo.toml
-        shell: bash
-        run: |
-          # Account for sha256_compress not being interpreted by miri
-          # https://github.com/rust-lang/miri/issues/3066
-          sed -i -e 's/^sha2 = { version.*/sha2 = "0"/g' ./Cargo.toml
-      - name: Run Cargo Miri Setup
-        run: cargo +${{ matrix.rust }} miri setup # keep output clean
+  #     - name: Patch Cargo.toml
+  #       shell: bash
+  #       run: |
+  #         # Account for sha256_compress not being interpreted by miri
+  #         # https://github.com/rust-lang/miri/issues/3066
+  #         sed -i -e 's/^sha2 = { version.*/sha2 = "0"/g' ./Cargo.toml
+  #     - name: Run Cargo Miri Setup
+  #       run: cargo +${{ matrix.rust }} miri setup # keep output clean
 
-      - name: Run Cargo Miri
-        env:
-          MIRIFLAGS: -Zmiri-disable-isolation
-        run: cargo +${{ matrix.rust }} miri test -- --nocapture
+  #     - name: Run Cargo Miri
+  #       env:
+  #         MIRIFLAGS: -Zmiri-disable-isolation
+  #       run: cargo +${{ matrix.rust }} miri test -- --nocapture
 
   result:
     name: Result (Careful CI)
     runs-on: ubuntu-latest
     needs:
       - careful
-      - miri
+      # - miri  # FIXME: don't run miri for now, due to addition of workspace
     steps:
       - name: Mark the job as successful
         run: exit 0

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -71,7 +71,7 @@ jobs:
           toolchain: ${{ matrix.rust }}
       - uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2
       - name: Run Cargo Test
-        run: cargo +${{ matrix.rust }} test -r --all-targets --all-features --workspace
+        run: cargo +${{ matrix.rust }} test -r --all-targets --all-features --workspace --examples
 
   docs:
     name: Build docs
@@ -109,6 +109,7 @@ jobs:
     strategy:
       matrix:
         rust: [stable, beta, nightly]
+        crate: [rustic_core, rustic_backend] # if we use a workspace, we also check all examples/*
         job:
           - os: macos-latest
           - os: ubuntu-latest
@@ -135,7 +136,7 @@ jobs:
           tool: cargo-hack
       - uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2
       - name: Run Cargo Hack
-        run: cargo +${{ matrix.rust }} hack check --feature-powerset --no-dev-deps
+        run: cargo +${{ matrix.rust }} hack check --feature-powerset --no-dev-deps -p ${{ matrix.crate }}
 
   result:
     name: Result (CI)

diff --git a/.github/workflows/msrv.yml b/.github/workflows/msrv.yml
@@ -18,6 +18,9 @@ jobs:
   msrv:
     name: Check MSRV
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        crate: [rustic_core, rustic_backend]
     steps:
       - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
       - name: Install cargo-hack
@@ -26,7 +29,7 @@ jobs:
           tool: cargo-hack
 
       - name: Run Cargo Hack
-        run: cargo hack check --rust-version
+        run: cargo hack check --rust-version -p ${{ matrix.crate }}
 
   result:
     name: Result (MSRV)

diff --git a/Cargo.toml b/Cargo.toml
@@ -1,148 +1,14 @@
-[package]
-name = "rustic_core"
-version = "0.1.2"
-authors = ["Alexander Weiss"]
-categories = [
-  "Algorithms",
-  "Compression",
-  "Cryptography",
-  "Data structures",
-  "Filesystem",
-]
-documentation = "https://docs.rs/rustic_core"
-edition = "2021"
-homepage = "https://rustic.cli.rs/"
-include = ["src/**/*", "LICENSE-*", "README.md", "examples/**/*"]
-keywords = ["backup", "restic", "deduplication", "encryption", "library"]
-license = "Apache-2.0 OR MIT"
-publish = true
-readme = "README.md"
-repository = "https://github.com/rustic-rs/rustic_core"
+[workspace]
+members = ["crates/core", "crates/backend", "examples/*"]
 resolver = "2"
-rust-version = "1.70.0"
-description = """
-rustic_core - library for fast, encrypted, deduplicated backups that powers rustic-rs
-"""
 
-[lib]
-path = "src/lib.rs"
-name = "rustic_core"
-test = true
-doctest = true
-bench = true
-doc = true
-harness = true
-edition = "2021"
+[workspace.package]
+rust-version = "1.71.1"
 
-[features]
-default = []
-cli = ["merge", "clap"]
-merge = ["dep:merge"]
-clap = ["dep:clap"]
-
-[package.metadata.docs.rs]
-all-features = true
-rustdoc-args = ["--document-private-items", "--generate-link-to-definition"]
-
-[dependencies]
-# errors
-displaydoc = "0.2.4"
-thiserror = "1.0.56"
-
-# macros
-derivative = "2.2.0"
-derive_more = "0.99.17"
-derive_setters = "0.1.6"
-
-# logging
-log = "0.4.20"
-
-# parallelize
-crossbeam-channel = "0.5.10"
-pariter = "0.5.1"
-rayon = "1.8.0"
-
-# crypto
-aes256ctr_poly1305aes = "0.2.0"
-rand = "0.8.5"
-scrypt = { version = "0.11.0", default-features = false }
-
-# chunker / packer
-integer-sqrt = "0.1.5"
-
-# serialization
-binrw = "0.13.3"
-hex = { version = "0.4.3", features = ["serde"] }
-serde = { version = "1.0.195" }
-serde-aux = "4.3.1"
-serde_derive = "1.0.195"
-serde_json = "1.0.111"
-serde_with = { version = "3.4.0", features = ["base64"] }
-
-# other dependencies
-bytes = "1.5.0"
-chrono = { version = "0.4.31", default-features = false, features = ["clock", "serde"] }
-enum-map = "2.7.3"
-enum-map-derive = "0.17.0"
-zstd = "0.13.0"
-
-# local backend
-aho-corasick = "1.1.2"
-cached = { version = "0.47.0", default-features = false, features = ["proc_macro"] }
-filetime = "0.2.23"
-ignore = "0.4.22"
-nix = { version = "0.27.1", default-features = false, features = ["user", "fs"] }
-walkdir = "2.4.0"
-
-# rest backend
-backoff = "0.4.0"
-reqwest = { version = "0.11.23", default-features = false, features = ["json", "rustls-tls-native-roots", "stream", "blocking"] }
-url = "2.5.0"
-
-# rclone backend
-# semver = "1.0.18" # FIXME: unused, remove?
-
-# cache
-cachedir = "0.3.1"
-dirs = "5.0.1"
-
-# cli
-clap = { version = "4.4.13", optional = true, features = ["derive", "env", "wrap_help"] }
-
-bytesize = "1.3.0"
-directories = "5.0.1"
-dunce = "1.0.4"
-gethostname = "0.4.3"
-humantime = "2.1.0"
-itertools = "0.12.0"
-merge = { version = "0.1.0", optional = true }
-path-dedot = "3.1.1"
-shell-words = "1.1.0"
-
-[target.'cfg(not(windows))'.dependencies]
-sha2 = { version = "0.10", features = ["asm"] }
-
-[target.'cfg(all(windows, not(target_env="gnu")))'.dependencies]
-# unfortunately, the asm extensions do not build on MSVC, see https://github.com/RustCrypto/asm-hashes/issues/17
-sha2 = "0.10"
-
-[target.'cfg(all(windows, target_env="gnu"))'.dependencies]
-sha2 = { version = "0.10", features = ["asm"] }
-
-[target.'cfg(not(any(windows, target_os="openbsd")))'.dependencies]
-xattr = "1"
-
-[dev-dependencies]
-expect-test = "1.4.1"
-pretty_assertions = "1.4.0"
-public-api = "0.33.1"
-quickcheck = "1.0.3"
-quickcheck_macros = "1.0.0"
-rstest = "0.18.2"
-rustdoc-json = "0.8.8"
-rustup-toolchain = "0.1.6"
+[workspace.dependencies]
+rustic_backend = { path = "crates/backend" }
+rustic_core = { path = "crates/core" }
 simplelog = "0.12.1"
-tempfile = "3.9.0"
 
 # see: https://nnethercote.github.io/perf-book/build-configuration.html
 [profile.dev]

diff --git a/README.md b/README.md
@@ -189,7 +189,7 @@ Please make sure, that you read the
 
 ## Minimum Rust version policy
 
-This crate's minimum supported `rustc` version is `1.68.2`.
+This crate's minimum supported `rustc` version is `1.71.1`.
 
 The current policy is that the minimum Rust version required to use this crate
 can be increased in minor version updates. For example, if `crate 1.0` requires