Unable to use sshfs (FUSE) to backup files with rustic.

[MarlonJMejia]

Version of rustic

rustic --version
rustic 0.7.0

What I am trying to accomplish

I am trying to mount a directory via sshfs to backup from a remote source.
The mount is successful and I am able to view the files for the remote host. I've also tried to do a more bare bones mount with no options. (There is no error on the output or success after an hour)

user is root for both local and remote host to execute the commands and mountfiles.

sshfs -o 'reconnect,cache=no,noauto_cache,entry_timeout=0' user@remotehost:/directory /localdirectory/.backup_sshfs/

The issue

I receive multiple warnings (see below), I initially though this warnings were simply warnings but it seems that backups are not sucessfull if there is a huge amount of files (or there is an actual error with sshfs) in the directory of the remote source.

errors

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`
[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

wc of files in the remote directorys to backup

fdfind -u . /mnt/storage/appdata/ -t f -t d | wc 
  49496   49501 4091785

fdfind -u . /var/lib/vz/dump/ -t f -t d | wc
    15      15     929

Command to execute backup

rustic -r . backup /tmp/.backup_sshfs/mnt/storage/appdata /tmp/.backup_sshfs/var/lib/vz/dump/

[00:00:00] backing up...                  ████████████████████████████████████████        0 B            0 B/s                            [WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

[WARN] ignoring error `Os { code: 95, kind: Uncategorized, message: "Operation not supported" }`

Notes

This same process works fine with Kopia, Duplicacy and Borg.

Backup process seems to work completely fine with a lower of total amount of files.

Please let me know if I can give you guys any descriptive logs.

[nardoor]

Hello,

I reproduced something similar with a minimal linux setup (ubuntu 22)

How I reproduced

To reproduced, I hosted a "linux server" using a dockerfile, and used sshfs on a test directory.

# on host:
docker pull linuxserver/openssh-server
docker run --rm -e PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)" -it linuxserver/openssh-server bash

# on docker container:
su linuxserver.io
cd /config
mkdir test-dir
echo "foobar" > test-dir/foobar

# on host: 
mkdir test-dir
sshfs -p 2222 [email protected]:/config/test-dir test-dir

The error I reproduce

Using getfattr utilitary on my host in the sshfs mounted directory, I encounter a failed operation:

getfattr -d -m '' -- test-dir/foobar 
# getfattr: test-dir/foobar: Operation not supported 

The error I have is slightly different from your logs @MarlonJMejia:

[WARN] ignoring error Error getting xattrs for "test-dir/foobar": Os { code: 95, kind: Uncategorized, message: "Operation not supported" }

This difference in our errors make me doubt that my solution fixes your problem.
Do you think you would be able to test the "patch" (see my pull request) and provide some feedback on whether or not it fixes your issue?

The fix I propose

The idea is to ignore this error that come directly from a failed syscall, and proceed without any "extended-attributes" for this backup.

See pull request: #233

[MarlonJMejia]

Compiled and used the patch created by @nardoor with his help.

Was able to finally create a backup of the files, but it does take quite an amount of time (during the initial backup) around 1HR30M~+ and almost the same amount while creating a second snapshot.

I've gone ahead and capture the output of the session, the logs contain quite an amount of lines and they are a bit wonky due to the color encoding. less -R -N might help to view the file.

Command executed

./rustic/target/debug/rustic -r . backup /tmp/.backup_sshfs/mnt/storage/appdata /tmp/.backup_sshfs/var/lib/vz/dump/ --password . --log-level debug

Tmux session logs

https://0x0.st/Xfxj.log - Initial backup
https://0x0.st/Xfxy.log - Second backup

Notes

I do understand that the wikijs directory contains way too many files, specifically it seems to take very long under the node_modules directory and I do understand that I could in fact create an ignore file to ignore the directory, therefore if the fix is to simple ignore I will simply do so.

But I am curious as to why others don't seem to have this issue (rsync, borg, kopia) and I will retry with restric as well.

[aawsome]

@MarlonJMejia thanks a lot for testing the patch of@nardoor. I think this can be added soon to rustic main.

About you behavior: I actually also miss an explanation why the scanning (and obviously it is just the scanning) is so slow in your case.

I'm also curious what your experiences are using other tools, especially restic.

You could test the following;

• run without --log-level debug. I don't think this will improve something, just to rule out that a slow logging output is the reason.
• run with the --no-scan option. This should half the time in case the scanning is the cause.

[MarlonJMejia]

Hey aawesome, I tested it again with restic and as per your advice (--no-scan on rustic).

I think this issue is mainly with rustic/restic in how they handle their file searching as both have the same issue.

I won't post the entire log this time but rustic (this was on an incremental backup)

initial date/time

• Thu Aug 1 02:48:14 EDT 2024
  ending date/tome
• Thu Aug 1 03:44:27 EDT 2024

Again, I don't believe it matters much as I should not be having that many files being backup, therefore I'll be closing the issue.