uploading and downloading file errors

[alecristia]

Disclaimer! this is not a question about childproject...

We've set up a datalad repo that holds 60k clips annotated in zooniverse, that will be analyzed acoustically on https://gin.g-node.org/LAAC-LSCP/zoo-clips.

When uploading, although largely the clips uploaded (and they can be downloaded from the web interface), we saw a range of errors:

Transfer data to 'gin': 50%|██████████████████████████████████████████ | 2.00/4.00 [00:00<00:00, 6.12k Steps/s]Connection to gin.g-node.org closed by remote host.
[ERROR ] Lost connection (fd:16: hGetChar: end of file)
[ERROR ] Transfer failed

a little later (without relaunching the command):

Transfer data to 'gin': 50%|████████████████████████████████████████ | 2.00/4.00 [4:16:46<4:16:46, 7.70ks/ StepsCommandError: 'git annex copy --batch -z --to gin --fast --json --json-error-messages --json-progress -c annex.dotfiles=true -c 'remote.gin.annex-ssh-options=-o ControlMaster=auto -S /Users/acristia/Library/Caches/datalad/sockets/3beae3e8' -c annex.retry=3' failed with exitcode 1 under /Users/acristia/Documents/zoo-clips
git-annex: copy: 6 failed

We relaunched the command, and got:

$ datalad push --to gin

Transfer data to 'gin': 50%|██████████████████████████████████████████ | 2.00/4.00 [00:00<00:00, 7.84k Steps/s]Connection to gin.g-node.org closed by remote host.
copy(ok): recordings/raw/batch1_2020_batch_19/6249401870.wav (file) [to gin...]
copy(ok): recordings/raw/batch1_2020_batch_19/6252629657.wav (file) [to gin...]
copy(ok): recordings/raw/batch1_2020_batch_20/1488932221.wav (file) [to gin...]
copy(ok): recordings/raw/batch1_2020_batch_20/1504055687.wav (file) [to gin...]
copy(ok): recordings/raw/batch1_2020_batch_20/1556449434.wav (file) [to gin...]
copy(ok): recordings/raw/batch1_2020_batch_20/1558783343.wav (file) [to gin...]
Update availability for 'gin': 75%|█████████████████████████████████████████████████████████▊ | 3.00/4.00 [00:00<00:00, 13.6k Steps/s]Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Connection to gin.g-node.org closed by remote host.█▍ | 185k/513k [00:07<00:20, 15.8k Objects/s]
CommandError: 'git push --progress --porcelain gin master:master git-annex:git-annex' failed with exitcode 128 under /Users/acristia/Documents/zoo-clips
Delta compression using up to 6 threads
CommandError: 'ssh -o ControlPath=/Users/acristia/Library/Caches/datalad/sockets/3beae3e8 [email protected] 'git-receive-pack '"'"'/LAAC-LSCP/zoo-clips.git'"'"''' failed with exitcode 255
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

And this happens when we relaunch it, though of course this may be a connection problem on my end.

On the other end, when we tried to download the clips (with datalad get .), permission errors emerge. @ninohdasilva can you please add an example?

[lucasgautheron]

This sure looks like network errors.
Normally, pushing again will solve the problem;

you can also try git annex copy . --to gin

[ninohdasilva]

Hi, concerning the permission errors, this is what I try to run

ninoh@pop-os:~/STAGES/CHILD JUNE 2021/gitgin/zoo-clips$ datalad get recordings/raw/batch1_2020_batch_19/
Total (0 ok, 58 failed out of 58): 0%| | 0.00/931k [00:01<?, ?B/s][WARNING] Running get resulted in stderr output: git-annex: get: 58 failed

and I get this type of error for every file in the folder :

[ERROR ] not available; Try making some of these repositories available:; 3119dfa8-7c9f-4d8c-9e51-2645ad6633e6 -- git@82709b2ed170:/data/repos/laac-lscp/zoo-clips.git; 419a7fc9-191c-40f9-91a7-51fe6e89460b -- [email protected]:/Documents/zoo-clips; ; (Note that these git remotes have annex-ignore set: origin) [get(/home/ninoh/STAGES/CHILD JUNE 2021/gitgin/zoo-clips/recordings/raw/batch1_2020_batch_19/0004414454.wav)]
get(error): /home/ninoh/STAGES/CHILD JUNE 2021/gitgin/zoo-clips/recordings/raw/batch1_2020_batch_19/0004414454.wav (file) [not available; Try making some of these repositories available:; 3119dfa8-7c9f-4d8c-9e51-2645ad6633e6 -- git@82709b2ed170:/data/repos/laac-lscp/zoo-clips.git; 419a7fc9-191c-40f9-91a7-51fe6e89460b -- [email protected]: /Documents/zoo-clips; ; (Note that these git remotes have annex-ignore set: origin)]

We checked that the version of Datalad is correct and I have also tried to pull with git but there are similar errors.

[lucasgautheron]

Hello,

I did:

$ datalad install [email protected]:/LAAC-LSCP/zoo-clips.git
Clone attempt:   0%|                                                                                                                                          | 0.00/1.00 [00:00<?, ? Candidate locations/s]
Enumerating: 0.00 Objects [00:00, ? Objects/s]

[INFO   ] Scanning for unlocked files (this may take some time)  
$ cd zoo-clips
$ datalad get . -J 8
get(ok): recordings/raw/batch1_2020_batch_19/0047415554.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0054249077.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0004414454.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0045905491.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0020688591.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0191605195.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0209558836.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0115853089.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0153655961.wav (file) [from origin...]                                                                                                                         
get(ok): recordings/raw/batch1_2020_batch_19/0189845676.wav (file) [from origin...]                                                                                                                         
  [48 similar messages have been suppressed]                                                                                                                                                                
action summary:                                                                                                                                                                                             
  get (ok: 58)                                         

And it worked just fine.

Well, except I only got the files into recordings/raw/batch1_2020_batch_19, as the others have not be been pushed to GIN apparently.

[alecristia]

any idea why @ninohdasilva cannot download? perhaps I set up the permissions incorrectly? he's not part of our team on GIN, but instead I directly added him to this repo...

[lucasgautheron]

No, I cannot say much without more information; can he try installing the dataset again like I did ?

If I can make a few recommandations:

• if you're committing/pushing so many files, you should do it with a reliable Internet, so maybe try this from the cluster
• Do you need all the chunks to be split ? It's possible to read bytes from a specific location in a large wav file, and it will make the dataset much faster to install/manage.

[ninohdasilva]

I tried installing like you did :

ninoh@pop-os:~/STAGES/CHILD JUNE 2021$ datalad install [email protected]:/LAAC-LSCP/zoo-clips.git
The authenticity of host 'gin.g-node.org (141.84.41.219)' can't be established.]
ECDSA key fingerprint is SHA256:E35RRG3bhoAm/WD+0dqKpFnxJ9+yi0uUiFLi+H/lkdU.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gin.g-node.org,141.84.41.219' (ECDSA) to the list of known hosts.
[email protected]: Permission denied (publickey,keyboard-interactive).
[ERROR ] Failed to clone from all attempted sources: ['[email protected]:/LAAC-LSCP/zoo-clips.git'] [install(/home/ninoh/STAGES/CHILD JUNE 2021/zoo-clips)]
install(error): /home/ninoh/STAGES/CHILD JUNE 2021/zoo-clips (dataset) [Failed to clone from all attempted sources: ['[email protected]:/LAAC-LSCP/zoo-clips.git']]

As far as I know, we need the chunks to be split unless there is a simple way to split them after the download.

[lucasgautheron]

The Permission denied (publickey,keyboard-interactive) error makes it look like your ssh key has not been properly configured in GIN. Just to make sure, have you done all the steps here ? https://childproject.readthedocs.io/en/latest/reuse.html#setup-your-gin-account.

Re splitting:
It is not necessary to have them split at all in theory; you could just have some file relating each chunk to their position in the source wav, and then do something like:

import librosa

chunk_n = 100 # 100th chunk
offset = chunk_n*0.5
chunk_signal, sample_rate = librosa.load("bigfile.wav", offset = offset, duration = 0.5, sr = None)

It is just a suggestion, you may work with the clips directly if it makes your life easier, BUT operations on the dataset will be slower.

[ninohdasilva]

The problem was indeed the ssh key configuration.

I have successfully downloaded the files now and I will keep your advice in mind if the operations are too slow.

Thank you for your help.