Fix/236 (fix Zenodo blocking downloads)

[leoisl]

The main change in this PR is dealing with Zenodo blocking downloads (see #236 for details). This is solved by adding a random sleep time between 0 and (attempt-1) minutes (or 0 and (attempt-1)*60 seconds) and retrying the download for up to 3 times.
Two tests in the EBI cluster succeeded, where Zenodo always blocks downloads after some time. In the first test, Zenodo blocked 85 downloads but the retries were successful. In the second test, 71 downloads were blocked but the retries were also successful. As such I think this PR solves this issue and closes #236.
As a secondary change, this PR also contains some refactor to the download code.

[karel-brinda]

Thanks for the PR, will look at it

[leoisl]

Hey @karel-brinda

I've created a download section in config.yaml and added two new params with default values:

# how many times to retry a download if it fails
download_retries: 3

# how many seconds to wait between retries
download_retry_wait: 10

These parameters are self-explanatory, but now we don't wait a random amount of time. We simply wait the amount of seconds specified in download_retry_wait times the attempt (e.g. for the values above, if a download fail we will try to redownload 3 times. In the first attempt, there is no wait (sleep time 0); in the second attempt, 10 seconds of sleep; in the 3rd, 20 seconds; and in the 4th, 30 seconds).

This approach has worked on EBI cluster, where I had 8 downloads failing due to Zenodo blocks.

Would this now be a fix to #236 or is there sth else that should be changed?

[karel-brinda]

Ok; actually I liked the randomness there as I thought the issue would just get postponed by the waiting time. But if just this solves the problem, then it's probably ok.

[karel-brinda]

Thanks for the fix!!