Can't resume if download link changes

[iripu]

Your DownThemAll! and Browser Version and Language and Operating System

DTA 3.0.6, Firefox 48.0.2 32bit, WIndows 8.1 core 64bit, en-GB both FF and OS

Expected Behavior

There are many websites that changes download links periodically. The files allow pausing and continue. But the download links stop working after that period. Suppose I am downloading a 1 GB file. Due to some reason I can't complete it and it was paused at 80%. I come back after a day and try to resume it. But, the download link has already stopped working. The file is now available at a different link. DTA will now show an error and all that 800mb will be deleted. I wish DTA to keep it at the paused state and may be show a message that the file is not available at that location now. And if I enter a new download link, it will continue from that 80% just like normal.

Actual Behavior

After error, can't resume download. It deletes whatever amount has been downloaded and deletes the chunk bits information.

Steps To Reproduce

Example download: start downloading any Youtube video, pause at about 10%, close Firefox if you want. After about 12 hrs, try to resume that file. It will show error and you will not be able to continue even if you enter a new download link.

[iripu]

I tried it today with the latest DTA and the issue is still there. It shows 403 error. The only way to solve it right now is to add a new download link as mirror before resuming the download. But, it's not safe as I can't determine whether the link changed before resuming and seeing the error, after which I can't do anything.

LOG FILE
log (403 error).txt

[neurobin]

It's bugging me too.

[Nedimar]

The issue still exists and it is infuriating. Changing DTA to not delete incomplete files shouldn't be that difficult, right?

[BooSN]

Changing DTA to not delete incomplete files shouldn't be that difficult, right?

Sure, why not just submit a patch? Mozilla effectively killed DTA! by announcing removal of XUL. Is it so hard to understand that the motivation to work on a called dead Addon is at point zero?

[444gina]

I was testing DTA and "solution" of resuming broken downloads (when ISP drop connections, change / give dynamic IP, and target server change link of downloading file, can be in "pause local download".

1. So for example frist link of downloading .iso file is:
   http://down2.chinaunix.net/data1/disc/mandriva/2010.0/mandriva-linux-free-2010.0-x86_64.iso?st=9mqRTZO0-m-N6udAKEUWUjw&e=1502713765

2. Downloading file has total size:
   mandriva-linux-free-2010.0-x86_64.iso = 4.3 GB (4612685824 bytes)

3. And DTA reserves whole 4.3 GB (4612685824 bytes) on your local hard drive, even if downloaded only 1 % in file:
   *.dtapart = (always 4.3 GB (4612685824 bytes) even with empty sectors)

4. After process of download connection was dropping , IPS changed IP and on server old link is death. So DTA even lost % progress of downloading file and display as unknown, but on hard drive still exists file:
   *.dtapart = 4.3 GB 4612685824 bytes

5. So renamed file *.dtapart to: mandriva-linux-free-2010.0-x86_64.iso

6. And mounted .iso file to see real downloaded size:
   sudo mount -o loop '/media/3/mandriva/tmp-old3-iso/mandriva-linux-free-2010.0-x86_64.iso' /media/3/mandriva/tmp-old3-iso/DIR

7. Total size of mounted .iso file in directory DIR is about 10%:

mounted mandriva-linux-free-2010.0-x86_64.iso = 464.9 MB (487444480 bytes)

8. But DTA can't resume .iso file download, even if new link on server is added by:
   Right click > Advanced > Manage Mirrors
   and still displays unknown progress

9. So copied partially downloaded .iso file to other local location, and reastarted web browser and DTA, and started local download of file:
   mandriva-linux-free-2010.0-x86_64.iso = whole 4.3 GB (real data size 464.9 MB)
   This create the same *.dtapart = 4.3 GB 4612685824 bytes file and display % progress of download

10. But DTA has disabled "pause" buttons and shortcuts, so there is not possible to pause local download before size 464.9 MB, and there is not possible to pause any local download ( but downloads from internet can be paused).

11. Even if .iso file is downloading from local usb drive , and usb is unpluging from computer, DTA still continue download (empty bytes) to the final value:
    *.dtapart = 4.3 GB 4612685824 bytes

12. So the question is how to enable "pause" local downloads, and then easy pause "real downloaded bytes" ( in this sample before size 464.9 MB) and resume from internet (without losing earlier downloaded bytes)?

Firefox by built downloader also do not pause local downloads, but firefox can resume (broken file without losing earlier downloaded bytes) from internet any partially downloaded file by adding it as:
*.part

But DTA even if add again file:
*.dtapart = 4.3 GB 4612685824 bytes
does not see this file (is unknown size in DTA after whole local download) , and even if download it from local drive, not let to pause that download. And from new link start download again from 0 bytes (doesn't see earlier downloaded bytes).

[mojshlevi]

This is test on local download of 1 file from 2 local locations. If DTA succesfully changes between local mirrors , it should also change local mirror on internet mirror.

TEST 1:

1. Prepare two partitions on hard drive type ext4 (Linux 0x83)

best option for file operations on Linux:

/media/1/local-dir/
/media/2/local-dir/

2. Prepare 2 local mirrors of the same .iso file from two hard drives:

/media/1/local-dir/test1.iso # 436.2 MB (457422848 bytes)
/media/2/local-dir/test1.iso # 436.2 MB (457422848 bytes)

3. Untick (unselect) removing downloaded files !!!

Tools > DownThemAll! Tools > Preferences > Privacy > untick Remove Completed Downloads

4. Select "Diagnostic loggin"

Tools > DownThemAll! Tools > Preferences > Privacy > select Enable Diagnostic loggin

5. Set limt for local download (minimum is 10 KB/s):

Tools > DownThemAll! Tools > Manager > Limit: 10 KB/s

6. Set tmp direstory and download directory in DTA:

/media/1/tmp-dir/ # here will be *.dtapart file
/media/1/download-dir/ # here will be completed (locally downloaded test1.iso file)

So in web browser:

Tools > DownThemAll! Tools > Preferences > Advanced > Save downloads segments in a temporary directory: /media/1/tmp-dir/

Tools > DownThemAll! Tools > Manager > + > Save files in: /media/1/download-dir/

7. Start download from 1st local mirror:

Tools > DownThemAll! Tools > Manager > + > Download: file:///media/1/local-dir/test1.iso

you can change limit of speed to achieve needed size of downloading locally file

then save login file (for steeps till point 7):

cd ~/./.mozilla/firefox/default
cat dta_log.txt > /media/1/dta_log7.txt

8. Drop local downloading connection to test *.dtapart file:

File > select Work Offline

Sometimes DTA displays offline and last % progress but sometimes

DTA can has broken download and unknown % progress, and can remove file *.dtapart

so really would be better to have "local downloads pause" button active

then use browser's "Work Offline"

but if temporary file *.dtapart still exist, recovery can be possible, then continue

then save login file (for steeps till point 8):

cd ~/./.mozilla/firefox/default
cat dta_log.txt > /media/1/dta_log8.txt

9. Copy tmp file to new directory: "resume"

mkdir /media/1/resume/
cp -R /media/1/tmp-dir/*.dtapart /media/1/resume/

10. Copy dtatmp* directory to new directory: "resume"

sudo cp -R /tmp/dtatmp* /media/1/resume/

11. Rename *.dtapart on .iso

mv /media/1/resume/*.dtapart /media/1/resume/test1.iso

12. Mount .iso to estimate real size:

mkdir /media/1/mnt
sudo mount -o loop '/media/1/resume/test1.iso' /media/1/mnt

Then umount:

sudo umount -O loop '/media/1/resume/test1.iso' /media/1/mnt

13. Close/ Remove broken download from DTA:

Tools > DownThemAll! Tools > Manager > Right click > Remove Downloads

You can also close DTA and browser open again (then 1st *.dtapart file will be dalated)

14. Set 1 segment download

Tools > DownThemAll! Tools > Preferences > Advanced > Max. Number of Segments Per Download = 1

15. Set work online:

File > uselect Work Offline

16. Start download from 1st local (resume) mirror:

Tools > DownThemAll! Tools > Manager > + > Download: file:///media/1/resume/test1.iso

Then before real downloaded size add 2nd mirror

17. If you used higher limit, change on smallest 10 KB/s and add 2nd local mirror:

Rigt click (on row with local downloading file) > Advanced > Manage Mirrors > + > Location:
file:///media/2/local-dir/test1.iso

18. Then remove 1st local mirror:

Rigt click (on row with local downloading file) > Advanced > Manage Mirrors > Prioryty: minimum
Click on: file:///media/1/resume/test1.iso and press "-" then OK

But for sure if DTA still not download 1st file (from 1st location) remove it:

sudo rm -rf /media/1/resume/test1.iso

19. If DTA displays % progress of downloadning, you can also test other options / change them like add more downloading segments (multi threads) etc. And finally add unlimited speed for fastest download:

Tools > DownThemAll! Tools > Manager > Limit: Unlimited

20. If in download directory will be final file with correct size (DTA should be OK)

/media/1/download-dir/test1.iso # 436.2 MB (457422848 bytes)

21. But for sure mount final .iso file check size and some files inside if they are the same with origin .iso file:

mkdir /media/1/mnt
sudo mount -o loop '/media/1/download-dir/test1.iso' /media/1/mnt

Even if size of files are the same you must to check if they open correct and are not damaged.

During some tests DTA was creating (downloading) final: test1.iso file with correct size

but some files inside .iso although they had correct size , were damages (not readable).

Sometime DTA also hangs and doesn't want to run after changing miror described in this sample

It looks like DTA store information about downloads not only in files:

/media/1/tmp-dir/.dtapart
~/./.mozilla/firefox/default/dta_log.txt
/tmp/dtatmp

because even fully replacement of these files from previously downloading, not change DTA Manager window, and DTA not displays previous % progress of download, and not see previous *.dtapart file although is replaced in DTA tmp directory.

---

TEST 2:

1 - 10. Points do the same as above

11. BUT NOT Rename *.dtapart on .iso !!!

12. Point omit (do not execute)

13. Close/ Remove broken download from DTA: (closed at 74 % progress)

Tools > DownThemAll! Tools > Manager > Right click > Remove Downloads

12. Point omit (do not execute)

17. Add 2nd local mirror (start new download from this new mirror):

Rigt click (on row with local downloading file) > Advanced > Manage Mirrors > + > Location:
file:///media/2/local-dir/test1.iso

18. Start 2nd mirror local download (start from 0%)

File > unselect Work Offline

19. Set work Offline: (unfortunately pause local download is disabled) stopped at 2%

File > select Work Offline

20. Replace *.dtapart files and names:

mkdir /media/1/resume2/
cp -R /media/1/tmp-dir/.dtapart /media/1/resume2/ &&
rm -rf /media/1/tmp-dir/.dtapart &&
cp -R /media/1/resume/*.dtapart /media/1/tmp-dir/ &&
cd /media/1/resume2/
NAME="$(ls)"
cd /media/1/tmp-dir/
mv * $NAME

21. Start again 2nd mirror local download

File > unselect Work Offline

During tests DTA not see previous size of downloaded file, but continue from level of 2nd paused mirror (so continue from 2% not from 74% of replaced and renamed file *.dtapart)

Even if remove *.dtapart file from /media/1/tmp-dir/ during downloading by DTA , progress still in DTA window show the same downloaded size and recreate *.dtapart file (in fact from 0%) but show in DTA window 2%. It looks that DTA not check real data in file *.dtapart but displays only transfer sent through browser.

But replacement of file *.part in built downloads window of firefox, let on using previously downloaded partially file, and firefox upgrade % progress in downloads window to data size in file *.part

So how user can not lose transfer if DTA even for local download can not continue download from previous *.dtapart file.

Someone can say that user can pause download from internet and then add 2nd mirror, but who stop / pause working process of download?

There is no possibility to resuming data from previously downloaded *.dtapart file especially from broken download, because people are not Holy Spirit to know when it will hapen and to pause before.

File: ~/./.mozilla/firefox/default/dta_log.txt is not helpful because not contain size of downloaded file *.dtapart , so from that log, user do not know what is real size of downloaded (also partially) file.

Other users should also do some local tests to save your time and internet transfer

---

TEST 3:

Solution can be find a way how to extract only data without empty sectors from *.dtapart file
And then that *.dtapart file (converted to real downloaded size) can be use for continue download by:
firefox (renamed to *.part)
wget (renamed to *.iso origin name of downloaded file)

But some tests not found good conversion to extract real size of file *.dtapart

mv *.dtapart 1.iso
cat 1.iso > 2.iso # copy (clone) everything also with empty sectors

mv *.dtapart 1.iso
dd if=1.iso of=2.iso # copy (clone) everything also with empty sectors

file-roller -e /media/1/mnt 1.iso # lose ID of 1.iso and can extract also not completely downloaded files

sudo mount -o loop '1.iso' /media/1/mnt # can show incorrect size of files, and folders which are in fact (not completely downloaded) so they are not readable.

So how user can convert *.dtapart file to file with size of real downloaded data?

??????????

[mojshlevi]

???