Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gphotosdl stops responding to requests from rclone - Google photo transfers drop to zero #4

Open
SteakmanAus opened this issue Jan 15, 2025 · 7 comments
Open

gphotosdl stops responding to requests from rclone - Google photo transfers drop to zero #4

SteakmanAus opened this issue Jan 15, 2025 · 7 comments
Labels
bug Something isn't working

Comments

@SteakmanAus
Copy link

SteakmanAus commented Jan 15, 2025 •
edited by ncw
Loading

First of all great idea - appreciate the effort to get a solution to the google photos issue.

The proxy solution has a few limitations. First and foremost is the speed, with only one transfer at a time and waiting for the proxy browser to pull info it will take weeks to transfer my photos out of google (hence google takeout isnt an option). The multiple tab idea might be worth persuing to make this much faster.

There does seem to be an issue with the proxy browser stopping responding randomly to rclone download requests. To solve this I am running gphotosdl in a bat file that kills and restarts gphotosdl after 15 mins. Thankfully rclone waits patently and reconnects to the proxy once it starts.

Suggest a little more debugging is required to make this more stable. Let me know how I can assist.

I'm using the following command to start rclone which is configured to run gPHOTOS remote as a separate user in Google Cloud:

rclone copy -vvP --disable copy --tpslimit 5  --tpslimit-burst=5 --transfers 10 --checkers 10 --fast-list --gphotos-proxy "http://localhost:8282" gPHOTOS:media/by-month "\\truenas.local\Storage\Photos\Google Photos\media\by-month" 

Transferred:      397.833 MiB / 397.833 MiB, 100%, 0 B/s, ETA 0s
Transferred:          424 / 10444, 4%
Elapsed time:     46m36.5s
Transferring:
-*                     2003/2003-02/Scan0046.jpg: transferring
-*                     2003/2003-02/Scan0051.tif: transferring
-*                2003/2003-02/Scan0052 copy.jpg: transferring
-*                2003/2003-02/Scan0053 copy.jpg: transferring
-*                2003/2003-02/Scan0054 copy.jpg: transferring
-*                2003/2003-02/Scan0055 copy.jpg: transferring
-*                2003/2003-02/Scan0056 copy.jpg: transferring
-*                2003/2003-02/Scan0057 copy.jpg: transferring
-*                2003/2003-02/Scan0058 copy.jpg: transferring
-*                2003/2003-02/Scan0059 copy.jpg: transferring"

For the gphotosdl my batch file looks like this for others benefit:

:loop
start C:\rclone\gphotosdl.exe
timeout /t 900 >null
taskkill /f /im gphotosdl.exe >nul
timeout /t 10 >null
goto loop

Output from gphotosdl seems to be ok when its starts working, however it stops showing download photo notifications when it stops responding and rclone slowing drops transfer speed to 0 B/s as seen above:

2025/01/15 13:12:27 INFO Downloaded photo id=AB0N6ed1-7bpgTi844UACO-N7MO1WyYONI9C355CPk_m4AaG-s-qgrpseL0C3qDgQsBMMVrn1TlPSkfEXaTcNUt9GtUqGQTObA path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\8d2dbd4a-ed58-48a6-82ec-80905eacc555
2025/01/15 13:12:36 INFO got photo request id=AB0N6eet4FmD8w6BdtvJZYhqN5tf-B_kHUOVbjJ4KZXtB2TtRDYe7nl2apfuvB8_1qzpXNxqryqWsW52Zaoidll_B8Qn60vtDQ
2025/01/15 13:12:36 INFO got photo request id=AB0N6edm-Jgnr5sGelQCYeQckcrdp_hDNNMYnbhYZoBGTpxPtMg6BW1yCKR1eguPzHJh7FqMxyPh6Ufp0dn5zMAV3YTmOzMd0w
2025/01/15 13:12:37 INFO got photo request id=AB0N6eeUURDJ6MBqV5dNcTb9QV4Ww3kbp7CVxIkNAyvIYYIfFH795xowoHZL-uOZ9Eakaufgdckl_s_9bPKWy8kEgogypZhs5g
2025/01/15 13:12:37 INFO got photo request id=AB0N6ecOG0PfZgHtQgksE2Z_ImA80O8DL7nCvJBFSu3WfR2rTqCG2-JPLlk8caj4p8kTPOEC_aTYYJXzfK64NREm7uzbZT9bJA
2025/01/15 13:12:37 INFO got photo request id=AB0N6eck26gttFWCwNiAxrUDcJScjRu6IUCljLZCraN8K6aSX83dlAuVZDdbOgSjIophcF4mNt4TFEh25g6BVqk2x9ML7JLBlQ
2025/01/15 13:12:37 INFO got photo request id=AB0N6edyHXawCNlRzZobBXBJtiJvU5OVrT2LkbG0iVpYZsYeqcGJoJ4y9MLHFn9gr2gHf8mpiuaSejpGTgTfapQKNGgTn6KTNQ
2025/01/15 13:12:37 INFO got photo request id=AB0N6edSHlrgCGVQ8_GsKXrke6N34k2Fzqw9XV-LRfXYLl8GjR1oTsnIvZcvHO4w-sZ-1qL4tsxB1M7ySUaoVG1Xt7tUed6DEA
2025/01/15 13:12:37 INFO got photo request id=AB0N6ed3vXKEyn-mH8dxBAbx8X8urj5U-ycq2V5C6N5azRR_y9SPeSyVwj6k4Ro-2wOtiROJULT3lHDTHWeiKg2m3tsMkYXPzA
2025/01/15 13:12:37 INFO got photo request id=AB0N6eeHXzXiXg0u5vPdd3XxvNvfA5r3o9TpXTuZUHMQ-wXF0VZ5LRkWRpTzn43OY-o-MV3yv1l_pYQgD-UkNupQ3mo8Z8ry0Q
2025/01/15 13:12:37 INFO got photo request id=AB0N6ec2EwkKdmRE0VkJk7c0UHBWQuQwenhgsyU1lpzdnDd7vG8p3ANirokZyBSJY8hveTFVi_xoacmh3qOanNCKjmighgi7Ig
2025/01/15 13:12:45 INFO Downloaded photo id=AB0N6eet4FmD8w6BdtvJZYhqN5tf-B_kHUOVbjJ4KZXtB2TtRDYe7nl2apfuvB8_1qzpXNxqryqWsW52Zaoidll_B8Qn60vtDQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\63ecb457-1294-4ec8-97b6-02c37053ae43
2025/01/15 13:12:45 INFO got photo request id=AB0N6eeX8CUl7C41zzSUlpHDkxcvreElPlFegQ27HZllcpPGL7pCbq2NSurNySpwc7B47p2K4Z0trEs_LeajcihCObF8Th0d8w
2025/01/15 13:12:56 INFO Downloaded photo id=AB0N6edm-Jgnr5sGelQCYeQckcrdp_hDNNMYnbhYZoBGTpxPtMg6BW1yCKR1eguPzHJh7FqMxyPh6Ufp0dn5zMAV3YTmOzMd0w path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\960446ea-3a6c-4181-9a67-ccde44eaab5a
2025/01/15 13:12:56 INFO got photo request id=AB0N6efdkD8t0x4psQizPGZ4gjPsBwDWlASE4rT_ySff-y0jp-GtmkYSZZhGLHbQGMYf3je7lrVoWVS7hJ6kozvy0SnQTomvOw
2025/01/15 13:13:01 INFO Downloaded photo id=AB0N6eeUURDJ6MBqV5dNcTb9QV4Ww3kbp7CVxIkNAyvIYYIfFH795xowoHZL-uOZ9Eakaufgdckl_s_9bPKWy8kEgogypZhs5g path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\5cc95267-c591-4abf-b61b-afdc13d80c89
2025/01/15 13:13:01 INFO got photo request id=AB0N6ed-yw6glDK2W1Bg7IFbzPOBWFHgXxy4K73HuRUe1-yfmDuYRYudgIYEI1w2kuoisr6qwxVWgprSLkFArqJMilt3yIzBCQ
2025/01/15 13:13:11 INFO Downloaded photo id=AB0N6ecOG0PfZgHtQgksE2Z_ImA80O8DL7nCvJBFSu3WfR2rTqCG2-JPLlk8caj4p8kTPOEC_aTYYJXzfK64NREm7uzbZT9bJA path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\d457ad76-ed7c-441d-bd95-9418ac2c29dd
2025/01/15 13:13:11 INFO got photo request id=AB0N6eeBYtMQOFDFI59lZ4dPjNW15eSLKQGnIsMSDZXj_w5DorYCB3A4DvPJHNu40nDBIhO5yzQ5xzyTOhSn9mH0tuuwyeYX4w
2025/01/15 13:13:19 INFO Downloaded photo id=AB0N6eck26gttFWCwNiAxrUDcJScjRu6IUCljLZCraN8K6aSX83dlAuVZDdbOgSjIophcF4mNt4TFEh25g6BVqk2x9ML7JLBlQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\49cb23e9-1043-4ed9-9f64-163968dbce7a
2025/01/15 13:13:19 INFO got photo request id=AB0N6eebwYWO6bMC_a8irRRWokufSD_cuGtHd8dSHPswOciefOPClBL7eW_5DPO_IhkO2dADq-tEFlQgPKY4e51UAGnRTEMwZw
2025/01/15 13:13:27 INFO Downloaded photo id=AB0N6edyHXawCNlRzZobBXBJtiJvU5OVrT2LkbG0iVpYZsYeqcGJoJ4y9MLHFn9gr2gHf8mpiuaSejpGTgTfapQKNGgTn6KTNQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\6e49817b-ba7d-4504-ba24-e3d2f5515f7e
2025/01/15 13:13:27 INFO got photo request id=AB0N6ecvoiLZZfj9KKsheM5sEPbf5fc4rdid1WvRHEuMur8oGI4t9ORc7zkD5uFgirIEgfMmdpLyTQm0AyPwrFF24AvA__3z
@ncw
Copy link
Member

ncw commented Jan 15, 2025

First of all great idea - appreciate the effort to get a solution to the google photos issue.

Thanks!

The proxy solution has a few limitations. First and foremost is the speed, with only one transfer at a time and waiting for the proxy browser to pull info it will take weeks to transfer my photos out of google (hence google takeout isnt an option). The multiple tab idea might be worth persuing to make this much faster.

I think we need to make it reliable first, but yes I think multiple tabs is probably the way to go.

There does seem to be an issue with the proxy browser stopping responding randomly to rclone download requests. To solve this I am running gphotosdl in a bat file that kills and restarts gphotosdl after 15 mins. Thankfully rclone waits patently and reconnects to the proxy once it starts.

I didn't see this in my testing. It is possible that something has changed at google.

Suggest a little more debugging is required to make this more stable. Let me know how I can assist.

The best thing to do would be to run gphotosdl with the -show flag and have a look at the browser to see if you see anything unusual when it stops. Look in the javascript console also.

It would be worth running gphotosdl with the -debug flag also - that might give a clue. If you could attach a gphotosdl log with -debug when it stops that will help.

@ncw ncw added the bug Something isn't working label Jan 15, 2025
@SteakmanAus
Copy link
Author

SteakmanAus commented Jan 17, 2025 •
edited
Loading

Command prompt debug output.txt

Ok get an out of memory issue by the look of it but plenty of memory available

Image Image Image Image

@ncw ncw closed this as completed in d43cde5 Jan 17, 2025
@ncw
Copy link
Member

ncw commented Jan 17, 2025

Great - we are getting somewhere :-)

I tried that locally and yes, I can see that the browser memory just keeys going up and up.

I made a small change to the code to make it download each photo in a new tab. This should help with the memory use.

Let me know what you think?

This is half way to downloading multiple photos at once with multiple tabs, but I decided not to go that far just yet.

Please try release v0.0.2 with this fix.

Thanks

@SteakmanAus
Copy link
Author

SteakmanAus commented Jan 18, 2025 via email •
edited
Loading

@ncw
Copy link
Member

ncw commented Jan 18, 2025

Sorry, forgot to push the tag!

The release is here: https://github.com/rclone/gphotosdl/releases/tag/v0.0.2

@SteakmanAus
Copy link
Author

SteakmanAus commented Jan 18, 2025 via email •
edited
Loading

@ncw
Copy link
Member

ncw commented Jan 18, 2025

Can you show me a task manager graph of the memory increasing with the chrome process?

@ncw ncw reopened this Jan 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ncw @SteakmanAus