LA Times puzzles are being downloaded from 2018

[jgok]

#173
This issue started recurring again, starting with the week of April 26, 2021.

The LA Times puzzle for April 25, 2021 was downloaded currently, but since then I have been seeing puzzles from 2018 being downloaded.

[merlinstardust]

I'm seeing the same thing again unfortunately

[jdesai61]

On my Samsung Tab A7, I see puzzles from 2020 (since last couple of months) instead of 2021. I was assuming that this is not an app issue (perhaps issue with site where the files are downloaded from) but since I am seeing 2020 puzzles and others are seeing 2018 puzzles, now I am suspecting the app may be a factor too.

[jdesai61]

I looked at LATimesDownLoader.java and did tests and indeed the issue seems to be with the source where files are coming from. The app is downloading the proper filename with year, date, month but the file returned contains 2018, 2020 or 2021 puzzles (depending on the date range).

[jdesai61]

Looks like LATimes has moved to a different platform (PuzzleMe by AmuseLabs) and while one can get to Puzzles using https://cdn4.amuselabs.com/lat/crossword?id=tcaYYMMDD&set=latimes&embed=0 (e.g. https://cdn4.amuselabs.com/lat/crossword?id=tca210726&set=latimes&embed=0). The puzzle is returned as HTML/JS code which will likely have to be parsed to get it into data format that can be used by the app. The old website that this app uses (arkadium.com) seems to be recycling old puzzles for now.

[jbbdude]

LATimes has moved to a different platform (PuzzleMe by AmuseLabs)

I just opened an issue to add support for AmuseLabs/PuzzleMe, which would fix this issue and enable adding many new crossword services. Hopefully the dev or someone can figure out a simple way to scrape the puzzle pages. Unfortunately not every service uses simple puzzle IDs named with dates, but there is also a standardized selector to access the IDs.

[jgok]

Thanks for doing the research on this, but I fear that the developer(s) for this app is no longer active.

[armchairlinguist]

Appreciate the comments here - I was having the same issue (reported to the dev in late April by email) and now don't need to do the code research on what's going on.

I wonder how cruciverb.com is getting the PUZ files - it still has the recent ones. Might be easier to figure that out than parse the result from AmuseLabs.

[jdesai61]

I have found a method to download LATimes puz files and put them in folder where shortyz expects every night. It is not simple, but end result is automated so the crossword just shows up.
WARNING If you are not familiar with Linux commands etc, you may have some difficulty following this

These steps below are from a combination of following websites

https://github.com/thisisparker/xword-dl
https://wiki.termux.com/wiki/Termux:Boot
https://termux.com/
https://www.reddit.com/r/termux/comments/i27szk/how_do_i_crontab_on_termux/
https://wiki.termux.com/wiki/Termux-setup-storage

Step 1:
Install termux app (from here https://play.google.com/store/apps/details?id=com.termux or https://github.com/termux/termux-app/releases)

Step 2:
Launch termux (you may want to read the Wiki https://wiki.termux.com/wiki/Main_Page)

Step 3:
Termux should give you a bash prompt (after enabling proper Android permissions etc). Play around with keyboard (use Ton-screen Tab key for command completion).
Issue following commands at bash prompt (load some packages, enable crond, provide storage permission).

apt update
pkg install python3
pkg install cronie
pkg install termux-services
sv-enable crond
pip3 install xword-dl
termux-setup-storage

The last command will need you to provide Android storage permission and will create folder ~/storage/shared/crosswords will be where your shortyz puzzles will be

Now exit termux app (type exit or kill the app using Android UI) and restart termux for crontab to take effect

Step 4:
If xword-dl installed properly, try to see if it can download by typing
xword-dl lat
This should result in xword-dl printing a message with a filename of puzzle that was downloaded. Now determine where shortyz crosswords folder is by following command (for me it was in ~/storage/shared/crosswords)

ls -l ~/storage/shared
ls -l ~/storage/shared/crosswords
cd ~/storage/shared/crosswords
xword-dl lat -o `date+%F`-LATimes.puz 

###This should put file where shortyz can find it - Now Launch Shortyz and make sure it sees the new puzzle

Step 5:
To automate the above every day add following lines in crontab file using
crontab -e

Lines to add to crontab

### run this every night at 2:30am - your folder location may vary from the one below - must escape percent sign
30 2 * * * (cd ~/storage/shared/crosswords; xword-dl lat -o `date+\%F`-LATimes.puz > ~/xword-dl.log 2>&1)

Step 6:
I just launch the Termux app once when my tablet restarts - but if you want, you can launch Termux on Androd startup by following instructions here
https://wiki.termux.com/wiki/Termux:Boot

Hope this helps someone - it is a bit complicated to setup but end result justifies the effort for me...

[jdesai61]

I have added AmuseLabs support and modified code so LATimes and WaPo use AmuseLabs. I have also submitted a PR (#184). This is a major enough functional fix so that I hope the owner of the repo will be able to spare some time and push a new release to app store. Meanwhile, one can test the changes by downloading a debug APK (at your own risk) from my fork here: https://github.com/jdesai61/shortyz/releases

[jdesai61]

@jgok @jbbdude or others
If someone could try the debug build in above release, it will be appreciated. Here is the direct link: https://github.com/jdesai61/shortyz/releases/download/v4.4.4-debug-beta/shortyz-app-debug.apk.

[jbbdude]

I just tried it out. It seems to download current LA Times and Atlantic puzzles. I played through today's puzzles and checked them against the Atlantic and LA Times sites. They seem to be working right. I'm testing with an Android 10 device; I normally play on a Kindle running Fire OS 7.3.2.2/Android 9 and I'd be happy to test on that moving forward/as this new build/fork matures.

WaPo downloaded the Sunday puzzle for Sunday, and it seems correct based on their web version, but I don't see any of their daily puzzles. I assume adding those puzzles as a separate/new feed (and others I highlighted in my related issue which should probably get merged with this one, and any other PuzzleMe/Amuse Labs puzzles) would be a fairly simple process of swapping in their ID.

Not sure whether it would be worthy of its own issue as it's a whole different set of functionality though related, but this app would get way more flexible by adding an "add a source" option which could take as input an RSS feed of .puz files or a PuzzleMe/Amuse Labs link or ID.

[jdesai61]

Thanks. In the code, WaPo was only enabled for Sunday downloads, so I left it that way. But I will look further. Kindle test will be very useful. If you do have issues and are able to use adb to connect and get debug logs, that will help as I have left in a few log statements in.

Not sure whether it would be worthy of its own issue as it's a whole different set of functionality though related, but this app would get way more flexible by adding an "add a source" option which could take as input an RSS feed of .puz files or a PuzzleMe/Amuse Labs link or ID.

The app supports loading PUZ files emailed thru a gmail account, that may be a workaround instead of RSS feed. Getting people to add links has the potential to become a support nightmare.

[jbbdude]

Actually, I did note one minor bug: Emoji support. Not new, as prior emoji in clues have appeared broken. However, in case it's related to the scraping, The Atlantic's 6 across clue today is "❤️" which appears as tofus/.notdefs, i.e. "⛝⛝" (two white boxes with Xs through them).

[jdesai61]

@jbbdude I have created a new release https://github.com/jdesai61/shortyz/tree/v4.4.4-debug-beta-rel2 with support for old android devices and also to handle unicode - which will show Feb 27 Atlantic Clue for heart as one white box instead of two (due to utf-8 character set limitation)

Direct download link:
https://github.com/jdesai61/shortyz/releases/download/v4.4.4-debug-beta-rel2/shortyz-app-debug-4.4.4-rel2.apk

[ThatGitJer]

Thank you @jdesai61 - I was hoping someone would post a new build of Shortyz. Does this include all the 3rd-party pull requests?

[jdesai61]

I don't have write access to this repo. The debug build is in my fork and only contains my fixes. I don't have ability to create production builds or send app to app stores.

[jdesai61]

I have created a new debug release at https://github.com/jdesai61/shortyz/releases/tag/v4.4.4-debug-beta-rel3
This is to account for some changes AmuseLabs made from May 20th, 2022 onwards obfuscating some data, Can someone please try it out and post any issues here or on my fork at https://github.com/jdesai61/shortyz

[jdesai61]

The AmuseLabs support is broken again as of June 19th 2022. They changed the obfuscation scheme and I don't have the time to incorporate the fix. For those interested, you can use xword-dl project to get AmuseLabs based puzzles and manually copy them to your shortyz folders. I have documented using termux and xword-dl to achieve this in earlier post in this thread,