Hangs and causes browser lockup (Firefox) when dealing with large compendiums.

[MicahZoltu]

If you try to open a very large compendium (e.g., 1800 monsters imported from D&D Beyond), it will take a very long time. This wouldn't be a big deal if the results were cached, but this load time happens every time you open the Compendium Browser.

Spells:

loadAndFilterItems: 11423ms - timer ended

Feats:

loadAndFilterItems: 11694ms - timer ended

Items:

loadAndFilterItems: 4893ms - timer ended

NPCs:

loadAndFilterNpcs: 21465ms - timer ended

As a short term remediation, making it more clear that it is loading would help. A spinner icon and/or changing the wording from "Loaded 0 items" to "Loading..." would go a long way to letting the user know that something is happening.

For long term, I feel like the best solution is to add a GM button for "update cache" which would load everything, then build multiple indexes, then save the indexes to a permanent location on the server. When users (or GM) load the browser, it would use the appropriate index to search rather than trying to search the actual items. Anytime content is added to the Compendium, the GM would need to refresh the indexes but that is a one-time and fairly rare operation.

[spetzel2020]

Thanks for the testing and timing! I can definitely modify the Loading message to make it clearer what it's doing.

The original behavior of Compendium Browser was that it built in-memory indexes. That behavior caused out-of-memory and other problems on large Compendiums (because the indexes were built on initial Foundry load on the ready hook). The current rewrite "fixed" that by not caching anything.

I agree that a better long-term solution would be to build/save indexes and then update them and I did investigate that approach. However I don't know how to solve several challenges - would welcome any pointers:

• I'm not aware of any hooks that would fire on add/delete/changes to Compendium entries. It's possible that Foundry 0.8 adds something in this area
• I believe that "saving to the server" is not generally possible because of security issues; the closest you can get is to save flags on an entity which doesn't really help. I explored creating a hash table which would look like an entity, but that was beyond my ability.

[spetzel2020]

The architecture you suggest is the original one that caused out-of-memory and server crashes.

I have introduced a Module Setting that limits how many entities are loaded per search; if you exceed that limit you can either increase it, or use the filters to reduce the number queried.

Caching would obviously be an improvement on the current setup; especially if one is successively refining a search. Alternatively not starting the search until you pressed a [Search] button would also work.

[spetzel2020]

Enhancement: Add an optional [Search] button if that option was set in the CB Settings.