Basic performance improvements

[gsvarovsky]

Basic performance optimisations – in chaos testing, transactions seem to sometimes take 10ms or more.

In the browser, observing up to 100ms!

Need an example dataset - m-ld/m-ld-spec#17

Current checklist:

• Remove cross-graph quad indexes
• Keep the journal in memory
• Compact IRIs in storage
• Remove 4ms streaming delays in browser gone away with quadstore 7
• Remove transaction hashing (optional, keep relevant code e.g. stringified delta @delete & @insert)
• Other journal compaction options (ticked)
• Non-indexed buffers in quadstore (journal entry body) BACKEND: Bypass literal indexing? quadstorejs/quadstore#119
• Stop using (or fix) jsonld#fromRDF #3 fromRDF
• Remove string-to-binary translation for memdown (and IndexedDB) not obviously an issue any more
• Offer web persistence that is not hobbled by IndexedDB, e.g. streaming from memdown to web cache (https://web.dev/cache-api-quick-guide/)

[gsvarovsky]

Tried removing strange Indexeddb loop in Level.js https://github.com/Level/level-js/blob/cdacd61dba749f638123420a2ea2073e948f34be/index.js#L159

No effect.

[gsvarovsky]

Every quad is entered 6 times into the store as different indexes.

Quadstore uses encoding-down to do JSON value encoding. not any longer

Level.js encodes keys and values as Uint8Arrays.

There is a potential issue with IndexedDB storing the backing buffer: dexie/Dexie.js#665

For the demo, keys and values are approx. 200B.

[gsvarovsky]

Testing with https://github.com/nolanlawson/database-comparison with 50-100B key and 100-300B buffer values, it really does take ~150ms to put 120 entries. No difference using string keys/values.

[gsvarovsky]

https://stackoverflow.com/a/22315353 "I saw 60-70 entries/second"

[gsvarovsky]

Attempted compaction of IRIs in storage.
Branch: https://github.com/m-ld/m-ld-js/tree/compact-iri-storage
Requires: https://github.com/m-ld/node-quadstore/tree/compact-iris
Analysis: https://docs.google.com/spreadsheets/d/1xyYqS4FcB_MCguQSyLrfh1XhfCMOOJpy7_SKKML4BdA/edit?usp=sharing
Bottom Line: no performance benefit. 40% reduction in storage volume.

[gsvarovsky]

Reducing quad count for the journal, by packaging most state and entry properties into a single 'body', has improved transaction apply time by about 20%. However it also made a big difference to transaction prepare time.

[gsvarovsky]

When using Memdown (and maybe others) in the browser, setTimeout(x, 0) is used by the Browserify process shim. This is a performance problem because setTimeout is throttled to 4ms when deeply nested.

https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout#Reasons_for_delays_longer_than_specified

defunctzombie/node-process#86

---

Update: gone away with quadstore 7

[gsvarovsky]

Experiment using work from defunctzombie/node-process#88 removes 4ms delays and shows ~14ms transactions.
https://github.com/m-ld/m-ld-website/tree/node-process-shim
Profile-20200911T084202.json.zip

Significant time is in utf8ToBytes and blitBuffer, which are used to serialise keys and values in memdown. Could fix by using TextEncoder API in the browser feross/buffer#268