Add support for streaming ZIP packages over HTTP

[mickael-menu]

Added

Shared

• Support for streaming ZIP packages over HTTP. This lets you open a remote EPUB, audiobook, or any other ZIP-based publication without needing to download it first.

---

⚠️ This does not support LCP-protected publications yet, unless they are stored on the server with the license embedded in the ZIP package.

A significant part of this feature depends on the work done in our fork of ZIPFoundation:

• The access to ZIP data has been moved behind a new DataSource access layer, allowing us to integrate our Resource component.
• The APIs have been updated to be async for compatibility with HTTP requests.

Further optimization was done on the Readium toolkit to reduce the number of HTTP requests sent to the server.

To avoid sending hundreds of consecutive small-range requests when extracting a ZIP entry, we make larger requests and cache the remaining data using a new implementation of BufferingResource. The buffer was set to 512 kB which seems to grant good performance with HTTP requests.

Before streaming each ZIP entry, ZIPFoundation must read the End of Central Directory Record (EOCDR) located at the end of the ZIP archive. ZIPFoundation finds the EOCDR by iterating over each byte from the end of the resource, which generates numerous small requests. To optimize this process, the tail of the ZIP resource is cached using a new TailCachingResource decorator, allowing it to be read only once. The cache size is based on the maximum EOCDR size of 65,557 bytes, as specified in the ZIP specification, plus an additional 76 bytes for the ZIP 64 EOCDR, according to the ZIPFoundation implementation.

Finally, for ZIP packages smaller than 5 MB, which includes most EPUB files, we load them entirely into memory.

☝️ Note that this feature only works with an HTTP server supporting byte range requests.

Here are some URLs that can be used to test this feature:

• An audiobook containing a single audio file of 3+ hours, to test buffering

• A large audiobook split in several chapters

• A small audiobook split in several chapters

• An EPUB