Investigate possible performance gains of batching of requests for entries

[zspecza]

Intro

We use contentful.js to receive data from the official Contentful Delivery API.

A call to contentful.createClient() returns an instance of an authenticated client, with methods for consuming various data from an individual Contentful Space.

One such method, client.entries(), is used in roots-contentful.

How we fetch entries

First, we read all your defined content_types and their IDs in app.coffee, then we call client.entries({ content_type: <id> }) for each Content Type in parallel, which concurrently fetches all the entries for each individual content type. Due to the nature of this concurrency, the whole process will only take as long as the longest-running client.entries() call.

How we can improve fetching of entries

It's possible to batch all of this into one request using the Contentful Delivery API's search feature (notably, the inclusion flag). By aggregating all content types beforehand, we can then combine them into one HTTP request:

client.entries({ 'content_type[in]': '<content_type_1_id>,<content_type_2_id>' })

Then we can transform the response to split out the entries according to their content type so that we're back to multiple content_types, each with their own set of unique entries.

Potential gains

As of this moment, it is unclear to me if there might be any performance gains by doing this. Since the current requests are small in size and run concurrently, it might be the case that one giant HTTP request might take much longer to complete than several smaller ones that run in parallel. However, there is for sure some HTTP overhead associated with having multiple HTTP requests, so this functionality might be extremely useful for larger projects that consume a lot of Contentful data and are nearing the threshold for API request limits or just want to reduce excessive bandwidth consumption.

Therefore, if this theory proves to be of any benefit, it makes sense to offer a configuration option that alternates between the two request strategies (and choose a sensible default out of the two). This will convolute a bit of internal code and very likely look messy, but it might be well worth the effort.

This only applies to incremental compiles

Bare in mind that as of yesterday (Jan 27th 2016), we finally added the final test for some configurable logic that caches locals in the case of a second compile, for a significant performance gain (up to 5x!) - so this enhancement will only apply to the very first compile in a roots watch, as well as any compiles triggered by roots compile. (as long as you don't set cache to false in the options set for roots-contentful in your project's app.coffee)