Development | Release v0.22 - 12/12/21

[johanjanssens]

After months of working hard on Joomlatools Server, Pages new sidekick, I’m very happy and proud to announce Joomlatools Pages v0.22. This release brings some ground breaking changes and new features, the major focus of is on making sure that Pages delivers top-notch web vitals performance.

It also brings a key architecture change; moving from a PHP app to a full-stack PHP-centric approach. Combined with Joomlatools Server, Pages is evolving to an open source cloud native web publishing platform, the use of Swoole in Joomlatools Server ensure thats PHP remains front en center in the whole stack while bringing async and multi-threading support, replacing the need for other technologies like Nodejs or Go

Joomlatools Server is a cloud native LA(M)P stack and web publishing platform that is specially tailored for building and running standalone Joomlatools Pages sites and apps. It's supported on macOS, Linux, and Windows (via WSL2).

It’s both a local/remote development environment and a multi-site web server. You can run it locally using Docker Desktop, or remotely using Gitpod, or deploy it on any cloud hosting that supports Docker images: Fly.io, Google App Engine, Google Cloud Run, Digital Ocean App Plaform, AWS Fargate, ...

I'm very happy to announce Pages v0.22, its the first release that offers dedicated support and integrations with Joomlatools Server.

Lots of changes in this release

• Overview
• Releases
• Changelog
• Breaking

Overview

This release concludes all responsive images work I been working on over the past year, there are a number of breaking changes that I all included in this release. The breaking changes are all features that have been moved to Joomlatools Server to be able to make use of the performance and power it offers.

Responsive Images

Joomlatools Pages now offers out-of-the-box-support for responsive images through a new image template filter which transforms a standard html <img> element to a responsive image element.

It provides automatic lazy-loading, preloading of critical images, correct sizing across devices, and automatically supports modern formats with images from any source (including remote images).

This feature doesn't implement the actual image optimisation, it only transforms the image element (when the page is generated) and injects the necessary scripts, headers etc. Image optimisation is provided by using Joomlatools Server or can be provided by an external service.

This feature has been carefully crafted to offer 100% compatibility with all modern browsers. While the code provides forward compatibility for newer browser features such a Client Hints and Native Browser Lazy loading it doesn't rely on them as defaults.

Images on the Web

Images take up 50% of the total bytes on web pages. Half of all images we use on the web are over one megabyte in size, which means they aren't optimised to be displayed on the web. Images have a big impact on Largest Contentful Paint(LCP) as they're often the largest visible element when a page is loaded. LCP is a Core Web Vital that Google will be using in their search ranking very soon.

Nowadays users browse the web using their phones, tablets, and laptops, yet images are still as a one size fits all. For example: sites load a 2000 by 2000 pixel image, but phones are only displaying it as 100 by 100 pixels. Tim Kadlec has estimated that — for users on low-res mobile devices — 72% of image bytes served up by responsive pages are wasted. Responsive pages are sending small-screened users nearly four times as much data as they need.

Furthermore, 30% of images on web pages are outside of the initial viewport, meaning the browser loads images that a user does not see until they scroll further down the page.

Images often don't have a width and height property, causing them to jump around when the page is loaded and 99.7% of images on websites don't use modern image formats like WebP. All this hurts the Cumulative Layout Shift Core Web Vital badly.

In order to use images on web pages in a performant way a lot of aspects have to be considered: size, weight, lazy loading, device preferences, connection quality, ... and modern image formats.

Developers have to set up complicated build tooling to optimise images, however those tools usually don't cover user-submitted images coming from an external data source, making it impossible to optimise all images.

This impossible development task inevitably leads to a frustrating end-user experience. Responsive images in Joomlatools Pages implements a solid solution to performant and optimised images on the web that takes care of all the above.

Features

Pages help reduce the complexity of maintaining and generating multiple image versions with the dynamic image transformation feature. Pages build the dynamic image URLs automatically for you. This means you don't have to pre-create the images, and your images are dynamically resized on-the-fly as needed.

1. Generation of responsive img markup

At it's most basic it is just a drop-in replacement for the HTML <img> element, evolved for the modern web. Turning this

<img src="/images/profile-picture.jpg" width="400" height="400" alt="Profile Picture">

into this (in its most basic form):

<img srcset="/images/profile-picture.jpg?w=400 1x, /images/profile-picture.jpg?w=400 2x, /images/profile-picture.jpg?w=400 2x" width="400" height="400" alt="Profile Picture">

Image dimensions are enforced, allowing browsers to immediately render the space needed for the image instead of having it jump in when loaded, preventing layout shift.

While width and height on the HTML <img> element can cause issues with responsive layouts, this is not the case when using Pages. The image is automatically made responsive based on the aspect ratio using the actual image size.

2. Smart responsive breakpoint calculation

Having too many image versions reduces the number of CDN cache hits for requested images and increases the average delivery time for images to your users. On the other hand, having too few image versions means delivering images to users that are larger than needed for the available width, and end up being scaled down by the browser.

To solve this Pages calculates the breakpoints for you based on the actual image filesize. It calculates the optimal number of versions needed for every image balancing the number of image versions generated vs. the file size reduction between each version. The set of breakpoints are thus calculated based on a difference in the actual image file size at different widths.

3. Lazy image loading

Images are automatically lazy-loaded using the Lazysizes js library meaning they're only rendered when the user is close to seeing the image. This prevents loading that 30% of images outside of the initial viewport.

4. Preloading images

You can mark images that are in the initial viewport to be automatically preloaded. Preloading images in the initial viewport has shown improvements to the Largest Contentful Paint by up to 50%.

5. Low Quality Image placeholder (LQI)

Pages can generate and insert a LQI placeholder image that will be loaded. LQIP’s logic is simple. In a sense, this is like loading progressive JPEGs. Initially load the page with low quality images. Once the page isloaded, replace them with the full quality images.

LQIP gives us the best of both worlds. On a slow connection, the user will load a fully usable page much faster, giving them a significantly better user experience. On a fast connection, the extra download of the low quality images – or the delay of the high quality ones – does not lead to a substantial delay. In fact, even on the fast connection the user will get a usable page faster, and they’ll get the full rich visuals a moment later.

6. Automatic Image Optimisation

Pages can automatically generate smaller sizes through built-in Image Optimisation by deploying it using Joomlatools Server. Instead of optimising images at build time, images are optimised on-demand, as users request them. Unlike static site generators and static-only solutions, your build times aren't increased, whether shipping 10 images or 10 million images.

Built-in Image Optimisation automatically serves the images in modern image formats like WebP, which is about 30% smaller than JPEG, if the browser supports it. It also allows to automatically adopt future image formats and serve them to browsers that support those formats.

Image Optimisation works with any image source. Even if images come from an external location they can be optimised, do take note that using this feature on remote images requires additional HTTP requests and will slow down page generation considerably, especially if the pages has multiple remote images.

7. Adapts to users device capabilities and preferences

Image Optimisation will consider device capabilities and user preferences to serve the most optimised image possible for the context of the device and or user. It makes use of:

• Save-Data
• The connection type

Rules:

• If the connection is 2g only lazy load nearby images (just outside of the viewport), and the image quality is reduced 2x
• If save-data is enabled only load visible images and the image quality is reduced 3x

Getting started

To use the image filter simple add the filter to your page or layout:

---
@layout: template://pages/document.html
@process:
    filters:
        - image
---

This is all you need to do, all images generated for the specific page, or layout will now be transformed to responsive image. Pages makes a difference between two types of images:

Fixed image size

A fixed size image require either the width or the height (or both) attribute(s) to be specified. Pages will then generate a responsive img element using srcset display density descriptors.

For example:

<img src="/images/profile-picture.jpg" width="400" alt="Profile Picture">
<img src="/images/profile-picture.jpg" height="400" alt="Profile Picture">
<img src="/images/profile-picture.jpg" width="400" height="400" alt="Profile Picture">

Flexible image size

For flexible images Pages generates srcset’s width descriptors using automatic generated sizes.

To optimise bandwidth as much as possible, the width descriptors are calculated by downscaling the image based on its actual filesize in steps of 30%, taking into account a max and min image size which can be configured.

Transparent images

To optimise bandwidth it best to generate none transparent images images as they are generally smaller than transparent images ones. To generate a none transparent images from a transparent image you need to specify the background colour to be used when generating the image, otherwise the background will be black.

<img src="/images/profile-picture.png?bg=white" width="400" alt="Profile Picture">

or

<img src="/images/profile-picture.png?bg=FFFFFF" width="400" alt="Profile Picture">

Joomlatools Server accepts any hex, rgb or css color name

Configuration

To provide additional configuration you either add additional data-[option] attributes to the <img> element or you can wrap a html fragment into a <ktml:images> block, and specific additional config option through the element attributes.

• lazyload: true|false|progressive|inline
• preload: true|false
• width: [int]
• max-width: [int]
• height: [int]
• max-height: [int]
• max-dpr: [int]

If lazyload="progressive"a Low Quality Image (LQI) will be generated and inserted.
If lazyload="progressive,inline"a Low Quality Image (LQI) will be generated and transformed into a data uri

For example:

<img src="/images/foo.jpg"  width="400" data-lazyload="progressive,inline" />

 <ktml:images lazyload="progressive,inline" width="400">
     <img src="/images/foo.jpg" / >
     <img src="/images/bar.jpg"  />
 </ktml:images>

Video Player

Joomlatools Pages now includes video player support through a new video template filter which transforms a standard html <video> element to a a full fledged video player. The template filter makes use of the popular https://plyr.io/ video player.

This feature doesn't provided video streaming, videos are still downloaded before they can be played. Video streaming using byte serving is provided through Joomlatools Server or adaptive bitrate streaming can be provided by any other external service.

Features

1. HTML5 video player

Plyr is a simple, lightweight, accessible and customisable HTML5, YouTube and Vimeo media player that supports modern browsers. For a complete list of features and customisation options please check: https://github.com/sampotts/plyr#features

2. Lazy video loading

The video player is automatically lazy-loaded using the Lazysizes js library meaning that it's only rendered when the user is close to seeing the video player. This prevents loading videos that are outside of the initial viewport.

3. Poster image support

Pages can generate an optimised poster image by making use of the features provided by Joomlatools Server. For more info please also see: #812

4. Adapts to users device capabilities and preferences

Pages will consider device capabilities and user preferences to serve the most optimised video format possible for the context of the device and or user. It makes use of:

• Save-Data
• The connection type

Rules:

• If the connection is 2g load lowest quality video (*)
• If save-data is enabled load lowest quality video (*)

(*) Requires multi quality videos to be available.

Getting started

To use the image filter simple add the filter to your page or layout:

---
@layout: template://pages/document.html
@process:
    filters:
        - video
---

Static HTTP Caching

The static HTTP cache support has been further improved and the implementation has been replaced by middleware in Joomlatools Server. The middleware uses Swoole and implements a cache proxy that understand the Pages http cache and removes the need to store additional static files and setup additional and complex Apache rewrite rules.

Joomlatools Pages active HTTP cache uses a file based storage implementation modelled after Apache's object cache module. The headers and bodies of cached responses are combined, and stored underneath in single file, where the filename is derived from the hash of the cached URL.

The static cache implements an RFC 7234 compliant content filter and provides a mechanism to verify whether stale or expired content is still fresh, and represent a significant performance boost by honouring the If-None-Match and Last-Modified HTTP request headers.

The Joomlatools Pages active HTTP cache is perpetual, meaning that Pages are only regenerated from scratch when their content has changed, and not when the cached entry expires.

In Joomlatools Server the static HTTP cache is placed in front of content originating from FastCGI. The cache is inserted as as far forward as possible bypassing all per request processing when returning content to the client. The cache may be thought of as a caching proxy server bolted to the front of the FasCGI service, while running within the Swoole HTTP server itself.

The static HTTP cache is little more basic then the Pages active cache, if it cannot figure out if a request can be served from cache it will forward the request through FastCGI to Pages which will handle it, if Pages decides that the cache is no longer valid the page will be regenerated.

Performance

The performance benefits remains the same. When running in Joomlatools Server, you now have 2 levels of caching:

• Joomlatools Server static HTTP cache: +/- 25msec TTFB
• Joomlatools Pages active HTTP cache: +/- 100msec TTFB

Distributed CDN

A great benefit of this new approach is that Joomlatools Server offers out of the box support for Fly.io's multi-region deployments, which allows you to very easily build your own active CDN!

Limitations

At present nor the Pages active and the Server static HTTP cache offer support for the caching of content negotiated responses containing the Vary header and for handling stale content as per RFC 5861. Following is a list of headers not supported by the HTTP cache:

• Vary:
• Stale-While-Revalidate
• Stale-If-Error

Active Cache Revalidation

The Mason Cache Revalidator which has been deprecated and removed and a new revalidate API endpoint: /__api/revalidate to handle on demand cache revalidation and regeneration has been introduced in Joomlatools Server.

Cache revalidation is the process of checking if the cache is stale or invalid, and if so regenerate the cache, or purge it if the page was deleted.

The cache revalidator requires that you use Joomlatools Server to revalidate the Pages cache, you however don't need to run Pages inside Joomlatools Server, support for remote cache revalidation is also available.

Cache JSONAPI endpoint

The cache revalidator communicates with a JSONAPI endpoint /cache.json that exposes the Pages HTTP cache through the cache collection for introspection. By default the endpoint path is /cache.json

To create the endpoint add a file called /joomlatools-pages/pages/cache.json using the following frontmatter.

---
@collection:
    model: cache
    state:
        sort: date
        order: desc
        limit: 0
@process:
    cache: false
---

Make sure you have caching enabled for your site by setting the http_cache => true in your site configuration

Introspect your cache by visiting http://localhost:8080/cache.jsonto make sure everything is working. To easily view the JSON output you can install the Chrome JSON Viewer plugin

Usage

The cache revalidator is implemented as an HTTP API. It can be triggered using an HTTP client. Example using curl:

curl -X "POST" "http://localhost:8080/__api/revalidate" \
     -H 'Authorization: Bearer [API_KEY]' \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
     --data-urlencode "concurrency=10" \
     --data-urlencode "force=false" 

Modes

The revalidator offers two validation modes: selective (default) and forced. In selective mode cache is revalidated only for pages that are no longer valid, eg who's data has changed. In forced mode pages are revalidated regardless if they are valid or not.

If you make a data change, e.g. a change to your content or change to one of the collections used by a page the page will be marked invalid. However, if you make code changes, like a change to a data file, partial, layout or theme asset a forced revalidation is necessary to update the cache.

To use forced mode add the force=true parameter, or change the config default.

Examples

Revalidate all pages

curl -X "POST" "http://localhost:8080/__api/revalidate" \
     -H 'Authorization: Bearer [API_KEY]' 

Regenerate all pages

curl -X "POST" "http://localhost:8080/__api/revalidate" \
     -H 'Authorization: Bearer [API_KEY]' \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
     --data-urlencode "force=true" 

Send revalidation requests to a remote

This allows to perform cache revalidation on a remote server. For example in a scenario where you run a dedicated server to handle cache revalidation for other remotes.

curl -X "POST" "http://localhost:8080/__api/revalidate" \
     -H 'Authorization: Bearer [API_KEY]' \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
       --data-urlencode "base=https://foo.com"

Documentation

For more info about the HTTP parameters and more examples please see: joomlatools/joomlatools-server#46

Releases

Released v0.21.0 v0.22.3 More info:

• v0.22.0: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.22.0
• v0.22.1: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.22.1
• v0.22.2: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.22.2
• v0.22.3: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.22.2

Changelog

Responsive Images and Video support

1. Responsive image support #812

Implemented responsive image support through a new image template filter which transforms a standard html <img> element to a responsive image element. The template filter offers following features:

• Generation of responsive img markup
• Smart responsive breakpoint calculation
• Lazy image loading
• Preloading images
• Low Quality Image placeholder (LQI)
• Automatic Image Optimisation
• Adapts to users device capabilities and preferences

To use the image filter simple add the filter to your page or layout:

---
@layout: template://pages/document.html
@process:
    filters:
        - image
---

Note: This feature doesn't implement the actual image optimisation, it only transforms the image element (when the page is generated) and injects the necessary scripts, headers etc. Image optimisation is provided by using Joomlatools Server or can be provided by any other external service.

2. Asset prefix support for remote image base paths #836

Make it possible to use asset prefixes in remote origin base paths. For example:

'com:pages.template.helper.image' =>
 [
      'origins' => [
           'https://s3.amazonaws.com' => 'images://s3',
      ],
],

2. Video player #816

Implemented video player support through a new video template filter which transforms a standard html <video> element to a full fledged video player. The template filter makes use of the popular https://plyr.io video player. The template filter offers following features:

• HTML5 video player
• Lazy video loading
• Poster image support
• Adapts to users device capabilities and preferences

To use the image filter simple add the filter to your page or layout:

---
@layout: template://pages/document.html
@process:
    filters:
        - video
---

Note: This feature doesn't implement the actual video streaming, it only transforms the video element (when the page is generated) and injects the necessary scripts, headers etc. Video streaming is provided by using Joomlatools Server or can be provided by any other external service.

Caching

1. Add timing info to http caching #834

Added a timing attribute to the http cache to make it easier to find possible rendering bottlenecks by introspecting the cache. The timing stores the total time it took to generate the cache in mili-seconds.

2. Asset versioning and cache busting #814

Added an asset versioning filter for theme:// and media:// assets. The filter will calculate a content based version hash and add it to the asset url as version=[hash]. To use it simply include the filter in a page or layout as follows:

---
@layout: template://pages/document.html
@process:
    filters:
        - version
---

Other

1. Upgrade email processor for PHPMailer 6.5 support #841

The email processor has been refactored to support PHPMailer 6.5.x. Support for older versions of PHPMailer has been deprecated.

Also added a new debug config option to the processor to allow enabling debug and logging, by default this is disabled. To enable:

'com:pages.controller.processor.email' => [
      'debug' => \PHPMailer\PHPMailer\SMTP::DEBUG_LOWLEVEL,
 ],

The logs are stored in the log directory you have configured for your site, by default this is /joomlatools-pages/logs/phpmailer.log

For more info about PHPMailer debugging please see: https://github.com/PHPMailer/PHPMailer/wiki/SMTP-Debugging

Note: Joomla 3 still uses PHPMailer 5.2. To use this processor with Joomla 3, make sure to install PHPMailer 6.5 using composer.

Breaking

This release removes number of features that are now implemented by custom middleware in [Joomlatools Server]. To achieve this Joomlatools Server makes use of Swoole which allows to offer a more powerful and more performant solution compared to implementing these features through FastCGI.

1. Remove support for PHP built-in web server #818

Removed support for the PHP build-in web server. To make full use of Pages it's advised to use Joomlatools Server instead.

2. Remove static cache support #820

Removed the static cache support. The implementation has been replaced by the cache middleware in Joomlatools Server which offers the same performance benefits without the need to store an additional cache and setup Apache rewrite rules.

3. Remove mason cache revalidator #825

The Mason cache revalidator has been removed and replaced by the revalidate API endpoint in Joomlatools Server