- Magento Block Cache & Varnish extension
- Redis Cache & Sessions
- Varnish, ESI & Sessions
- Varnish Session Validation
- Config Cache Regeneration Locking
- FAQ
Few know that Magento out of the box actually doesn't cache any frontend blocks other than Navigation and Footer, which are basically static as they are. This module enhances performance by giving developers a simple interface for caching any block they want, and comes with good default settings.
- Quick & Versatile Performance Boost
- Varnish + ESI support
- High-performance Redis cache & session backends
- Configuration regeneration locking
- Unobtrusive & Future Proof
- Simple Configuration
Install this module either by:
-
Using modman
modman clone [email protected]:madepeople/Made_Cache.git
-
Using composer
-
Downloading a copy from Magento Connect (not always updated)
Most of the configuration is done via layout XML. In general it comes down to choosing which blocks to cache (or not), and which ones to fetch via ESI (or not).
For instance, to cache the products.list block on every cms_page for 7200 seconds:
<layout version="0.1.0">
<cms_page>
<cache>
<name lifetime="7200">products.list</name>
</cache>
<cms_page>
</layout>
These tags exist:
cache
: Used to group which blocks should be cachednocache
: Used to group which block should not be cachedesi (Varnish)
: Generates an ESI tag in place of the blocknoesi (Varnish)
: Excludes a block from the ESI tag generationname
: Used inside the above to determine which blocks should be used
See madecache.xml for more details.
In order to keep the block caching flexible and allow for custom key generation and timeouts, we're using so called Modifier classes. This lets us apply the same cache for the main product list as for a custom block with products in it, for instance. Modifiers typically build the final caching key, which defines how granular the block should be cached.
The default modifiers are:
cacheid
: The core cache id for the specific blockstore
: Cache one version per storecurrency
: Cache differently depending on currencygroupid
: Use the group IDssl
: SSL or no SSL, typically for blocks that include linksblocktype
: Custom built in modifier that uses different methods for different type of core blocks. See Model/Modifier/Blocktyperequest
: Use the request and its parameters
Modifiers are also a nice way to cache differently depending on layout handles and so on.
Set it up like this:
<layout version="0.1.0">
<default>
<cache>
<name modifiers="store currency">block_that_differs_depending_on_store_and_currency</name>
</cache>
<default>
</layout>
Custom modifiers can be defined like this.
In order to have full control over the caching, locking and sessions, I have developed custom implementations of these backends. The existing solutions suffer from locking timeouts and race conditions, as well as need garbage collection. This didn't suit me and gave me strange issues with load balancing.
To enable the cache and/or session handler, edit your local.xml:
<config>
<global>
<!-- ... -->
<!--
<session_save><![CDATA[files]]></session_save>
-->
<session_save><![CDATA[db]]></session_save>
<models>
<core_resource>
<rewrite>
<session>Made_Cache_Redis_Session</session>
</rewrite>
</core_resource>
</models>
<!-- Optional settings with defaults
<redis_session>
<hostname>127.0.0.1</hostname>
<database>2</database>
<prefix></prefix>
<port>6379</port>
</redis_session>
-->
<cache>
<backend>Made_Cache_Redis_Backend</backend>
<!-- Optional settings -->
<backend_options>
<hostname>127.0.0.1</hostname>
<database>0</database>
<prefix></prefix>
<port>6379</port>
</backend_options>
</cache>
<!-- For Enterprise Edition >= 1.11 -->
<full_page_cache>
<backend>Made_Cache_Redis_Backend</backend>
<!-- Optional settings -->
<backend_options>
<hostname>127.0.0.1</hostname>
<database>1</database>
<prefix></prefix>
<port>6379</port>
</backend_options>
</full_page_cache>
<!-- ... -->
</global>
</config>
It's recommended to set up the three different settings on completely different redis instances, since sessions should persist and cache generally shouldn't. Also, cache needs different memory limit/purge settngs. Also, you don't want a cache "FLUSHALL" to remove all sessions.
A custom magento.vcl file is available in the etc/ directory of the module. With Varnish in front and using this VCL, you can harness full page caching.
In order to handle dynamic user-dependent blocks, something called ESI (Edge Side Includes) is used. With this in place, Varnish makes an extra request to the backend for each dynamic block, such as the cart, compared items, etc.
- Use magento.vcl with your Varnish instance and modify its IP settings in the top
- For ESI to work properly, it's a good idea to add
-p esi_syntax=0x1
to the Varnish command line - Set up your Varnish server's IP in System / Configuration / Made People / Cache
- Enable "Varnish" in the Magento Cache Management page
- Flush everything
The layout handle varnish_enabled is added to every request when Varnish is in front.
Because Varnish sits in front of Magento itself, we need to have a way to validate sessions, otherwise Varnish has to pass every request to the backend as soon as a session is in place. Out of the box, this module adds a special cookie AJAX request to the bottom of the page which will send a new session if the visitor doesn't have one. This approach means that there will always be a request to the backend. The effect is that the user gets the idea of a super-fast loading page, but your backend still gets hit and might not be able to handle the thousands of requests you want it to.
It is technically possible for Varnish to check the actual session storage directly in the actual VCL, but this methodThere are different sessions storage mechanisms available for Magento. Each with their own drawbacks:
- Files - Hard to distribute on a network, and has locking issues
- Memcache - Fast, but no persistence and no locking
- MySQL - Has persistence and locking, but performance is subject to all query noise that Magento creates
Apart from these drawbacks, making Varnish to talk to the different storages isn't very straight forward.
Given the above, I have experimented with using Redis:
- Built-in optimistic locking
- In-memory with optional persistence
- A Varnish Redis client exists
- Very fast
We will be using Debian/Ubuntu steps for reference. First of all the sources to a built Varnish package need to exist since we want to build VMODs.
apt-get update
apt-get install build-essential dpkg-dev debhelper libedit-dev libncurses-dev libpcre3-dev python-docutils xsltproc libvarnishapi-dev autoconf automake autotools-dev libtool pkg-config
mkdir -p varnish/out
cd varnish
apt-get -b source varnish
IMPORTANT! If an apt-get upgrade also upgrades varnish, you have to recompile libvmod-curl again, using the whole procedure from apt-get -b source varnish
and forward.
First, install and configure my Redis session backend and make sure it's working. After this, Varnish needs libvmod-redis installed. For reference, here are Debian instructions:
apt-get install libhiredis-dev
git clone https://github.com/brandonwamboldt/libvmod-redis.git
cd libvmod-redis
./autogen.sh
./configure --prefix=$PWD/../out VARNISHSRC=../varnish-*
make
make install
With vmod-redis in place, search for "redis" in the magento.vcl file and uncomment and configure the affected lines. Then just restart Varnish and you should be good to go.
If you have a highly trafficked Magento store with many websites and store views, you're probably very afraid of flushing the cache. The reason for this is the time it takes to run this method combined with the race conditions here and here. The Config model can be rewritten since Magento 1.7 which is nice, but the App model has to be copied into app/code/local/. A version of the App model from 1.9.0.1/1.14.0.1 can be found here.
Also, the bottom of index.php needs to be modified to use the custom Config model, like this:
Mage::run($mageRunCode, $mageRunType, array(
'config_model' => 'Made_Cache_Model_Config'
));
The values of spin_timeout
and lock_timeout
can be adjusted to a level that works with the amount of visitors and the time it takes to regenerate the configuration tree.
So far this is a single instance lock in Redis, which does the job and lets us load balance. For super high performance with load balancing, a distributed lock should be implemented instead.
Hopefully not. Events are used instead of block rewrites, and only one core model is rewritten, in a non-aggressive way. This means that there will be less interference with other modules, and that manual block cache settings are preserved.
That's right. The nice thing with this implementation is automatic ESI tag generation and session invalidation. We try to cache as much as we can without messing with standard installations. It also supports caching ESI requests on a user-level, meaning the majority of the requests come directly from Varnish (super fast).
This project is licensed under the 4-clause BSD License, see LICENSE