Why does Magento need full page caching?

Why does Magento need full page caching?

The Magento Community Edition is a resource intensive application that, in its default configuration, needs a lot of CPU and memory resources to process the PHP scripts, create and execute the database queries and sustain all the Apache processes for all the website assets that must be served to each client.

This guide will explore caching as a way to reduce the memory and CPU requirements of Magento and thereby dramatically increase the amount of pages that a Magento instance is able serve from the same hardware.

What is caching?

Caching is where pre-generated dynamic assets (e.g. a PHP generated HTML pages) and static assets e.g. images, are retained in RAM after the first time they are requested by a client and served from RAM to the next client requesting the same page. Without caching the webserver will run the PHP same scripts and make the same database lookups every time a page is requested. This all requires a great deal of CPU and memory for what is, a great deal of the time, replicated work. When caching is employed the already generated HTML pages, images etc are stored in RAM and served directly to the visitor requesting the same page without the webserver having to do anything thereby saving a large amount of system resources and greatly speeding up the Magento site.

This guide will cover the installation of the excellent caching server Varnish Cache and configuration of the Turpentine extension. This extension enables Magento to fully utilize the Varnish cache and can be configured from within the Magento control panel.

Installation and configuration of Varnish Cache and Turpentine

This guide will uses standard Memset VM2000 miniserver (1xCPU, 1GB RAM, 60GB HDD) running Ubuntu LTS 14.04 64-bit . The server will run a standard LAMP (Linux, Apache, MySQL and PHP) stack which, along with the Varnish server, are installed directly from the Ubuntu repositories.

This guide is written for Magento Community Edition and therefore assumes a working install of Magento Community Edition 1.9.x. Magento. This is available from the Magento download page. The Magento installation guide can be found here.

Varnish Cache

Once the Magento instance is up and running Varnish needs installing and configuring. Varnish, and the nano text editor (needed to edit configuration files), are installed with the following command:

apt-get update
apt-get install varnish nano

Apache and Varnish both need stopping prior to editing their configuration files:

service apache2 stop && service varnish stop

Now Varnish needs to be configured to listen for the incoming page requests instead of Apache. In addition a couple of extra start up parameters need adding. This is all done by editing its initialisation file:

nano /etc/default/varnish

The section that needs editing is the section labelled Alternative 2 and looks like this before the edit:

DAEMON_OPTS="-a :6081 \ 
            -T localhost:6082 \ 
            -f /etc/varnish/default.vcl \ 
            -S /etc/varnish/secret \ 
            -s malloc,256m"

And must look like this after the edit:

DAEMON_OPTS="-a :80 \ 
            -T localhost:6082 \ 
            -f /etc/varnish/default.vcl \ 
            -S /etc/varnish/secret \ 
            -s malloc,256m \ 
            -p esi_syntax=0x2 \ 
            -p cli_buffer=16384"

The changes that need making are that the first line needs the port number changing from 6081 to 80 and the last two lines need adding.

Please pay close attention to the trailing slashes and the final “ at the end of the configuration lines as the config file will be rejected unless they are exactly as shown above.

Varnish can now be started:

service varnish start

Apache now needs to be re-configured to listen for page requests passed to it from Varnish on port 8080. The first file that needs editing is:

nano /etc/apache2/ports.conf

The line:

Listen 80

Need changing to:

Listen 8080

Now the Magneto virtualhost file needs re-configuring. This file will be found in the directory /etc/apache2/sites-available/. The name of the file will depend how you have configured your Apache server. The line:

<VirtualHost *:80>

needs changing to:

<VirtualHost *:8080>

Apache now needs starting:

service apache2 start

Varnish and Apache are now working so visiting your Magento site should work as normal if everything is configured correctly. In order to check that Varnish is indeed working run the following command:

varnishstat

and click around the site. You will see data about the requests passing through the Varnish server.

Varnish is now working and if left at this stage it will dramatically speed up the site. However, with the use of the Turpentine plugin it can be easily optimised further.

Before leaving the terminal take a copy of the contents of the file /etc/varnish/secret which contains the varnish authentication key:

cat /etc/varnish/secret

as we will need it later when configuring the Turpentine plugin.

Now that Varnish is installed, the Turpentine plugin must be installed in order to take full advantage of the Varnish cache.

Turpentine Extension

This is installed via the Magento administration backend at:

System > Magento Connect > Magento Connect Manager

by pasting:

http://connect20.magentocommerce.com/community/Nexcessnet_Turpentine

into the extension key field and installing the plugin. Once the install has completed return to the admin pages.

Now go to the Magento cache management section:

System > Cache Management

And clear the following caches:

• Config
• Layout
• Block HTML

Also, the Full Page Cache needs to be disabled if it is present and enabled. Finally logout and log back in.

Now the Turpentine plugin needs the Varnish authentication key added so that it can communicate with the Varnish server and update its configuration. This is done at:

System > Configuration > Turpentine > Varnish Options > Servers

The authentication key needs copy and pasting into the field marked Varnish Authentication Key and saving.

Note, here the key must be followed with \n immediately after the final character or it will not work. E.g. f46e9d51-e819-4716-9f67-eca89a948fa5\n

Now the Varnish cache needs enabling at:

System > Cache Management

By checking the two boxes next to:

• Varnish Pages
• Varnish ESI Blocks

and enabling them from the Action check list at the top right of the page.

Next the buttons labelled:

• Apply Vanish Config
• Save Varnish Config

need clicking in that order on the same page.

Finally a configuration change need to be made to maximise the cache hit rate. This is made at:

System > Configuration > Turpentine > Caching Option

The options that needs enabling is Normalize User-Agent.

The Varnish cache is now fully configured and optimised for your Magento instance.

Benchmarks

The benchmarking server was a separate machine on the same vLAN as the Magento shop using siege with a populated urls.txt file to simulate a client clicking around the shop. The Magento instance was loaded with the Magento sample data in order to provide reproducible and realistic figures.

Siege was run in benchmark mode for 1 minute at a time. These one-minute tests were run ten times and the results averaged.

Unmodified Magento installation

• Transactions per minute: 2491
• Transaction per second:   42.7

Magneto with Varnish cache and Turpentine extension

• Transactions per minute: 52857
• Transaction per second:   885.3

This represents a ~21x improvement in the number of transaction served. A transaction being any website asses served e.g. HTML page, image etc. Benchmarking is somewhat artificial so the real experience of a busy shop will no doubt be somewhat lower. In addition, bandwidth was not a limiting factor here either due both the Magento server and the benchmarking server being on the same vLAN. The Varnish+Turpentine benchmark required ~5MBps of bandwidth to hit those figures. However, the numbers indicate that the small investment in time to needed to install Varnish and Turpentine are well worth the effort.