Bitnami Varnish for AWS Cloud

Varnish is a web application accelerator (also known as a caching HTTP reverse proxy) that is installed and configured in front of any HTTP and takes care of caching its contents. Varnish is really fast, typically speeding up delivery with a factor of 300-1000x depending on the architecture.

How to block phpMyAdmin, phpPgAdmin and server-status for being cached?

It is advisable to block phpMyAdmin, phpPgAdmin and/or server-status from being cached and public. To do this. add the following lines of code to the end of the default Varnish configuration file at /opt/bitnami/varnish/etc/varnish/default.vcl or in the Varnish configuration file for your application:

sub vcl_recv {
    if (req.url ~ "^/phpmyadmin/.*$" || req.url ~ "^/phppgadmin/.*$" || req.url ~ "^/server-status.*$") {
        return (synth(403, "For security reasons, this URL is only accessible using localhost (127.0.0.1) as the hostname"));
    }
}

How to configure multiple applications with Varnish?

Varnish allows configuring multiple applications in the same server. To do this, follow the steps below:

  • Get the VCL files for each application (in this example, Ghost and WordPress):

     $ sudo cp ghost.vcl /opt/bitnami/varnish/etc/varnish/ghost.vcl
     $ sudo sed -i 's/port = "[^"]*"/port = "80"/g' /opt/bitnami/varnish/etc/varnish/ghost.vcl
         
     $ sudo cp wordpress.vcl /opt/bitnami/varnish/etc/varnish/
     $ sudo sed -i 's/port\s*=\s*"[^"]*"/port = "80"/g' /opt/bitnami/varnish/etc/varnish/wordpress.vcl
    
  • Append some code to the default.vcl file that will load the proper VCL file configuration depending on the application selected. This will depend on how you have your applications stored on the server.

    • If your applications are configured based on a URL prefix such as http://DOMAIN/foo corresponding to the foo application and http://DOMAIN/bar pointing to the bar application, use:

         if ( req.url ~ "^/foo/")
         {
            # /foo configuration
         }
         elsif ( req.url ~ "^/bar/")
         {
          # /bar configuration
         }
      
    • If your applications are configured with different domains such as http://wordpress.example.com pointing to your WordPress blog and http://ghost.example.com pointing to Ghost, use:

         if (req.http.Host == "wordpress.example.com")
         {
            # Wordpress configuration
         }
         elsif (req.http.Host == "ghost.example.com")
         {
           # Ghost configuration
         }
      
    • If your applications are accessible on different ports, use

         if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
                # Wordpress configuration
           } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {?
            # Ghost configuration
         }
      

    In this example, assume the second approach. Create the new configuration by merging the contents of the different sections (vcl_recv, vcl_fetch, vcl_deliver, vcl_**) inside *wordpress.vcl and ghost.vcl:

       sub vcl_recv {
         if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
            # Wordpress vcl_recv configuration
            ...
          } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {?
            # Ghost vcl_recv configuration
            ...
          }}
       sub vcl_fetch {
         if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
          # Wordpress vcl_fetch configuration
    
            ...
          } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {?
            # Ghost vcl_fetch configuration
    
            ...
          }
       }
       sub vcl_deliver {
         if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
            # Wordpress vcl_deliver configuration
            ...
          } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {?
            # Ghost vcl_deliver configuration
            ...
          }
       }
    

    If any of the sections are empty in any of the VCL files, omit them. In addition, if you find additional vcl_ subst sections in addition to vcl_recv, vcl_fetch and vcl_deliver, merge them in a similar way. In this scenario, you end up with:

       # Previous contents of default.vcl
       # ....
       # ....
    
       # Multi-apps configuration:
       # This is a merge of all the different sections of wordpres.vcl and ghost.vcl
           
       sub vcl_recv {
         if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
             # Contents of the section vcl_recv in wordpress.vcl      
             if (req.http.Accept-Encoding) {
               #revisit this list
               if (req.url ~ "\.(gif|jpg|jpeg|swf|flv|mp3|mp4|pdf|ico|png|gz|tgz|bz2)(\?.*|)$") {
                 remove req.http.Accept-Encoding;
               } elsif (req.http.Accept-Encoding ~ "gzip") {
                 set req.http.Accept-Encoding = "gzip";
               } elsif (req.http.Accept-Encoding ~ "deflate") {
                 set req.http.Accept-Encoding = "deflate";
               } else {
                 remove req.http.Accept-Encoding;
               }
             }
             if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
               unset req.http.cookie;
               set req.url = regsub(req.url, "\?.*$", "");
             }
             if (req.http.cookie) {
               if (req.http.cookie ~ "(wordpress_|wp-settings-)") {
                 return(pass);
               } else {
                 unset req.http.cookie;
               }
             }
           } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
             # Contents of the section vcl_recv in ghost.vcl
                    
                    
             # If the client uses shift-F5, get (and cache) a fresh copy. Nice for
             # systems without content invalidation. Big sites will want to disable
             # this.
             if (req.http.cache-control ~ "no-cache") {
               set req.hash_always_miss = true;
             }
             set req.http.x-pass = "false";
             # TODO: I haven't seen any urls for logging access. When the
             # analytics parts of ghost are done, this needs to be added in the
             # exception list below.
             if (req.url ~ "^/(api|signout)") {
               set req.http.x-pass = "true";
             } elseif (req.url ~ "^/ghost" && (req.url !~ "^/ghost/(img|css|fonts)")) {
               set req.http.x-pass = "true";
             }
                
             if (req.http.x-pass == "true") {
               return(pass);
             }
             unset req.http.cookie;
           }
       }
       sub vcl_fetch {
         if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
           # Contents of the section vcl_fetch in wordpress.vcl
             if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true" || req.url ~ "xmlrpc.php") {
               return (hit_for_pass);
             }
             if ( (!(req.url ~ "(wp-(login|admin)|login)")) || (req.request == "GET") ) {
               unset beresp.http.set-cookie;
             }
             if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
              set beresp.ttl = 365d;
             }
           } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
           # Contents of the section vcl_fetch in ghost.vcl
                    
           # Only modify cookies/ttl outside of the management interface.
             if (req.http.x-pass != "true") {
               unset beresp.http.set-cookie;
               if (beresp.status < 500 && beresp.ttl == 0s) {
                 set beresp.ttl = 2m;
               }
             }  
           }
        }
        sub vcl_deliver {
           if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
               # Contents of the section vcl_deliver in wordpress.vcl
                     
               # multi-server webfarm? set a variable here so you can check
               # the headers to see which frontend served the request
               #   set resp.http.X-Server = "server-01";
               if (obj.hits > 0) {
                   set resp.http.X-Cache = "HIT";
               } else {
                   set resp.http.X-Cache = "MISS";
               }
           } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
               # Contents of the section vcl_deliver in ghost.vcl
                  
           }   
        }
    
  • Restart Varnish:

     $ /opt/bitnami/ctlscript.sh restart varnish
    

You should now have both configurations working simultaneously depending on the hostname.

You can download the different files used in the example as a reference:

How to enable Varnish?

Varnish is disabled by default, so it cannot be started using the control script. To enable it, follow the steps below:

  • Rename the Varnish control script from ctl.sh.disabled to ctl.sh:

      $ sudo mv /opt/bitnami/varnish/scripts/ctl.sh.disabled /opt/bitnami/varnish/scripts/ctl.sh
      $ sudo /opt/bitnami/ctlscript.sh start varnish
    
  • Since Varnish with PageSpeed is not currently supported by Bitnami, ensure PageSpeed is disabled. Follow this guide for more information on this.

How to change the Varnish configuration?

Varnish is installed with a default configuration file, agnostic to the Web application being cached. Using this configuration file, although achieving high performance, could lead to some content not being properly refreshed in the Varnish cache. As a result, users would see an outdated version of the site.

The solution is to use a custom VCL configuration file. There are multiple sources on the Internet that provide customized configuration files for different applications. A good source is the Varnish example page.

This section discusses how to change the default configuration file to a WordPress-specific one. Follow the steps below:

  • Obtain the source file here.

  • The file requires some modification to register the port on which your Apache server will be running. This port can be read either from the Apache configuration file at /opt/bitnami/apache2/conf/httpd.conf in the Listen directive:

     ...
     # Change this to Listen on specific IP addresses as shown below to 
     # prevent Apache from glomming onto all bound IP addresses.
     # 
     #Listen 12.34.56.78:80
     Listen 80
     ...
    

    Or by executing this command in a console:

     $ egrep '^Listen ' /opt/bitnami/apache2/conf/httpd.conf
     Listen 80
    

    With this value (80), edit the downloaded file and update the section below with the port number:

     backend default {
         .host = "127.0.0.1";
         .port = "81";
     }      
    
    NOTE: For Bitnami stacks, Varnish is installed on the same server as Apache so the host can be configured as 127.0.0.1. You can also use Varnish to cache a remote server, by providing the host's IP address.
  • Copy the file to the Varnish directory:

     $ cp /path/to/the/wordpress.vcl  /opt/bitnami/varnish/etc/varnish/
    
  • Stop Varnish:

     $ cd /opt/bitnami
     $ ./ctlscript.sh stop varnish
    
  • Modify the Varnish control scripts to load the appropriate file. Edit the file /opt/bitnami/varnish/scripts/ctl.sh and change the VARNISH_CONFIG_FILE variable to point to the new file:

     #! /bin/sh
     ...
     VARNISH_CONFIG_FILE=/opt/bitnami/varnish/etc/varnish/wordpress.vcl
     ... 
    
  • Restart Varnish (and Apache if needed):

     $ cd /opt/bitnami
     $ ./ctlscript.sh start varnish
     $ ./ctlscript.sh start apache
    
IMPORTANT: Varnish will not cache content if Apache's PageSpeed module is enabled. Find out how to disable this module.

How to change the Varnish and Apache ports?

After checking all is working properly, you may want to change the Varnish port to a standard one, usually port 80. If it was free at installation time, it should already be in use by Apache.

Follow these steps:

  • Stop Apache and Varnish:

     $ cd /opt/bitnami
     $ ./ctlscript.sh stop apache
     $ ./ctlscript.sh stop varnish
    
  • The configuration of the ports will involve first changing the Apache port and then the Varnish port. Move Apache to a different port, by editing the Listen directive in the Apache configuration file at /opt/bitnami/apache2/conf/httpd.conf. Find the lines below:

     ...
     # Change this to Listen on specific IP addresses as shown below to 
     # prevent Apache from glomming onto all bound IP addresses.
     # 
     #Listen 12.34.56.78:80
     Listen 80
     ...
    

and change them so that Apache listens on a different port:

    ...
    # Change this to Listen on specific IP addresses as shown below to 
    # prevent Apache from glomming onto all bound IP addresses.
    # 
    #Listen 12.34.56.78:80
    Listen 81
    ...
  • Update your application configuration for Apache. For example, if your applications are configured for virtual hosting, change the port in the application configuration file for Apache as well as in the file at /opt/bitnami/apps/APP-NAME/conf/httpd-vhosts.conf. To know which applications are running under virtual hosts, check the file at /opt/bitnami/apache2/conf/bitnami/bitnami-apps-vhosts.conf, as this file contains the list of applications that you need to update, if any.

  • Configure Varnish to use the old Apache port (80) and specify the new port for Apache (81) in the configuration file. Edit the file at /opt/bitnami/varnish/scripts/ctl.sh and update it so that it looks like this:

     #! /bin/sh  
     ...
     VARNISH_PORT=80
     ...
     VARNISH_CONFIG_FILE=/opt/bitnami/varnish/etc/varnish/default.vcl
     ...
    
  • The Varnish configuration file at /opt/bitnami/varnish/etc/varnish/default.vcl contains the port on which Apache is listening. Update it to reflect the new Apache port (81):

     backend default {
         .host = "127.0.0.1";
         .port = "81";
     }      
    
  • Restart the servers.

     $ cd /opt/bitnami
     $ ./ctlscript.sh restart
    

Apache without caching should now be available at port 81 and Varnish at port 80 as a reverse proxy for Apache.

How to start Varnish?

By default, Varnish is configured to use the first free port after the selected Apache port. This should be enough to do a preliminary test and check all is in place. Follow these steps:

  • Execute these commands at the server console:

     $ /opt/bitnami/ctlscript.sh start varnish
    
  • Ensure Apache is running:

     $ /opt/bitnami/ctlscript.sh start apache
    

Now you should be able to access the index page on both ports 80 and 81.

  • Accessing the server through the 80 port will behave as if Varnish were not enabled, retrieving all the data from the server.

  • Accessing the server through the 81 port will use Varnish as a reverse proxy, serving cached contents and requesting Apache for non-cached content.

How to check if Varnish is working?

Check what Varnish is doing under the hood with the varnishlog command. To indicate which instance of Varnish you are interested in, specify the Varnish working directory which is located by default at /opt/bitnami/varnish/var/varnish/.

  0 CLI          - Rd ping
  0 CLI          - Wr 200 19 PONG 1340840690 1.0
  0 CLI          - Rd ping
  0 CLI          - Wr 200 19 PONG 1340840693 1.0
  0 CLI          - Rd ping
  0 CLI          - Wr 200 19 PONG 1340840696 1.0

If you visit your server URL though the configured Varnish port (81 in our example), you will see more interesting output:

 15 Hash         c /favicon.ico
 15 Hash         c 75.101.208.108
 15 VCL_return   c hash
 15 VCL_call     c pass pass
 15 Backend      c 14 default default
 15 TTL          c 1976586397 RFC 120 -1 -1 1340840847 0 1340840847 0 0
 15 VCL_call     c fetch
 15 TTL          c 1976586397 VCL 120 -1 -1 1340840847 -0
....
 15 TxResponse   c OK
 15 TxHeader     c Server: Apache
 15 TxHeader     c X-Powered-By: PHP/5.3.13
 15 TxHeader     c Content-Type: image/vnd.microsoft.icon
 15 TxHeader     c Content-Length: 0
 15 TxHeader     c Accept-Ranges: bytes
 15 TxHeader     c Date: Wed, 27 Jun 2012 23:47:27 GMT
 15 TxHeader     c X-Varnish: 1976586397

To get a clearer idea of what is happening, use the varnishstat command instead:

Hitrate ratio: 1 1 1
Hitrate avg: 0.0000 0.0000 0.0000
35 0.00 0.02 client_conn - Client connections accepted
23 0.00 0.01 client_req - Client requests received
8 0.00 0.00 cache_miss - Cache misses

The command shows much more information but a clear indication of whether it is working can be obtained by checking the Hitrate ratio (how often Varnish finds the contents in its cache) and the cache_misses (how many times it failed and had to contact Apache).

After browsing the site for a while, you may find something like the below:

Hitrate ratio: 10 62 62
Hitrate avg: 0.9990 0.9677 0.9677
7393 0.00 3.23 client_conn - Client connections accepted
7380 0.00 3.23 client_req - Client requests received
7354 0.00 3.22 cache_hit - Cache hits

There is also a Web page to check Varnish status and obtain information about the configuration, at http://www.isvarnishworking.com/.

What requirements does Varnish have?

Varnish requires a working compiler (such as gcc) to compile its configuration file, which is then dynamically linked into the server process.

aws