generalmastodon

Configure and use Varnish(TM)

Varnish(TM) is a web application accelerator (also known as a caching HTTP reverse proxy) that is installed and configured in front of any HTTP server and takes care of caching its contents. Varnish(TM) is fast, typically speeding up delivery with a factor of 300-1000x depending on the architecture.

Varnish(TM) requires a working compiler (such as gcc) to compile its configuration file, which is then dynamically linked into the server process.

Apache

Enable Varnish(TM)

Varnish(TM) is disabled by default, so it cannot be started using the control script. To enable it, follow the steps below:

  • Enable the Varnish(TM) service:

      $ [ -f /opt/bitnami/scripts/varnish/start.sh.disabled ] && sudo mv /opt/bitnami/scripts/varnish/start.sh.disabled /opt/bitnami/scripts/varnish/start.sh
  $ sudo mv /etc/monit/conf.d/varnish.conf.disabled /etc/monit/conf.d/varnish.conf
  $ sudo gonit reload

  • Since Varnish(TM) with PageSpeed is not currently supported by Bitnami, ensure PageSpeed is disabled. Follow this guide for more information on this.

Disable Varnish(TM)

IMPORTANT: Before disabling Varnish(TM), make sure that all services are stopped and that Apache is running on port 80 as described in the instructions to customize Varnish(TM) guide.

In some cases it is necessary to disable Varnish(TM). One example would be when trying to force HTTPS redirection with Apache (although you can also configure Varnish(TM) with SSL). To disable Varnish(TM), follow these steps:

  • Stop the Varnish(TM) service:

      $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • Disable the Varnish(TM) service:

      $ sudo mv /opt/bitnami/scripts/varnish/start.sh /opt/bitnami/scripts/varnish/start.sh.disabled
  $ sudo ln -s /bin/true /opt/bitnami/scripts/varnish/start.sh
  $ sudo mv /etc/monit/conf.d/varnish.conf /etc/monit/conf.d/varnish.conf.disabled
  $ sudo gonit reload

  • Restart Apache:

      $ sudo /opt/bitnami/ctlscript.sh start apache

Start Varnish(TM)

By default, Varnish(TM) is configured to use TCP port 81. Follow these steps to start the Varnish(TM) service:

  • Execute these commands at the server console:

      $ sudo /opt/bitnami/ctlscript.sh start varnish

  • Ensure Apache is running:

      $ sudo /opt/bitnami/ctlscript.sh start apache

Now you should be able to access the index page on both ports 80 and 81, for Apache and Varnish(TM), respectively.

  • Accessing the server through the 80 port will behave as if Varnish(TM) was not enabled, retrieving all the data from the server.

  • Accessing the server through the 81 port will use Varnish(TM) as a reverse proxy, serving cached contents and requesting Apache for non-cached content.

Check Varnish(TM) status

Check what Varnish(TM) is doing under the hood with the varnishlog command. To indicate which instance of Varnish(TM) you are interested in, specify the Varnish(TM) working directory which is located by default at /opt/bitnami/varnish/var/varnish/.

0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840690 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840693 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840696 1.0

If you visit your server URL though the configured Varnish(TM) port (81 in our example), you will see a more interesting output message:

15 Hash         c /favicon.ico
15 Hash         c 75.101.208.108
15 VCL_return   c hash
15 VCL_call     c pass pass
15 Backend      c 14 default default
15 TTL          c 1976586397 RFC 120 -1 -1 1340840847 0 1340840847 0 0
15 VCL_call     c fetch
15 TTL          c 1976586397 VCL 120 -1 -1 1340840847 -0
....
15 TxResponse   c OK
15 TxHeader     c Server: Apache
15 TxHeader     c X-Powered-By: PHP/5.3.13
15 TxHeader     c Content-Type: image/vnd.microsoft.icon
15 TxHeader     c Content-Length: 0
15 TxHeader     c Accept-Ranges: bytes
15 TxHeader     c Date: Wed, 27 Jun 2012 23:47:27 GMT
15 TxHeader     c X-Varnish: 1976586397

To get a clearer idea of what is happening, use the varnishstat command instead:

NAME CURRENT CHANGE AVERAGE AVG_10 AVG_100 AVG_1000
MGT.uptime 0+00:07:02
MAIN.uptime 0+00:07:03
MAIN.sess_conn 8 0.00 0.02 0.03 0.01 0.01
MAIN.client_req 197 0.00 0.47 0.03 0.01 0.01
MAIN.cache_miss 7 0.00 0.02 0.03 0.01 0.01

The command shows much more information but a clear indication of whether it is working can be obtained by checking the MAIN.backend_reuse (how often Varnish(TM) finds the contents in its cache) and the MAIN.cache_miss (how many times it failed and had to contact the web server).

After browsing the site for a while, you may find something like the below:

NAME CURRENT CHANGE AVERAGE AVG_10 AVG_100 AVG_1000
MGT.uptime 0+00:19:34
MAIN.uptime 0+00:19:35
MAIN.sess_conn 26 0.00 0.03 0.00 0.08 0.03
MAIN.client_req 593 0.00 0.63 0.04 1.77 0.65
MAIN.cache_hit 6 0.00 0.01 0.00 0.03 0.03
MAIN.cache_hit_grace 3 0.00 0.00 0.00 0.00 0.00
MAIN.cache_miss 12 0.00 0.01 0.00 0.02 0.01
MAIN.backend_conn 21 0.00 0.02 0.00 0.05 0.02
MAIN.backend_reuse 569 0.00 0.60 0.04 1.70 0.62

Use Varnish(TM) with SSL

Varnish(TM) is not compatible with HTTPS and needs an SSL terminator in front of it. However, it is possible to configure Apache to proxy all HTTPS requests to Varnish(TM). Follow the steps below:

  • Enable and start Varnish(TM).

  • Open the /opt/bitnami/apache/conf/bitnami/bitnami-ssl.conf Apache HTTPS virtual host configuration file in a text editor:

  • Modify the Apache HTTPS virtual host (which uses port 443 by default) and add the lines shown below to proxy all its requests to Varnish(TM).

      <VirtualHost _default_:443>
      ...
      ProxyPreserveHost On
      RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
      ProxyPass "/"  "http://127.0.0.1:81/"
      ProxyPassReverse "/"  "http://127.0.0.1:81/"

      ...

      # Bitnami applications installed with a prefix URL (default)
      # Include "/opt/bitnami/apache/conf/bitnami/bitnami-apps-prefix.conf"
  </VirtualHost>

NOTE: If the default Varnish(TM) port has been modified to port 80, adjust the configuration shown above and replace all references to port 81 with port 80.

  • Add the same lines in any file ending with the prefix -https-vhost.conf in the /opt/bitnami/apache/conf/vhosts/ directory.

  • Configure the application to use the correct domain name. The steps to do this will vary for each application. As an example, if you are using WordPress, you would need to follow these steps.

  • Restart Apache:

      $ sudo /opt/bitnami/ctlscript.sh restart apache

Customize Varnish(TM)

Varnish(TM) is installed with a default configuration file, agnostic to the Web application being cached. Using this configuration file, although achieving high performance, could lead to some content not being properly refreshed in the Varnish(TM) cache. As a result, users would see an outdated version of the site.

The solution is to use a custom VCL configuration file. There are multiple sources on the Internet that provide customized configuration files for different applications. A good source is the Varnish(TM) example page.

This section discusses how to change the default configuration file to a WordPress-specific one. Follow the steps below:

  • Obtain the source file here.

  • The file requires some modification to register the port on which your Apache server will be running. This port can be read either from the Apache configuration file at /opt/bitnami/apache/conf/httpd.conf in the Listen directive:

      ...
  # Change this to Listen on specific IP addresses as shown below to
  # prevent Apache from glomming onto all bound IP addresses.
  #
  #Listen 12.34.56.78:80
  Listen 80
  ...

    Or by executing this command in a console:

      $ egrep '^Listen ' /opt/bitnami/apache/conf/httpd.conf
  Listen 80

    With this value (80), edit the downloaded file and update the section below with the port number:

      backend default {
      .host = "127.0.0.1";
      .port = "80";
  }

    NOTE: For Bitnami stacks, Varnish(TM) is installed on the same server as Apache so the host can be configured as 127.0.0.1. You can also use Varnish(TM) to cache a remote server, by providing the host’s IP address.

  • Copy the file to the Varnish(TM) directory:

      $ cp /path/to/the/wordpress.vcl  /opt/bitnami/varnish/etc/varnish/

  • Stop Varnish(TM):

      $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • Configure Varnish(TM) to load the appropriate file:

    • Edit the file at /opt/bitnami/scripts/varnish-env.sh to add a new line specifying the path to the new configuration file:

        export VARNISH_CONF_FILE="/opt/bitnami/varnish/etc/varnish/wordpress.vcl"

  • Restart Varnish(TM) (and Apache if needed):

      $ sudo /opt/bitnami/ctlscript.sh start varnish
  $ sudo /opt/bitnami/ctlscript.sh start apache

IMPORTANT: Varnish(TM) will not cache content if Apache’s PageSpeed module is enabled. Find out how to disable this module.

Modify the default Varnish(TM) and Apache ports

After checking all is working properly, you may want to change the Varnish(TM) port to a standard one, usually port 80. If it was free at installation time, it should already be in use by Apache. Follow these steps to change the default port number:

  • Stop Apache and Varnish(TM):

      $ sudo /opt/bitnami/ctlscript.sh stop apache
  $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • The configuration of the ports will involve first changing the Apache port and then the Varnish(TM) port. Move Apache to a different port, by editing the Listen directive in the Apache configuration file at /opt/bitnami/apache/conf/httpd.conf. Find the lines below:

      ...
  # Change this to Listen on specific IP addresses as shown below to
  # prevent Apache from glomming onto all bound IP addresses.
  #
  #Listen 12.34.56.78:80
  Listen 80
  ...

    and change them so that Apache listens on a different port:

      ...
  # Change this to Listen on specific IP addresses as shown below to
  # prevent Apache from glomming onto all bound IP addresses.
  #
  #Listen 12.34.56.78:81
  Listen 81
  ...

  • Update your application configuration for Apache. For example, if your applications are configured for virtual hosts, change the port in the Apache /opt/bitnami/apache/conf/httpd.conf configuration file, the /opt/bitnami/apache/conf/bitnami/bitnami.conf file as well as any application-specific virtual hosts:

    • Edit any virtual host file inside the /opt/bitnami/apache/conf/vhosts/ directory.

  • Configure Varnish(TM) to use the old Apache port (80) and specify the new port for Apache (81) in the configuration file.

    • Edit the file at /opt/bitnami/scripts/varnish-env.sh to add a new line to specify the value for the new port number:

        $ echo "export VARNISH_PORT_NUMBER=80" | sudo tee -a /opt/bitnami/scripts/varnish-env.sh

  • The Varnish(TM) configuration file at /opt/bitnami/varnish/etc/varnish/APPNAME.vcl contains the port that Apache is listening on. Update it to reflect the new Apache port (81):

    NOTE: Replace the APPNAME.vcl placeholder with the appropriate value for the Varnish(TM) configuration file name.

      backend default {
      .host = "127.0.0.1";
      .port = "81";
  }

  • Restart the servers.

      $ sudo /opt/bitnami/ctlscript.sh restart varnish

Apache without caching should now be available at port 81 and Varnish(TM) at port 80 as a reverse proxy for Apache.

NOTE: You can disable Varnish(TM) at any time by following these instructions.

Block specified URLs from being cached by Varnish(TM)

It is advisable to block phpMyAdmin and/or server-status from being cached and public. To do this. add the following lines of code to the end of the default Varnish(TM) configuration file at /opt/bitnami/varnish/etc/varnish/default.vcl or in the Varnish(TM) configuration file for your application:

sub vcl_recv {
    if (req.url ~ "^/phpmyadmin/.*$" || req.url ~ "^/server-status.*$") {
        return (synth(403, "For security reasons, this URL is only accessible using localhost (127.0.0.1) as the hostname"));
    }
}

Varnish is a registered trademark of Varnish Software AB and its affiliates.

NGINX

Enable Varnish(TM)

Varnish(TM) is disabled by default, so it cannot be started using the control script. To enable it, follow the steps below:

  • Enable the Varnish(TM) service:

      $ sudo mv /etc/monit/conf.d/varnish.conf.disabled /etc/monit/conf.d/varnish.conf
  $ sudo gonit reload

  • Since Varnish(TM) with PageSpeed is not currently supported by Bitnami, ensure PageSpeed is disabled. Follow this guide for more information on this.

Start Varnish(TM)

By default, Varnish(TM) is configured to use TCP port 81. Follow these steps to start the Varnish(TM) service:

  • Execute these commands at the server console:

      $ sudo /opt/bitnami/ctlscript.sh start varnish

  • Ensure NGINX is running:

      $ sudo /opt/bitnami/ctlscript.sh start nginx

Now you should be able to access the index page on both ports 80 and 81, for NGINX and Varnish(TM), respectively.

  • Accessing the server through the 80 port will behave as if Varnish(TM) was not enabled, retrieving all the data from the server.

  • Accessing the server through the 81 port will use Varnish(TM) as a reverse proxy, serving cached contents and requesting NGINX for non-cached content.

Check Varnish(TM) status

Check what Varnish(TM) is doing under the hood with the varnishlog command. To indicate which instance of Varnish(TM) you are interested in, specify the Varnish(TM) working directory which is located by default at /opt/bitnami/varnish/var/varnish/.

0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840690 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840693 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1340840696 1.0

If you visit your server URL though the configured Varnish(TM) port (81 in our example), you will see a more interesting output message:

15 Hash         c /favicon.ico
15 Hash         c 75.101.208.108
15 VCL_return   c hash
15 VCL_call     c pass pass
15 Backend      c 14 default default
15 TTL          c 1976586397 RFC 120 -1 -1 1340840847 0 1340840847 0 0
15 VCL_call     c fetch
15 TTL          c 1976586397 VCL 120 -1 -1 1340840847 -0
....
15 TxResponse   c OK
15 TxHeader     c X-Powered-By: PHP/5.3.13
15 TxHeader     c Content-Type: image/vnd.microsoft.icon
15 TxHeader     c Content-Length: 0
15 TxHeader     c Accept-Ranges: bytes
15 TxHeader     c Date: Wed, 27 Jun 2012 23:47:27 GMT
15 TxHeader     c X-Varnish: 1976586397

To get a clearer idea of what is happening, use the varnishstat command instead:

NAME CURRENT CHANGE AVERAGE AVG_10 AVG_100 AVG_1000
MGT.uptime 0+00:07:02
MAIN.uptime 0+00:07:03
MAIN.sess_conn 8 0.00 0.02 0.03 0.01 0.01
MAIN.client_req 197 0.00 0.47 0.03 0.01 0.01
MAIN.cache_miss 7 0.00 0.02 0.03 0.01 0.01

The command shows much more information but a clear indication of whether it is working can be obtained by checking the MAIN.backend_reuse (how often Varnish(TM) finds the contents in its cache) and the MAIN.cache_miss (how many times it failed and had to contact the web server).

After browsing the site for a while, you may find something like the below:

NAME CURRENT CHANGE AVERAGE AVG_10 AVG_100 AVG_1000
MGT.uptime 0+00:19:34
MAIN.uptime 0+00:19:35
MAIN.sess_conn 26 0.00 0.03 0.00 0.08 0.03
MAIN.client_req 593 0.00 0.63 0.04 1.77 0.65
MAIN.cache_hit 6 0.00 0.01 0.00 0.03 0.03
MAIN.cache_hit_grace 3 0.00 0.00 0.00 0.00 0.00
MAIN.cache_miss 12 0.00 0.01 0.00 0.02 0.01
MAIN.backend_conn 21 0.00 0.02 0.00 0.05 0.02
MAIN.backend_reuse 569 0.00 0.60 0.04 1.70 0.62

Disable Varnish(TM)

IMPORTANT: Before disabling Varnish(TM), make sure that all services are stopped and that NGINX is running on port 80 as described in the Varnish(TM) guide.

In some cases it is necessary to disable Varnish(TM). One example would be when trying to force HTTPS redirection with NGINX. To disable Varnish(TM), follow these steps:

  • Stop the Varnish(TM) service:

              $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • Disable the Varnish(TM) service:

          $ sudo mv /etc/monit/conf.d/varnish.conf /etc/monit/conf.d/varnish.conf.disabled
      $ sudo gonit reload

  • Restart NGINX:

      $ sudo /opt/bitnami/ctlscript.sh start nginx

Customize Varnish(TM)

Varnish(TM) is installed with a default configuration file, agnostic to the Web application being cached. Using this configuration file, although achieving high performance, could lead to some content not being properly refreshed in the Varnish(TM) cache. As a result, users would see an outdated version of the site.

The solution is to use a custom VCL configuration file. There are multiple sources on the Internet that provide customized configuration files for different applications. A good source is the Varnish(TM) example page.

This section discusses how to change the default configuration file to a WordPress-specific one. Follow the steps below:

  • Obtain the source file here.

  • The file requires some modification to register the port on which your NGINX server will be running. This port can be read from the NGINX configuration file at /opt/bitnami/nginx/conf/nginx.conf in the Listen directive:

      ...
  server {
    listen 80;
  ...

    With this value (80), edit the downloaded file and update the section below with the port number:

      backend default {
      .host = "127.0.0.1";
      .port = "81";
  }

    NOTE: For Bitnami stacks, Varnish(TM) is installed on the same server as NGINX so the host can be configured as 127.0.0.1. You can also use Varnish(TM) to cache a remote server, by providing the host’s IP address.

  • Copy the file to the Varnish(TM) directory:

      $ cp /path/to/the/wordpress.vcl  /opt/bitnami/varnish/etc/varnish/

  • Stop Varnish(TM):

      $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • Configure Varnish(TM) to load the appropriate file.

    • Edit the file at /opt/bitnami/scripts/varnish-env.sh and add the following lines to the bottom of the file:

        export VARNISH_CONF_FILE="/opt/bitnami/varnish/etc/varnish/wordpress.vcl"

  • Restart Varnish(TM) (and NGINX if needed):

      $ sudo /opt/bitnami/ctlscript.sh start varnish
  $ sudo /opt/bitnami/ctlscript.sh start nginx

IMPORTANT: Varnish(TM) will not cache content if NGINX’s PageSpeed module is enabled. Find out how to disable this module.

Modify the default Varnish(TM) and NGINX ports

After checking all is working properly, you may want to change the Varnish(TM) port to a standard one, usually port 80. If it was free at installation time, it should already be in use by NGINX.

Follow these steps:

  • Stop NGINX and Varnish(TM):

      $ sudo /opt/bitnami/ctlscript.sh stop nginx
  $ sudo /opt/bitnami/ctlscript.sh stop varnish

  • The configuration of the ports will involve first changing the NGINX port and then the Varnish(TM) port. Move NGINX to a different port, by editing the Listen directive in the NGINX configuration file at /opt/bitnami/nginx/conf/nginx.conf. Find the lines below:

      ...
  server {
    listen 80;
  ...

    and change them so that NGINX listens on a different port:

      ...
  server {
    listen 81;
  ...

  • Update your application configuration for NGINX. For example, if your applications are configured for server blocks, change the port in the NGINX /opt/bitnami/nginx/conf/nginx.conf configuration file, the /opt/bitnami/nginx/conf/nginx.conf file as well as any application-specific server blocks:

    • Edit any server block file inside the /opt/bitnami/nginx/conf/server_blocks/ directory.

  • Configure Varnish(TM) to use the old NGINX port (80) and specify the new port for NGINX (81) in the configuration file:

    • Edit the file at /opt/bitnami/scripts/varnish-env.sh and add the following lines to the bottom of the file:

        export VARNISH_CONF_FILE="/opt/bitnami/varnish/etc/varnish/default.vcl"
  export VARNISH_PORT_NUMBER="80"

  • The Varnish(TM) configuration file at /opt/bitnami/varnish/etc/varnish/default.vcl contains the port on which NGINX is listening. Update it to reflect the new NGINX port (81):

      backend default {
      .host = "127.0.0.1";
      .port = "81";
  }

  • Restart the servers.

      $ sudo /opt/bitnami/ctlscript.sh restart varnish

NGINX without caching should now be available at port 81 and Varnish(TM) at port 80 as a reverse proxy for NGINX.

NOTE: You can disable Varnish(TM) at any time by following these instructions.

Block specified URLs from being cached by Varnish(TM)

It is advisable to block phpMyAdmin and/or server-status from being cached and public. To do this. add the following lines of code to the end of the default Varnish(TM) configuration file at /opt/bitnami/varnish/etc/varnish/default.vcl or in the Varnish(TM) configuration file for your application:

sub vcl_recv {
    if (req.url ~ "^/phpmyadmin/.*$" || req.url ~ "^/server-status.*$") {
        return (synth(403, "For security reasons, this URL is only accessible using localhost (127.0.0.1) as the hostname"));
    }
}

Varnish is a registered trademark of Varnish Software AB and its affiliates.

Last modification February 9, 2023