Getting Started

In this chapter, you will:

  • learn about the Varnish distribution,
  • install Varnish, and
  • cover basic Varnish configuration.

Most of the commands you will type in this course require root privileges. You can get temporary root privileges by typing sudo <command>, or permanent root privileges by typing sudo -i.

In Varnish terminology, a backend is the origin server. In other words, it is whatever server Varnish talks to fetch content. This can be any sort of service as long as it understands HTTP. Most of the time, Varnish talks to a web server or an application frontend server. In this book, we use backend, origin server, web server or application frontend server depending the context.

Varnish Distribution

Utility programs part of the Varnish distribution:

  • varnishd
  • varnishtest
  • varnishadm
  • varnishlog
  • varnishstat
  • and more

The Varnish distribution includes several utility programs that you will use in this course. You will learn how to use these programs as you progress, but it is useful to have a brief introduction about them before we start.

The central block of Varnish is the Varnish daemon varnishd. This daemon accepts HTTP requests from clients, sends requests to a backend, caches the returned objects and replies to the client request. varnishd is further explained in the Varnish Architecture section.

varnishtest is a script driven program used to test your Varnish installation. varnishtest is very powerful because it allows you to create client mock-ups, fetch content from mock-up or real backends, interact with your actual Varnish configuration, and assert the expected behavior. varnishtest is also very useful to learn more about the behavior of Varnish.

varnishadm controls a running Varnish instance. The varnishadm utility establishes a command line interface (CLI) connection to varnishd. This utility is the only one that may affect a running instance of Varnish. You can use varnishadm to:

  • start and stop varnishd,
  • change configuration parameters,
  • reload the Varnish Configuration Language (VCL),
  • view the most up-to-date documentation for parameters, and
  • more.

The front-end varnishadm of the Varnish Command Line Interface (CLI) section explains in more detail this utility.

The Varnish log provides large amounts of information, thus it is usually necessary to filter it. For example, “show me only what matches X”. varnishlog does precisely that. You will learn more about varnishlog in the Examining Varnish Server’s Output chapter.

varnishstat is used to access global counters. It provides overall statistics, e.g the number of total requests, number of objects, and more. varnishstat is particularly useful when using it together with varnishlog to analyze your Varnish installation. The varnishstat section explains in detail this utility.

In addition, there are other utility programs such as varnishncsa, varnishtop and varnishhist. Appendix B: Varnish Programs explains them.

Exercise: Install Varnish

  • Install Varnish
  • Use packages provided by
    • varnish-software.com for Varnish Cache Plus
    • varnish-cache.org for Varnish Cache
  • When you are done, verify your Varnish version, run varnishd -V

For official training courses, a varnish-plus package should already be available for installation. When in doubt, ask your instructor to confirm which package should be installed.

You may skip this exercise if already have a well configured environment to test Varnish. In case you get stuck, you may look at the proposed solution.

Table 2 Different Locations of the Varnish Configuration File
SysV SysV systemd systemd
Ubuntu/Debian RHEL/CentOS Ubuntu/Debian Fedora/RHEL 7+/CentOS 7+
/etc/default/varnish /etc/sysconfig/varnish [2] /etc/varnish/varnish.params
/etc/default/varnishlog [1] [3] [5]
/etc/default/varnishncsa [1] [4] [5]

[1] There is no configuration file. Use the command chkconfig varnishlog/varnishncsa on/off instead.

[2] Create a drop-in systemd service file in /etc/systemd/system/varnish.service.d/customexec.conf:

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f \
/etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

This file overrides the ExecStart option of the default configuration shipped with Varnish Cache. Run systemctl daemon-reload to make sure systemd picks up the new configuration before restarting Varnish.

[3] Create a drop-in systemd service file in /etc/systemd/system/varnishlog.service.d/customexec.conf to customize your varnishlog configuration.

[4] Create a drop-in systemd service file in /etc/systemd/system/varnishncsa.service.d/customexec.conf. In this file you can for example set VARNISHNCSA_ENABLED=1.

[5] There is no configuration file. Use the command systemctl start/stop/enable/disable/ varnishlog/varnishncsa instead.

The configuration file is used to give parameters and command line arguments to the Varnish daemon. This file also specifies the location of the VCL file. Modifications to this file require to run service varnish restart for the changes to take effect.

The location of the Varnish configuration file depends on the operating system and whether it uses the init system of SysV, or systemd. Table 2 shows the locations for each system installation.

To install packages on Ubuntu or Debian, use the command apt-get install <package>, e.g., apt-get install varnish. For CentOS, RHEL or Fedora, use yum install <package>.

You might want to look at Solution: Install Varnish, if you need help.

If the command service varnish restart fail, try to start Varnish manually to get direct feedback from the shell. Command example:

$ sudo /usr/sbin/varnishd -j unix,user=varnish,ccgroup=varnish \
-P /var/run/varnish.pid -f /etc/varnish/default.vcl -a :80 -a :6081,PROXY \
-T 127.0.0.1:6082 -t 120 -S /etc/varnish/secret \
-s malloc,256MB -F

Exercise: Configure Varnish

  • Configure listening ports for client requests and Varnish administration

  • In CentOS 7:

    • /etc/varnish/varnish.params

    • Variable substitution in /usr/lib/systemd/system/varnish.service:

      -a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT}
      -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}
      
  • Configure one backend in VCL file /etc/varnish/default.vcl

See Table 2 and locate the Varnish configuration file for your installation. Open and edit that file to listen to client requests on port 80 and have the management interface on port 1234.

In Ubuntu and Debian, this is configured with options -a and -T of variable DAEMON_OPTS. In CentOS, RHEL, and Fedora, use VARNISH_LISTEN_PORT and VARNISH_ADMIN_LISTEN_PORT respectively.

In order for changes in the configuration file to take effect, varnishd must be restarted. The safest way to restart Varnish is by using service varnish restart.

The default VCL file location is /etc/varnish/default.vcl. You can change this location by editing the configuration file. The VCL file contains the backend definitions.

In this book, we use Apache as backend. Before continuing, make sure you have Apache installed and configured to listen on port 8080. See Appendix F: Apache as Backend if you do not know how to do it.

Edit /etc/varnish/default.vcl to use Apache as backend:

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

Varnish Cache Plus supports SSL/TLS encryption. To encrypt connections between Varnish and the backend, you specify it as follows:

backend default {
  .host = "host.name";
  .port = "https";       # This defaults to https when SSL
  .ssl = 1;              # Turns on SSL support
  .ssl_nosni = 0;        # Disable SNI extension
  .ssl_noverify = 1;     # Don't verify peer
}

For Varnish to accept incoming encrypted connections, you need a terminator for encrypted connections such as hitch https://github.com/varnish/hitch. Varnish Plus 4.1 has integrated this functionality and you can easily configure it as detailed in SSL/TLS frontend support with hitch.

VCL Reload

  • varnishd can reload VCL programs without restart
service varnish reload

or:

systemctl reload varnish

or:

varnishadm vcl.load vcl01 /etc/varnish/default.vcl
varnishadm vcl.use vcl01

service varnish reload is a shortcut to reload VCL programs. varnishadm vcl.load <compiledVCL> <VCLsourcecode> compiles the VCL program you specify. You can have multiple compiled files in Varnish. To see them, run:

varnishadm vcl.list

To apply a compiled VCL program, type:

varnishadm vcl.use <compiledVCL>

This command does not restart varnishd, it only reloads the compiled VCL code.

The result of your configuration is resumed in Table 3.

Table 3 Varnish and Backend Configuration
Server Result Configuration file
Varnish Listens on port 80 /etc/default/varnish *
Varnish Uses backend on localhost:8080 /etc/varnish/default.vcl
Apache Listens on port 8080 /etc/apache2/ports.conf * and /etc/apache2/sites-enabled/000-default *

* These files are for a SysV Ubuntu/Debian configuration

You can get an overview over services listening on TCP ports by issuing the command netstat -nlpt. Within the result, you should see something like:

tcp     0     0 0.0.0.0:80         0.0.0.0:*     LISTEN     9223/varnishd
tcp     0     0 127.0.0.1:1234     0.0.0.0:*     LISTEN     9221/varnishd

Warning

If you have Security-Enhanced Linux (SELinux), be aware that SELinux defines ports 6081 and 6082 for varnishd. If you need to use another port number, you need either to disable SELinux or set the boolean varnishd_connect_any variable to 1. You can do that by executing the command sudo setsebool varnishd_connect_any 1.

Tip

Issue the command man vcl to see all available options to define a backend.

Tip

You can also configure Varnish via the Varnish Administration Console (VAC).

../_images/vac_config.png

Fig. 3 GUI to configure Varnish via the Varnish Administration Console (VAC).

Test Varnish Using Apache as Backend

  • Run http -p Hh localhost

  • Your output should look as:

    # http -p Hh localhost
    GET / HTTP/1.1
    Accept: */*
    Accept-Encoding: gzip, deflate, compress
    Host: localhost
    User-Agent: HTTPie/0.8.0
    
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Age: 0
    Connection: keep-alive
    Content-Encoding: gzip
    Content-Length: 3256
    Content-Type: text/html
    Date: Wed, 18 Mar 2015 13:55:28 GMT
    ETag: "2cf6-5118f93ad6885-gzip"
    Last-Modified: Wed, 18 Mar 2015 12:53:59 GMT
    Server: Apache/2.4.7 (Ubuntu)
    Vary: Accept-Encoding
    Via: 1.1 varnish-plus-v4
    X-Varnish: 32770
    

You can test your Varnish installation by issuing the command http -p Hh localhost. If you see the HTTP response header field Via containing varnish, then your installation is correct.

The X-Varnish HTTP header field contains the Varnish Transaction ID (VXID) of the client request and if applicable, the VXID of the backend transaction that stored in cache the object delivered. X-Varnish is useful to find the correct log entries in the Varnish log. For a cache hit, X-Varnish contains both the ID of the current request and the ID of the request that populated the cache. You will learn more about VXIDs in the Transactions section.

You can also define and test connectivity against any backend in varnishtest. Learn how to it by doing the Exercise: Test Apache as Backend with varnishtest.

The front-end varnishadm of the Varnish Command Line Interface (CLI)

You can use the varnishadm utility to:

  • start and stop the cacher (aka child) process
  • change configuration parameters without restarting Varnish
  • reload the Varnish Configuration Language (VCL) without restarting Varnish
  • view the most up-to-date documentation for parameters
  • varnishadm help and man varnishadm

Varnish has a command line interface (CLI) which can control and change most of the operational parameters and the configuration of the Varnish daemon, without interrupting the running service. varnishadm is front-end to the Varnish interface. In practice, it is a utility program with a set of functions that interact with the Varnish daemon varnishd. If there are many Varnish instances running in one machine, specify the instance with the -n option of varnishadm. Keep the following in mind when issuing commands to the Varnish daemon:

  1. Changes take effect on the running Varnish daemon instance without need to restart it.
  2. Changes are not persistent across restarts of Varnish. If you change a parameter and you want the change to persist after you restart Varnish, you need to store your changes in the configuration file of the boot script. The location of the configuration file is in Table 2

varnishadm uses a non-encrypted key stored in a secret file to authenticate and connect to a Varnish daemon. You can control access by adjusting the read permission on the secret file. varnishadm looks for the secret file in /etc/varnish/secret by default, but you can use the -S option to specify another location. The content of the file is a shared secret, which is a string generated under Varnish installation.

varnishadm authenticates with a challenge-response mechanism. Therefore, the shared secret is never transmitted, but a challenge and the response to the challenge. This authentication mechanism offers a reasonably good access control, but it does not protect the data transmitted over the connection.

In order to avoid eavesdroppers like in the man-in-the-middle attack, we recommend that you configure the management interface listening address of varnishd to listen only on localhost (127.0.0.1 IP address). You configure this address with the -T option of the varnishd command.

For convenience, varnishadm has an embedded command line tool. You can access it by simply issuing varnishadm in the terminal.

Tip

Varnish provides many on-line reference manuals. To learn more about varnishadm, issue man varnishadm. To check the Varnish CLI manual page, issue man varnish-cli.

Tip

You can also access the varnishadm via the Varnish Administration Console (VAC). To do that, you just have to navigate to the CONFIGURE tab and click on the Varnish server you want to administrate. Then, varnishadm is ready to use in a terminal emulator right in your web browser.

../_images/vac_varnishadm.png

Fig. 4 Access to varnishadm by clicking on the Varnish server that you want to administrate.

../_images/vac_cli.png

Fig. 5 Terminal emulator in your web browser.

More About Varnish Configuration

Table 4 Varnish Configuration Types
Configuration Type Restart Required Persistence at next restart
Command line options Yes If stored in the configuration file as part of DAEMON_OPTS
Tunable parameters No (if changed in varnishadm) If stored in the configuration file as part of DAEMON_OPTS
Configuration in VCL No Yes

The location of the configuration file is in Table 2.

Table 5 How to reload Varnish
Command Result
service varnish restart Restarts Varnish using the operating system mechanisms. Caches are flushed.
service varnish reload Only reloads the VCL. Caches are not affected.
varnishadm vcl.load <configname> <filename> and varnishadm vcl.use <configname> Can be used to manually reload VCL. The service varnish reload command does this for you automatically.
varnishadm param.set <param> <value> Can be used to set parameters without restarting Varnish.

Using the service commands is recommended, safe and fast.

Command line options and tunable parameters are used to: 1) define how Varnish should work with operating system and hardware, and 2) set default values. Configuration in VCL defines how Varnish should interact with web servers and clients.

Almost every aspect of Varnish can be reconfigured without restarting Varnish. Notable exceptions are: 1) allocated memory size for caching, 2) cache file location, 3) ownership (for user and group privileges) of the Varnish daemon, and 4) the hashing algorithm.

Some parameters changes require to restart Varnish to take effect. For example, the modification of the listening port. Other changes might not take effect immediately, but restart is not required. Changes to cache time-to-live (TTL), for instance, take effect only after the current cached objects expire. In this example, the value of the TTL parameter is only applicable to caches fetched after the TTL modification.

param.show <parameter> outputs a description of parameter. The description includes when and how modifications takes effect, and the default and current value of the parameter.

There are other ways to reload VCL and make parameter-changes take effect, mostly using the varnishadm tool. However, using the service varnish reload and service varnish restart commands is a good habit.

Note

If you want to know how the service varnish-commands work, look at the script that runs behind the scenes. The script is in /etc/init.d/varnish.

Warning

The varnish script-configuration (located under /etc/default/ or /etc/sysconfig/) is directly sourced as a shell script. Pay close attention to any backslashes (\) and quotation marks that might move around as you edit the DAEMON_OPTS environmental variable.

Command Line Configuration

Relevant options for the course are:

-a <[hostname]:port>
 listening address and port for client requests
-f <filename> VCL file
-p <parameter=value>
 set tunable parameters
-S <secretfile>
 shared secret file for authorizing access to the management interface
-T <hostname:port>
 listening address and port for the management interface
-s <storagetype,options>
 where and how to store objects

All the options that you can pass to the varnishd binary are documented in the varnishd(1) manual page (man varnishd). You may want to take a moment to skim over the options mentioned above.

For Varnish to start, you must specify a backend. You can specify a backend by two means: 1) declare it in a VCL file, or 2) use the -b to declare a backend when starting varnishd.

Though they are not strictly required, you almost always want to specify a -s to select a storage backend, -a to make sure Varnish listens for clients on the port you expect and -T to enable a management interface, often referred to as a telnet interface.

For both -T and -a, you do not need to specify an IP, but can use :80 to tell Varnish to listen to port 80 on all IPs available. Make sure you do not forget the colon, as -a 80 tells Varnish to listen to the IP with the decimal-representation “80”, which is almost certainly not what you want. This is a result of the underlying function that accepts this kind of syntax.

You can specify -p for parameters multiple times. The workflow for tuning Varnish parameters usually is that you first try the parameter on a running Varnish through the management interface to find the value you want. Then, you store the parameter and value in a configuration file. This file is read every time you start Varnish.

The -S option specifies a file which contains a secret to be used for authentication. This can be used to authenticate with varnishadm -S as long as varnishadm can read the same secret file – or rather the same content: The content of the file can be copied to another machine to allow varnishadm to access the management interface remotely.

Note

Varnish requires at least one backend, which is normally specified in the VCL file. The VCL file is passed to varnishd with the -f <filename.vcl> option. However, it is possible to start Varnish without a VCL file. In this case, the backend is passed directly to varnishd with the -b <hostname:port> option. -f and -b are mutually exclusive.

Tip

Type man varnishd to see all options of the Varnish daemon.

Defining a Backend in VCL

/etc/varnish/default.vcl

vcl 4.0;

backend default {
    .host = "localhost";
    .port = "8080";
}

The above example defines a backend named default, where the name default is not special. Varnish uses the first backend you specify as default. You can specify many backends at the same time, but for now, we will only specify one to get started.

Tip

You can also add and edit your VCL code via the Varnish Administration Console (VAC). This interface also allows you to administrate your VCL files.

../_images/vac_vcl.png

Fig. 6 GUI of Varnish Administration Console (VAC) with command line interface to edit your VCL code.

Exercise: Use the administration interface to learn, review and set Varnish parameters

  1. Use param.show and param.set from varnishadm to see and set the value of default_ttl and default_grace
  2. Refer to Fig. 2 to see the object’s timeline
Parameters can also be set in varnishtest as explained in Setting Parameters in varnishtest. You will learn more about how to tune parameters in the Tunable Parameters section.

Exercise: Fetch Data Through Varnish

  • Execute http -p hH http://localhost/ on the command line
  • Pay attention to the Age response header field
  • Compare and discuss the results from multiple executions

-p hH specifies HTTPie to print only request and response headers, but not the content. The typical HTTP response is 200 OK or 404 File not found. Feel free to try removing some of the options and observe the effect. For more information about the HTTPie command, type man http.

Testing Varnish with a web browser can be confusing, because web browsers have their own cache. Therefore, it is useful to double-check web browsers requests with HTTPie or varnishtest as explained in Fetch Data with varnishtest. For more information about the Age response header field refer to the Age subsection.