Tuning

This chapter is for the system administration course only

This section covers:

  • Architecture
  • Best practices
  • Parameters

Perhaps the most important aspect of tuning Varnish is writing effective VCL code. For now, however, we will focus on tuning Varnish for your hardware, operating system and network. To be able to do that, knowledge of Varnish architecture is helpful.

It is important to know the internal architecture of Varnish for two reasons. First, the architecture is chiefly responsible for the performance, and second, it influences how you integrate Varnish in your own architecture.

There are several aspects of the design that were unique to Varnish when it was originally implemented. Truly good solutions, regardless of reusing ancient ideas or coming up with something radically different, is the aim of Varnish.

Varnish Architecture

../_images/architecture.svg

Fig. 15 Varnish Architecture

Fig. 15 shows a block diagram of the Varnish architecture. The diagram shows the data flow between the principal parts of Varnish.

The main block is the Manager process, which is contained in the varnishd binary program. The task of the Manager process is to delegate tasks, including caching, to child processes. The Manager process ensures that there is always a process for each task. The main driver for these design decisions is security, which is explain at Security barriers in Varnish https://www.varnish-cache.org/docs/trunk/phk/barriers.html.

The Manager’s command line interface (CLI) is accessible through: 1) varnishadm as explained in The front-end varnishadm of the Varnish Command Line Interface (CLI) section, 2) the Varnish Agent vagent2, or 3) the Varnish Administration Console (VAC) (via vagent2)

The Varnish Agent vagent2 is an open source HTTP REST interface that exposes varnishd services to allow remote control and monitoring. vagent2 offers a web UI as shown in Fig. 16, but you can write your own UI since vagent2 is an open interface. Some features of vagent2 are:

  • VCL uploading, downloading, persisting (storing to disk).
  • parameter viewing, storing (not persisting yet)
  • show/clear of panic messages
  • start/stop/status of varnishd
  • banning
  • varnishstat in JSON format
../_images/vagent2.png

Fig. 16 Varnish Agent’s HTML interface; designed to showcase the various features of the Varnish Agent.

For more information about vagent2 and installation instructions, please visit https://github.com/varnish/vagent2.

Varnish Software has a commercial offering of a fully functional web UI called Varnish Administration Console (VAC). For more information about VAC, refer to the Varnish Administration Console (VAC) section.

The Parent Process: The Manager

The Manager process is owned by the root user, and its main functions are:

  • apply configuration changes (from VCL files and parameters)
  • delegate tasks to child processes: the Cacher and the VCL to C Compiler (VCC)
  • monitor Varnish
  • provide a Varnish command line interface (CLI)
  • initialize the child process: the Cacher

The Manager checks every few seconds whether the Cacher is still there. If the Manager does not get a reply within a given interval defined in ping_interval, the Manager kills the Cacher and starts it up again. This automatic restart also happens if the Cacher exits unexpectedly, for example, from a segmentation fault or assert error. You can ping manually the cacher by executing varnishadm ping.

Automatic restart of child processes is a resilience property of Varnish. This property ensures that even if Varnish contains a critical bug that crashes the child, the child starts up again usually within a few seconds. You can toggle this property using the auto_restart parameter.

Note

Even if you do not perceive a lengthy service downtime, you should check whether the Varnish child is being restarted. This is important, because child restarts introduce extra loading time as varnishd is constantly emptying its cache. Automatic restarts are logged into /var/log/syslog.

To verify that the child process is not being restarted, you can also check its lifetime with the MAIN.uptime counter in varnishstat.

Varnish Software and the Varnish community at large occasionally get requests for assistance in performance tuning Varnish that turn out to be crash-issues.

The Child Process: The Cacher

Since the Cacher listens on public IP addresses and known ports, it is exposed to evil clients. Therefore, for security reasons, this child process is owned by an unprivileged user, and it has no backwards communication to its parent, the Manager.

The main functions of the Cacher are:

  • listen to client requests
  • manage worker threads
  • store caches
  • log traffic
  • update counters for statistics

Varnish uses workspaces to reduce the contention between each thread when they need to acquire or modify memory. There are multiple workspaces, but in Varnish 4 and later, the most important ones are workspace_client and workspace_backend. Memory in workspace_client is used to manipulate data at the frontend side of the Varnish state machine, i.e., request data. workspace_backend is used to manipulate data fetched from the backend side. As an example, think about the memory needed to normalize an object’s Host header from www.example.com to example.com before it is stored in the cache.

If you have 5 MB of workspace and are using 1000 threads, the actual memory usage is not 5 GB. The virtual memory usage will indeed be 5GB, but unless you actually use the memory, this is not a problem. Your memory controller and operating system will keep track of what you actually use.

To communicate with the rest of the system, the child process uses the VSL accessible from the file system. This means that if a thread needs to log something, all it has to do is to grab a lock, write to a memory area and then free the lock. In addition to that, each worker thread has a cache for log-data to reduce lock contention. We will discuss more about the Threading Model later in this chapter.

The log file is usually about 80MB, and split in two. The first part is counters, the second part is request data. To view the actual data, a number of tools exist that parses the VSL.

Since the log data is not meant to be written to disk in its raw form, Varnish can afford to be very verbose. You then use one of the log-parsing tools to extract the piece of information you want – either to store it permanently or to monitor Varnish in real-time.

If something goes wrong in the Cacher, it logs a detailed panic message to syslog. For testing, you can induce panic to varnishd by issuing the command varnishadm debug.panic.worker or by pressing the Induce Panic button in the Varnish Agent web interface.

VCL Compilation

  • Command to print VCL code compiled to C language and exit:

    varnishd -C -f <vcl_filename>
    
  • Useful to check whether your VCL code compiles correctly.

Configuring the caching policies of Varnish is done in the Varnish Configuration Language (VCL). Your VCL is then translated by the VCC process to C, which is compiled by a normal C compiler – typically gcc, and linked into the running Varnish instance. Since the VCL compilation is done outside of the child process, there is no risk of affecting the running Varnish instance by accidentally loading an ill-formatted VCL.

As a result, changing configuration while running Varnish is very cheap. Policies of the new VCL takes effect immediately. However, objects cached with an older configuration may persist until they have no more old references or the new configuration acts on them.

A compiled VCL file is kept around until you restart Varnish completely, or until you issue vcl.discard from the management interface. You can only discard compiled VCL files after all references to them are gone. You can see the amount of VCL references by reading the parameter vcl.list.

Storage Backends

  • The storage option -s defines the size of your cache and where it is stored
  • Use varnishd -s followed by one of the following methods to allocate space for the cache:
    • malloc
    • file
    • persistent (deprecated)
    • mse Varnish Massive Storage Engine (MSE) in Varnish Plus only

The -s <malloc[,size]> option calls malloc() to allocate memory space for every object that goes into the cache. If the allocated space cannot fit in memory, the operating system automatically swaps the needed space to disk.

Varnish uses the jemalloc implementation. Although jemalloc emphasizes fragmentation avoidance, fragmentation still occurs. Jemalloc worst case of memory fragmentation is 20%, therefore, expect up to this percentage of additional memory usage. In addition to memory fragmentation you should consider an additional 5% overhead as described later in this section.

Another option is -s <file,path[,size[,granularity]]>. This option creates a file on a filesystem to contain the entire cache. Then, the operating system maps the entire file into memory if possible.

The -s file storage method does not retain data when you stop or restart Varnish! For persistence, use the option -s persistent. The usage of this option, however, is strongly discouraged mainly because of consistency issues that might arise with it.

The Varnish Massive Storage Engine (MSE) option -s <mse,path[,path...]]> is an improved storage method for Varnish Plus only. MSE main improvements are decreased disk I/O load and lower storage fragmentation. MSE is designed to store and handle over 100 TB with persistence, which makes it very useful for video on demand setups.

MSE uses a hybrid of two cache algorithms, least recently used (LRU) and least frequently used (LFU), to manage memory. Benchmarks show that this algorithm outperforms malloc and file. MSE also implements a mechanism to eliminate internal fragmentation.

The latest version of MSE requires a bookkeeping file. The size of this bookkeeping file depends on the cache size. Cache sizes in the order of gigabytes require a bookkeeping file of around 1% of the storage size. Cache sizes in the order of terabytes should have a bookkeeping file size around 0.5% of storage size.

For detailed instructions on how to configure MSE, please refer to the Varnish Plus documentation. For more details about its features and previous versions, please visit https://info.varnish-software.com/blog/varnish-mse-persistence.

When choosing storage backend, use malloc if your cache will be contained entirely or mostly in memory. If your cache will exceed the available physical memory, you have two options: file or mse. We recommend you to use MSE because it performs much better than file storage backend.

There is a storage overhead in Varnish, so the actual memory footprint of Varnish exceeds what the -s argument specifies if the cache is full. The current estimated overhead is 1kB per object. For 1 million objects, that means 1GB extra memory usage. This estimate might slightly vary between Varnish versions.

In addition to the overhead per object, Varnish requires memory to manage the cache and handle its own operation. Our tests show that an estimate of 5% of overhead is accurate enough. This overhead applies equally to malloc, file or mse options.

For more details about memory usage in Varnish, please refer to https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage.

Note

As a rule of thumb use: malloc if the space you want to allocate fits in memory, if not, use file or mse. Remember that there is about 5% memory overhead and do not forget to consider the memory needed for fragmentation in malloc or the disk space for the bookkeeping file in mse.

The Varnish Shared memory Log (VSL)

  • Avoid I/O operations
  • Mount the shared memory log as tmpfs
  • VSL is not persistent

The Varnish Shared memory Log (VSL), sometimes called shm-log or SHMLOG, is used to log most data. VSL operates on a circular buffer. Therefore, there is no a start or an end of it, but you can issue varnishlog -d to see old log entries.

VSL is 80MB large by default and is not persistent, unless you instruct Varnish to do otherwise. VSL is memory space mapped under /var/lib/varnish/. To change the size of the VSL, see the option -l in the man page of varnishd.

There is not much you have to do with the VSL, except ensure that it does not cause I/O operations. You can avoid I/O by mounting the VSL as a temporary file storage (tmpfs). This is typically configured in /etc/fstab, and the shm-log is normally kept under /var/lib/varnish/ or equivalent locations. You need to restart varnishd after mounting it as tmpfs.

Warning

Some Varnish distribution setup the file storage backend option -s file by default. Those distribution set a path that puts the storage file in the same directory as the shm-log. We discourage this practice.

Tunable Parameters

  • In the CLI:

    param.show -l
    
  • Do not fall for the copy/paste tips

  • Test the parameters in CLI, then store them in the configuration file

Varnish has many different parameters which can be adjusted to make Varnish act better under specific workloads or with specific software and hardware setups. They can all be viewed with param.show in the management interface varnishadm.

You can set up parameters in two different ways. In varnishadm, use the command param.set <param> <value>. Alternatively, you can issue the command varnishd -p param=value.

Remember that changes made in the management interface are not persistent. Therefore, unless you store your changes in a startup script, they will be lost when Varnish restarts.

The general advice with regards to parameters is to keep it simple. Most of the defaults are optimal. If you do not have a very specific need, it is generally better to use the default values.

A few debug commands exist in the CLI, which can be revealed with help -d. These commands are meant exclusively for development or testing, and many of them are downright dangerous.

Tip

Parameters can also be configured via the Varnish Administration Console (VAC) as shown in the figure below.

../_images/vac_parameters.png

Fig. 17 GUI to configure parameters via the Varnish Administration Console (VAC).

Threading Model

  • The child process runs multiple threads in two thread pools
  • Threads accept new connections and delegate them
  • One worker threads per client request – it’s common to use hundreds of worker threads
  • Expire-threads evict old content from the cache
Table 8 Relevant threads in Varnish
Thread-name Amount of threads Task
cache-worker 1 per active connection Handle requests
cache-main 1 Startup
ban lurker 1 Clean bans
acceptor 1 Accept new connections
epoll/kqueue Configurable, default: 2 Manage thread pools
expire 1 Remove old content
backend poll 1 per backend poll Health checks
The child process runs multiple threads in two thread pools. The threads of these pools are called worker threads. Table 8 presents relevant threads.

Threading Parameters

  • Thread pools can safely be ignored
  • Start threads better sooner than later
  • Maximum and minimum values are per thread pool
Table 9 Threads parameters
Parameter Default Description
thread_pool_add_delay 0.000 [seconds] Period of time to wait for subsequent thread creation.
thread_pool_destroy_delay 1 second Added time to thread_pool_timeout.
thread_pool_fail_delay 0.200 [seconds] Period of time before retrying the creation of a thread. This after the creation of a thread failed.
thread_pool_max 5000 [threads] Maximum number of worker threads per pool.
thread_pool_min 100 [threads] Minimum number of worker threads per pool.
thread_pool_stack 48k [bytes] Worker thread stack size.
thread_pool_timeout 300.000 [seconds] Period of time before idle threads are destroyed.
thread_pools 2 [pools] Number of worker thread pools.
thread_queue_limit 20 requests Permitted queue length per thread-pool.
thread_stats_rate 10 [requests] Maximum number of jobs a worker thread may handle before it is forced to dump its accumulated stats into the global counters.
workspace_thread 2k [bytes] Bytes of auxillary workspace per thread.

When tuning Varnish, think about the expected traffic. The most important thread setting is the number of cache-worker threads. You may configure thread_pool_min and thread_pool_max. These parameters are per thread pool.

Although Varnish threading model allows you to use multiple thread pools, we recommend you to do not modify this parameter. Based on our experience and tests, we have seen that 2 thread pools are enough. In other words, the performance of Varnish does not increase when adding more than 2 pools.

Note

If you run across the tuning advice that suggests to have a thread pool per CPU core, rest assured that this is old advice. We recommend to have at most 2 thread pools, but you may increase the number of threads per pool.

Details of Threading Parameters

  • Default values have proved to be sufficient in most cases
  • thread_pool_min and thread_pool_max are the most common threading parameters to tune.
  • Run extra threads to avoid creating them on demand

Varnish runs one thread per session, so the maximum number of threads is equal to the number of maximum sessions that Varnish can serve concurrently. If you seem to need more threads than the default, it is very likely that there is something wrong in your setup. Therefore, you should investigate elsewhere before you increase the maximum value.

You can observe if the default values are enough by looking at MAIN.sess_queued through varnishstat. Look at the counter over time, because it is fairly static right after startup.

When tuning the number of threads, thread_pool_min and thread_pool_max are the most important parameters. Values of these parameters are per thread pool. The thread_pools parameter is mainly used to calculate the total number of threads. For the sake of keeping things simple, the current best practice is to leave thread_pools at the default 2 [pools].

Varnish operates with multiple pools of threads. When a connection is accepted, the connection is delegated to one of these thread pools. Afterwards, the thread pool either delegates the connection request to an available thread, queue the request otherwise, or drop the connection if the queue is full. By default, Varnish uses 2 thread pools, and this has proven sufficient for even the most busy Varnish server.

Varnish has the ability to spawn new worker threads on demand, and remove them once the load is reduced. This is mainly intended for traffic spikes. It’s a better approach to keep a few threads idle during regular traffic, than to run on a minimum amount of threads and constantly spawn and destroy threads as demand changes. As long as you are on a 64-bit system, the cost of running a few hundred threads extra is very low.

The thread_pool_min parameter defines how many threads run for each thread pool even when there is no load. thread_pool_max defines the maximum amount of threads that could be used per thread pool. That means that with the minimum defaults 100 [threads] and 5000 [threads] of minimum and maximums threads per pool respectively, you have:

  • at least 100 [threads] * 2 [pools] worker threads at any given time
  • no more than 5000 [threads] * 2 [pools] worker threads ever

Warning

New threads use preallocated workspace, which should be enough for the required task. If threads have not enough workspace, the child process is unable to process the task and it terminates. To avoid this situation, evaluate your setup and consider to increase the workspace_client or workspace_backend parameter.

Time Overhead per Thread Creation

  • thread_pool_add_delay: Wait at least this long after creating a thread.
  • thread_pool_timeout: Thread idle threshold.
  • thread_pool_fail_delay: After a failed thread creation, wait at least this long before trying to create another thread.

Varnish can use several thousand threads, and has had this capability from the very beginning. However, not all operating system kernels were prepared to deal with this capability. Therefore the parameter thread_pool_add_delay was added to ensure that there is a small delay between each thread that spawns. As operating systems have matured, this has become less important and the default value of thread_pool_add_delay has been reduced dramatically, from 20 ms to 2 ms.

There are a few, less important parameters related to thread timing. The thread_pool_timeout is how long a thread is kept around when there is no work for it before it is removed. This only applies if you have more threads than the minimum, and is rarely changed.

Another less important parameter is the thread_pool_fail_delay. After the operating system fails to create a new thread, thread_pool_fail_delay defines how long to wait for a re-trial.

System Parameters

As Varnish has matured, fewer and fewer parameters require tuning. The workspace_client and workspace_backend are parameters that could still be relevant.

  • workspace_client – incoming HTTP header workspace from the client
  • workspace_backend – bytes of HTTP protocol workspace for backend HTTP req/resp
  • Tune it if you have many big headers or have a VMOD that uses too much memory
  • Remember: it is virtual, not physical memory

Workspaces are some of the things you can change with parameters. Sometimes you may have to increase them to avoid running out of workspace.

The workspace_client parameter states how much memory can be allocated for each HTTP session. This space is used for tasks like string manipulation of incoming headers. The workspace_backend parameter indicates how much memory can be allocated to modify objects returned from the backend. After an object is modified, its exact size is allocated and the object is stored read-only.
As most of the parameters can be left unchanged, we will not go through all of them.
You can take a look at the list of parameter by issuing varnishadm param.show -l to get information about what they can do.

Timers

Table 10 Timers
Parameter Default Description Scope
connect_timeout 3.500 [seconds] OS/network latency Backend
first_byte_timeout 60.000 [seconds] Web page generation Backend
between_bytes_timeout 60.000 [seconds] Hiccoughs Backend
send_timeout 600.000 [seconds] Client-in-tunnel Client
timeout_idle 5.000 [seconds] keep-alive timeout Client
timeout_req 2.000 [seconds] deadline to receive a complete request header Client
cli_timeout 60.000 [seconds] Management thread->child Management

The timeout-parameters are generally set to pretty good defaults, but you might have to adjust them for unusual applications. The default value of connect_timeout is 3.500 [seconds]. This value is more than enough when having the Varnish server and the backend in the same server room. Consider to increase the connect_timeout value if your Varnish server and backend have a higher network latency.

Keep in mind that the session timeout affects how long sessions are kept around, which in turn affects file descriptors left open. It is not wise to increase the session timeout without taking this into consideration.

The cli_timeout is how long the management thread waits for the worker thread to reply before it assumes it is dead, kills it and starts it back up. The default value seems to do the trick for most users today.

Warning

If connect_timeout is set too high, it does not let Varnish handle errors gracefully.

Note

Another use-case for increasing connect_timeout occurs when virtual machines are involved as they can increase the connection time significantly.

Exercise: Tune first_byte_timeout

  • Access a slow resource on the backend
  • Set first_byte_timeout to 1 second
  • Check how Varnish times out the request to the backend

For the purpose of this exercise we use a CGI script that waits 5 seconds before responding:

http localhost/cgi-bin/sleep

To check how first_byte_timeout impacts the behavior of Varnish, analyze the output of varnishlog and varnishstat.

Alternatively, you can use delay in a mock-up backend in varnishtest and assert VSL records and counters to verify the effect of first_byte_timeout. The subsection Solution: Tune first_byte_timeout and test it against mock-up server shows you how to do it.

Exercise: Configure Threading

  • Change the thread_pool_min and thread_pool_max parameters to get 10 threads running at any given time, but never more than 20.
  • Execute varnishadm param.show <parameter> to see parameter details.

These exercises are for educational purposes, and not intended as an encouragement to change the values. You can learn from this exercise by using varnishstat, varnishadm and varnishstat

If you need help, see Solution: Configure Threading with varnishadm and varnishstat or Solution: Configure Threading with varnishadm and varnishstat.