Appendix G: Solutions

This appendix contains the solutions of exercises throughout the book.

Solution: Install Varnish

• Solution for systemd Ubuntu and Debian
• Solution for systemd Fedora/RHEL7+/CentOS 7+

Varnish is already distributed in many package repositories, but those packages might contain an outdated Varnish version. Therefore, we recommend you to use the packages provided by varnish-software.com for Varnish Cache Plus or varnish-cache.org for Varnish Cache. Please be advised that we only provide packages for LTS releases, not all the intermediate releases. However, these packages might still work fine on newer releases.

All software related to Varnish Cache Plus including VMODs are available in RedHat and Debian package repositories. These repositories are available on http://repo.varnish-software.com/, using your customer specific username and password.

All the following commands are for systemd Ubuntu or Debian and must be executed with root permissions. First, make sure you have apt-transport-https:

$ apt-get install apt-transport-https

To use the varnish-software.com repository and install Varnish Cache Plus 4.0 or 4.1 on Ubuntu 14.04 trusty:

$ curl https://<username>:<password>@repo.varnish-software.com/GPG-key.txt \
  | apt-key add -

To use the varnish-cache.org repository and install Varnish Cache 4.0 or 4.1 on Ubuntu 14.04 trusty:

$ curl https://repo.varnish-cache.org/ubuntu/GPG-key.txt | apt-key add -
$ echo "deb https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.0" >> \
  /etc/apt/sources.list.d/varnish-cache.list

If you are installing Varnish Cache 4.1, replace varnish-4.0 for varnish-4.1 in the command above.

If you are installing Varnish Cache Plus 4.0 or 4.1, add the repositories for VMODs in /etc/apt/sources.list.d/varnish-4.0-plus.list or /etc/apt/sources.list.d/varnish-4.1-plus.list respectively:

# Remember to replace 4.x, DISTRO and RELEASE with what applies to your system.
# 4.x=(4.0|4.1)
# DISTRO=(debian|ubuntu),
# RELEASE=(precise|trusty|wheezy|jessie)

# Varnish Cache Plus 4.x and VMODs
deb https://<username>:<password>@repo.varnish-software.com/DISTRO RELEASE \
varnish-4.x-plus



# non-free contains VAC, VCS and proprietary VMODs.
deb https://<username>:<password>@repo.varnish-software.com/DISTRO RELEASE \
non-free

Re-synchronize the package index files of your repository:

$ apt-get update

To install Varnish Cache Plus and VMODs:

$ apt-get install varnish-plus
$ apt-get install varnish-plus-addon-ssl
$ apt-get install varnish-plus-vmods-extra

To install Varnish-Cache:

$ apt-get install varnish

Finally, verify the version you have installed:

$ varnishd -V

To use Varnish Cache Plus 4.0 or 4.1 repositories on systemd Fedora/RHEL7+/CentOS 7+, put the following in /etc/yum.repos.d/varnish-4.0-plus.repo or /etc/yum.repos.d/varnish-4.1-plus.repo, and change 4.x for the version you want to install:

[varnish-4.x-plus]
name=Varnish Cache Plus
baseurl=https://<username>:<password>@repo.varnish-software.com/redhat
/varnish-4.x-plus/el$releasever
enabled=1
gpgcheck=0

[varnish-admin-console]
name=Varnish Administration Console
baseurl=
https://<username>:<password>@repo.varnish-software.com/redhat
/vac/el$releasever
enabled=1
gpgcheck=0

Then, execute the following:

$ yum update
$ yum install varnish-plus
$ yum install varnish-plus-vmods-extra

Finally, verify the version you have installed:

$ varnishd -V

Note

More details on Varnish Plus installation can be found at http://files.varnish-software.com/pdfs/varnish-cache-plus-manual-latest.pdf

Solution: Test Apache as Backend with varnishtest

vtc/b00002.vtc

varnishtest "Apache as Backend"

varnish v1 -arg "-b 127.0.0.1:8080" -start

client c1 {
      txreq
      rxresp

      expect resp.http.Server ~ "Apache"
      expect resp.http.Via ~ "varnish"
} -run

Solution: Assert Counters in varnishtest

vtc/b00005.vtc

varnishtest "Counters"

server s1 {
   rxreq
   txresp
} -start

varnish v1 -arg "-b ${s1_addr}:${s1_port}" -start

client c1 {
   txreq
   rxresp
}

varnish v1 -expect cache_miss == 0
varnish v1 -expect cache_hit == 0
varnish v1 -expect n_object == 0

client c1 -run

varnish v1 -expect cache_miss == 1
varnish v1 -expect cache_hit == 0
varnish v1 -expect n_object == 1

client c1 -run
client c1 -run

varnish v1 -expect cache_miss == 1
varnish v1 -expect cache_hit == 2
varnish v1 -expect n_object == 1

Solution: Tune first_byte_timeout and test it against mock-up server

vtc/b00006.vtc

varnishtest "Check that the first_byte_timeout works from parameters"

feature SO_RCVTIMEO_WORKS

server s1 {
        rxreq
        delay 1.5
        txresp -body "foo"
} -start

varnish v1 -vcl+backend {} -start
varnish v1 -cliok "param.set first_byte_timeout 1"

client c1 {
        txreq
        rxresp
        expect resp.status == 503
} -run


server s1 {
        rxreq
        delay 0.5
        txresp -body "foo"
} -start

client c1 {
        txreq
        rxresp
        expect resp.status == 200
} -run

In this example, we introduce -vcl+backend and feature in VTC. -vcl+backend is one way to pass inline VCL code and backend to v1. In this example, v1 receives no inline VCL injects declaration of the backend s1. Thus, -vcl+backend{} is equivalent to -arg "-b ${s1_addr}:${s1_port}" in this case.

feature checks for features to be present in the test environment. If the feature is not present, the test is skipped. SO_RCVTIMEO_WORKS checks for the socket option SO_RCVTIMEO before executing the test.

b00006.vtc is copied from Varnish-Cache/bin/varnishtest/tests/b00023.vtc We advise you to take a look at the many tests under Varnish-Cache/bin/varnishtest/tests/. You will learn so much about Varnish when analyzing them.

Solution: Configure vcl_recv to avoid caching all requests to the URL /admin

VCL code:

vcl 4.0;

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

sub vcl_recv {
  if (req.url ~ "^/admin") {
    return(pass);
  }
}

Command to compile and visualize result in less:

varnishd -C -f recv.vcl 2>&1 | less

In this suggested solution, the backend is configured to a local IP address and port. Since we are not running this code, you can configure it as you want.

Note the use of the match comparison operator ~ in regular expression.

In the output of the compiler you should be able to find your VCL code and the built-in VCL code appended to it.

Solution: Configure Threading with varnishadm and varnishstat

• Use varnishadm param.set to set the value of thread_pool_min and thread_pool_max.
• Remember that the values are per thread pool, the default value for thread_pools is 2.
• Monitor the MAIN.threads counter in varnishstat to see how many threads are running while performing this exercise.

Solution: Configure Threading with varnishtest

c00001.vtc

varnishtest "Configure Number of Worker-threads"

server s1 {
        rxreq
        txresp
} -start

varnish v1 \
-arg "-p vsl_mask=+WorkThread" \
-arg "-p thread_pool_min=10" \
-arg "-p thread_pool_max=15" \
-arg "-p thread_pools=1" \
-vcl+backend {}
varnish v1 -start

varnish v1 -expect threads == 10

logexpect l1 -v v1 -g raw {
        expect * 0 WorkThread {^\S+ start$}
        expect * 0 WorkThread {^\S+ end$}
} -start

varnish v1 -cliok "param.set thread_pool_min 11"

# Herder thread might sleep up to 5 seconds. Have to wait longer than that.
delay 6

varnish v1 -expect threads == 11

varnish v1 -cliok "param.set thread_pool_min 10"
varnish v1 -cliok "param.set thread_pool_max 10"

# Herder thread might sleep up to 5 seconds. Have to wait longer than that.
delay 6

varnish v1 -expect threads == 10

# Use logexpect to see that the thread actually exited
logexpect l1 -wait

The test above shows you how to set parameters in two ways; passing the argument -p to varnishd or calling param.set. -p vsl_mask=+WorkThread is used to turn on WorkThread debug logging.

The test proves that varnishd starts with the number of threads indicated in thread_pool_min. Changes in thread_pool_min and thread_pool_max are applied by the thread herder, which handles the thread pools and adds or removes threads if necessary. To learn more about other maintenance threads, visit https://www.varnish-cache.org/trac/wiki/VarnishInternals.

c00001.vtc is a simplified version of Varnish-Cache/bin/varnishtest/tests/r01490.vtc.

Solution: Rewrite URL and Host Header Fields

sub vcl_recv {

    set req.http.x-host = req.http.host;
    set req.http.x-url = req.url;

    set req.http.host = regsub(req.http.host, "^www\.", "");

    /* Alternative 1 */
    if (req.http.host == "sport.example.com") {
        set req.http.host = "example.com";
        set req.url = "/sport" + req.url;
     }

     /* Alternative 2 */
     if (req.http.host ~ "^sport\.") {
       set req.http.host = regsub(req.http.host,"^sport\.", "");
       set req.url = regsub(req.url, "^", "/sport");
     }
}

You can test this solution via HTTPie or varnishtest.

Using HTTPie:

http -p hH --proxy=http:http://localhost sport.example.com/index.html

Then you verify your results by issuing the following command and analyzing the output:

varnishlog -i ReqHeader,ReqURL

varnishtest solution:

varnishtest "Rewrite URL and Host Header Fields"

server s1 {
        rxreq
        expect req.http.Host == "example.com"
        expect req.http.ReqURL == "/sport/index.html"
        txresp
}

varnish v1 -vcl {
   backend default {
      .host = "93.184.216.34"; # example.com
      .port = "80";
   }
   sub vcl_recv {
       set req.http.x-host = req.http.host;
       set req.http.x-url = req.url;
       set req.http.host = regsub(req.http.host, "^www\.", "");

       /* Alternative 1 */
       if (req.http.host == "sport.example.com") {
           set req.http.host = "example.com";
           set req.url = "/sport" + req.url;
       }

       /* Alternative 2 */
       # if (req.http.host ~ "^sport\.") {
       #     set req.http.host = regsub(req.http.host,"^sport\.", "");
       #     set req.url = regsub(req.url, "^", "/sport");
       # }
   }
} -start

client c1 {
        txreq -url "/index.html" -hdr "Host: sport.example.com"
        rxresp
} -run

Solution: Avoid caching a page

// Suggested solution A
sub vcl_recv {
     if (req.url ~ "^/index\.html" || req.url ~ "^/$") {
             return(pass);
     }
}

// Suggested solution B
sub vcl_backend_response {
    if (bereq.url ~ "^/index\.html" || bereq.url ~ "^/$") {
       set beresp.uncacheable = true;
    }
}

Usually it is most convenient to do as much as possible in vcl_recv. The usage of bereq.uncacheable in vcl_backend_fetch creates a hit-for-pass object. See the hit-for-pass section for detailed description about this type of object.

Solution: Either use s-maxage or set TTL by file type

sub vcl_backend_response {
    if (beresp.http.cache-control !~ "s-maxage") {
        if (bereq.url ~ "\.jpg(\?|$)") {
            set beresp.ttl = 30s;
            unset beresp.http.Set-Cookie;
        }
        if (bereq.url ~ "\.html(\?|$)") {
            set beresp.ttl = 10s;
            unset beresp.http.Set-Cookie;
        }
    } else {
        if (beresp.ttl > 0s) {
            unset beresp.http.Set-Cookie;
        }
    }
}

There are many ways to solve this exercise, and this solution is only one of them. The first condition checks the presence of s-maxage and handles .jpg and .html files to make them cacheable for 30 and 10 seconds respectively. If s-maxage is present with a positive TTL, we consider the response cacheable by removing beresp.http.Set-Cookie.

Solution: Modify the HTTP response header fields

sub vcl_deliver {
    set resp.http.X-Age = resp.http.Age;
    unset resp.http.Age;

    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Solution: Change the error message

vcl/customized_error.vcl

/* Change your backend configuration to provoke a 503 error */
backend default {
    .host = "127.0.0.1";
    .port = "8081";
}

/* Customize error responses */
sub vcl_backend_error {
    if (beresp.status == 503){
        set beresp.status = 200;
        synthetic( {"
            <html><body><!-- Here goes a more friendly error message. -->
            </body></html>
        "} );
        return (deliver);
    }
}

The suggested solution forces a 503 error by misconfiguring .port in the default backend. You can also force a 503 response by using ${bad_ip} in varnishtest. The macro ${bad_ip} translates to 192.0.2.255.

vtc/b00011.vtc

varnishtest "Force 503 error"

varnish v1 -vcl {
   backend foo {
           .host = "${bad_ip}";
           .port = "9080";
   }
   /* Customize error responses */
   sub vcl_backend_error {
       if (beresp.status == 503){
          set beresp.status = 200;
          synthetic( {"
              <html><body><!-- Here goes a more friendly error message. -->
              </body></html>
          "} );
          return (deliver);
       }
   }
} -start

client c1 {
        txreq -url "/"
        rxresp
        expect resp.status == 200
} -run

Note that in the proposed solution the client receives a 200 response code.

Solution: PURGE an article from the backend

purgearticle.php

<?php
header( 'Content-Type: text/plain' );
header( 'Cache-Control: max-age=0' );
$hostname = 'localhost';
$port     = 80;
$URL      = '/article.php';
$debug    = true;

print "Updating the article in the database ...\n";
purgeURL( $hostname, $port, $URL, $debug );

function purgeURL( $hostname, $port, $purgeURL, $debug )
{
    $finalURL = sprintf(
        "http://%s:%d%s", $hostname, $port, $purgeURL
    );

    print( "Purging ${finalURL}\n" );

    $curlOptionList = array(
        CURLOPT_RETURNTRANSFER    => true,
        CURLOPT_CUSTOMREQUEST     => 'PURGE',
        CURLOPT_HEADER            => true ,
        CURLOPT_NOBODY            => true,
        CURLOPT_URL               => $finalURL,
        CURLOPT_CONNECTTIMEOUT_MS => 2000
    );

    $fd = false;
    if( $debug == true ) {
        print "\n---- Curl debug -----\n";
        $fd = fopen("php://output", 'w+');
        $curlOptionList[CURLOPT_VERBOSE] = true;
        $curlOptionList[CURLOPT_STDERR]  = $fd;
    }

    $curlHandler = curl_init();
    curl_setopt_array( $curlHandler, $curlOptionList );
    curl_exec( $curlHandler );
    curl_close( $curlHandler );
    if( $fd !== false ) {
        fclose( $fd );
    }
}
?>

solution-purge-from-backend.vcl

acl purgers {
    "127.0.0.1";
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (!client.ip ~ purgers) {
            return (synth(405, "Not allowed."));
        }
        return (purge);
    }
}

Solution: Write a VCL program using purge and ban

sub vcl_recv {
    if (req.method == "PURGE") {
        return (purge);
    }

    if (req.method == "BAN") {
        ban("obj.http.x-url ~ " + req.http.x-ban-url +
            " && obj.http.x-host ~ " + req.http.x-ban-host);
        return (synth(200, "Ban added"));
    }

    if (req.method == "REFRESH") {
        set req.method = "GET";
        set req.hash_always_miss = true;
    }
}

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;
}

sub vcl_deliver {
    # We remove resp.http.x-* HTTP header fields,
    # because the client does not neeed them
    unset resp.http.x-url;
    unset resp.http.x-host;
}

Solution: Handle Cookies with Vary in varnishtest

vtc/c00003.vtc

varnishtest "Purge objects with Vary: Cookie"

server s1 {
   rxreq
   expect req.url == "/cookie.php"
   txresp -hdr "Vary: Cookie"
   rxreq
   expect req.url == "/cookie.php"
   txresp -hdr "Vary: Cookie"
   rxreq
   expect req.url == "/article.html"
   txresp -hdr "Vary: Cookie"
   rxreq
   expect req.url == "/cookie.php"
   txresp -hdr "Vary: Cookie"
   rxreq
   expect req.url == "/article.html"
   txresp -hdr "Vary: Cookie"
} -start

varnish v1 -vcl+backend {
   sub vcl_recv{
     if (req.method == "PURGE") {
        return (purge);
     }
     else if (req.http.Cookie){
        # Forces Varnish to cache requests with cookies
        return (hash);
     }
   }
   sub vcl_backend_response{
      # Uncomment to remove effect from Vary
      # unset beresp.http.Vary;
   }
} -start

client c1 {
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1001"
   txreq -url "/cookie.php" -hdr "Cookie: user: Bob"
   rxresp
   expect resp.http.X-Varnish == "1003"
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1005 1002"
   txreq -url "/article.html" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1006"
   txreq -url "/article.html" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1008 1007"
} -run

varnish v1 -expect n_object == 3

client c1 {
   txreq -req PURGE -url "/cookie.php"
   rxresp
} -run

varnish v1 -expect n_object == 1

client c1 {
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1012"
   txreq -url "/article.html" -hdr "Cookie: user: Bob"
   rxresp
   expect resp.http.X-Varnish == "1014"
} -run

varnish v1 -expect n_object == 3

Vary and hash_data() might behave very similar at first sight and they might even seem like alternatives for handling cookies. However, cached objects are referenced in different ways.

If Varnish is forced to store responses with cookies, Vary ensures that Varnish stores resources per URL and Cookie. If Vary: Cookie is used, objects are purged in this way:

txreq -req PURGE -url "/cookie.php"

but something different is needed when using hash_data(req.http.Cookie), as you can see it in the next suggested solution.

Solution: Handle Cookies with hash_data() in varnishtest

vtc/c00004.vtc

varnishtest "Purge objects after hash_data(cookie)"

server s1 {
   rxreq
   expect req.url == "/cookie.php"
   txresp
   rxreq
   expect req.url == "/cookie.php"
   txresp
   rxreq
   expect req.url == "/article.html"
   txresp
   rxreq
   expect req.url == "/cookie.php"
   txresp
   rxreq
   expect req.url == "/article.html"
   txresp
} -start

varnish v1 -vcl+backend {
   sub vcl_recv{
     if (req.method == "PURGE") {
        return (purge);
     }
     else if (req.http.Cookie){
        # Forces Varnish to cache requests with cookies
        return (hash);
     }
   }
   sub vcl_hash{
      hash_data(req.http.Cookie);
   }
} -start

client c1 {
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1001"
   txreq -url "/cookie.php" -hdr "Cookie: user: Bob"
   rxresp
   expect resp.http.X-Varnish == "1003"
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1005 1002"
   txreq -url "/article.html" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1006"
   txreq -url "/article.html" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1008 1007"
} -run

varnish v1 -expect n_object == 3

client c1 {
   txreq -req PURGE -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
} -run

varnish v1 -expect n_object == 2

client c1 {
   txreq -url "/cookie.php" -hdr "Cookie: user: Alice"
   rxresp
   expect resp.http.X-Varnish == "1012"
   txreq -url "/article.html" -hdr "Cookie: user: Bob"
   rxresp
   expect resp.http.X-Varnish == "1014"
} -run

varnish v1 -expect n_object == 4

hash_data(req.http.Cookie) adds the request header field Cookie to the hash key. So Varnish can discern between backend responses linked to a specific request header field.

To purge cached objects in this case, you have to specify the header field used in hash_data():

txreq -req PURGE -url "/cookie.php" -hdr "Cookie: user: Alice"

Solution: Write a VCL that masquerades XHR calls

vcl/solution-vcl_fetch-masquerade-ajax-requests.vcl

vcl 4.0;

backend localhost{
    .host = "127.0.0.1";
    .port = "8080";
}

backend google {
    .host = "173.194.112.145";
    .port = "80";
}

sub vcl_recv{
    if (req.url ~ "^/masq") {
        set req.backend_hint = google;
        set req.http.host = "www.google.com";
        set req.url = regsub(req.url, "^/masq", "");
        return (hash);
    } else {
        set req.backend_hint = localhost;
    }
}

Note that the getMasqueraded() works now after being processed in vcl_recv().