HTTP

This chapter is for the web-developer course only

This chapter covers:

• Protocol basics
• Requests and responses
• HTTP request/response control flow
• Statelessness and idempotence
• Cache related headers

Varnish is designed to be used with HTTP semantics. These semantics are specified in the version called HTTP/1.1. This chapter covers the basics of HTTP as a protocol, its semantics and the caching header fields most commonly used.

Protocol Basics

Fig. 18 HTTP request/response control flow diagram

• Hyper-Text Transfer Protocol, HTTP, is at the core of the web
• Specified by the IETF, there are two main versions: HTTP/1.1 and HTTP/2
• Varnish 4.0 supports HTTP/1.1

HTTP is a networking protocol for distributed systems. It is the foundation of data communication for the web. The development of this standard is done by the IETF and the W3C. In 2014, RFCs 723X were published to clarify HTTP/1.1 and they obsolete RFC 2616.

A new version of HTTP called HTTP/2 has been released under RFC 7540. HTTP/2 is an alternative to HTTP/1.1 and does not obsolete the HTTP/1.1 message syntax. HTTP’s existing semantics remain unchanged.

The protocol allows multiple requests to be sent in serial mode over a single connection. If a client wants to fetch resources in parallel, it must open multiple connections.

Resources and Representations

• Resource: target of an HTTP request
• A resource may have different representations
• Representation: a particular instantiation of a resource
• A representation may have different states: past, current and desired

Each resource is identified by a Uniform Resource Identifier (URI), as described in Section 2.7 of [RFC7230]. A resource can be anything and such a thing can have different representations. A representation is an instantiation of a resource. An origin server, a.k.a. backend, produces this instantiation based on a list of request field headers, e.g., User-Agent and Accept-encoding.

When an origin server produces different representations of one resource, it includes a Vary response header field. This response header field is used by Varnish to differentiate between resource variations. More details on this are in the Vary subsection.

An origin server might include metadata to reflect the state of a representation. This metadata is contained in the validator header fields ETag and Last-Modified.

In order to construct a response for a client request, an algorithm is used to evaluate and select one representation with a particular state. This algorithm is implemented in Varnish and you can customize it in your VCL code. Once a representation is selected, the payload for a 200 (OK) or 304 (Not Modified) response can be constructed.

Requests and Responses

• A request is a message from a client to a server that includes the method to be applied to a requested resource, the identifier of the resource, the protocol version in use and an optional message body
• A method is a token that indicates the method to be performed on a URI
• Standard request methods are: GET, POST, HEAD, OPTIONS, PUT, DELETE, TRACE, or CONNECT
• Examples of URIs are /img/image.png or /index.html
• Header fields are allowed in requests and responses
• Header fields allow client and servers to pass additional information
• A response is a message from a server to a client that consists of a response status, headers and an optional message body

The first line of a request message is called Request-Line, whereas the first line of a response message is called Status-Line. The Request-Line begins with a method token, followed by the requested resource (URI) and the protocol version.

A request method informs the web server what sort of request this is: Is the client trying to fetch a resource (GET), update some data (POST) at the backend, or just get the headers of a resource (HEAD)? Methods are case-sensitive.

After the Request-Line, request messages may have an arbitrary number of header fields. For example: Accept-Language, Cookie, Host and User-Agent.

Message bodies are optional but they must comply to the requested method. For instance, a GET request should not contain a request body, but a POST request may contain one. Similarly, a web server cannot attach a message body to the response of a HEAD request.

The Status-Line of a response message consists of the protocol version followed by a numeric status code and its associated textual phrase. This associated textual phrase is also called reason. Important is to know that the reason is intended for the human user. That means that the client is not required to examine the reason, as it may change and it should not affect the protocol. Examples of status codes with their reasons are: 200 OK, 404 File Not Found and 304 Not Modified.

Responses also include header fields after the Status-Line, which allow the server to pass additional information about the response. Examples of response header fields are: Age, ETag, Cache-Control and Content-Length.

Note

Requests and responses share the same syntax for headers and message body, but some headers are request- or response-specific.

Request Example

GET / HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; fr; rv:1.9.2.16) \
Gecko/20110319 Firefox/3.6.16
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cache-Control: max-age=0

The above example is a typical HTTP request that includes a Request-Line, and headers, but no message body. The Request-Line consists of the GET request for the / resource and the HTTP/1.1 version. The request includes the header fields Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset, Keep-Alive, Connection and Cache-Control.

Note that the Host header contains the hostname as seen by the browser. The above request was generated by entering http://localhost/ in the browser. Most browsers automatically add a number of headers.

Some of the headers will vary depending on the configuration and state of the client. For example, language settings, cached content, forced refresh, etc. Whether the server honors these headers will depend on both the server in question and the specific header.

The following is an example of an HTTP request using the POST method, which includes a message body:

POST /accounts/ServiceLoginAuth HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; fr; rv:1.9.2.16) \
Gecko/20110319 Firefox/3.6.16
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: https://www.google.com/accounts/ServiceLogin
Cookie: GoogleAccountsLocale_session=en;[...]
Content-Type: application/x-www-form-urlencoded
Content-Length: 288

ltmpl=default[...]&signIn=Sign+in&asts=

Response Example

HTTP/1.1 200 OK
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.7
Cache-Control: public, max-age=86400
Last-Modified: Mon, 04 Apr 2011 04:13:41 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie,Accept-Encoding
ETag: "1301890421"
Content-Type: text/html; charset=utf-8
Content-Length: 23562
Date: Mon, 04 Apr 2011 09:02:26 GMT
X-Varnish: 1886109724 1886107902
Age: 17324
Via: 1.1 varnish
Connection: keep-alive

[data]

The example above is an HTTP response that contains a Status-Line, headers and message body. The Status-Line consists of the HTTP/1.1 version, the status code 200 and the reason OK. The response status code informs the client (browser) whether the server understood the request and how it replied to it. These codes are fully defined in https://tools.ietf.org/html/rfc2616#section-10, but here is an overview of them:

• 1xx: Informational – Request received, continuing process
• 2xx: Success – The action was successfully received, understood, and accepted
• 3xx: Redirection – Further action must be taken in order to complete the request
• 4xx: Client Error – The request contains bad syntax or cannot be fulfilled
• 5xx: Server Error – The server failed to fulfill an apparently valid request

HTTP Characteristics

• HTTP is a stateless protocol
• Common methods: safe, idempotent and cacheable
• Most common cacheable request methods are GET and HEAD

HTTP is by definition a stateless protocol meaning that each request message can be understood in isolation. Hence, a server MUST NOT assume that two requests on the same connection are from the same user agent unless the connection is secured and specific to that agent.

HTTP/1.1 persists connections by default. This is contrary to most implementations of HTTP/1.0, where each connection is established by the client prior to the request and closed by the server after sending the response. Therefore, for compatibility reasons, persistent connections may be explicitly negotiated as they are not the default behavior in HTTP/1.0 [https://tools.ietf.org/html/rfc7230#appendix-A.1.2]. In practice, there is a header called Keep-Alive you may use if you want to control the connection persistence between the client and the server.

A method is “safe” if it is read-only; i.e., the client request does not alter any state on the server. GET, HEAD, OPTIONS, and TRACE methods are defined to be safe. An idempotent method is such that multiple identical requests have the same effect as a single request. PUT, DELETE and safe requests methods are idempotent.

Cacheable methods are those that allow to store their responses for future reuse. RFC7231 specifies GET, HEAD and POST as cacheable. However, responses from POST are very rarely treated as cacheable. [https://tools.ietf.org/html/rfc7231#section-4.2]

Cache-related Headers Fields

• HTTP provides a list of headers fields to control cache behavior

• The most important caching header fields are:

  • Expires
  • Cache-Control
  • Etag
  • Last-Modified
  • If-Modified-Since
  • If-None-Match
  • Vary
  • Age

A cached object is a local store of HTTP response messages. These objects are stored, controlled, retrieved and deleted by a subsystem, in this case Varnish. For this purpose, Varnish uses the caching header fields defined in https://tools.ietf.org/html/rfc7232 and https://tools.ietf.org/html/rfc7234.

If a matched cache is valid, Varnish constructs responses from caches. As a result, the origin server is freed from creating and transmitting identical response bodies.

Constructing Responses from Caches

When to serve a cached object?

• The cached object is properly matched: cache-hit
• The requested method and its matched object allows it
• The freshness of the cached object is acceptable

When Varnish matches a request with a cached object (aka cache-hit), it evaluates whether the cache or origin server should be used to construct the response. There are many rules that should be taken into consideration when validating a cache. Most of those rules use caching header fields.

This subsection describes first the concept of cache-hit and cache-miss. After that, it describes three header fields commonly used to effectively match caches. These fields are Vary, Etag and If-None-Match.

Cache Matching

Fig. 19 Cache-hit control flow diagram

• Cache-hits are used to reuse, update or invalidate caches
• Objects may have variants (Vary and Etag)

Fig. 20 Cache-miss control flow diagram

Fig. 19 shows the flow diagram of a cache-hit. A cache-hit occurs when the requested object (URI) matches a stored HTTP response message (cache). If the matched stored message is valid to construct a response for the client, Varnish serves construct a response and serves it without contacting the origin server.

Fig. 20 shows the flow diagram of a cache-miss. A cache-miss happens when Varnish does not match a cache. In this case, Varnish forwards the request to the origin server.

Vary

• Selects a representation of a resource
• Be careful when using Vary
• Wrong usage can create a very large number of cached objects and reduce efficiency

If the origin server sends Vary in a response, Varnish does not use this response to satisfy a later request unless the later request has the same values for the listed fields in Vary as the original request. As a consequence, Vary expands the cache key required to match a new request to the stored cache entry.

Vary is one of the trickiest headers to deal with when caching. A caching server like Varnish does not necessarily understand the semantics of a header, or what part triggers different variants of a response. In other words, an inappropriate use of Vary might create a very large number of cached objects, and reduce the efficiency of your cache server. Therefore, you must be extremely cautious when using Vary.

Caching objects taking into consideration all differences from requesters creates a very fine-grained caching policy. This practice is not recommended, because those cached objects are most likely retrieved only by their original requester. Thus, fine-grained caching strategies do not scale well. This is a common mistake if Vary is not used carefully.

An example of wrong usage of Vary is setting Vary: User-Agent. This tells Varnish that for absolutely any difference in User-Agent, the response from the origin server might look different. This is not optimal because there are probably thousands of User-Agent strings out there.

Another example of bad usage is when using Vary: Cookie to differentiate a response. Again, there could be a very large number of cookies and hence a very large number of cached objects, which are going to be retrieved most likely only by their original requesters.

The most common usage of Vary is Vary: Accept-Encoding, which tells Varnish that the content might look different depending on the request Accept-Encoding header. For example, a web page can be delivered compressed or uncompressed depending on the client. For more details on how to use Vary for compressions, refer to https://www.varnish-cache.org/docs/trunk/users-guide/compression.html.

One way to assist Vary is by building the response body from cached and non-cached objects. We will discuss this further in the Content Composition chapter.

Varnish Test Cases (VTC) in varnishtest can also help you to understand and isolate the behavior of Vary. For more information about it, refer to the subsection Understanding Vary in varnishtest.

Note

Varnish can handle Accept-Encoding and Vary: Accept-Encoding, because Varnish has support for gzip compression.

ETag

• An Entity Tag (ETag) is metadata to differentiate between multiple states of a resource’s representation
• A differentiator key of presentations in addition to Vary
• An ETag is a validator header field
• Response header field

Origin servers normally add metadata to further describe the representation of a resource. This metadata is used in conditional requests. “Conditional requests are HTTP requests that include one or more header fields indicating a precondition to be tested before applying the method semantics to the target resource” [RFC7232]. ETag is a validator header field.

The ETag header field provides a state identifier for the requested variant (resource representation). ETags are used to differentiate between multiple states based on changes over time of a representation. In addition, ETags are also used differentiate between multiple representations based on content negotiation regardless their state.

Example of an ETag header:

ETag: "1edec-3e3073913b100"

The response ETag field is validated against the request If-None-Match header field. We will see the details of If-None-Match later in this subsection, but before, we learn about the other validator header field: Last-Modified.

Last-Modified

• Time-based state of presentations
• Validator header field
• Response header field

The Last-Modified response header field is a timestamp that indicates when the variant was last modified. This headers may be used in conjunction with If-Modified-Since and If-None-Match.

Example of a Last-Modified header:

Last-Modified: Wed, 01 Sep 2004 13:24:52 GMT

ETag and Last-Modified are validator header fields, which help to differentiate between representations. Normally, origin servers send both fields in successful responses. Whether you use one, another or both, depends on your use cases. Please refer to Section 2.4 in https://tools.ietf.org/html/rfc7232#section-2.4 for a full description on when to use either of them.

If-None-Match

• Precondition Header Field
• Request header field
• Validates local caches against ETag

A client that has obtained a response message and stored it locally, may reuse the obtained ETag value in future requests to validate its local cache against the selected cache in Varnish. The obtained value from an ETag is sent from the client to Varnish in the request If-None-Match header field. In fact, a client may have stored multiple resource representations and therefore a client may send an If-None-Match field with multiple ETag values to validate.

The purpose of this header field is to reuse local caches without compromising its validity. If the local cache is valid, Varnish replies with a 304 (Not Modified) response, which does not include a message body. In this case, the client reuses its local cache to construct the requested resource.

Example of an If-None-Match header:

If-None-Match: "1edec-3e3073913b100"

Fig. 21 If-None-Match control diagram.

If-Modified-Since

• Validates local caches by modification date
• Precondition Header Field
• Request header field

A request containing an If-Modified-Since header field indicates that the client wants to validate one or more of its local caches by modification date. If the requested representation has not been modified since the time specified in this field, Varnish returns a 304 (not modified) response. A 304 response does not contain message body. This behavior is similar to as when using If-None-Match.

Example of an If-Modified-Since header:

If-Modified-Since: Wed, 01 Sep 2004 13:24:52 GMT

Fig. 22 If-Modified-Since control flow diagram.

Tip

The subsection Understanding Last-Modified and If-Modified-Since in varnishtest explains further these concepts with a practical VTC example.

Allowance

• How to control which caches can be served?
• Cache-Control and Pragma (for backwards compatibility only)

Varnish allows you to validate whether the stored response (cache) can be reused or not. Validation can be done by checking whether the presented request does not contain the no-cache directive. This subsection reviews two common header fields, Cache-Control and Pragma, to check caching allowance.

Cache-Control

The Cache-Control header field specifies directives that must be applied by all caching mechanisms (from proxy cache to browser cache).

Table 11 Most common cache-control argument for each context

Directives	Request	Response
no-cache	X	X
no-store	X	X
max-age	X	X
s-maxage		X
max-stale	X
min-fresh	X
no-transform	X	X
only-if-cached	X
public		X
private		X
must-revalidate		X
proxy-revalidate		X

Table 11 summarizes the directives you may use for each context. The most relevant directives of Cache-Control are:

• public: The response may be cached by any cache.
• no-store: The response body must not be stored by any cache mechanism.

• no-cache: Authorizes a cache mechanism to store the response in its cache but it must not reuse it without validating it with the origin server first.

  In order to avoid any confusion with this argument think of it as a “store-but-do-no-serve-from-cache-without-revalidation” instruction.

• max-age: Specifies the period in seconds during which the cache is considered fresh.
• s-maxage: Like max-age but it applies only to public caches.

Example of a Cache-Control header:

Cache-Control: public, max-age=2592000

A more hands-on explanation as VTC can be found in the subsection Understanding Cache-Control in varnishtest.

Note

Cache-Control always overrides Expires.

Note

By default, Varnish does not care about the Cache-Control request header. If you want to let users update the cache via a force refresh you need to do it yourself.

Pragma

• Only for legacy
• Treat Pragma: no-cache as Cache-Control: no-cache

The Pragma request header is a legacy header and should no longer be used. Some applications still send headers like Pragma: no-cache but this is for backwards compatibility reasons only. Any proxy cache should treat Pragma: no-cache as Cache-Control: no-cache, and should not be seen as a reliable header especially when used as a response header.

Freshness

• Fresh object: age has not yet exceeded its freshness lifetime
• Stale object: has exceeded its freshness lifetime, i.e., expired object

When reusing a stored response (cached object), you should always check its freshness and evaluate whether to deliver expired objects or not. A response’s freshness lifetime is the length of time between its generation by the origin server and its expiration time. A stale (expired) object can also be reused, but only after further validation with the origin server.

As defined in RFC7234 [https://tools.ietf.org/html/rfc7234#section-4.2]:

A response’s age is the time that has passed since it was generated by, or successfully validated with, the origin server. The primary mechanism for determining freshness is for an origin server to provide an explicit expiration time in the future, using either the ``Expires`` header field or the ``max-age`` response directive.

Age

• Response header field calculated at the cache server, i.e., Varnish
• Varnish sends an additional response header field, Age, to indicate the age of the cache
• Clients (and Varnish) will use the Age header field to determine the freshness of a cache
• max-age-based equation: cache duration = max-age - Age
• Age can be used to disallow caches at the client side

Consider what happens if you let Varnish cache content for a week. If Varnish does not calculate the age of a cached object, Varnish might happily inform clients that the content is fresh, but the cache could be older than the maximum allowed max-age. By age we mean an estimate amount of time since the response was generated or successfully validated at the origin server.

Client browsers calculate a cache duration based on the Age header field and the max-age directive from Cache-Control. If this calculation results in a negative number, the browser does not cache the response locally. Negative cache duration times, however, do not prevent browsers from using the received object. Varnish does the same, if you put one Varnish server in front of another.

Exercise: Use article.php to test Age

1. Copy article.php from Varnish-Book/material/webdev/ to /var/www/html/
2. Send a request to article.php via Varnish, and see the response Age header field in varnishlog -g request -i ReqHeader,RespHeader
3. Click the link several times and refresh your browser. Can you identify patterns?
4. Analyze the output of varnishstat -f MAIN.client_req -f MAIN.cache_* in addition to varnishlog
5. Can you use the Age field to determine whether Varnish made a cache hit?
6. What is the difference between caching time in Varnish and the client?
7. Use different browsers and analyze whether their internal caches work different

You might encounter that different browsers have different behaviors. Some of them might cache content locally, and their behavior when refreshing might be different, which can be very confusing. This just highlights that Varnish is not the only part of your web-stack that parses and honors cache-related headers. There might also be other caches along the way which you do not control, like a company-wide proxy server.

Since browsers might interpret cache headers differently, it is a good idea to control them in your cache server. In the next chapters, you will learn how to modify the response headers Varnish sends. This also allows your origin server to emit response headers that should be seen and used by Varnish only, not in your browser.

When browsers decide to load a resource from their local cache, requests are never sent. Therefore this exercise and these type of tests are not possible to be simulated in varnishtest.

Expires

• Used to stale objects
• Response header field only

The Expires response header field gives the time after which the response is considered stale. Normally, a stale cache item should not be returned by any cache (proxy cache or client cache). The syntax for this header is:

Expires: GMT formatted date

It is recommended not to define Expires too far in the future. Setting it to 1 year is usually enough. The use of Expires does not prevent the cached object from being updated. For example, if the name of the resource is updated.

Expires and Cache-Control do more or less the same job, but Cache-Control gives you more control. The most significant differences between these two headers is:

• Cache-Control uses relative times in seconds
• Cache-Control is both a request and a response header field.
• Expires always returns an absolute time
• Expires is a response header field only

Tip

To learn more about the behavior of Expires, refer to the subsection Understanding Expires in varnishtest.

Availability of Header Fields

Table 12 Summary of HTTP header fields and their scope

Header	Request	Response
Expires		X
Cache-Control	X	X
Last-Modified		X
If-Modified-Since	X
If-None-Match	X
Etag		X
Pragma	X	X
Vary		X
Age		X

Exercise: Test Various Cache Headers Fields with a Real Browser

Against a real backend:

1. Use httpheadersexample.php via your Varnish server to experiment and get a sense of what it is all about.
2. Copy the PHP file from Varnish-Book/material/webdev/ to /var/www/html/
3. Use varnishstat -f MAIN.client_req -f MAIN.cache_hit and varnishlog -g request -i ReqHeader,RespHeader to analyze the responses.
4. Try every link several times by clicking on them and refreshing your browser.
5. Analyze the response in your browser and the activity in your Varnish server.
6. Discuss what happens when having the Cache-Control and Expires fields in the third link.
7. When testing Last-Modified and If-Modified-Since, does your browser issue a request to Varnish? If the item was in cache, does Varnish query the origin server?
8. Try the Vary header field from two different browsers.

When performing this exercise, try to see if you can spot the patterns. There are many levels of cache on the web, and you have to consider them in addition to your Varnish installation. If it has not happened already, it is likely that the local cache of your browser will confuse you at least a few times through this course. When that happens, pull up varnishlog, varnishstat and another browser, or use client mock-ups in varnishtest instead of browsers.