Apache sending “Vary: Host” making things uncacheable for Varnish

Apache sending “Vary: Host” making things uncacheable for Varnish

Author: seven February 13, 2017

TLDR;
Using %{HTTP_HOST} in .htaccess, will cause Apache to included a “Vary: Host” field in response.
Subsequently “Vary: Host” header from Apache will force Varnish not to cache otherwise cacheable content.

HTTP Vary is not a trivial concept. It is by far the most misunderstood HTTP header. (Varnish Docs)

On a project I’ve been working, I could not make Varnish hard cache the site no matter what I did.
It was mostly read-only site and I wanted to achieve that in case of backend failures – site would still run from the cache. To avoid surprises, I was using my own configuration template which was working exactly as I wanted it on different project.

But, no matter what I did – Varnish was not caching everything. Instead I got ‘per-user‘ cache: if user visited eg, homepage, in case of backend failure – user could reload homepage and Varnish would serve homepage from stale cache to the user. Visiting any other page user did not visit before backend failure (and therefore not in his cache), would result in Varnish freaking that backend is down.

What I noticed was strangely large Vary-Header in Varnish response

Vary:Host,Accept-Encoding,User-Agent

After spending hours of tunneling, debugging Varnish configuration, analysing header responses, comparing server configurations, hunting for cookies that could have been somehow magically slip through… last place I looked was my main .htaccess. I was using mod_deflate there, but as it checked out everything was fine.

On a side note, I have been experimenting with per host configuration in .htaccess, so Basic Authentication would not kick in dev enviroment. It was working great up until now, and it goes something like this:

<if "%{HTTP_HOST} == 'dev.nivas.hr'">

Require valid-user
...
Allow from facebook.com

</if>

What I had to find out the hard way, is that if special Apache enviroment variable %{HTTP_HOST} was used in a eg. .htaccess, Apache would change response header. Server was returning a header which included a “Vary: Host” field, which means that the server didn’t serve a regular static page, but its reply depends on the “Host” field in the HTTP request. Browsers interpret this as “the content returned is dynamic, don’t cache it (source).

curl -I http://localhost/
HTTP/1.1 200 OK
Date: Mon, 13 Feb 2017 14:48:14 GMT
Server: Apache/2.4.6 (CentOS)
Vary: Host,User-Agent
Content-Type: text/html; charset=UTF-8

It is really strange I did not hit this before because RewriteCond can add “Host” to the Vary-Header as well. eg. a RewriteCond that evaluates %{HTTP_HOST} automatically adds “Host” to the Vary-Header. This is unnecessary and not permitted according to https://tools.ietf.org/html/rfc7231#section-7.1.4. The issue was reported and has been sitting in Apache bugtraq for a while without clear resolution.

I can understand why Varnish is not caching, but cannot understand Apache logic. I do have VirtualHost defined, so therefore my request do vary on Host. There is no need in forcing this out in response.

After removing %{HTTP_HOST} from .htaccess, site was cacheable as we wanted.

HTTP/1.1 200 OK
Date: Mon, 13 Feb 2017 22:18:33 GMT
Content-Type: text/html; charset=UTF-8
Vary: Accept-Encoding
Age: 3707
X-Nivas-Crew: loves you :) http://www.nivas.hr
X-Backend: backend_app1
X-Cache: HIT
X-Cache-Hits: 786332
X-Vudu-Url-Cache: hit

Don’t forget to normalize your Vary in Varnish, chance are without normalization it will never see a cache hit.

Happy caching!

Have a cool Varnish project you need help on? Contact us.

Author
seven
CEO/CTO at Nivas®
Neven Jacmenović has been passionately involved with computers since late 80s, the age of Atari and Commodore Amiga. As one of internet industry pioneers in Croatia, since 90s, he has been involved in making of many award winning, innovative and successful online projects. He is an experienced full stack web developer, analyst and system engineer. In his spare time, Neven is transforming retro-futuristic passion into various golang, Adobe Flash and JavaScript/WebGL projects.

    Leave a Reply

    Your email address will not be published. Required fields are marked *