Thanks to scoof for documenting the Varnish VCL changes.

What is this for non-technical folks?

WordPress sucks at delivering the same content over and over again, actually, I should rephrase that, wordpress rocks 99% of the time but if you serve a page over and over again it will quickly exhaust your servers resource which will mean wordpress will go slow.  Varnish Cache speeds up WordPress by serving pages from memory instead of doinmagicg a bunch of hard work.

Why should I use this VCL?

I have done a number of different VCLs that can be used with varnish and wordpress but this is the final revision for now. It is designed for Varnish 2 but should work on 3 with no to very little modifications, I have tweaked this VCL over a few years and I’m finally happy with it.

Features:

• Supports multiple back-ends
• Supports round-robin
• Supports Purging using the Varnish WordPress plugin
• Supports logged in users
• Supports password protected pages
• Supports Mobile devices
• Provides long client side caching
• Forwards the correct client IP to the web daemon.
• Doesn’t cache 404s, 503, 500 etc.
• Doesn’t cache wp-admin, login, preview, signup
• Caches static objects such as images
• Supports Multisite
• Includes debug/log messages.
• Really clean code, proper tabbing etc.

Requirements

Varnish, WordPress, The Varnish WordPress plugin installed and working, mod_rpaf installed in apache or an nginx equivalent.

Why have I made this?

I was going to make a VCL generator but then I remembered most people will use this VCL as a point of reference and I’m lazy, let’s face it, being lazy is the biggest factor in me backing out of making a generator. The varnish configs I have done before have been overly verbose for what they did and rewriting them and cleaning them up means this config is much easier to understand and modify. Adding logging means that you can easily use varnishlog to debug any problems you have.

What isn’t included?

I didn’t include Custom error messages in this VCL.  It’s not because I’m lazy, it’s because custom error messages put a lot of cruft into the VCL and if you want custom error messages you should see this article.

Let me at it!

Varnish v3
Varnish v3.03
Which version of Varnish am I running?

varnishd -V

What do I need to change?

Search and replace myFirstServer and mySecondServer with your server names adding new backends where required. Make sure new backends are added to the round-robin cluster and the purge list.

Once you are happy with your VCL save it in as /etc/varnish/default.vcl (remember to make a backup of your original file) and restart Varnish.  Any problems try to debug yourself but if you are stuck just give me a shout, I will be happy to help!

Add this line to your varnish VCL in sub_recv:

set req.http.X-Forwarded-For = client.ip;

Grab mod_rpaf:

wget http://ftp.debian.org/debian/pool/main/liba/libapache2-mod-rpaf/libapache2-mod-rpaf_0.6-7_amd64.deb

Install mod_rpaf

dpkg -i libapache2-mod-rpaf_0.6-1_amd64.deb

Enable mod_rpaf:

a2enmod rpaf

Your rpaf config (/etc/apache2/mods-enabled/rpaf.conf) should look awesome like this:

<IfModule mod_rpaf.c>
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 123.123.123.123 10.0.0.2 127.0.0.1
RPAFheader HTTP_X_FORWARDED_FOR
</IfModule>

Note: RPAFproxy_ips is the ips of your varnish cache servers. Varnish is awesome.

Reload varnish and Apache the cool way.

/etc/init.d/varnish reload
/etc/init.d/apache reload

Test it by looking at your remote_addr variable:

print_r($_SERVER);

My Varnish config only included support for POST requests to xmlrpc.php

I updated my config to read:

// set backend, pipe, strip cookies from xmlrpc 
  if (req.request == "POST" && req.url ~ "xmlrpc.php") {remove req.http.cookie;set req.backend = mainserver;return(pipe);}
  if (req.request == "GET" && req.url ~ "xmlrpc.php") {remove req.http.cookie;set req.backend = mainserver;return(pipe);}

and it sorted it! Huraa

Look at an incoming client request of a specific URL:

varnishlog -c -m RxURL:"readysetlearn/readysetlearn.htm"

Look at a a backend request of a specific URL:

varnishlog -b -m TxURL:"readysetlearn/readysetlearn.htm"

See requests for one specific Hostname:

varnishlog -c -m RxHeader:"Host: etherpad.org"

See the age of the cache objects for a specific hostname:

varnishlog -c -m RxHeader:"Host: etherpad.org" | grep Age

Look at an incoming client request of a specific URL:

varnishlog -c -o RxURL readysetlearn/readysetlearn.htm

Look at a a backend request of a specific URL:

varnishlog -b -o TxURL readysetlearn/readysetlearn.htm

See requests for one specific Hostname:

varnishlog -c -o RxHeader "Host: etherpad.org"

See the age of the cache objects for a specific hostname:

varnishlog -c -o RxHeader "Host: etherpad.org" | grep Age

If an item has a high age it means that varnish is hitting the cache for it.

Thanks to Mithrandir from the #Varnish IRC channel for helping out and thanks to Tomnomnom for V3 help!.

yum install libxml2-devel
#OR
apt-get intall libxml2-dev
wget https://gist.github.com/raw/805710/9f34a18e528c20eff1c92672c6f1856ed849f5ea/wurfl.c
wget https://gist.github.com/raw/805710/b9272d8a1d32d29034574c88b81fc79eb050e21b/wurfl.h
gcc -c -o wurfl.o wurfl.c -I/usr/include/libxml2 -fPIC
gcc -shared -Wl,-soname,libwurfl.so.1 -o libwurfl.so.1.0.1 wurfl.o -lxml2
wget http://downloads.sourceforge.net/project/wurfl/WURFL/latest/wurfl-latest.zip
unzip wurfl-latest.zip
mv wurfl.xml /etc/wurfl.xml
cp libwurfl.so /usr/lib
ln -s /usr/lib/libwurfl.so /usr/lib64/libwurfl.so

Edit your /etc/varnish/mobile.vcl using this as a guide

/etc/init.d/varnish stop
/usr/sbin/varnishd -s malloc,32M -a 0.0.0.0:80 -f /etc/varnish/mobile.vcl   -p 'cc_command=exec cc -fpic -shared -Wl,-x -lwurfl -o %o %s'

Test by looking in your server heads.. IE in PHP.. print_r ($_SERVER);

I have been tweaking a varnish vcl config for WordPress for quite some time and I wanted to share it..  Thanks to everyone(especially DocWilco)  in #varnish on Linpro IRC for helping

Features:

1. Load balancing
2. Probing
3. Does not cache wp-admin
4. Puts all uploads/content requests onto one server
5. Purging
6. Long timeout for file uploads
7. XML RPC support
8. Custom 404 and 500 message
9. Forwards user IP address for comments

First of all.. Let’s define some backends..

// BACKEND CONFIGS
backend server1 {
  .host = "server1.example.com";
  .port = "8080";
  .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1 s;
                .window = 5;
                .threshold = 3;
  }
// we include time outs so uploads don't time out
 .connect_timeout = 600s;
 .first_byte_timeout = 600s;
 .between_bytes_timeout = 600s;
}

backend server2 {
  .host = "server2.example.com;
  .port = "8080";
  .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1 s;
                .window = 5;
                .threshold = 3;
  }
// we include time outs so uploads don't time out
 .connect_timeout = 600s;
 .first_byte_timeout = 600s;
 .between_bytes_timeout = 600s;
}

// define round-robin for backends
director cluster round-robin {
        {.backend = server1;}
        {.backend = server2;}
}

// set the servers wordpress can purge from
acl purge {
        "server1.example.com";
        "server2.example.com";
}

sub vcl_fetch {
 if (req.http.host ~ "ourdomain.com"
     || req.http.host ~ "ourotherdomain.com"
// don't cache wp-admin ever cause that's not cool
     && req.url !~ "wp-admin")
{
// we cache these domains for 8 hours unless they are purged.
        set beresp.ttl = 8h;
        set beresp.grace = 600s;
// don't cache 404 or 500 errors
        if (beresp.status == 404 || beresp.status >= 500) {
                  set beresp.ttl = 0s;
        }
}
// tell all of the files to use server1
if (req.url ~ "files") {set req.backend = server;set beresp.ttl = 8h;}
}

sub vcl_recv {
// Purge WordPress requests for purge
  if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                purge("req.url == " req.url " && req.http.host == " req.http.host);
                error 200 "Purged.";
        }

// forward the client IP so comments show up properly
set req.http.X-Forwarded-For = client.ip;

// let server2 handle all feeds
    if (req.url ~ "/feed/")
    {set req.backend = server2;}

// server1 must handle file uploads
    if (req.url ~ "media-upload.php"
    || req.url ~ "file.php"
    || req.url ~ "async-upload.php")
    {set req.backend = server1;return(pass);}

// server1 can serve all files.
    if (req.url ~ "/files/")
    {set req.backend = server1;}

// do not cache xmlrpc.php
    if (req.url ~ "xmlrpc.php")
    {return(pass);}

// strip cookies from xmlrpc
    if (req.request == "GET" && req.url ~ "xmlrpc.php")
    remove req.http.cookie;return(pass);}

// caching these files is fine
if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|lzma|tbz)(\?.*|)$") {
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            remove req.http.Accept-Encoding;
        }
}

// Remove cookies and query string for real static files
    if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.*$", "");
    }

// Remove cookies from front page
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }

// if the request is for our domain and not for wp-admin then load balance it to a server that is responding or send it to server1
if (req.http.host ~ "ourdomain.com"
    || req.http.host ~ "ourotherdomain.com"
    && req.url !~ "wp-admin")
{
        set req.http.X-Forwarded-For = client.ip;
        set req.backend = cluster;
        } else {
        set req.http.X-Forwarded-For = client.ip;
        set req.backend = server1;
}

// Custom error message
sub vcl_error {
if(obj.status == 404) {
        set obj.ttl = 0s;
  set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {" <!--?xml version="1.0" encoding="utf-8"?-->

    "} obj.status " " obj.response {"
</pre>
<div style="background-color: white;"><center>
 <img src="http://whatever.com/heavyload.jpg" alt="" width="600px" /></center>
<h1>This page is unavailable</h1>
If you are seeing this page, either maintenance is being
performed or you are trying to access a file that doesn't exist. Please <a href="http://whatever.com/contact.html">contact us</a> if you believe this is an error
<h2>Error "} obj.status " " obj.response {"</h2>
"} obj.response {" on server "} req.backend {"
<address><a href="http://whatever.com/">Us.</a></address></div>
<pre>
  "};
    return (deliver);
error 404 "Not found";
 }
else
{
    set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {" <!--?xml version="1.0" encoding="utf-8"?-->  
  
    "} obj.status " " obj.response {"
  

</pre>
<div style="background-color: white;"><center>
 <img src="http://whatever.com/heavyload.jpg" alt="" width="600px" /></center>
<h1>This website is unavailable</h1>
If you are seeing this page, either maintenance is being
performed
 or something really bad has happened. Try returning in a few
minutes. If you still see this error in a few minutes please <a href="http://whatever.com/contact.html">contact us</a>
<h2>Error "} obj.status " " obj.response {"</h2>
"} obj.response {" on server "} req.backend {"
<address><a href="http://whatever.com/">Us.</a></address></div>
<pre>
  "};
    return (deliver);
}
}

In the vcl_recv part of your varnish config add this

set req.http.X-Forwarded-For = client.ip;

Then edit your WordPress wp-includes/comment.php and replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR
Then edit your wp-content/plugins/akismet/akismet.php and replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR

Restart varnish

/etc/init.d/varnish restart

Finished, test by commenting.

Note: Users experiencing spam on PrimaryBlogger should now experience less spam and more accurate comment filtering.

If you are running Varnish 2.1.2 RPCXML will not work, you need to upgrade to 2.1.3..  You can find out your version by typing

varnishd -V