Creating a frontend

Creating and running frontends can be done on pretty much any machine. The frontend can be seen as a passthrough server, which relays HTTP requests from clients to an available backend in one or more pool(s), based on a config file. It will health check the backends and remove them from the pool(s) if they are unreachable, or unhealthy.

Depending on how complicated and feature rich the frontend should be (it's always a trade off between complexity/features, speed, cost and reliability), one can choose for a simple HA Proxy or a more complicated NGINX (this document).

As with most howto's in the user guide, this one assumes that you have a host set up using one of the methods described in the Host Setup guide. Once the machine is up and has the necessary access, we install the required packages and introduce the machine into our provisioning system, which is represented on your admin machine by an RCS, of which you can read more here.

We installed a few OpenBSD and Ubuntu LTS machines and put them at three different hosting providers in Europe: BIT bv in Ede, the Netherlands; Coloclue in Amsterdam, the Netherlands; IP Max in Zurich/Geneva, Switzerland; and Saitis in Lausanne, Switzerland. You can use existing machines as long as (a) you have root access to this machine using ssh(1), and (b) you are able and allowed to bind port 80 (and optionally 443 if you would like to terminate HTTPS) on at least one IP address of this machine.

A1) Using OpenBSD

OpenBSD before 5.2 has a port, from 5.3 onwards we use the stock nginx in base. OpenBSD by default wants to force nginx to chroot() into the homedir of the user running it (normally _nginx). This doesn't work for us and all other versions of nginx will not have this behavior, so we turn it off using the -u flag in rc.conf.local.

1. Put nginx in rc.conf.local

sudo su -
echo nginx=\"YES\" >> /etc/rc.conf.local
echo nginx_flags=\"-u\" >> /etc/rc.conf.local

mkdir -p /etc/nginx/conf.d
ln -sf /etc/nginx/conf.d/nginx.conf /etc/nginx/nginx.conf
mkdir -p /paphosting/nginx/cache /paphosting/nginx/logs /paphosting/nginx/sites
chown -R paphosting:paphosting /etc/nginx/conf.d/
chown _nginx:paphosting /paphosting/nginx
chown paphosting:paphosting /paphosting/nginx/{logs,sites}/
chown _nginx:paphosting /paphosting/nginx/cache/

A2) Using Ubuntu

Note: IPv6 was added in nginx relatively recently, and LTS 8.04 (possibly Debian Stable) does not have a recent IPv6 enabled package. Jeff Waugh has recent builds for Debian/Ubuntu on his PPA. Another good document is on the nginx wiki. This is not an issue for LTS 10.04 and 12.04 so there we simply use the maintainer supplied version.

1. Install needed packages

sudo su -
apt-get install nginx rsync

2. Enable nginx

sudo su -
mkdir -p /etc/nginx/conf.d
ln -sf /etc/nginx/conf.d/nginx.conf /etc/nginx/nginx.conf
mkdir -p /paphosting/nginx/logs
ln -sf /paphosting/nginx/logs /etc/nginx/logs
chown -R paphosting:paphosting /etc/nginx/conf.d/ /paphosting/nginx/

A3) Staging Hosts

Staging hosts don't get synced using the normal push method, as such, one needs to check out their svn directory in a useful place and then: 
export SVNROOT=/your/path/to/svnroot/paphosting/
mkdir -p /paphosting/nginx/
ln -s ${SVNROOT}nginx/sites/ /paphosting/nginx/sites
cd /etc/nginx/conf.d/
ln -s ${SVNROOT}nginx/config/host/* .
ln -s ${SVNROOT}nginx/config/default/* .
rm *.regtest
This way, the config is actually running out of the live SVN directory instead of being pushed. Of course, when adding files one has to update them there too.

B) Configuring NGINX

1. Add the machine to config/nginx{,.$CLUSTER}.hosts

(for the SixXS cluster this is performed with the config update) On your client, add the hostname (any hostname or IPv4 or IPv6 address to which you can connect on the ssh port: 
CLUSTER=.ipng
svn update
echo ${HOSTNAME} >> config/nginx${CLUSTER}.hosts
mkdir -p files/${HOSTNAME}/etc/logrotate.d
ln -s ../../../common/logrotate.d/nginx-paphosting \
  files/${HOSTNAME}/etc/logrotate.d/
svn commit

The NGINX configuration consists of statements in nginx/config/fe-* (frontend configs), and nginx/config/$HOSTNAME/ (the machine-specific configs). The configuration builder will read nginx/config/default/ and the machine-specific configs, bundle them and copy them out to /etc/nginx/conf.d/. As an example, see this machine: 

$ ls nginx/config/$HOSTNAME/
STAR.ipng.nl.cert -> ../fe-ipng-ssl0/STAR.ipng.nl.cert
STAR.ipng.nl.key -> ../fe-ipng-ssl0/STAR.ipng.nl.key
STAR.sixxs.cctld.cert -> ../fe-sixxs-ssl0/STAR.sixxs.cctld.cert
STAR.sixxs.cctld.key -> ../fe-sixxs-ssl0/STAR.sixxs.cctld.key
STAR.sixxs.net.cert -> ../fe-sixxs-ssl0/STAR.sixxs.net.cert
STAR.sixxs.net.key -> ../fe-sixxs-ssl0/STAR.sixxs.net.key
class3-cacert.cert -> ../fe-sixxs-ssl0/class3-cacert.cert
listen-ipng-ssl0-default.inc
listen-ipng-ssl0.inc
listen-sixxs-ssl0-default.inc
listen-sixxs-ssl0.inc
nginx.ipng.regtest -> ../fe-ipng-ssl0/nginx.ipng.regtest
nginx.sixxs.regtest -> ../fe-sixxs-ssl0/nginx.sixxs.regtest
server-ipng.conf -> ../fe-ipng-ssl0/server-ipng.conf
server-sixxs.conf -> ../fe-sixxs-ssl0/server-sixxs.conf
sixxs.org.cert -> ../fe-sixxs-ssl0/sixxs.org.cert
sixxs.org.key -> ../fe-sixxs-ssl0/sixxs.org.key
www.sixxs.net.cert -> ../fe-sixxs-ssl0/www.sixxs.net.cert
www.sixxs.net.key -> ../fe-sixxs-ssl0/www.sixxs.net.key
As can be seen from this example, this particular machine is serving frontend traffic for two clusters: ipng and sixxs. The only thing specific to the machines are the IP addresses they bind in their server {} blocks, so these are their own files. The rest is shared between all nginx cluster members, and therefor symlinked to the frontend configs nginx/config/fe-*.

2. Ensure you can SSH into the machine as paphosting

From your client, try to SSH as paphosting into the machine. Your SSH keys should be in config/ssh-keyring.pub, and those should be in ~paphosting/.ssh/authorized_keys.
Now that you're here, you need to setup sudo access for the paphosting user, so that it can restart the NGINX: 
cat << EOF >> /etc/sudoers
paphosting ALL = NOPASSWD: /usr/sbin/nginx
paphosting ALL = NOPASSWD: /usr/local/sbin/pap_cleancache nginx
EOF

3. Force a push of the NGINX configs

On your client, try to do a nginx push 
scripts/nginx-push.sh -v -n $HOSTNAME
# If this looks good, then:
scripts/nginx-push.sh -f $HOSTNAME
# If you changed config/fe-ipng-ssl0/ data, then:
scripts/nginx-push.sh -f -file config/nginx.ipng.hosts

4. Put the machine in DNS

Add IPv4 and IPv6 addresses of the machine to the http0 label, which will put your nginx into the rotation within $TTL seconds (probably 300). Note: your frontend will go live as soon as DNS propagates! 
$EDITOR dns/zones/paphosting/http0.inc
# (or ipng.inc, or sixxs.inc, or ...)
scripts/dns-push.sh -v -n
# If this looks good, then:
scripts/dns-push.sh -f
EOF :)