Index

About this document
About CARP and pfsync
Installing OpenBSD
DHCP server setup
TFTP server setup
Netboot the soekris
Install OpenBSD 4.0
Initial system configuration
Configuring CARP
Configuring pfsync
Other references

 

About this document

This page describes the setup of the soekris net4801 with OpenBSD as redundant firewall. First we'll describe the basic setup of OpenBSD on the net4801. After that we'll describe how to setup the 2 boxen to be a failover redundant firewall. This is in no way the only method of installing such a setup, but it worked for us.

 

About CARP and pfsync

On most networks, the firewall is a single point of failure. When the firewall goes down, inside users are unable to surf the web, the website goes dead to the outside world, and email grinds to a halt. Since version 3.5, OpenBSD has included a number of components which can be used to solve this problem, by placing two firewalls in parallel. All traffic passes through the primary firewall; when it fails the backup firewall assumes the identity of the primary firewall, and continues where it left off. Existing connections are preserved, and network traffic continues as if nothing had happened.

We are going to use this to better protect our hosting platform.

The tools

The two main components provided by OpenBSD are CARP (the Common Address Redundancy Protocol), which allows a backup host to assume the identity of the primary, and pfsync, which ensures that firewall states are synchronised so that the backup can take over exactly where the master left off and no connections will be lost.

CARP

The Common Address Redundancy Protocol manages failover at the intersection of Layers 2 and 3 in the OSI Model (link layer and IP layer). Each CARP group has a virtual MAC (link layer) address, and one or more virtual host IP addresses (the common address). CARP hosts respond to ARP requests for the common address with the virtual MAC address, and the CARP advertisements themselves are sent out with this as the source address, which helps switches quickly determine which port the virtual MAC address is currently "at".

The master of the address sends out CARP advertisement messages via multicast using the CARP protocol (IP Protocol 112) on a regular basis, and the backup hosts listen for this advertisement. If the advertisements stop, the backup hosts will begin advertising. The advertisement frequency is configurable, and the host which advertises most frequently is the one most likely to become master in the event of a failure.

pfsync

pfsync transfers state insertion, update, and deletion messages between firewalls. Each firewall sends these messages out via multicast on a specified interface, using the PFSYNC protocol (IP Protocol 240). It also listens on that interface for similar messages from other firewalls, and imports them into the local state table.

In order to ensure that pfsync meets the packet volume and latency requirements, the initial implementation has no built-in authentication. An attacker who has local (link layer) access to the subnet used for pfsync traffic can trivially add, change, or remove states from the firewalls. It's possible to run the pfsync protocol on one of the "real" networks, but because of the security risks, it is strongly recommended that a dedicated, trusted network be used for pfsync. This can be as simple as a crossover cable between interfaces on two firewalls.

 

Installing OpenBSD

There are many ways to install OpenBSD on a soekris net4801. Most popular are flashdist and OpenSoekris. They aim at smaller installations (~32MB). The method we use will give you a fullblown OpenBSD system. Because of this you will need a medium or big CF card. This document uses a 1GB CF card as storage.

The soekris net4801 comes with PXEboot enabled network interfaces. We are using this to get the OpenBSD installer running on it. To get to this point we need a couple of services running in our network:

• DHCP server
• TFTP server

1. DHCP server setup

   Any DHCP server will do, as long as they can send some PXEboot options to clients. We use the default dhcp3-server package on a Debian machine. The relevant part in dhcpd.conf is:

   host soekris001 {
    	hardware ethernet 00:00:24:XX:XX:XX;
   	fixed-address ip.outside.range.of.dhcp.but.inside.same.subnet;
   	filename "pxeboot";
   	next-server ip.of.tftp.server;
   } 				

2. TFTP server setup

   Any tftp server will do. We used the default tftpd package on a Debian machine. The root directory for the tftp server is /srv/tftp. If yours is different adopt the following instructions.
   
   To PXEboot the soekris there should be a couple of files in /srv/tftp:
   pxeboot from the OpenBSD distribution
   bsd.rd from the OpenBSD distribution
   etc/boot.conf with the following content:
   

   set tty com0
   stty com0 19200
   boot bsd.rd 				

3. Netboot the soekris

   When the soekris is booting, hit CTRL+p to go into the Manager. No settings need to be changed, we only want the soekris to netboot this one time. To netboot the soekris refer to the soekris manual. On the 1.28 firmware it is:
   boot f0
   If all is ok you will see the DHCP prompt, it will get an ip address and download and run PXEboot. PXEboot will download bsd.rd and load it. This is the normal i386 OpenBSD installer.

4. Install OpenBSD 4.0

   Refer to the OpenBSD documentation on how to do this.
   A couple of things differ from that description. The soekris is running on CF disk. Therefore you don't need a swap partition. The installer however won't continue if you don't have a swap partition. So we create one with size 1.
   Also, everything is going to be mounted read-only, so no need to devide the disk into a lot of partitions. 3 is used in our setup.

   # disklabel wd0
   # Inside MBR partition 3: type A6 start 63 size 2046177
   # /dev/rwd0c: 
   type: ESDI 
   disk: ESDI/IDE disk 
   label: SILICONSYSTEMS I 
   flags: 
   bytes/sector: 512 
   sectors/track: 63 
   tracks/cylinder: 16 
   sectors/cylinder: 1008 
   cylinders: 2030 
   total sectors: 2046240 
   rpm: 3600 
   interleave: 1 
   trackskew: 0 
   cylinderskew: 0 
   headswitch: 0           # microseconds 
   track-to-track seek: 0  # microseconds 
   drivedata: 0  
   
   16 partitions: 
   #             size        offset  fstype [fsize bsize  cpg]   
   a:       2016945            63  4.2BSD   2048 16384  328 # Cyl     0*-  2000   
   b:             1       2017008    swap                   # Cyl  2001 -  2001*   
   c:       2046240             0  unused      0     0      # Cyl     0 -  2029   
   d:         29231       2017009  4.2BSD   2048 16384   28 # Cyl  2001*-  2029  				

   The d partition is used to store our configuration files like hostname.if and pf.conf. You are free to keep them inside /etc but this is how we did it.
   
   When asked what sets to install type in:
   -*
   +bsd
   +base40.tgz
   +etc40.tgz
   done
   When everything is loaded the installer will instruct you to halt the system. Don't try to be smart, let's follow them.

5. Initial system configuration

   Boot into single user mode. When the OpenBSD boot prompt appears type boot -s. The system will boot and prompt for a shell. Hit enter to accept the default sh
   
   We are now going to setup a memory filesystem. We do this because CF is slow and will fail sooner when a lot of writes are performed on it. First mount the root filesystem and create root for memory filesystem:

   # mount -o rw / 
   # mkdir /mfs 				

   Now modify /etc/rc to setup and populate /mfs. We add the following lines after rm -f /fastboot. Line 203 on default 4.0 install.

   #mfs related stuff 
   echo 'mfs: mounting /mfs...' 
   mount_mfs -s 16384 /dev/wd0b /mfs 
   mkdir -p /mfs/var/run 
   mkdir -p /mfs/var/tmp 
   chmod 1777 /mfs/var/tmp 
   cp -Rp /var/log.template /mfs/var/log 
   cp -Rp /var/spool/mqueue /mfs/mqueue 
   cp -Rp /var/spool/clientmqueue /mfs/clientmqueue 
   cp -Rp /var/mail.template /mfs/mail 				

   Edit /etc/mail/sendmail.cf and /etc/mail/submit.cf and change the entries for clientmqueue and mqueue to point at /mfs/{clientmqueue,mqueue}
   
   Now we move some directories and create symbolic links to the /mfs locations.

   # mv /var/log /var/log.template 
   # mv /var/mail /var/mail.template 
   # rm -rf /tmp 
   # rm -rf /var/tmp 
   # rm -rf /var/run 
   # ln -s /mfs/var/tmp /tmp 
   # ln -s /mfs/var/tmp /var/tmp 
   # ln -s /mfs/var/log /var/log 
   # ln -s /mfs/var/run /var/run 
   # ln -s /mfs/mail /var/mail 				

   Now we are ready to reboot our system.
   
   When the system is booted and everything works we can alter /etc/fstab to mount the root filesystem read-only:

    /dev/wd0a / ffs ro,noatime 1 1 				

   Next time the system is booted, everything will be read-only except /mfs
   
   Remember the d partition? This is used for the configuration files that will change most often. This small partition is mounted read-write.

   # mkdir /mnt/vanbaak 
   # mount /dev/wd0d /mnt/vanbaak 
   # mkdir /mnt/vanbaak/confs 
   # mv /etc/hostname.* /mnt/vanbaak/confs/ 
   # mv /etc/pf.conf /mnt/vanbaak/confs/ 
   # cd /etc 
   # ln -s /mnt/vanbaak/confs/hostname.* 
   # ln -s /mnt/vanbaak/confs/pf.conf 				

   Now edit /etc/fstab and add this mountpoint. The file should look like this:

   /dev/wd0a / ffs ro,noatime 1 1 /dev/wd0d /mnt/vanbaak ffs rw,nodev,nosuid,noatime 1 2 				

   Reboot once more and your system is ready to be configured as redundant firewall.
   
   A df -h of the installed system looks like:

   # df -h Filesystem     Size    Used   Avail Capacity  Mounted on 
   /dev/wd0a      967M    171M    748M    19%    / 
   /dev/wd0d     13.8M   16.0K   13.1M     0%    /mnt/vanbaak 
   mfs:19171      7.7M    150K    7.2M     2%    /mfs 				

 

Configuring CARP

Before we can use CARP, we have to enable it in /etc/sysctl.conf. Add the following two lines to it:

net.inet.carp.allow=1 net.inet.carp.preempt=1 		

Now you can reboot to activate it or run the following two commands:

# sysctl -w net.inet.carp.allow=1 
# sysctl -w net.inet.carp.preempt=1 		

Now we need to redo the network setup. The soekris comes with 3 interfaces: sis0 sis1 and sis2.
sis0 will be connected to the outside world.
sis1 will be connected to the internal network.
sis2 will be used to share state information. See Configuring pfsync.
The virtual ip address will be realized by CARP. Both the internal and the external side need a virtual ip so we will configure two CARP interfaces.
carp1 on the outside using sis0 as physical interface.
carp2 on the inside using sis1 as physical interface.
To create a carp interface you can use ifconfig to setup things on a running system. Later we'll discuss the files we need to create to setup CARP during system boot.
Creating a CARP interface goes:

# ifconfig sis0 ip.address netmask some.mask 
# ifconfig carp1 create 
# ifconfig carp1 vhid 1 carpdev sis0 pass somepass virtual.ip.address netmask some.mask 		

On the second, standby firewall the commands are:

# ifconfig sis0 ip.address+1 netmask some.mask 
# ifconfig carp1 create 
# ifconfig carp1 vhid 1 carpdev sis0 pass somepass advskew 100 virtual.ip.address netmask some.mask 		

The keyword that's interesting is the advskew. The higher you set this, the lower priority the firewall has. The machine with the lowest advskew will be the master firewall.

For our setup we assume the following ip addresses. Note that these are used in the test setup. Replace the ip information with your real information.

         +----| WAN/Internet |----+
         |                        |
     sis0|                        |sis0
 +------------+                  +------------+
 | soekris001 |-sis2--------sis2-| soekris002 |
 +------------+                  +------------+
      sis1|                        |sis1
          |                        |
       ---+-------Shared LAN-------+---  

soekris001 is the master firewall.
soekris002 is our standby failover firewall

• soekris001 sis0: 192.168.2.14
• soekris001 sis1: 10.0.0.14
• soekris001 sis2: 172.16.0.14
• soekris002 sis0: 192.168.2.15
• soekris002 sis1: 10.0.0.15
• soekris002 sis2: 172.16.0.15
• external shared IP: 192.168.2.13
• internal shared IP: 10.0.0.13

Configuration on soekris001:

# ifconfig sis0 192.168.2.14 netmask 255.255.255.0 up 
# ifconfig sis1 10.0.0.14 netmask 255.255.255.0 up 
# ifconfig sis2 172.14.0.14 netmask 255.255.255.0 up 
# ifconfig carp1 create 
# ifconfig carp1 vhid 1 carpdev sis0 pass somethingsecret 192.168.2.13 netmask 255.255.255.0 
# ifconfig carp2 create 
# ifconfig carp2 vhid 2 carpdev sis1 pass somethingother 10.0.0.13 netmask 255.255.255.0 		

Be sure to put the following line on top of your /etc/pf.conf

pass out on { sis0, sis1 } proto carp keep state

Configuration on soekris002:

# ifconfig sis0 192.168.2.15 netmask 255.255.255.0 up 
# ifconfig sis1 10.0.0.15 netmask 255.255.255.0 up 
# ifconfig sis2 172.14.0.15 netmask 255.255.255.0 up 
# ifconfig carp1 create 
# ifconfig carp1 vhid 1 carpdev sis0 pass somethingsecret advskew 100 192.168.2.13 netmask 255.255.255.0 
# ifconfig carp2 create 
# ifconfig carp2 vhid 2 carpdev sis1 pass somethingother advskew 100 10.0.0.13 netmask 255.255.255.0 		

Be sure to put the following line on top of your /etc/pf.conf

pass out on { sis0, sis1 } proto carp keep state

Everything should work now. ssh into 192.158.2.13 and run hostname. It should print soekris001 If you unplug power and ssh to 192.168.2.13 again it should print soekris002.

To keep this setup whenever the machine boots we need to edit/create a couple of configuration files.
hostname.sis0
hostname.sis1
hostname.sis2
hostname.carp1
hostname.carp2

On soekris001:

# cat /etc/hostname.sis0 
inet 192.168.2.14 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description External 
# cat /etc/hostname.sis1 
inet 10.0.0.14 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description Internal 
# cat /etc/hostname.sis2 
inet 172.14.0.14 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description pfsync 
# cat /etc/hostname.carp1 
inet 192.168.2.13 255.255.255.0 192.168.2.255 vhid 1 carpdev sis0 pass somethingsecret 
# cat /etc/hostname.carp2 
inet 10.0.0.13 255.255.255.0 10.0.0.255 vhid 2 carpdev sis1 pass somethingother 		

On soekris002:

# cat /etc/hostname.sis0 
inet 192.168.2.15 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description External 
# cat /etc/hostname.sis1 
inet 10.0.0.15 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description Internal 
# cat /etc/hostname.sis2 
inet 172.14.0.15 255.255.255.0 192.168.2.255 media 100baseTX mediaopt full-duplex description pfsync 
# cat /etc/hostname.carp1 
inet 192.168.2.13 255.255.255.0 192.168.2.255 vhid 1 carpdev sis0 pass somethingsecret advskew 100 
# cat /etc/hostname.carp2 
inet 10.0.0.13 255.255.255.0 10.0.0.255 vhid 2 carpdev sis1 pass somethingother advskew 100 		

Now the soekris002 will takeover when soekris001 dies. Only annoying thing is that established connections will be terminated. Here's where pfsync comes to rescue.

 

Configuring pfsync

pfsync syncs pf states between the two soekris machines. That way established connections will continue in the case of a failover. Customers wont notice a thing.
We are going to use sis2 for this. This interface already has an IP address. All we have to do is create the pfsync0 interface and attach it to sis2.
On both machines:

ifconfig pfsync0 syncdev sis2 		

To keep this setup whenever the machine boots we need to create /etc/hostname.pfsync0

# cat /etc/hostname.pfsync0 
up syncdev sis2 		

Be sure to put the following line on top of your /etc/pf.conf

pass on sis2 proto pfsync

/etc/pf.conf tips

Remember: filtering takes place on the physical interface, NOT the virtual carpX interface. The virtual interface can however be used to match addresses:

pass in on sis0 from any to carp1 port ssh 		

replacing sis0 with carp1 wont work.

 

Other references

Please see these other sources for more information:

• carp(4)
• pfsync(4)
• ifconfig(8)
• hostname.if(5)
• pf.conf(5)
• ifstated(8)
• ifstated.conf(5)

 

---

 Feedback on this page or article:

This article has no feedback yet

 Give feedback:

Copyright (c) 2006-2008 Michiel van Baak.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License".