Configuring IPSec VPN Connection Between FreeBSD and OpenBSD

General Information

Although there are some good tutorials on setting VPN connections using IPSec in FreeBSD-to-FreeBSD or OpenBSD-to-OpenBSD configurations, only tutorials on setting FreeBSD-to-OpenBSD IPSec tunnels are at least partly outdated, mostly due to changes introduced in OpenBSD 3.8.

In this tutorial I’ll demonstrate how to configure such a VPN connection (tunneling mode) using racoon and isakmpd IKE daemons along with x509 certificates. The IP addresses used throughout this tutorial are:

- gateway A running FreeBSD with internal IP address:, and external IP:,

- gateway B running OpenBSD with internal IP address:, and external IP:

For the purpose of this tutorial I have used FreeBSD 6.2-RELEASE and OpenBSD 4.1-RELEASE operating systems, but the configuration covered here should be applicable in any reasonably recent version as well.


  1. Root access to both FreeBSD and OpenBSD machines.
  2. Recompiled FreeBSD kernel with IPSec support according to FreeBSD Handbook.
  3. IPsec-Tools port (/usr/ports/security/ipsec-tools) installed on FreeBSD host.


As written above, further in this tutorial I’ll assume that you have an IPSec enabled FreeBSD along with /security/ipsec-tools port installed already. If not, please refer to FreeBSD Handbook on doing so.

As the ipsec-tools port doesn’t create configuration directory in /usr/local/etc subtree, you’ll have to create it manually:

# mkdir -p /usr/local/etc/racoon/certs

OpenBSD already contain required applications and IPSec support compiled into the GENERIC kernel, so there’s no need to install anything.


x509 Certificates

The first step toward setting the VPN over IPSec is to generate x509 certificates. In the following example I have generated all the keys and certificates on gateway B — the OpenBSD box — and copied to appropriate hosts afterwards. If you’ll decide to generate them on FreeBSD box instead, please make sure that you have a ”x509v3.conf” file.

Creating CA (Certificate Authority) and generating peer certificates is quite simple. The OpenBSD’s isakmpd(8) manual page contains some very useful informations on doing so.

1. Create self-signed CA certificate:

# openssl req -x509 -days 365 -newkey rsa:1024 -keyout /etc/ssl/private/ca.key -out /etc/ssl/ca.crt

2. Generate private key for the gateway A:

# cd /etc/ssl
# openssl genrsa -out 1024

3. Create Certificate Signing Request for the new key:

# openssl req -new -key -out

4. Sign the CSR:

# env CERTIP= openssl x509 -req -days 365 \
-in -CA /etc/ssl/ca.crt -CAkey /etc/ssl/private/ca.key \
-CAcreateserial -extfile /etc/ssl/x509v3.cnf -extensions x509v3_IPAddr -out

Repeat steps 2-4 for another peer — gateway B, in this case. The important part is to put propper IP address into the ”subjectAltName” extension field, for which we use the ”CERTIP” variable. The ”subjectAltName” is actually required by OpenBSD’s isakmpd daemon only, but racoon will not mind if you’ll add this to all the certificates created for use with IPSec.

You can do some cleaning afterwards, removing obsolete *.csr files. It is also a good thing to make sure the certificates contain proper information in x509 extensions field. To do so type:

# openssl x509 -text -noout -in

You should be looking for something like this:

X509v3 extensions:
    X509v3 Subject Alternative Name:
        IP Address:

Now, place OpenBSD certificates in ”/etc/isakmpd” subdirectories:

- ca.crt in ”/etc/isakmpd/ca”,

- and in ”/etc/isakmpd/certs”,

- in ”/etc/isakmpd/private” renaming to local.key and setting the file permissions to readable only by the owner (root that is).

You may place all FreeBSD certificates in one directory — it actually doesn’t matter — let’s say ”/usr/local/etc/racoon/certs” which you have created previously. The files to copy to FreeBSD host are:

- ca.crt CA certificate,

- FreeBSD certificate,

- OpenBSD certificate,

- FreeBSD private key.


I think one of the most common errors when configuring IPsec connection between racoon and isakmpd is “ERROR: no suitable proposal found” (racoon) or “dropped message from port 500 due to notification type NO_PROPOSAL_CHOSEN” (isakmpd). The reason is that both IKE daemons uses different default crypto transforms.

Currently, default racoon configuration available from FreeBSD’s port uses 3des cipher for encryption and hmac_sha1 algorithm for authentication in both phases. The default settings for isakmpd are: aes and hmac-sha1 for phase 1 and aes and hmac-sha2-256 for phase 2. Of course you could define a group of algorithms IKE daemons could choose from, but I think it’d be better to define one set of algorithms to be sure both daemons uses what you want. The OpenBSD’s ipsec.conf(5) manual page lists available authentication types and ciphers.

My proposition for this tutorial is:

phase 1 phase 2

authentication hmac-sha1 hmac-sha2-256

encryption blowfish blowfish

group/pfs modp1024 modp1024


Although setting gif(4) interface is not required in OpenBSD it might be useful when testing IPSec configuration. This can be done using following three commands. Note that the IP address of machine you’re currently configuring goes first:

# ifconfig gif0 create
# ifconfig gif0 tunnel
# ifconfig gif0 inet netmask 0xffffffff

Starting from OpenBSD 3.8 the ipsecctl(8) utility is used for IPSec and IKE daemon configuration, so in the end it comes down to modifying the ”/etc/ipsec.conf” file only:

ike esp from to peer
  main auth hmac-sha1 enc blowfish group modp1024
  quick auth hmac-sha2-256 enc blowfish group modp1024

As far as the first line is quite obvious, the rest demands a word of explanation. Second line actually defines method and crypto transforms used in phase 1 of automatic keying (that is what IKE daemons do). The main mode and group modp1024 are the default parameters and we’re going to stick to them now. Third line defines the second phase. For details, see ipsec.conf(5) manual page. All the hard work of setting flows and SADs for both communication directions — inbound and outbound — will be done by ipsecctl.


First, create the gif(4) interface the same way as with OpenBSD — remember to type IP addresses in propper order.

The ”/etc/ipsec.conf” configuration in FreeBSD is a bit different from OpenBSD’s and require to enter policies for both directions:

spdadd ipencap
  -P out ipsec esp/tunnel/;
spdadd ipencap
  -P in ipsec esp/tunnel/;

Put the following in ”/usr/local/etc/racoon/racoon.conf” file (create it if necessary). The phase 1 configuration is located within ”proposal { }” block. Second phase is in ”sainfo { }”. For explanation of what is what and further customization please refer to racoon.conf(5) manual.

path include "/usr/local/etc/racoon";
path certificate "/usr/local/etc/racoon/certs";

# "padding" defines some padding parameters.
# You should not touch these.
        maximum_length 20;      # maximum padding length.
        randomize off;          # enable randomize length.
        strict_check off;       # enable strict check.
        exclusive_tail off;     # extract last one octet.


remote anonymous
        exchange_mode main;
        doi ipsec_doi;
        situation identity_only;

        my_identifier asn1dn;
        certificate_type x509 "" "";
        peers_certfile x509 "";

        nonce_size 16;
        initial_contact on;
        proposal_check obey;

        proposal {
                encryption_algorithm blowfish;
                hash_algorithm sha1;
                authentication_method rsasig;
                dh_group modp1024;

sainfo anonymous
        pfs_group modp1024;
        encryption_algorithm blowfish;
        authentication_algorithm hmac_sha256;
        compression_algorithm deflate;

Now you should be able to establish the IPSec tunnel. For the first start-up make both daemons display messages directly to screen, saving yourself digging through log files.


Use one of consoles/terminal windows to start the isakmpd daemon and keep him in foreground:

# isakmpd -Kdv

On the other one initialize IPSec:

# ipsecctl -f /etc/ipsec.conf


Flush and initialize IPSec SA/SP database:

# setkey -FP
# setkey -f /etc/ipsec.conf

Start racoon and keep him in foreground:

# racoon -F


If all is well you should see something like this in a while.

on OpenBSD host:

195801.577434 Default isakmpd: phase 1 done: initiator id c0a85601:, responder id /C=PL/ST=Mazowieckie/O=therekNET, src: dst:
195801.693699 Default isakmpd: quick mode done: src: dst:

… and on FreeBSD machine:

Foreground mode.
2007-07-03 19:58:00: INFO: @(#)ipsec-tools 0.6.6 (
2007-07-03 19:58:00: INFO: @(#)This product linked OpenSSL 0.9.7e-p1 25 Oct 2004 (
2007-07-03 19:58:00: INFO:[500] used as isakmp port (fd=5)
2007-07-03 19:58:07: INFO: respond new phase 1 negotiation:[500]<=>[500]
2007-07-03 19:58:07: INFO: begin Identity Protection mode.
2007-07-03 19:58:07: INFO: received Vendor ID: draft-ietf-ipsec-nat-t-ike-02

2007-07-03 19:58:07: INFO: received Vendor ID: draft-ietf-ipsec-nat-t-ike-03
2007-07-03 19:58:07: INFO: received Vendor ID: RFC 3947
2007-07-03 19:58:07: INFO: received Vendor ID: DPD
2007-07-03 19:58:07: WARNING: ignore INITIAL-CONTACT notification, because it is only accepted after phase1.
2007-07-03 19:58:07: INFO: ISAKMP-SA established[500]-[500] spi:4bebaadc52d2ab88:24bd26983016e810
2007-07-03 19:58:07: INFO: respond new phase 2 negotiation:[0]<=>[0]
2007-07-03 19:58:07: INFO: IPsec-SA established: ESP/Tunnel[0]->[0] spi=196753108(0xbba36d4)
2007-07-03 19:58:07: INFO: IPsec-SA established: ESP/Tunnel[0]->[0] spi=4211671499(0xfb0901cb)

Yes, that’s it! The IPSec is working correctly!

To take a look into established connection’s entries in Security Association Database (SAD) and Security Policy Database (SPD) do…

on OpenBSD:

# ipsecctl -sa # dump both entries

on FreeBSD:

# setkey -D  # dump SAD entries
# setkey -DP # dump SPD entries

All that is left to do is to make your settings permanent, so that IPSec tunnel would be brought up every time when either host will reboot.

On OpenBSD add following to ”/etc/rc.conf.local” file:


…and to ”/etc/rc.local”:

route add 192.168.0/24

On FreeBSD place following in ”/etc/rc.conf” file:

ifconfig_gif0="inet netmask 0xffffffff"
route_vpn=" netmask 0xffffff00"



For sake of this tutorial, I’ll assume that FreeBSD uses IPFilter (ipf) and OpenBSD uses it’s native Packet Filter (pf) as firewalls.

IPSec utilizes protocol UDP port 500 for key exchange — and port 4500/UDP for NAT-Traversal, which I left out of this article intentionally but is commonly used in practice — as well as protocols ESP and IPENCAP. The interfaces taking part in communication are gif0 plus an external interface — in this example this is dc0 — on FreeBSD, and enc0 plus external interface — this time sk0 — on OpenBSD machine.


The PF configuration for IPSec traffic is explained quite clearly in ipsec.conf(5) manual. Assuming that our packet filtering policy is to block all non-IPSec related traffic, the firewall configuration file pf.conf(5) should look like this:

set skip on { lo enc0 }

block on sk0

pass  in on sk0 proto udp from to port 500
pass out on sk0 proto udp from to port 500

pass  in on sk0 proto esp from to
pass out on sk0 proto esp from to

As you propably have noticed the pf is configured to simply skip packet filtering on the enc0 interface. For information on limitting communication on this interface as well as for queuing please refer to aforementioned ipsec.conf(5) manual page. IPENCAP communication goes through enc0 interface, so in this case there’s no need to menion this protocol in firewall configuration.


The FreeBSD’s ipf(5) configuration should be quite similar. All there’s to do is pass 500/UDP, ESP and — additionally — IPENCAP traffic.

pass in quick on dc0 proto udp from port = isakmp
  to port = isakmp
pass in quick on dc0 proto esp from to
pass in quick on dc0 proto ipencap from to

pass out quick on dc0 proto udp from port = isakmp
  to port = isakmp
pass out quick on dc0 proto esp from to
pass out quick on dc0 proto ipencap from to

block quick on dc0


Note: Browsing the Web for IPSec related howtos and other information you may stumble upon packet filtering definitions using protocol numbers instead of names. Just keep in mind that ESP is protocol 50, AH (not used in this configuration) is 51, IPENCAP is protocol 4 and UDP is 17.

Speak Your Mind