This report was written by Artem Belan as part of Google Summer of Code 2026.
\nracoon2 is a system to exchange and install security parameters for IPsec.\nIt consists of an IKEv1/IKEv2 key exchange daemon, a security policy\nmanagement daemon, and a Kerberos-based key exchange daemon. The project\nlives at zoulasc/racoon2 and is being\nmodernized so that it can serve as a reliable L2TP/IPsec or IKEv2 VPN\nserver on NetBSD and Linux for built-in Windows, iOS and Android VPN\nclients.
\nThe goal of this GSoC project was to work through that TODO list:\nimplement NAT traversal properly for both IKEv1 and IKEv2 according to the\nrelevant RFCs, make IPv6 works well so racoon2 can be tested with both IPv4 IPv6 properly,\nclean up the address-macro mess, and build a unit test framework so that the fixes stay fixed. In the end,\nthe daemon can sit behind a NAT device and serve modern VPN clients\nwithout any workaround selectors in its configuration, IPv6 works out of\nthe box, and the changes are protected by an automated test suite.
All work was done on the gsoc2026\nbranch and merged upstream through pull requests #13–#39 between June and\nSeptember 2026.
The fragmentation code was, in the words of the TODO, "old/incomplete":\nreassembled messages could be misrecognized by the daemon, and oversized\npackets could crash it. The result of the project is reliable RFC 7383\nfragmentation for both IKEv1/IKEv2 on both the sending and the receiving side: large IKE\nmessages are split and put back together correctly, packets that exceed\nthe allowed fragment size are rejected with a clear log message instead of\na crash, and exchanging big messages — certificates, large proposals —\nbetween the peers no longer silently fails or takes the daemon down.
At the beginning of the summer the daemon ignored NAT original address\npayloads on input and never sent them to the peer. The consequence was\nthat the kernel had no idea which addresses the client actually used\nbehind the NAT, so it had to recompute checksums over entire packets —\nslow, and not what RFC 3947 prescribes. Now the NAT-OA payloads match the\nRFC's structure, are constructed and sent in Quick Mode, and are parsed\npositionally on receipt (local address first, peer address second), with\na missing payload from the peer tolerated rather than treated as an\nerror. The original addresses are also passed down to the kernel so it can\ndo incremental checksum fixup instead of full recomputation.
\nThis was the heart of the project and the direct answer to a long-standing\nTODO item: when a NAT device was in the path, the addresses the peer\nproposed in phase 2 did not match the responder's configured selectors, so\nresponders needed extra selectors that were valid only on the initiator's\nside just to pass the selector check — configurations like\ntransport_ike_natt.conf carried them as a workaround, and connections\nfrom real clients stayed fragile. The final behaviour follows RFC 7296\n§2.23.1: when NAT-T is enabled and the addresses observed on the wire do\nnot match the configured selectors, the observed addresses are substituted\ninto the negotiated selectors, on both IKEv1 and IKEv2. The workaround\nselectors are no longer needed, and the daemon now properly handles\nNAT-T traffic on the dedicated UDP port instead of only the initial one.
The wildcard address macros were, in the TODO's words, "the cause of\nmany configuration-related bugs": what worked in an IKEv1 configuration\ncould silently misbehave in IKEv2, one macro behaved as a wildcard in some\ncode paths and as "unknown, wait until we know" in others, and\nmisconfigurations were ignored rather than reported. The project\nunified them into a single consistent wildcard notion that behaves the\nsame way everywhere, works for both IKEv1 and IKEv2 configurations, and\nproduces clear diagnostics when an address lookup or expansion fails,\ninstead of failing silently and leaving the user to guess why the policy\nnever got installed.
IPv6 support had existed in racoon2 for years, but it worked crooked: when\nthe interface was configured to use all addresses, the daemon still\nbehaved as if only IPv4 addresses existed — as though the interface were\nset to use IPv4 only — and prefix-length edge cases were broken, so in\npractice racoon2 could not be tested with IPv6 at all. During the project\nthese bugs were fixed — address matching, interface address validation on Linux,\nprefix-length handling — and IPv6 became what a modern daemon\nshould have been: used by default, with no configuration switch to\nremember. After the project, the same configuration files simply work\nwith IPv6 addresses, and the TODO item about IPv6 is closed on the\nconfiguration side; only end-to-end testing with real IPv6 traffic\nremains.
Before the project, transport mode policies were generated from the\ngeneric SA endpoints rather than the addresses configured for the SA,\ntunnel mode policies could reference the wrong endpoints, and the daemon\nassembled the negotiated IPsec proposal itself instead of accepting what\nthe peer actually proposed — a frequent source of failed negotiations with\nreal clients, which all bring their own proposals. Now policies in both\ntransport and tunnel modes are generated from the configured SA addresses,\nand the negotiated proposal is built from the peer's proposal, so\ninteroperability with stock Windows, iOS and Android clients no longer\ndepends on the local configuration guessing everything right.
\nThe project had no automated tests at all; every change had to be\nvalidated by reading code and running the daemon by hand. The last part of\nthe summer went into changing that: a small self-contained test framework\nwith no external dependencies is now integrated into the standard build\nand its tests run under sanitizers. On top of it, four new test programs\ncover exactly the logic this project introduced — NAT original address\npayload handling, traffic selector address substitution, wildcard address\nhandling in configurations, and the address matching used when a peer is\npurged — and the pre-existing crypto self-test was fixed so the whole\nsuite passes cleanly. This matters beyond the project itself: the\nframework makes it straightforward to add regression tests for the parts\nof the daemon that still have none, whoever tackles them next.
\nDocumentation and sample configurations still described the old defaults,\nso the manual contradicted the shipped behaviour: it still described the\nold IPv6 behaviour and the pre-substitution selector workarounds. The\nproject synced the docs and samples with the new\ndefaults — IPv6 used by default, fragmentation handled automatically, tunnel endpoints\nand wildcard addresses used consistently — and added a NEWS entry\nsummarizing all of the GSoC 2026 work for users and downstream packagers.
Everything described above is on the gsoc2026 branch of\nssszcmawo/racoon2 (merged\nupstream into zoulasc/racoon2). A quick tour for anyone who wants to try\nit:
Build and install. The repository ships only the autotools sources,\nso the configure script has to be regenerated first — this is the same\nflow documented in doc/INSTALL and used by the project's CI:
autoreconf -fi\n./configure\nmake\nmake installYou will need the usual developer toolchain (a C compiler, GNU make,\nautoconf, automake and libtool, plus flex and bison) and the OpenSSL\nheaders; if you want the kinkd daemon you also need a Kerberos 5\nlibrary (MIT krb5 or Heimdal), and the optional packet-dump debug mode\nadditionally wants libpcap. On Debian/Ubuntu the CI installs exactly\nautomake libtool libssl-dev libkrb5-dev libpcap-dev flex bison before\nrunning the commands above. Useful configure options are\n--disable-iked / --disable-kinkd to skip a daemon, --with-krb5=<dir>\nfor a non-standard Kerberos, and --enable-pcap for the packet dumps;\n--prefix=$NEWDIR moves the installation away from the default\n/usr/local.
Configure. Everything needed ships in the repository under\nsamples/. The .in files are templates that get their install paths\nfilled in during installation, and the scenario fragments are plain\nfiles. For an L2TP/IPsec transport-mode server — the scenario this\nproject was about — the flow is:
Take samples/vals.conf.in as your vals.conf and set your
environment in it: MY_IPADDRESS and PEERS_IPADDRESS (or the\n IP_ANY wildcard when the peer's address is not known in advance),\n MY_PUBLIC_IPADDRESS if you are behind NAT, and the shared-key\n settings. Generate the key itself with pskgen.
Take samples/racoon2.conf.in as your racoon2.conf. It includes
vals.conf and default.conf and defines the interfaces; the sample\n ships with NAT-T commented out, with instructions in place:
\n interface\n {\n ike {\n MY_IP port 500;\n # MY_IP port 4500; # uncomment to enable NAT-T\n };\n ...\n };\n
Uncomment the one include line for the scenario you want — for this
case transport_ike.conf. That file holds the remote section\n (IKEv1 and IKEv2, pre-shared key), the selectors and the policy:
\n selector ike_trans_sel_out {\n direction outbound;\n src "${MY_IPADDRESS}" port 1701;\n dst "${PEERS_IPADDRESS}" port any;\n upper_layer_protocol "udp";\n policy_index ike_trans_policy;\n };\n ...\n policy ike_trans_policy {\n action auto_ipsec;\n remote_index ike_trans_remote;\n ipsec_mode transport;\n ipsec_level require;\n ipsec_index { ipsec_esp; };\n };\n
and, if you are behind NAT, its own commented include line pulls in\n transport_ike_natt.conf with the additional selectors for common\n private address ranges.
Two details worth noting: the wildcard\naddress in vals.conf (IP_ANY) now behaves consistently, so a server\ncan accept a peer whose address is unknown in advance; and IPv6 now\nworks in the same files as is — put IPv6 addresses in and they are used,\nwith no switch to flip.
The other samples follow the same pattern: tunnel_ike.conf and\ntunnel_ike_natt.conf for tunnel mode, transport_kink.conf and\ntunnel_kink.conf for Kerberos-based key exchange, and local-test.conf\nfor trying two daemons against each other on one machine.
Run it. Start spmd first (it owns the policy channel to the kernel),\nthen iked with your configuration — ./configure puts both in place,\nand samples/rc.d/ has service scripts for NetBSD-style systems. Once\nboth are up, traffic matching a selector with action auto_ipsec (like\nthe one above) triggers the negotiation automatically, so on a client you\njust connect with the built-in Windows, iOS or Android VPN client using\nthe server address and the shared key. The daemon logs the NAT detection\nand the substitution it performs, which makes it easy to see what\nhappened if a client does not come up.
Follow the work. All changes are in pull requests #13–#39 on the\ngsoc2026 branch, and the NEWS file has a single consolidated entry\ndescribing the user-visible changes. The top-level TODO file lists what\nis still open — see the "What remains" section below.
Old, untested code. racoon2 has been around since the WIDE project
days and had no tests. Every change had to be validated mostly by\nreading code, running the daemon by hand under a debugger, and watching\nthe sanitizers — which is exactly why the test framework became part of\nthe project rather than an afterthought.
RFC compliance vs. working configurations. Implementing address
substitution meant first understanding why responders needed those\nextra initiator-only selectors before being able to remove the need for\nthem, and getting the substitution to happen at the right point of the\nIKEv1 negotiation took several attempts.
Address macros are deceptively deep. The wildcard macros looked like
a simple rename, but in some places they mean "the address is not known\nyet and the kernel policy must wait", while in others they do not.\nUnifying them without breaking either case required careful tracing\nthrough configuration parsing, selector matching and policy updates.
Build system mistakes. An optional-build patch for the daemons
imported from elsewhere looked harmless but broke the build and had to\nbe reverted — a good reminder to actually build the tree, not just read\nthe diff.
Sanitizer-driven debugging. Building with sanitizers turned up
memory-safety bugs in paths that had "worked" for years; fixing them\nwithout changing observable behaviour meant carefully tracking ownership\nof every duplicated buffer.
All work merged upstream into the gsoc2026 branch through pull
requests #13–#39.
Every item on the TODO list is addressed except racoon2ctl: phase 2
SA management (1), policy generation for IKEv1 (3), IPv6 (4), NAT-OA\npayloads (5), address substitution (6), fragmentation (7) and the\naddress macro confusion (8) are done; only item 2, the graceful\nconnect/disconnect control tool, remains open.
A self-contained unit test framework with four new test programs
covering the NAT-OA, address-substitution and wildcard-address logic.\nThe full suite — including the pre-existing crypto self-test — passes\ncleanly under sanitizers: 5 out of 5 test programs, 0 failures.
A new NEWS entry, and documentation and sample configurations matching
the new defaults.
The following items are recorded in the top-level TODO file and are\ndeliberately left for future work:
A racoon2ctl tool. There is still no way for an initiator to
connect and disconnect gracefully: sending SIGTERM to the daemon does\nnot delete the security associations or send a delete notification to\nthe peer the way an IKEv2 connection would, and for IKEv1 this gap is\nespecially noticeable. A small control tool (and the delete-on-shutdown\nbehaviour behind it) would make racoon2 behave like a proper long-running\nservice instead of something you kill and re-run.
Tests for the functionality that still has none. This project added
tests only for the logic it introduced. The rest of the daemon — the\nphase 2 SA lifecycle, policy generation, rekeying, fragmentation paths,\nthe configuration parser as a whole — is still only exercised by\nrunning it by hand. Extending the framework with tests for those areas\nis the natural next step, and the purge-related test added here is\nmeant as a starting point for exactly that.
End-to-end IPv6 testing. IPv6 now works on the configuration side,
but it still needs much more thorough testing in real scenarios on\nNetBSD and Linux: full IPv6 tunnels, NAT-T interactions, and\nlong-running connections on both initiator and responder sides have not\nbeen exercised yet.
This project started as "work down the TODO list" and grew into a\nsubstantially more robust racoon2: NAT traversal now follows the relevant\nRFCs for both IKE versions, IPv6 actually works and is used by default,\nconfigurations behave consistently, and — perhaps most importantly for the\nlong term — the\ndaemon finally has an automated test suite running under sanitizers.
The open items are collected in the "What remains" section above. I\nbelieve the combination of RFC-level fixes and a regression test framework\nmakes racoon2 meaningfully closer to being a dependable VPN server for\nreal clients on NetBSD and Linux.
\nI'd like to thank my mentor, Christos Zoulas, for reviewing this stream of\npull requests, for patiently explaining the history behind this project and quirks like the\naddress-macro mess. Thanks also to\nthe other mentors and students of GSoC 2026 for a pleasant summer of\ndigging into an old codebase and making it a little less scary.
","headings":[{"level":2,"text":"Introduction & Description","id":"introduction-description"},{"level":2,"text":"What was done","id":"what-was-done"},{"level":3,"text":"IKE fragmentation (RFC 7383)","id":"ike-fragmentation-rfc-7383"},{"level":3,"text":"NAT-OA payloads in IKEv1 Quick Mode (RFC 3947)","id":"nat-oa-payloads-in-ikev1-quick-mode-rfc-3947"},{"level":3,"text":"Address substitution in NAT-T transport mode (RFC 7296 §2.23.1)","id":"address-substitution-in-nat-t-transport-mode-rfc-7296-2-23-1"},{"level":3,"text":"Address macros in configurations","id":"address-macros-in-configurations"},{"level":3,"text":"IPv6 support fixed and enabled by default","id":"ipv6-support-fixed-and-enabled-by-default"},{"level":3,"text":"Policy and proposal negotiation","id":"policy-and-proposal-negotiation"},{"level":3,"text":"A unit test framework for the IKE daemon","id":"a-unit-test-framework-for-the-ike-daemon"},{"level":3,"text":"Documentation, samples and NEWS","id":"documentation-samples-and-news"},{"level":2,"text":"How to Use","id":"how-to-use"},{"level":2,"text":"Challenges","id":"challenges"},{"level":2,"text":"Results","id":"results"},{"level":2,"text":"What remains","id":"what-remains"},{"level":2,"text":"Conclusion","id":"conclusion"}]}}