---
title: "Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD"
slug: big-fish-exploiting-a-37-year-old-kernel-bug-in-openbsd
url: https://listedarticles.com/articles/big-fish-exploiting-a-37-year-old-kernel-bug-in-openbsd
canonical_url: https://calif.io/research/big-fish
content_type: research
language: en
published_at: 2026-08-10T00:00:00.000Z
updated_at: 2026-10-16T17:13:20.374Z
authored_by: human
publisher: "Calif"
publisher_url: https://calif.io
topics: ["Security", "OpenBSD", "Operating Systems", "Vulnerability Research"]
license: all-rights-reserved
word_count: 1101
reading_minutes: 5
citation: "Calif. \"Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD.\" 10 Aug 2026. https://calif.io/research/big-fish (all-rights-reserved)"
# The full text follows. The web page shows an extract and sends readers
# to the source above; quote the citation and link the canonical URL.
---

# Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD

> Calif describes vulnerabilities it reported to OpenBSD, including a shared-memory attachment counter defined as a signed 16-bit short in the original BSD sources roughly 37 years ago. Overflowing it enabled a use-after-free that, combined with a separate kernel out-of-bounds write, gave local root despite KARL and SMAP; fixes shipped as errata 026 and 018.

# Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD

Big fish. Small counter. Root shell.

## Contents

Calif frequently publishes our offensive research and work, but we otherwise don't draw much attention to the defensive work we do — assisting open source projects with remediation, patch generation, cryptographic improvements, and internal engineering projects we carry out for our clients.

Earlier this year, we reported some vulnerabilities, often along with suggested patches to OpenBSD. One traced back roughly [37 years to the original BSD sources](https://github.com/weiss/original-bsd/blame/master/sys/sys/shm.h#L28-L33), where a shared memory attachment counter was defined as a signed 16-bit short. Repeatedly forking a process with shared memory attachment records could push that counter past its maximum value of 32,767, causing it to wrap to a negative value. This integer overflow could lead to a use-after-free via a race condition; combined with a separate out-of-bounds write in the kernel, it allowed local privilege escalation to root. The issues were addressed in [errata 026](https://ftp.openbsd.org/pub/OpenBSD/patches/7.9/common/026_sysvipc.patch.sig) and [018](https://ftp.openbsd.org/pub/OpenBSD/patches/7.9/common/018_shmat.patch.sig), respectively.

## The exploit

We managed to get local root in OpenBSD with [`KARL`](https://isopenbsdsecu.re/mitigations/karl/) / [`SMAP`](https://en.wikipedia.org/wiki/Supervisor_Mode_Access_Prevention) / [`SMEP`](https://edc.intel.com/content/www/us/en/design/products/platforms/processor-and-core-i3-n-series-datasheet-volume-1-of-2/001/intel-supervisor-mode-execution-protection-smep/) enabled without the use of an information disclosure vulnerability via the following path:

- Map 127 SysV shared memory segments, unmap them via `munmap()` , and fork 259 processes to wrap the shared memory attach count to -32,643.
- Perform heap grooming and layout, using metrics from sysctl(`KERN_MALLOCSTATS)` (such as`kb_total` and`kb_totalfree` ) to track allocations and determine when new pages are allocated.
- Win the race condition using `IPC_RMID` in one process while attaching a new size-confused shared memory segment in a helper process.
- Align IP socket allocations and set `IP_MULTICAST_TTL` to a non-default value of 2 to allocate a persistent 16-byte[`struct ip_moptions`](https://github.com/openbsd/src/blob/1b90eb3a568a1476ae3aba0f237490961fc422a0/sys/netinet/ip_var.h#L164-L170) . Its final word packs two 16-bit counters:`imo + 12 = imo_num_memberships | imo_max_memberships << 16` .
- Trigger the conditional `-1` out-of-bounds write to overwrite both the number of memberships and maximum memberships fields.
- `close()` the socket to traverse the group members, triggering[atomic decrements](https://github.com/openbsd/src/blob/1b90eb3a568a1476ae3aba0f237490961fc422a0/sys/netinet/in.c#L945-L984) that modify the PTE self-referential mapping twice, switching it from user to supervisor accessible mapping, and achieving kernel code execution by overwriting a predictable kernel pointer.
- Clean up shared memory accounting and restore the overwritten kernel pointer to obtain a root shell.

That's the [short of the matter](https://github.com/openbsd/src/blob/1b90eb3a568a1476ae3aba0f237490961fc422a0/sys/sys/shm.h#L92-L99), the longer version is [more interesting](https://github.com/califio/publications/tree/main/MADBugs/openbsd/big-fish.c), including the history of the vulnerability, as a short `shm_nattch` can be traced back [~37 years ago](https://github.com/weiss/original-bsd/blame/master/sys/sys/shm.h#L28-L33), in the original BSD sources.

Taking a step back, this vulnerability ended up getting code execution via a `write what where` primitive, where the `write` was caused by a [`TAILQ_REMOVE`](https://github.com/openbsd/src/blob/master/sys/sys/queue.h), the `what` was our userland memory mapping that was made supervisor mode accessible to bypass `SMAP`/`SMEP`, and the `where` was a predictable kernel pointer location.

## Exploit mitigation

While OpenBSD's KARL relinks the kernel binary to randomize both function and data addresses, the kernel's memory allocators remain deterministic. Both the [boot-time allocator](https://github.com/openbsd/src/blob/1b90eb3a568a1476ae3aba0f237490961fc422a0/sys/uvm/uvm_addr.c#L701-L734) and the [best-fit allocator](https://github.com/openbsd/src/blob/1b90eb3a568a1476ae3aba0f237490961fc422a0/sys/uvm/uvm_init.c#L179-L190) lack base address randomization. Consequently, early boot allocations consistently land at predictable locations across boots on the same hardware. When these predictable structure allocations contain function pointers, they become ideal targets for exploitation. For instance, the exploit reliably targets the `lo0` and `enc0` `ioctl` handlers, overwriting one to gain control flow and using the other to restore the overwritten pointer.

Aside from the aforementioned errata fixes, there's several other approaches that can be taken to either eliminate or reduce the likelihood of those avenues being used again.

Firstly, we can implement safe-unlinking, a technique that checks doubly linked list invariants before performing the unlink memory writes. If the invariants are violated, then a suitable `abort()` or `panic()` is called, depending on if user mode or kernel mode, preventing memory corruption from occurring. An example implementation is [here](https://github.com/CalANDif/openbsd-src/commit/dc16d6afe2d897c5e2be8227f737c0d2ebdbfc86). [FreeBSD](https://github.com/freebsd/freebsd-src/blob/main/sys/sys/queue.h) and [NetBSD](https://github.com/NetBSD/src/blob/trunk/sys/sys/queue.h) have an implementation of safe-unlinking in their `sys/queue.h` implementation.

Secondly, we can randomize the self-referential PTE map allocation address so that the targeted atomic decrement locations would not be possible. This is done by using a random byte (`.openbsd.randomdata`) and selecting a [PML4](https://wiki.osdev.org/X86_Page_Tables) slot (`320 + (random_byte & 127)`) at boot time, and installing the recursive entry at that location. The `PTE_BASE` definition becomes a runtime expression, yielding bases from `0xffffa00000000000` through `0xffffdf8000000000` at 512-GiB intervals. An example implementation is [here](https://github.com/CalANDif/openbsd-src/commit/89db398d53e6698a80eaf72f32d1a3e544d2e95f). In case randomization is not suitable, other options might suffice, such as marking the PTE as read-only, and changing CR0.WP during modification.

Thirdly, the kernel allocator can be made non-deterministic using two approaches; by [randomizing the base address](https://github.com/CalANDif/openbsd-src/commit/94c29e3432ad016ce34004d52594382a166ee707) where memory allocations occur at boot time, and additionally [randomizing the allocation order at run time](https://github.com/CalANDif/openbsd-src/commit/dd6cd7b9655b1a7fdfabfa97f82ead646e15333a). The base address randomization is effectively free as the gap it creates can still be used later on; while runtime randomization does incur some overhead during allocation - it may be possible that more in-depth allocator changes could alleviate this impact. Either approach would be sufficient to break the predictable pointer location without additional vulnerabilities, be it software or hardware.

Furthermore, there are existing and upcoming compiler features that can be used to mitigate against entire vulnerabilities, from [array bounds sanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) (a subset of Undefined Behaviour sanitizer) which can eliminate out of bounds fixed size array accesses. Calif has [suggested to FreeBSD](https://github.com/freebsd/freebsd-src/pull/2404) that the array bounds sanitizer may make a useful exploit mitigation that can be enabled in release builds, along with privately reporting 21 issues that the array bounds sanitizer would have caught; some with local privilege escalation security impact. An upcoming compiler feature that can be used to help mitigate against heap issues is [Alloc Tokens](https://clang.llvm.org/docs/AllocToken.html), which can be used to isolate pointer containing allocations from non-pointer allocations. This isolation can make overflows from the non-pointer containing allocation much more difficult to exploit by preventing pointer access, along with making vulnerability classes like Use-after-Free unable to use the opposite type of allocation as the replacement memory allocation.

Since its inception, OpenBSD has consistently championed proactive security through source code auditing, the replacement of insecure APIs, advanced password hashing, cryptography, privilege separation, sandboxing, along with memory safety via [`ASLR`](https://en.wikipedia.org/wiki/Address_space_layout_randomization) and [`W^X`](https://en.wikipedia.org/wiki/W%5EX); frequently setting the standard for system and network hardening.

By proactively hardening, and adapting new and existing mitigation options, we can often systematically neutralize entire classes of exploitation primitives; or at the very least, increase the attacker's cost (i.e. additional time investment, additional vulnerabilities needed to exploit an issue, find alternative avenues, etc). Investing in defensive improvements before vulnerabilities are actively targeted raises the bar for attackers, helping ensure robust protection and long-term resilience. In the age of AI-assisted vulnerability discovery, mitigating against entire vulnerability classes, combined with increasing attack complexity only becomes more critical.