i've used containers before. you run something, it works, and you don't really think about what is happening underneath. you just know that somehow this thing called a "container" makes your application run in its own environment.
\nand honestly, i didn't really like that. i wanted to know what is actually happening when you run a container. what makes it isolated? how does it get its own filesystem? how does it get its own network? how does it stop a process from using all the memory on the machine?
\nso instead of just reading about it, i decided to build one myself.
\nthat's how forge started.
\nforge is a small container runtime that i built from scratch in go. the source is on github ↗
\nat a high level, you give forge something to run and it creates an isolated environment for it. something like:
\nforge run alpine:3.20 /bin/sh\nand you get a shell running inside a container.
but forge isn't using docker underneath. i built the main pieces myself using linux primitives.
\n*6*stages, each one a complete working system
\n12internal packages, one per kernel concept
\n*1*dependency outside the standard library (golang.org/x/sys)
\n*0*shell-outs to docker, runc, ip or iptables
\nthe goal was never to build something that replaces docker. i wanted to understand what a container actually is and what has to happen underneath that one simple command.
\nthis was probably the biggest thing i wanted to understand.
\nwhen you hear "container", it can sound like you're starting some kind of tiny virtual computer. you're not.
\nat the end of the day, a container is still a normal process running on your computer. the difference is that the process is given boundaries.
\nit can be made to see its own processes, have its own hostname, have its own filesystem and network, and have limits on how many resources it is allowed to use.
\nso the container feels like its own little machine, even though it's actually running on the same linux kernel as everything else.
\nthe idea sounds pretty simple when you say it like that. actually making it work is a different story.
\ni built forge in six stages, with each stage adding another important part of what we normally think of as a container.
\nthe first thing i needed was isolation.
\nforge uses linux namespaces to give processes their own view of certain parts of the system. for example, a process inside the container can have its own pid namespace, which means it can see itself as pid 1 instead of seeing every process running on my laptop.
\nit can also have its own hostname and mount namespace.
\nthe part that actually asks the kernel for this is tiny. a Config says which namespaces you want, and that turns into flags for clone(2):
internal/namespace/namespace.go
// CloneFlags returns the clone(2) flags that create the requested namespaces.\n//\n// This is pure computation and is deliberately separated from Apply so the\n// mapping from Config to kernel flags is unit-testable without root.\nfunc (c Config) CloneFlags() uintptr {\n var flags uintptr\n if c.PID {\n flags |= syscall.CLONE_NEWPID\n }\n if c.UTS {\n flags |= syscall.CLONE_NEWUTS\n }\n if c.Mount {\n flags |= syscall.CLONE_NEWNS\n }\n if c.Net {\n flags |= syscall.CLONE_NEWNET\n }\n return flags\n}that was the first thing that surprised me. asking for isolation is one line per namespace. making the isolation actually hold is the rest of the project.
\nthe very first example of that is the mount namespace. CLONE_NEWNS gives you a copy of the host's mount table, and the copy inherits each mount's propagation type. on any systemd host / is shared, so a mount made inside the container would still travel back out to the host. so the child has to detach the tree itself, from inside:
func makeMountTreePrivate() error {\n const (\n source = "none"\n target = "/"\n fstype = ""\n data = ""\n )\n if err := syscall.Mount(source, target, fstype, syscall.MS_REC|syscall.MS_PRIVATE, data); err != nil {\n return fmt.Errorf("making mount tree private: %w", translatePermission(err))\n }\n return nil\n}a new mount namespace is not an empty one. it starts as a copy of its parent's, propagation types and all. without the recursive MS_PRIVATE above, every mount the container makes would still show up on the host. the namespace would exist, and the isolation still wouldn't.
this was the first point where forge started feeling like an actual container instead of just another go program starting a process.
\nnext was the filesystem.
\nif i start a container, i don't want the process to simply have access to my entire computer.
\nforge creates an isolated filesystem environment for the container and uses things like mounts and pivot_root to give the process its own view of the filesystem.
so when you're inside the container, / is no longer simply the / from my laptop. it's the container's root filesystem.
internal/mount/apply.go
// PivotRoot makes newRoot the calling process's root filesystem and detaches\n// the old one.\nfunc PivotRoot(newRoot string) error {\n newRoot = filepath.Clean(newRoot)\n // The kernel refuses a new root that is not a mount point, with a bare\n // EINVAL that explains nothing. Say what it means instead.\n mounted, err := IsMountPoint(newRoot)\n if err != nil {\n return err\n }\n if !mounted {\n return fmt.Errorf("%w: %q is a plain directory; bind it onto itself first", ErrRootNotMountPoint, newRoot)\n }\n putOld := filepath.Join(newRoot, oldRootDirName)\n if err := os.Mkdir(putOld, oldRootPerm); err != nil && !os.IsExist(err) {\n return fmt.Errorf("creating %q for pivot_root: %w", putOld, err)\n }\n // Entering the new root first means the process holds no reference to the\n // old one when it is detached below.\n if err := unix.Chdir(newRoot); err != nil {\n return fmt.Errorf("entering the new root %q: %w", newRoot, err)\n }\n if err := unix.PivotRoot(newRoot, putOld); err != nil {\n return fmt.Errorf("pivot_root to %q: %w", newRoot, translatePermission(err))\n }\n // "/" now means the new root, and the old one hangs off it.\n if err := unix.Chdir("/"); err != nil {\n return fmt.Errorf("entering the pivoted root: %w", err)\n }\n oldRoot := string(filepath.Separator) + oldRootDirName\n if err := unix.Unmount(oldRoot, unix.MNT_DETACH); err != nil {\n return fmt.Errorf("detaching the old root at %q: %w", oldRoot, translatePermission(err))\n }\n // ... the empty directory the old root hung from is removed here.\n return nil\n}chroot only changes the calling process's root directory. the old root stays mounted, stays listed in the process's own mount table, and is still reachable through the classic mkdir tmp; chroot tmp; chdir(../../..) walk. after pivot_root and the detach above, there is nothing left to walk back to.
this was also where i started seeing how many little details are involved in something that looks extremely simple from the outside.
\none of those details is that a mount destination is a path inside a root filesystem i didn't create. resolving /etc/hosts the way the host would is exactly how a bind mount ends up writing to the host's /etc, so forge resolves every path component itself, rebasing absolute symlinks against the container's root instead of following them out.
isolation isn't enough.
\nwhat happens if a process inside the container decides to use as much memory as possible? or creates thousands of processes? or tries to use all of the cpu?
\nthat's where cgroups come in.
\nforge uses cgroups to put limits on resources such as memory, cpu and the number of processes.
\nwith cgroups v2 a "limit" is genuinely just a string written into a file, so the whole thing splits cleanly into a pure function that decides what the kernel is told, and the write itself:
\ninternal/cgroup/cgroup.go
func (l Limits) Files() []File {\n var files []File\n if l.MemoryMax != nil {\n files = append(files,\n File{Name: "memory.max", Value: l.MemoryMax.String()},\n // Swap is limited alongside memory, never independently.\n File{Name: "memory.swap.max", Value: swapFor(*l.MemoryMax), Optional: true},\n )\n }\n if l.CPU != nil {\n files = append(files, File{Name: "cpu.max", Value: l.CPU.String()})\n }\n if l.CPUWeight != nil {\n files = append(files, File{Name: "cpu.weight", Value: l.CPUWeight.String()})\n }\n if l.PIDsMax != nil {\n value := Unlimited\n if *l.PIDsMax >= 0 {\n value = strconv.FormatInt(*l.PIDsMax, 10)\n }\n files = append(files, File{Name: "pids.max", Value: value})\n }\n return files\n}and joining the cgroup is one write too:
\ninternal/cgroup/apply.go
// addProc makes the process a member of the cgroup at dir by writing its PID\n// to cgroup.procs.\n//\n// Writing a PID moves the whole thread group, and every process it forks from\n// then on is a member too. It does not move processes it has *already* forked,\n// which is why internal/runtime attaches the container's init before the init\n// is allowed to proceed past its handshake.\nfunc addProc(dir string, pid int) error {\n return writeControlFile(dir, fileProcs, strconv.Itoa(pid))\n}that comment is the whole timing problem in stage 3. limits have to be in place before the container runs its first instruction, otherwise there is a window where an unlimited process already exists.
\nso the container isn't just isolated. it also has limits on what it is allowed to consume.
\nthen came networking.
\na container isn't very useful if it can't communicate with anything.
\nforge creates isolated network namespaces and sets up networking using things like veth pairs and a bridge. it also handles ip allocation and nat so that containers can communicate and reach the outside world.
\nthis is the part i underestimated the most. everything the kernel is told here goes over a raw netlink socket, with no netlink library and no shelling out to ip or iptables. which means a netlink message is just a byte layout you have to get exactly right:
internal/network/network.go
// nlAttr encodes one netlink attribute: a length, a type, the payload, and\n// enough padding to align the next attribute.\n//\n// The encoded length covers the header and the payload but *not* the padding,\n// which is why this cannot be a simple append of a header to a body.\nfunc nlAttr(typ uint16, payload []byte) []byte {\n length := nlAttrHdrLen + len(payload)\n buf := make([]byte, nlAlign(length))\n order.PutUint16(buf[0:2], uint16(length))\n order.PutUint16(buf[2:4], typ)\n copy(buf[nlAttrHdrLen:], payload)\n return buf\n}and once you can build attributes, an operation that sounds enormous ("move this interface into the container's network namespace") turns out to be one message:
\ninternal/network/namespace.go
func moveLinkToNetns(c *nlConn, index int32, pid int) error {\n body := concat(\n ifInfoMsg(index, 0, 0),\n nlAttrU32(unix.IFLA_NET_NS_PID, uint32(pid)),\n )\n if err := c.execute(unix.RTM_NEWLINK, 0, body); err != nil {\n return fmt.Errorf("moving interface %d into the network namespace of pid %d: %w", index, pid, err)\n }\n return nil\n}forge never enters the container's network namespace to configure it. setns(2) changes the namespace of the calling thread, and in go that means locking an os thread and hoping nothing migrates. get it wrong and forge itself is left sitting inside a container's network namespace. so the parent does the one thing only it can do, which is move the interface into a namespace it can name by pid, and the container configures its own interface from a plain description that arrived over a pipe.
this was probably one of the parts where the command:
\nforge run ...\nhides the most work.
behind that one command, forge has to create and connect a whole network environment for the container.
\nat this point i could create an isolated environment, but what exactly am i going to run inside it?
\nthat's where container images come in.
\nforge supports oci images. so when i run something like:
\nforge run alpine:3.20 /bin/sh\nforge can pull the image, verify it, unpack its layers, and use that filesystem as the container's root filesystem.
verification is the part i cared about most here. every document and every blob is named by the sha-256 of its own bytes, so it can be checked at every boundary the bytes cross, and forge checks at all of them: in flight as the registry streams them, again on the write to the cache, and again when a layer is decompressed for use.
\ninternal/image/blob.go
// FetchBlob streams one blob into w, verifying it as the bytes pass (FR-5.2).\n//\n// Both the digest and the length are checked. A hash mismatch alone would catch\n// a truncated response, but "the registry sent 3 MB of the 5 MB it promised" is\n// the sentence an operator can act on, and "the hash was wrong" is not.\nfunc (c *Client) FetchBlob(ctx context.Context, ref Reference, d Descriptor, w io.Writer) error {\n hasher, err := newHasher(d.Digest)\n if err != nil {\n return err\n }\n resp, err := c.get(ctx, ref, c.endpoint(ref, "blobs", d.Digest), nil)\n if err != nil {\n return err\n }\n defer drain(resp, c.logger)\n written, err := io.Copy(io.MultiWriter(w, hasher), resp.Body)\n if err != nil {\n if ctxErr := ctx.Err(); ctxErr != nil {\n return ctxErr\n }\n return fmt.Errorf("%w: downloading %s after %d bytes: %w",\n ErrRegistryUnavailable, d.Digest, written, err)\n }\n if d.Size > 0 && written != d.Size {\n return fmt.Errorf("%w: %s sent %d bytes for %s, but its descriptor says %d",\n ErrDigestMismatch, ref.Host(), written, d.Digest, d.Size)\n }\n if computed := formatDigest(hasher); computed != d.Digest {\n return fmt.Errorf("%w: %d bytes from %s hash to %s, but were requested as %s",\n ErrDigestMismatch, written, ref.Host(), computed, d.Digest)\n }\n return nil\n}it also has image caching so that already downloaded content doesn't have to be downloaded again every time. layers are keyed by digest, so the second run of the same image downloads nothing.
\nthis is integrity, not authenticity. it proves the bytes are the bytes that digest names. it does not prove who made them. signature and provenance verification is out of scope for forge.
\nthis was the part that made the project feel much closer to an actual container runtime.
\nthe final stage was taking all of these pieces and turning them into an actual runtime.
\ninstead of having a collection of things that can create namespaces or configure networking, forge now has a container lifecycle.
\nyou can:
\nruncreate and start a container, attached
\npswhat is running, from the on-disk state store
\nexeca second process inside its namespaces
\nlogscaptured stdout/stderr, attached or not
\nstopsigterm, then sigkill after the timeout
\nrmremove it and everything still held for it
\nyou can start a container, see which containers are running, execute a command inside an existing container, read its logs, stop it and finally remove it.
\nsudo forge run --keep alpine:3.20 /bin/sh -c 'while :; do date; sleep 1; done'\n# in another terminal\nsudo forge ps\nCONTAINER ID IMAGE COMMAND STATUS CREATED PID\n7f3c9a1b2d04 alpine:3.20 /bin/sh -c … running 12 seconds ago 48213\nsudo forge exec 7f3c9a1b2d04 /bin/ps\n# the container's processes, not the host's\nthat's when forge finally felt like a complete project to me.
\nby stage 6 there are six primitives that all have to happen in a particular order, split across two processes, because most of what makes a container a container can only be done by code already running inside the new namespaces. so forge doesn't start the container's binary directly. it starts itself again, and that second copy does the rest to itself.
\n1resolve the image reference (pure, no i/o)
2fetch the manifest, resolving the tag to an immutable digest
3download only the layers the blob cache is missing
4verify digests: on the wire, on the write, and at use
5construct the rootfs: layers unpacked, base first
6build the mount plan its init will apply
7create the cgroup leaf and write the limits
8bridge, nat and an address, claimed before there is a container to leak it
9start the child, attach its pid to the cgroup, move the interface into its namespace, plug the host end into the bridge
10send the payload down the pipe, releasing the child
11wait: supervise until the container exits
12unwind the cleanup stack, in reverse
·blocked on the payload pipe, running nothing
a
namespace.Apply: mount tree made private first, or every mount below it propagates to the host\nb
network.Configure: the pushed-in interface brought up and routed, while netlink is all that is needed\nc
mount.Apply: made while the host filesystem is still reachable, because bind sources are host paths\nd
mount.PivotRoot: "/" becomes the container's root, the old one is detached\ne
chdir: the working directory, now a container path\nf
execve: the process becomes the container\nthe child side of that is small enough to read in one go, and the order is load-bearing at every single line:
internal/runtime/init.go
// Init is the container's entry point, executed by the re-exec'd forge binary\n// inside the new namespaces created by clone(2).\nfunc Init() error {\n payload, err := readInitPayload()\n if err != nil {\n return err\n }\n if err := namespace.Apply(payload.Namespace); err != nil {\n return err\n }\n if err := configureNetwork(payload); err != nil {\n return err\n }\n if payload.Mount != nil {\n if err := mount.Apply(*payload.Mount); err != nil {\n return err\n }\n if err := mount.PivotRoot(payload.Mount.Root); err != nil {\n return err\n }\n }\n if err := enterWorkingDir(payload.WorkingDir); err != nil {\n return err\n }\n path, err := resolveCommand(payload.Command[0], payload.Env)\n if err != nil {\n return err\n }\n if err := syscall.Exec(path, payload.Command, payload.Env); err != nil {\n return fmt.Errorf("executing %s: %w", path, err)\n }\n // Unreachable: a successful execve never returns.\n return errors.New("execve returned without an error")\n}on success this function never returns. execve replaces the process with the container's binary, which inherits its pid. and inside a pid namespace, that pid is 1.
because that wasn't the point.
\nif i just wanted to run containers, i would use docker. the whole reason i made forge was to understand what docker and other container runtimes are actually doing underneath.
\nusing something is one thing. building a smaller version yourself is a completely different experience.
\nwhen you're forced to implement the pieces yourself, you can't just say "docker handles that."
\nyou have to figure out what "that" actually means.
\nhow does the process get isolated? how does the filesystem change? how does networking get connected? how are resources limited? how does the image become a filesystem? how does everything get cleaned up when the container stops?
\nthose were the things i wanted answers to.
\nthe biggest thing i learned is that a container isn't really one big complicated thing. it's a bunch of smaller linux features working together.
\nnamespaces handle isolation. cgroups handle resource limits. the filesystem gives the process its own environment. networking connects it to the outside world. oci images provide the filesystem and configuration. and the runtime puts all of those pieces together.
\nonce you break it down like that, the word "container" becomes a lot less mysterious.
\nthis wasn't just me writing some code and everything magically working.
\nthere were plenty of moments where something looked correct but didn't actually work the way i expected, especially with things like networking, cleanup, process handling and filesystem setup.
\nand that's actually one of the reasons i wanted to build this instead of just reading about containers.
\nwhen something breaks, you can't skip over the details. you have to figure out exactly which part of the system is responsible.
\nthat ended up teaching me a lot more than just reading a diagram of how containers work.
\ni also didn't want forge to be one of those projects where the readme says it works because one command worked once.
\nthe project has unit tests and integration tests covering the different parts of the runtime. i also use race detection and linting as part of the validation.
\nmake test # unit tests, with -race, no root required\nmake test-integration # privileged integration tests (root, linux)\nmake lint # golangci-lintthe split matters more than it looks. everything that is pure (clone flags, cgroup limit files, netlink byte layouts, subnet arithmetic, mount-path resolution) is a function from values to bytes, so it can be asserted without a kernel. everything that actually touches the kernel lives behind a build tag and runs as root.
\nthe final testing isn't just about checking whether a container starts. i need to know that processes are actually isolated, the filesystem is actually isolated, resource limits actually work, networking actually works, images are handled correctly, commands like exec, logs, stop and rm work, multiple containers can coexist, and cleanup actually happens.
because a container that starts but leaves half its resources behind isn't really a finished runtime.
\njust to be clear, forge isn't trying to be a replacement for docker or any other production container runtime.
\nit's a learning project.
\ni wanted something small enough that i could actually understand the whole thing.
\nsomething where i could go from:
\nprocessstill just a normal process on the host
\nnamespaceits own pids, hostname, mounts, network
\nfilesystempivot_root, bind mounts, its own "/"
\ncgroupmemory, cpu and pid ceilings
\nnetworkveth, bridge, an address, nat
\nimageoci layers, verified and unpacked
\nruntimea lifecycle that ties all six together
\nand understand what each part is doing.
\nthat's what forge is.
\nforge is an educational systems project. it is not production-hardened and should not be used to run untrusted workloads. there is no seccomp, no apparmor or selinux integration, no rootless mode, no image building and no orchestration. those are deliberate non-goals, not a todo list.
\nif everything above sounded confusing, here's the easiest way i can explain it.
\nso basically imagine you have a game on your laptop. it works perfectly. you send that same game to your friend, and they try to run it but it doesn't work.
\nmaybe their laptop has different software installed. maybe they're missing something your laptop has. maybe their settings are different.
\nthat's the classic:
\n"it works on my laptop."
\na container is kind of like giving that game its own little room with the things it needs inside.
\ninstead of depending completely on what the rest of the laptop looks like, it gets its own controlled environment.
\nforge is something i built to create those little rooms.
\nthat's probably the simplest way i can explain the entire project.
\nthanks for reading.
\nA container runtime built from scratch in Go using Linux primitives and raw kernel interfaces.
","headings":[{"level":1,"text":"Inside Forge | Writing a Container Runtime From Scratch","id":"inside-forge-writing-a-container-runtime-from-scratch"},{"level":1,"text":"i built my own container runtime in go","id":"i-built-my-own-container-runtime-in-go"},{"level":2,"text":"so, what is forge?","id":"so-what-is-forge"},{"level":2,"text":"but what actually is a container?","id":"but-what-actually-is-a-container"},{"level":2,"text":"what forge does","id":"what-forge-does"},{"level":3,"text":"stage 1: process isolation","id":"stage-1-process-isolation"},{"level":3,"text":"stage 2: filesystem isolation","id":"stage-2-filesystem-isolation"},{"level":3,"text":"stage 3: resource limits","id":"stage-3-resource-limits"},{"level":3,"text":"stage 4: networking","id":"stage-4-networking"},{"level":3,"text":"stage 5: images","id":"stage-5-images"},{"level":3,"text":"stage 6: the runtime","id":"stage-6-the-runtime"},{"level":3,"text":"the whole thing, in one picture","id":"the-whole-thing-in-one-picture"},{"level":2,"text":"why didn't i just use docker?","id":"why-didn-t-i-just-use-docker"},{"level":2,"text":"what i learned from building it","id":"what-i-learned-from-building-it"},{"level":2,"text":"and obviously, things went wrong","id":"and-obviously-things-went-wrong"},{"level":2,"text":"how i tested it","id":"how-i-tested-it"},{"level":2,"text":"forge isn't docker","id":"forge-isn-t-docker"},{"level":2,"text":"if you're not a tech person","id":"if-you-re-not-a-tech-person"},{"level":3,"text":"Related Project","id":"related-project"}]}}