{"article":{"slug":"hooking-into-the-go-toolchain","title":"Hooking into the Go Toolchain","subtitle":null,"summary":"A hands-on tour of go build -toolexec: how the go command invokes the compiler, assembler and linker, and how to time, rewrite and instrument every run, from a simple stopwatch to injecting packages via importcfg and keeping the build cache honest, as used by garble, Orchestrion and OpenTelemetry's otelc.","content_type":"tutorial","language":"en","canonical_url":"https://internals-for-interns.com/posts/hooking-into-the-go-toolchain/","author":{"name":"Kemal Akkoyun and Jesús Espino","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Internals for Interns","url":"https://internals-for-interns.com/","listing_slug":null,"listing":null},"topics":[{"name":"Go","slug":"go","url":"https://listedarticles.com/topics/go"},{"name":"Compilers","slug":"compilers","url":"https://listedarticles.com/topics/compilers"},{"name":"Toolchains","slug":"toolchains","url":"https://listedarticles.com/topics/toolchains"},{"name":"OpenTelemetry","slug":"opentelemetry","url":"https://listedarticles.com/topics/opentelemetry"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":5364,"reading_minutes":23,"published_at":"2026-10-05T07:00:00.000Z","added_at":"2026-10-06T11:14:01.653Z","updated_at":"2026-10-06T11:14:01.653Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/hooking-into-the-go-toolchain","markdown_url":"https://listedarticles.com/articles/hooking-into-the-go-toolchain.md","example":false,"citation":"Kemal Akkoyun and Jesús Espino, Internals for Interns. \"Hooking into the Go Toolchain.\" 5 Oct 2026. https://internals-for-interns.com/posts/hooking-into-the-go-toolchain/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://internals-for-interns.com/posts/hooking-into-the-go-toolchain/"},"body_markdown":"Today we’re going to take a look at the Go toolchain, and more specifically at how we can take part in the compilation process with our own code. We won’t patch the compiler. We’ll stand right next to it while it works, watch what it does, and every now and then hand it something it didn’t ask for. Think of it as photobombing the compiler, politely. 📸\n\nThe way in is a single flag, and the cheapest experiment I know looks like this:\n\n```\n$ go build -a -o /tmp/app -toolexec=/usr/bin/time ./app\n# internal/unsafeheader\n 0.02 real 0.00 user 0.00 sys\n# internal/goarch\n 0.03 real 0.00 user 0.00 sys\n# internal/coverage/rtcov\n 0.02 real 0.00 user 0.00 sys\n...\n```\nEach of those little `real user sys` receipts is one program the `go` command\nran on our behalf. For this small program there are 108 of them: 59 compiler\nruns, 48 assembler runs and one linker run. `-toolexec` tells `go build` to run\neach of those programs through a program we pick, and here we picked\n`/usr/bin/time`, which makes it a stopwatch. A chatty stopwatch, but a\nstopwatch. ⏱️\n\nTiming wasn’t the plan, though. Russ Cox\n[added the flag in January 2015](https://github.com/golang/go/commit/83c10b204d619d18100716c9588404200acdf6e0)\nso the toolchain could run under tools like valgrind, or with a stashed copy of\nthe compiler swapped in by toolstash, a tool he wrote for working on the\ncompiler itself. A few weeks later toolstash learned to time every command it\nran. Years afterwards he described `-toolexec=time` as\n[“an accident - a mostly happy one”](https://github.com/golang/go/issues/27628#issuecomment-702251208)\n.\nThis post is about that happy accident, a corner of Go that Daniel Martí once\ncalled\n[“a space that hasn’t been explored much so far”](https://golab.io/talks/diving-into-the-go-toolchain-to-obfuscate-builds)\n.\nHe had been exploring it with [garble](https://github.com/burrowers/garble)\n, an\nobfuscator, and his work turns up all over this post.\n\nWe’ll follow the stopwatch the whole way. First we’ll build our own. Then we’ll\nteach it to rewrite code before the compiler sees it, sneak in packages the\nbuild never asked for, and call code it isn’t allowed to import. Along the way\nthe build cache will lie to us. At the end we’ll look at\n[otelc](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation)\n,\na real tool that does all of this for a living, and time it with the same\nstopwatch.\n\nThanks to Jesús for inviting me. His\n[series on the Go compiler](https://internals-for-interns.com/series/understanding-the-go-compiler/)\nexplains what the compiler does with our code; today is about how we get\nbetween the `go` command and the compiler in the first place. Full disclosure:\nI work at Datadog and help maintain otelc, so keep that in mind for the last\npart, and feel free to roll your eyes at the appropriate moment. All the code we’ll write lives in a\n[companion repository](https://github.com/kakkoyun/hooking-into-the-go-toolchain)\n.\n\n*For the sake of simplicity, everything below was run on macOS with Go 1.27.1. Your\nnumbers will differ, and that’s half the fun :)*\n\nBefore we can hook into anything, though, we need to know what we’re hooking\ninto. Let’s see what `go build` actually does when nobody’s watching.\n\n## What `go build` actually runs\n\nWe can get in front of every step of the build. Great! But what are those\nsteps? If we run `go build` with `-x`, it narrates every command it runs, and\n`-a` makes sure it rebuilds everything instead of reusing the cache:\n\n```\n$ go build -a -x -o /tmp/app ./app\n...\n$GOROOT/pkg/tool/darwin_arm64/compile -o $WORK/b001/_pkg_.a -trimpath \"$WORK/b001=>\" \\\n -p main -lang=go1.25 -complete -buildid kIewNeZLieJMDX0sK6uO/kIewNeZLieJMDX0sK6uO \\\n -goversion go1.27.1 -c=16 -shared -nolocalimports \\\n -importcfg $WORK/b001/importcfg -pack ./app/main.go\n...\n$GOROOT/pkg/tool/darwin_arm64/link -o $WORK/b001/exe/a.out \\\n -importcfg $WORK/b001/importcfg.link -buildmode=pie ... $WORK/b001/_pkg_.a\n```\nThat’s almost 800 lines for our little program, so I’ve kept just two of them.\nYou’re welcome.\nThe first compiles our `main` package. The compiler gets the package path\n(`-p main`), the list of `.go` files at the end, and a file passed with\n`-importcfg`. The second line, the last step of the build, links everything\ninto a binary.\n\nIn other words, `go build` is running a lot of compile commands and linking the results\ntogether at the end. That makes sense, but how does it know what needs to be\ncompiled, and what goes into the link? The build is actually a graph of\nactions, one or more per package, and each action gets its own numbered\ndirectory under `$WORK`, a temporary directory the `go` command creates for the\nbuild. `b001` is our `main` package. Before running the compiler, the `go`\ncommand writes that `importcfg` file into the action’s directory. For our\npackage it looks like this:\n\n```\n# import config\npackagefile bytes=$WORK/b002/_pkg_.a\npackagefile fmt=$WORK/b042/_pkg_.a\npackagefile github.com/kakkoyun/hooking-into-the-go-toolchain/greet=$WORK/b060/_pkg_.a\npackagefile os=$WORK/b049/_pkg_.a\npackagefile time=$WORK/b054/_pkg_.a\npackagefile runtime=$WORK/b009/_pkg_.a\n```\nOne line for each package `main.go` imports, plus `runtime`, each pointing at\nthe compiled archive of that package. That’s the compiler’s whole view of the\noutside world. It doesn’t search a `GOPATH` or read `go.mod`; if a package isn’t\nin this file, it doesn’t exist. (If you’re curious what’s inside those\narchives, Jesús’s post on the\n[unified IR format](https://internals-for-interns.com/posts/go-compiler-unified-ir/)\nopens one up.) The linker gets a similar file listing every package in the\nprogram. Keep the `importcfg` in mind, because it’s going to bite us later. (Yes, that’s\nforeshadowing. 👀)\n\nMost of the other lines in that log are the `go` command doing things itself,\nlike writing those files, creating directories or copying archives around. The\nlines that matter to us start a tool from `$GOROOT/pkg/tool/` (`compile`, `asm`\nand `link`), and those are exactly the lines `-toolexec` steps into. This is how\n`go help build` describes the flag:\n\n```\n-toolexec 'cmd args'\n\ta program to use to invoke toolchain programs like vet and asm.\n\tFor example, instead of running asm, the go command will run\n\t'cmd args /path/to/asm '.\n\tThe TOOLEXEC_IMPORTPATH environment variable will be set,\n\tmatching 'go list -f {{.ImportPath}}' for the package being built.\n```\nAnd that’s the entire interface. Our program receives the tool’s path as its\nfirst argument and the tool’s arguments after it, and it can do whatever it\nlikes before, after or instead of running the tool. The `TOOLEXEC_IMPORTPATH`\nvariable tells it which package is being built. That one is\n[Daniel’s work too](https://github.com/golang/go/commit/de74ea5d740ccc69dbb146578dc8a965351a3d6b)\n,\nadded in Go 1.16; before that, wrappers had to guess the package from the flags.\n\nNow that we know exactly where `-toolexec` steps in, let’s put something of\nour own there, starting with a better stopwatch, one that can at least tell\npackages apart.\n\n## Building our own stopwatch\n\n`/usr/bin/time` gives us numbers, but it doesn’t tell us which package each\nnumber belongs to. Let’s write a wrapper that does. The heart of it is small:\n\n```\nfunc main() {\n\ttool, args := os.Args[1], os.Args[2:]\n\tcmd := exec.Command(tool, args...)\n\tcmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr\n\t// ... forward SIGINT, SIGTERM, SIGHUP and SIGQUIT to the tool ...\n\tstart := time.Now()\n\tif err := cmd.Start(); err != nil {\n\t\tfmt.Fprintf(os.Stderr, \"stopwatch: %v\\n\", err)\n\t\tos.Exit(1)\n\t}\n\terr := cmd.Wait()\n\tlogLine(tool, args, time.Since(start)) // appends to $STOPWATCH_LOG\n\tos.Exit(exitCode(err))\n}\n```\nIt runs the real tool with the same standard input and output, measures how\nlong it took, and appends one line to a log file: the tool, the import path, the\nmilliseconds. Then it exits with the tool’s own status, so `go build` never\nnotices we’re there.\n\nLet’s build it and point `go build` at it:\n\n```\ngo build -o .bin/stopwatch ./cmd/stopwatch\nSTOPWATCH_LOG=/tmp/sw.tsv go build -a -o /tmp/app -toolexec=$PWD/.bin/stopwatch ./app\n```\nHere are a few lines from the log, one per tool call:\n\n```\ncompile | runtime | 860 | files=182\ncompile | fmt | 80 | files=5\ncompile | github.com/kakkoyun/hooking-into-the-go-toolchain/app | 16 | files=1\nlink | github.com/kakkoyun/hooking-into-the-go-toolchain/app | 59 | files=0\n```\nEach line is one tool call: which tool ran, which package it was working on, how many milliseconds it took, and how many source files it got. The whole log has 111 lines. Counting them by tool, and sorting the compiles by time, gives us this:\n\n```\ntool runs -V=full\nasm 48 1\ncompile 59 1\nlink 1 1\n ms package\n 860 runtime\n 379 reflect\n 261 internal/abi\n 230 syscall\n 195 math\n```\nNo surprise that `runtime` is the slowest package to compile; it has the\nhardest job in the building. (Don’t add the\nmilliseconds up and call it the build time, though: the tools ran in parallel,\nso their sum is larger than the time we actually waited.)\n\nThe interesting part is the last column of the first table. Before building\nanything, the `go` command ran each tool once with a single argument,\n`-V=full`, and it did that through our wrapper. In our log it looks like this:\n\n```\ncompile | - | 8 | -V=full\n```\nThat’s the `go` command asking each tool “who are you?”. A tiny identity\ncrisis, once per tool, every single build. The compiler answers\n`compile version go1.27.1`, and that answer becomes the tool’s ID. The ID ends\nup in the cache key of every package the tool compiles. That key, the action\nID, is a hash of the package’s source files, its flags, the tool ID and what\nits dependencies produced. Notice what isn’t in it: the `-toolexec` flag itself.\n\nWhy ask the wrapper instead of just reading the compiler binary? The\n[`go` command’s source](https://github.com/golang/go/blob/go1.27.1/src/cmd/go/internal/work/buildid.go#L115-L185)\nexplains: “we want ‘-toolexec toolstash’ to continue working”. If a wrapper\nswaps in a different compiler, the cache key should know.\n\nHere’s the whole picture, from the `-V=full` question to the cache:\n\nThis has a funny consequence for our wrapper: its standard output isn’t really\nours anymore. During that `-V=full` question, stdout *is* the answer. Since Go\n1.10 the `go` command\n[ignores whatever a tool prints to stderr](https://github.com/golang/go/issues/22588)\nwhile answering, as long as stdout has the right line, but stdout gets no such\npass. If our stopwatch says hello on stdout before running the tool, the build\nstops right there. Rude, but fair:\n\n```\ngo: parsing buildID from go tool compile -V=full: unexpected output:\n\tstopwatch: starting compile\ncompile version go1.27.1\n```\nIf it prints after the tool instead, things get sneakier. The build works, but our log line contains a millisecond count, so the answer, and with it the tool ID, changes on every build. A second build recompiles all 59 packages again, and nothing tells us why.\n\nOur well-behaved wrapper writes only to its log file. Let’s run the build\nagain, without `-a` this time, and look at the log:\n\n```\ncompile | - | 6 | -V=full\nasm | - | 4 | -V=full\nlink | - | 5 | -V=full\n```\nThree questions and nothing else. Every package came straight from the cache,\nso no tool ran and our stopwatch had nothing to time. Best build ever,\nterrible demo. 🤷 A `-toolexec` wrapper only\nsees what the cache lets through. Hold on to that thought; it’ll come back.\n\nWatching is fun, but our wrapper is sitting in a much more interesting spot than that. Let’s see what happens when it stops being a polite spectator.\n\n## Rewriting code before the compiler sees it\n\nNow that we’re sitting between the `go` command and the compiler, we can do\nmore than watch. The compiler gets its source files as arguments, and we see\nthose arguments first. What if we handed it different files?\n\nChanging Go source from a program sounds like something you’d only do on a\ndare, but Go makes it surprisingly friendly. The compiler has its own parser, the one Jesús takes apart in his\n[parser post](https://internals-for-interns.com/posts/the-go-parser/)\n, but the\nstandard library ships a second set of packages just for tools:\n[`go/token`](https://pkg.go.dev/go/token)\nkeeps track of positions,\n[`go/parser`](https://pkg.go.dev/go/parser)\nturns source into a syntax tree,\n[`go/ast`](https://pkg.go.dev/go/ast)\ndescribes every node in that tree, and\n[`go/printer`](https://pkg.go.dev/go/printer)\nand\n[`go/format`](https://pkg.go.dev/go/format)\nturn a tree back into code. These are\nthe packages\n[`gofmt` is built on](https://github.com/golang/go/blob/go1.27.1/src/cmd/gofmt/gofmt.go#L12-L16)\n,\nand `go vet`’s checks run on them through the\n[analysis framework](https://pkg.go.dev/golang.org/x/tools/go/analysis)\nthat\nmost Go linters use. If you’ve ever written a linter, you’ve already done the\nfirst half of what we need: find the code you care about. As Jesús puts it at\nthe end of his post, many Go developers use `go/ast`\n[“to parse Go code programmatically and build powerful tools”](https://internals-for-interns.com/posts/the-go-parser/#using-the-ast-in-your-own-code)\n.\n\nThat’s exactly what our second toy wrapper, `toyhook`, does. It looks for functions\nmarked with a `//demo:log` comment, like this one in our app:\n\n```\n//demo:log\nfunc countLines(path string) int {\n\tdata, err := os.ReadFile(path)\n\t...\n```\nFinding them takes the same three steps every linter starts with: parse the\nfile, walk the tree, and check each node. Trimmed down a little, the heart of\n`toyhook` looks like this:\n\n```\nfset := token.NewFileSet()\nfile, err := parser.ParseFile(fset, abs, src, parser.ParseComments)\nif err != nil {\n\treturn nil, 0, err\n}\nfor _, decl := range file.Decls {\n\tfn, ok := decl.(*ast.FuncDecl)\n\tif !ok || fn.Body == nil || !hasDirective(fn) {\n\t\tcontinue\n\t}\n\tlbrace := fset.Position(fn.Body.Lbrace)\n\t// ... insert our statement right after lbrace.Offset ...\n}\n```\n`parser.ParseFile` reads the file into an `*ast.File`, and `ParseComments`\nasks it to keep the comments, which we need because our marker is one. Then we\nloop over the file’s top-level declarations, keep the functions, and\n`hasDirective` checks each function’s doc comment for `//demo:log`. The\n`token.FileSet` is what turns a node back into a file, line and byte offset, so\n`lbrace` tells us exactly where the function’s opening brace is.\n\nNow we need to add our log statement. The textbook way is to build it as more\ntree: every call, identifier and literal becomes a struct, and we splice them\ninto the function body. It works, but it’s wordy, in the way tax forms are\nwordy. Here’s just `start := time.Now()` as AST nodes, from the\n[injector I wrote for a talk](https://github.com/kakkoyun/otel-night-berlin-2026/blob/65fc5bb0559331235704dc5707e165175f8c28a2/demo/toolchain/cmd/loginjector/main.go#L113-L124)\n:\n\n```\nstartDecl := &ast.AssignStmt{\n\tLhs: []ast.Expr{ast.NewIdent(\"start\")},\n\tTok: token.DEFINE,\n\tRhs: []ast.Expr{\n\t\t&ast.CallExpr{\n\t\t\tFun: &ast.SelectorExpr{\n\t\t\t\tX: ast.NewIdent(\"time\"),\n\t\t\t\tSel: ast.NewIdent(\"Now\"),\n\t\t\t},\n\t\t},\n\t},\n}\n```\nThat injector ended up at 306 lines for two log statements. Two. Log.\nStatements. 😩 Printing the tree\nback out has a catch too: `go/ast` comments “are stored by their byte offset\ninstead of attached to nodes, so re-arranging nodes breaks the output”. That’s\nstraight from the README of [dst](https://github.com/dave/dst)\n, a third-party Go\npackage (the name stands for Decorated Syntax Tree) built to fix exactly this\nproblem. It keeps comments attached to the nodes they belong to, which is why\nserious tools like otelc rewrite with it.\n\nOur toy takes a shortcut. It uses the tree only to find where each marked\nfunction’s body starts, and inserts the new statement as plain text right after\nthat brace, so every byte we didn’t touch stays where it was. When the compile\nfor our package comes through, `toyhook` writes the result into the action’s own\n`$WORK` directory, which it finds from the compiler’s `-o` flag, swaps the new\nfile into the argument list, and runs the real compiler. Let’s build it and use\nit the same way as the stopwatch:\n\n```\ngo build -o .bin/toyhook ./cmd/toyhook\nTOYHOOK_MODE=rewrite go build -o /tmp/app -toolexec=$PWD/.bin/toyhook ./app\n```\nHere’s the difference between what we wrote and what the compiler actually gets:\n\n```\n--- app/main.go\n+++ $WORK/b001/main.go\n@@ -21,6 +21,8 @@\n //\n //demo:log\n func countLines(path string) int {\n+\tfmt.Fprintf(os.Stderr, \"→ %s at %s\\n\", \"countLines\", time.Now().Format(time.TimeOnly))\n+//line $REPO/app/main.go:24\n \tdata, err := os.ReadFile(path)\n \tif err != nil {\n \t\tfmt.Fprintln(os.Stderr, err)\n```\nAnd the program now narrates itself:\n\n```\nhello, toolchain\n→ countLines at 14:17:40\ngo.mod has 3 lines\ndone in 0s\n```\nThe second line we inserted, the `//line` comment, is easy to miss but\nimportant. It’s a\n[compiler directive](https://github.com/golang/go/blob/go1.27.1/src/cmd/compile/doc.go#L184-L196)\nthat resets the file name and line number, so compiler errors, panics and\nstack traces after our insertion still point at `app/main.go:24`, and not at a\ntemporary file that’s long gone by the time anyone reads the error. Without it,\nevery line below our insertion would be off by one, which is a great way to\nmake people distrust instrumentation.\n\nThe wrapper is called for every package in the build, but it only rewrites one; for the other 58 compiles it gets out of the way and runs the compiler untouched. Each call is a fresh process that sees a single package, and it can only change the files of that package. That limitation is exactly what the next two experiments run into.\n\n## Packages the build never asked for\n\nOur app already imported `fmt`, `os` and `time`, so the code we inserted only\nused packages the compiler knew about. Let’s get more ambitious and log with\n`log/slog`, which our app never imports. Adding one import sounds simple\nenough, right? Well, with `TOYHOOK_MODE=slog`, `toyhook` adds the import and the\ncall, and the compiler says:\n\n```\n$WORK/b001/main.go:3:8: could not import log/slog (open : no such file or directory)\n```\nRemember the `importcfg`? The `go` command wrote it from the imports in the\n*original* file, before it ever called us. The compiler looks up `log/slog` in\nthat file, finds nothing, and tries to open an empty path. Told you it would\nbite.\n\nEvery tool that adds imports hits this wall. Julio Guerra\n[asked about it in 2019](https://github.com/golang/go/issues/35204)\n, and Ian\nLance Taylor’s [answer](https://github.com/golang/go/issues/35204#issuecomment-547168404)\nwas that “the `-toolexec` option is not powerful enough to\nsupport arbitrary source code rewriting.” Julio\n[did it anyway](https://github.com/golang/go/issues/35204#issuecomment-633996403)\n,\nand so will we.\n\nThe trick is to write the missing lines ourselves. The `go` command will happily\ntell us where the compiled archive of any package lives:\n\n```\ngo list -deps -export -f '{{if .Export}}packagefile {{.ImportPath}}={{.Export}}{{end}}' log/slog\n```\nWith `TOYHOOK_IMPORTCFG=patch`, `toyhook` adds the lines the compiler’s\n`importcfg` is missing. Then, when the link command comes through at the end of\nthe build, it does the same for the linker’s file. That’s a separate process,\nso `toyhook` saves the `go list` answer in a small state directory the first\ntime and reads it back here. The linker needs those lines too, because it needs\nevery package that ends up in the binary, including all of `log/slog`’s own\ndependencies. The compiler’s file\ngrows from 7 to 79 lines, the linker’s from 60 to 80, and our program logs\nthrough `slog`:\n\n```\nhello, toolchain\n2026/10/01 14:17:52 INFO enter func=countLines\ngo.mod has 3 lines\ndone in 2ms\n```\nOne thing to be careful about: that `go list` call runs in the middle of our\nbuild. If it inherits our `-toolexec`, through `GOFLAGS` for example, it goes\nthrough our wrapper too, and a wrapper that runs `go list` again from there\ncalls itself forever, which is a fun way to heat up your laptop. 🔥 `toyhook` clears `GOFLAGS` before calling it.\n\nNow that we can bring in any package we like, there’s one kind of code we still can’t reach: the code we didn’t write. Let’s go after it (politely, of course).\n\n## Calling code you are not allowed to import\n\nUntil now we’ve only touched our own package. Real instrumentation has to reach\ncode we didn’t write. Let’s make every call to `os.ReadFile`, in the\nstandard library, report to a function in our own module:\n\n```\npackage hooks\nfunc OnReadFile(name string) {\n\tfmt.Fprintf(os.Stderr, \"hooks.OnReadFile(%q)\\n\", name)\n}\n```\nHere’s the problem: package `os` can’t import `hooks`. `hooks` imports `fmt`,\n`fmt` imports `os`, and Go doesn’t allow import cycles. No amount of\n`importcfg` patching gets us around that.\n\nThe way out is a bit of sorcery called `//go:linkname`. It’s a\n[compiler directive](https://github.com/golang/go/blob/go1.27.1/src/cmd/compile/doc.go#L268-L300)\nthat tells the compiler “this name refers to a symbol defined somewhere else”,\nand leaves it to the linker to connect the two. That’s our way in. With\n`TOYHOOK_MODE=linkname`, `toyhook` adds one generated file to the compile of\n`os`:\n\n```\npackage os\nimport _ \"unsafe\"\n//go:linkname toyhookOnReadFile github.com/kakkoyun/hooking-into-the-go-toolchain/hooks.OnReadFile\nfunc toyhookOnReadFile(name string)\n```\nThis declares a function with no body, and the `//go:linkname` comment says its\nbody lives in our `hooks` package. Then `toyhook` inserts a call to it at the\ntop of `ReadFile`, the same way we did with `countLines`:\n\n```\n func ReadFile(name string) ([]byte, error) {\n+\ttoyhookOnReadFile(name)\n+//line $GOROOT/src/os/file.go:872\n \tf, err := Open(name)\n```\nAnd our app, which reads its own `go.mod`, now reports every read:\n\n```\nhello, toolchain\nhooks.OnReadFile(\"go.mod\")\ngo.mod has 3 lines\ndone in 0s\n```\nThe standard library just called into our module without importing it. Don’t\ntell anyone. 🤫 A\ncouple of things had to go right for that. First, `os` had to be compiled again,\nsince standard library packages come from the cache like everything else, so\nwe build with `-a`. Second, `hooks` had to end up in the binary at all.\nNothing imports it, so the linker has no reason to include it, and without help\nthe build fails:\n\n```\nos.ReadFile: relocation target github.com/kakkoyun/hooking-into-the-go-toolchain/hooks.OnReadFile not defined\n```\nThe fix is a blank import, `import _ \".../hooks\"`, in a file of our `main`\npackage. This is why instrumentation tools generate a file for your `main`\npackage.\n\nIf you’ve heard that Go 1.23\n[locked down `//go:linkname`](https://go.dev/doc/go1.23#linker)\n, don’t worry:\nthat rule stops code from reaching *into* standard-library internals that\naren’t marked for it.\nWe’re going the other way, from the standard library out to a package we own,\nand the linker leaves that alone.\n\nUp to now, every trick has worked the first time we tried it. That’s about to change, because there’s one part of the build we’ve been ignoring all along: the cache. It has been quietly judging us the whole time.\n\n## The cache will lie to you\n\nLet’s go back to the `/usr/bin/time` build from the beginning of the post, and\nrun a plain build afterwards, in the same cache and without `-toolexec`:\n\n```\n$ go build -o /tmp/app ./app\n# internal/godebugs\n 0.01 real 0.00 user 0.00 sys\n# internal/coverage/rtcov\n 0.01 real 0.00 user 0.00 sys\n...\n```\nIt prints all 108 timing receipts again, even though nothing was timed this time.\n\nWhat happened? The `go` command saves a tool’s output together with its cache\nentry, and replays it whenever it reuses that entry. Cherry Mui\n[reported exactly this case](https://github.com/golang/go/issues/27628)\nin\n2018, using `-toolexec=/usr/bin/time`, and the issue is still open. Our\nstopwatch only escapes it because it writes to a file.\n\nReplayed timings are just noise. Replayed object code is a real problem.\nRemember that the cache key contains the tool ID, but not the `-toolexec` flag.\nWhat happens, then, if our wrapper changes what the compiler produces, but answers\n`-V=full` exactly like the real compiler?\n\nOur demo has a small package, `greet`, shared by two programs, `app` and\n`other`. Let’s build `app` with `toyhook` rewriting `greet`, and then build\n`other` the normal way, without any wrapper, in the same cache:\n\n```\n$ export GOCACHE=$(mktemp -d) # keep the poisoned entries out of your real cache\n$ TOYHOOK_MODE=rewrite TOYHOOK_TARGET=github.com/kakkoyun/hooking-into-the-go-toolchain/greet \\\n go build -o /tmp/app -toolexec=$PWD/.bin/toyhook ./app\n$ go build -o /tmp/other ./other && /tmp/other\n→ Hello at 14:18:08\nhello, other\n```\n`other` was never built with `-toolexec`, and it’s instrumented anyway.\nSpooky action at a distance, build cache edition. 👻 Both\nbuilds computed the same cache key for `greet`, from the same sources, the same\nflags and the same tool ID, so the plain build happily reused our rewritten\nversion. Build them in the opposite order and it’s just as wrong, the other way\naround: the plain build fills the cache first, `toyhook` is never even called\nfor `greet`, and `app` ends up with no instrumentation at all. There’s the thought we held on\nto earlier: a wrapper only sees what the cache lets through. And the `toyhook`\nwe’ve been using all along has exactly this flaw.\n\nDaniel ran into this while building garble. In 2020 he\n[proposed a way](https://github.com/golang/go/issues/41145)\nfor `-toolexec`\ntools to opt into caching, and Russ\n[replied](https://github.com/golang/go/issues/41145#issuecomment-694612401)\nthat\n“the tool that is altering the\nbehavior of the compiler should be responsible for altering the -V=full output\nas well”. Daniel withdrew the proposal, and that’s exactly what garble does. Its\n[answer to the question](https://github.com/burrowers/garble/blob/v0.18.0/hash.go#L56-L90)\nis the compiler’s own answer with a garble hash appended:\n\n```\nfmt.Printf(\"%s +garble buildID=_/_/_/%s\\n\", line, encodeBuildIDHash(contentID))\n```\n`toyhook` can do the same thing. With `TOYHOOK_MARK=1`, it appends a shorter\nmarker, a hash of its own settings:\n\n```\ncompile version go1.27.1 toyhook@v1/0e25ba9b\n```\nThe answer is different, so the tool ID is different, so every cache key is\ndifferent, and `other` stays clean whichever order we build in. Every tool built\nthis way depends on that one line. Daniel later said that what garble does\nthere “is in\n[undocumented territory](https://github.com/golang/go/issues/41145#issuecomment-2405558244)\n”.\n\nEverything we’ve built so far is a toy, held together with environment variables and good intentions. Now that we know every trick, let’s see what it looks like when someone builds the real thing.\n\n## From toy to tool: otelc\n\nEverything `toyhook` does, badly and for one package at a time, otelc does for\na whole program. It’s OpenTelemetry’s compile-time instrumentation tool for Go,\nbuilt by a special interest group that\n[Alibaba, Datadog and Quesma started together](https://opentelemetry.io/blog/2025/go-compile-time-instrumentation/)\nin January 2025.\n\nLet’s point it at a small HTTP server that has no OpenTelemetry code at all. Its\n`/hello` handler calls `/world` on the same server, so one request makes two\nhops. We build it by putting `otelc` in front of the usual command:\n\n```\notelc go build -o hello .\n```\nThat one command does its work in two phases. First, otelc does a dry run of\nthe build with `go build -a -x -n`, which prints the same narration we read at\nthe start without running anything, so it can see which packages are going to\nbe compiled and match its rules against them. Then it runs the real build with\nitself as the `-toolexec` wrapper, rewriting the packages that matched.\n\nIf we run the server and send it a single request, we get three spans, all in the same trace:\n\n```\nGET /hello server trace f6411a6c… span 26d028cb… parent (root)\nGET client trace f6411a6c… span 6db51783… parent 26d028cb…\nGET /world server trace f6411a6c… span edab8b01… parent 6db51783…\n```\nThe incoming `/hello` request, the outgoing call to `/world`, and `/world`\nitself, each pointing at its parent. Nobody wrote a line of tracing code, and\nnobody had to sit through a meeting about it either. otelc\nkeeps its `$WORK` directory around, so we can open it and see what it did to\n`net/http`:\n\n```\nfunc (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {\n\t//line :1\n\tif hookContext4219161129, _ := OtelBeforeTrampoline_ServeHTTP4219161129(&sh, &rw, &req); false {\n\t} else {\n\t\tdefer OtelAfterTrampoline_ServeHTTP4219161129(hookContext4219161129)\n\t}\n\t//line server.go:3405:2\n\thandler := sh.srv.Handler\n\t...\n```\nThis is the method the standard library’s HTTP server runs for every request,\nand it now calls a “before” function on the way in and defers an “after”\nfunction for the way out. Those functions reach a hook package through the same\n`//go:linkname` trick we used for `os.ReadFile`, and the hooks themselves are\nordinary Go: the before hook starts a server span, and the after hook ends it.\n\nThe call doesn’t go straight to the hook, though. It goes through a small generated function called a trampoline, which builds the hook’s context and catches any panic, so a broken hook can’t take the request down with it. One request through the instrumented method looks like this:\n\nThat odd `if …; false {} else { defer … }` shape is deliberate too. In general\na hook can tell otelc to skip the original function entirely. When a hook\ndoesn’t need that, otelc rewrites the condition to `false` and leaves the rest\nto the compiler’s dead code elimination, one of the SSA passes Jesús walks\nthrough in his [SSA post](https://internals-for-interns.com/posts/the-go-ssa/)\n,\nwhich reduces the whole thing to a plain call and a `defer`.\n\nThe rest of our toy’s tricks are in there too. otelc patches the `importcfg`\nfiles, and the trampolines it generates\n[don’t import anything at all](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/v1.1.0/tool/internal/instrument/trampoline.go#L60-L71)\n,\nfor exactly the reason we ran into earlier. It answers the `-V=full` question\nwith its own marker, so instrumented builds never share cache entries with\nplain ones:\n\n```\ncompile version go1.27.1 otelc@v1.1.0/55ec54fb480c0c69\n```\nThe suffix is a hash of the rules that matched, so changing a rule changes\nevery cache key as well. And remember our `//demo:log` toy? In otelc, that\nwhole wrapper becomes a rule in a YAML file:\n\n```\ndemo_log:\n target: main\n where:\n directive: \"demo:log\"\n do:\n - expand_directive:\n template: |-\n start := time.Now()\n slog.Info(\"function entry\", \"func\", \"{{ .FuncName }}\")\n defer func() {\n slog.Info(\"function exit\", \"func\", \"{{ .FuncName }}\",\n \"duration\", time.Since(start))\n }()\n imports:\n slog: \"log/slog\"\n time: \"time\"\n```\nLook at the `imports` block at the bottom: that’s our `importcfg` problem,\nsolved with three lines of configuration. We build with\n`otelc --rules log.otelc.yml go build`, call the handler, and get:\n\n```\n2026/10/01 14:28:30 INFO function entry func=world\n2026/10/01 14:28:30 INFO function exit func=world duration=232.041µs\n```\nMy favourite rules reach into the runtime itself. One adds two fields to the\nruntime’s goroutine struct, and another copies them every time a new goroutine\nstarts. That way the trace context follows `go` statements even when nobody\npasses a `context.Context` along. Two small fields in a struct we were never\nmeant to touch, all from a wrapper sitting in front of the compiler. Purists,\nlook away. 🙈\n\nThat’s a lot of machinery for a few free spans. All that sorcery must cost something, right? Luckily, we built just the tool to find out.\n\n## Back to the stopwatch\n\nWe started with a stopwatch, so let’s use it one last time. Here’s a full rebuild of our HTTP server, with no wrapper, with our stopwatch, and with otelc:\n\n```\nplain, no wrapper: real 5.97 s\nplain, stopwatch: real 5.76 s\notelc --stats: real 18.16 s\n```\nIt’s one run on one machine, so take the exact numbers with a pinch of salt; the stopwatch run even came out faster than the plain one. Instrumentation that speeds up your build: I’ll take it, but I won’t put it on a slide. Starting an extra process for every tool call is cheap next to running a compiler, so a wrapper costs us almost nothing.\n\notelc takes about three times as long, but it isn’t building the same program.\nThe instrumented server pulls in the OpenTelemetry SDK, so the build compiles\n511 compiler runs instead of 188. And look at that `--stats` flag: it’s a hidden\notelc option that times every tool call from inside otelc’s own `-toolexec`\nwrapper. Our stopwatch, all grown up.\n\nBuilding on undocumented corners like these isn’t comfortable, and the people\nwho build these tools know it. The Orchestrion team at Datadog\n[asked the Go team for better hooks](https://github.com/golang/go/issues/69887)\nin 2024, and the answer so far is that dedicated support for source rewriting\nwould add a lot of complexity to the `go` command. For\nnow, `-toolexec` is the interface, and everything in this post is how we live\nwith it.\n\nEnough watching me do it. Your turn.\n\n## Try it yourself\n\nEvery output above comes from a real run. The\n[companion repository](https://github.com/kakkoyun/hooking-into-the-go-toolchain)\nhas the stopwatch, `toyhook`, the small programs and the HTTP server, with one\n`make` target per experiment:\n\n```\ngit clone https://github.com/kakkoyun/hooking-into-the-go-toolchain\ncd hooking-into-the-go-toolchain\nmake help\nmake step2 # the stopwatch\nmake step6-poison # watch the cache lie\nmake otelc-install # otelc v1.1.0 into ./.bin\nmake step7 # otelc on the HTTP server\n```\nEach target uses its own build cache and output path, so your real build cache stays clean (otelc’s modules still land in your module cache). You’ll need Go 1.25 or newer.\n\nThe quickest experiment, though, is still the one we started with. Point a stopwatch at a project you work on, sort the log by milliseconds, and see which package you’ve been waiting for all along.\n\nAnd then go further. You now know where the compiler gets its files and how to\nhand it different ones. You know how to sneak packages into the `importcfg`,\nhow to make the standard library call your code, and how to keep the build\ncache honest. That’s the whole toolkit behind garble, Orchestrion and otelc.\nRewriting source like this isn’t something the Go team signed up to support,\nbut nobody took the flag away either. Go hack your toolchain. 🛠️ Build\nsomething awesome, something weird, something that makes your colleagues ask\n“wait, how?”. Just remember to change your `-V=full` answer. 🚀\n","body_html":"

Today we’re going to take a look at the Go toolchain, and more specifically at how we can take part in the compilation process with our own code. We won’t patch the compiler. We’ll stand right next to it while it works, watch what it does, and every now and then hand it something it didn’t ask for. Think of it as photobombing the compiler, politely. 📸

\n

The way in is a single flag, and the cheapest experiment I know looks like this:

\n
$ go build -a -o /tmp/app -toolexec=/usr/bin/time ./app\n# internal/unsafeheader\n        0.02 real         0.00 user         0.00 sys\n# internal/goarch\n        0.03 real         0.00 user         0.00 sys\n# internal/coverage/rtcov\n        0.02 real         0.00 user         0.00 sys\n...
\n

Each of those little real user sys receipts is one program the go command\nran on our behalf. For this small program there are 108 of them: 59 compiler\nruns, 48 assembler runs and one linker run. -toolexec tells go build to run\neach of those programs through a program we pick, and here we picked\n/usr/bin/time, which makes it a stopwatch. A chatty stopwatch, but a\nstopwatch. ⏱️

\n

Timing wasn’t the plan, though. Russ Cox\nadded the flag in January 2015\nso the toolchain could run under tools like valgrind, or with a stashed copy of\nthe compiler swapped in by toolstash, a tool he wrote for working on the\ncompiler itself. A few weeks later toolstash learned to time every command it\nran. Years afterwards he described -toolexec=time as\n“an accident - a mostly happy one”\n.\nThis post is about that happy accident, a corner of Go that Daniel Martí once\ncalled\n“a space that hasn’t been explored much so far”\n.\nHe had been exploring it with garble\n, an\nobfuscator, and his work turns up all over this post.

\n

We’ll follow the stopwatch the whole way. First we’ll build our own. Then we’ll\nteach it to rewrite code before the compiler sees it, sneak in packages the\nbuild never asked for, and call code it isn’t allowed to import. Along the way\nthe build cache will lie to us. At the end we’ll look at\notelc\n,\na real tool that does all of this for a living, and time it with the same\nstopwatch.

\n

Thanks to Jesús for inviting me. His\nseries on the Go compiler\nexplains what the compiler does with our code; today is about how we get\nbetween the go command and the compiler in the first place. Full disclosure:\nI work at Datadog and help maintain otelc, so keep that in mind for the last\npart, and feel free to roll your eyes at the appropriate moment. All the code we’ll write lives in a\ncompanion repository\n.

\n

*For the sake of simplicity, everything below was run on macOS with Go 1.27.1. Your\nnumbers will differ, and that’s half the fun :)*

\n

Before we can hook into anything, though, we need to know what we’re hooking\ninto. Let’s see what go build actually does when nobody’s watching.

\n

What go build actually runs

\n

We can get in front of every step of the build. Great! But what are those\nsteps? If we run go build with -x, it narrates every command it runs, and\n-a makes sure it rebuilds everything instead of reusing the cache:

\n
$ go build -a -x -o /tmp/app ./app\n...\n$GOROOT/pkg/tool/darwin_arm64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" \\\n    -p main -lang=go1.25 -complete -buildid kIewNeZLieJMDX0sK6uO/kIewNeZLieJMDX0sK6uO \\\n    -goversion go1.27.1 -c=16 -shared -nolocalimports \\\n    -importcfg $WORK/b001/importcfg -pack ./app/main.go\n...\n$GOROOT/pkg/tool/darwin_arm64/link -o $WORK/b001/exe/a.out \\\n    -importcfg $WORK/b001/importcfg.link -buildmode=pie ... $WORK/b001/_pkg_.a
\n

That’s almost 800 lines for our little program, so I’ve kept just two of them.\nYou’re welcome.\nThe first compiles our main package. The compiler gets the package path\n(-p main), the list of .go files at the end, and a file passed with\n-importcfg. The second line, the last step of the build, links everything\ninto a binary.

\n

In other words, go build is running a lot of compile commands and linking the results\ntogether at the end. That makes sense, but how does it know what needs to be\ncompiled, and what goes into the link? The build is actually a graph of\nactions, one or more per package, and each action gets its own numbered\ndirectory under $WORK, a temporary directory the go command creates for the\nbuild. b001 is our main package. Before running the compiler, the go\ncommand writes that importcfg file into the action’s directory. For our\npackage it looks like this:

\n
# import config\npackagefile bytes=$WORK/b002/_pkg_.a\npackagefile fmt=$WORK/b042/_pkg_.a\npackagefile github.com/kakkoyun/hooking-into-the-go-toolchain/greet=$WORK/b060/_pkg_.a\npackagefile os=$WORK/b049/_pkg_.a\npackagefile time=$WORK/b054/_pkg_.a\npackagefile runtime=$WORK/b009/_pkg_.a
\n

One line for each package main.go imports, plus runtime, each pointing at\nthe compiled archive of that package. That’s the compiler’s whole view of the\noutside world. It doesn’t search a GOPATH or read go.mod; if a package isn’t\nin this file, it doesn’t exist. (If you’re curious what’s inside those\narchives, Jesús’s post on the\nunified IR format\nopens one up.) The linker gets a similar file listing every package in the\nprogram. Keep the importcfg in mind, because it’s going to bite us later. (Yes, that’s\nforeshadowing. 👀)

\n

Most of the other lines in that log are the go command doing things itself,\nlike writing those files, creating directories or copying archives around. The\nlines that matter to us start a tool from $GOROOT/pkg/tool/ (compile, asm\nand link), and those are exactly the lines -toolexec steps into. This is how\ngo help build describes the flag:

\n
-toolexec 'cmd args'\n    a program to use to invoke toolchain programs like vet and asm.\n    For example, instead of running asm, the go command will run\n    'cmd args /path/to/asm <arguments for asm>'.\n    The TOOLEXEC_IMPORTPATH environment variable will be set,\n    matching 'go list -f {{.ImportPath}}' for the package being built.
\n

And that’s the entire interface. Our program receives the tool’s path as its\nfirst argument and the tool’s arguments after it, and it can do whatever it\nlikes before, after or instead of running the tool. The TOOLEXEC_IMPORTPATH\nvariable tells it which package is being built. That one is\nDaniel’s work too\n,\nadded in Go 1.16; before that, wrappers had to guess the package from the flags.

\n

Now that we know exactly where -toolexec steps in, let’s put something of\nour own there, starting with a better stopwatch, one that can at least tell\npackages apart.

\n

Building our own stopwatch

\n

/usr/bin/time gives us numbers, but it doesn’t tell us which package each\nnumber belongs to. Let’s write a wrapper that does. The heart of it is small:

\n
func main() {\n    tool, args := os.Args[1], os.Args[2:]\n    cmd := exec.Command(tool, args...)\n    cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr\n    // ... forward SIGINT, SIGTERM, SIGHUP and SIGQUIT to the tool ...\n    start := time.Now()\n    if err := cmd.Start(); err != nil {\n        fmt.Fprintf(os.Stderr, "stopwatch: %v\\n", err)\n        os.Exit(1)\n    }\n    err := cmd.Wait()\n    logLine(tool, args, time.Since(start)) // appends to $STOPWATCH_LOG\n    os.Exit(exitCode(err))\n}
\n

It runs the real tool with the same standard input and output, measures how\nlong it took, and appends one line to a log file: the tool, the import path, the\nmilliseconds. Then it exits with the tool’s own status, so go build never\nnotices we’re there.

\n

Let’s build it and point go build at it:

\n
go build -o .bin/stopwatch ./cmd/stopwatch\nSTOPWATCH_LOG=/tmp/sw.tsv go build -a -o /tmp/app -toolexec=$PWD/.bin/stopwatch ./app
\n

Here are a few lines from the log, one per tool call:

\n
compile | runtime | 860 | files=182\ncompile | fmt | 80 | files=5\ncompile | github.com/kakkoyun/hooking-into-the-go-toolchain/app | 16 | files=1\nlink | github.com/kakkoyun/hooking-into-the-go-toolchain/app | 59 | files=0
\n

Each line is one tool call: which tool ran, which package it was working on, how many milliseconds it took, and how many source files it got. The whole log has 111 lines. Counting them by tool, and sorting the compiles by time, gives us this:

\n
tool           runs    -V=full\nasm              48          1\ncompile          59          1\nlink              1          1\n    ms  package\n   860  runtime\n   379  reflect\n   261  internal/abi\n   230  syscall\n   195  math
\n

No surprise that runtime is the slowest package to compile; it has the\nhardest job in the building. (Don’t add the\nmilliseconds up and call it the build time, though: the tools ran in parallel,\nso their sum is larger than the time we actually waited.)

\n

The interesting part is the last column of the first table. Before building\nanything, the go command ran each tool once with a single argument,\n-V=full, and it did that through our wrapper. In our log it looks like this:

\n
compile | - | 8 | -V=full
\n

That’s the go command asking each tool “who are you?”. A tiny identity\ncrisis, once per tool, every single build. The compiler answers\ncompile version go1.27.1, and that answer becomes the tool’s ID. The ID ends\nup in the cache key of every package the tool compiles. That key, the action\nID, is a hash of the package’s source files, its flags, the tool ID and what\nits dependencies produced. Notice what isn’t in it: the -toolexec flag itself.

\n

Why ask the wrapper instead of just reading the compiler binary? The\ngo command’s source\nexplains: “we want ‘-toolexec toolstash’ to continue working”. If a wrapper\nswaps in a different compiler, the cache key should know.

\n

Here’s the whole picture, from the -V=full question to the cache:

\n

This has a funny consequence for our wrapper: its standard output isn’t really\nours anymore. During that -V=full question, stdout is the answer. Since Go\n1.10 the go command\nignores whatever a tool prints to stderr\nwhile answering, as long as stdout has the right line, but stdout gets no such\npass. If our stopwatch says hello on stdout before running the tool, the build\nstops right there. Rude, but fair:

\n
go: parsing buildID from go tool compile -V=full: unexpected output:\n    stopwatch: starting compile\ncompile version go1.27.1
\n

If it prints after the tool instead, things get sneakier. The build works, but our log line contains a millisecond count, so the answer, and with it the tool ID, changes on every build. A second build recompiles all 59 packages again, and nothing tells us why.

\n

Our well-behaved wrapper writes only to its log file. Let’s run the build\nagain, without -a this time, and look at the log:

\n
compile | - | 6 | -V=full\nasm | - | 4 | -V=full\nlink | - | 5 | -V=full
\n

Three questions and nothing else. Every package came straight from the cache,\nso no tool ran and our stopwatch had nothing to time. Best build ever,\nterrible demo. 🤷 A -toolexec wrapper only\nsees what the cache lets through. Hold on to that thought; it’ll come back.

\n

Watching is fun, but our wrapper is sitting in a much more interesting spot than that. Let’s see what happens when it stops being a polite spectator.

\n

Rewriting code before the compiler sees it

\n

Now that we’re sitting between the go command and the compiler, we can do\nmore than watch. The compiler gets its source files as arguments, and we see\nthose arguments first. What if we handed it different files?

\n

Changing Go source from a program sounds like something you’d only do on a\ndare, but Go makes it surprisingly friendly. The compiler has its own parser, the one Jesús takes apart in his\nparser post\n, but the\nstandard library ships a second set of packages just for tools:\ngo/token\nkeeps track of positions,\ngo/parser\nturns source into a syntax tree,\ngo/ast\ndescribes every node in that tree, and\ngo/printer\nand\ngo/format\nturn a tree back into code. These are\nthe packages\ngofmt is built on\n,\nand go vet’s checks run on them through the\nanalysis framework\nthat\nmost Go linters use. If you’ve ever written a linter, you’ve already done the\nfirst half of what we need: find the code you care about. As Jesús puts it at\nthe end of his post, many Go developers use go/ast\n“to parse Go code programmatically and build powerful tools”\n.

\n

That’s exactly what our second toy wrapper, toyhook, does. It looks for functions\nmarked with a //demo:log comment, like this one in our app:

\n
//demo:log\nfunc countLines(path string) int {\n    data, err := os.ReadFile(path)\n    ...
\n

Finding them takes the same three steps every linter starts with: parse the\nfile, walk the tree, and check each node. Trimmed down a little, the heart of\ntoyhook looks like this:

\n
fset := token.NewFileSet()\nfile, err := parser.ParseFile(fset, abs, src, parser.ParseComments)\nif err != nil {\n    return nil, 0, err\n}\nfor _, decl := range file.Decls {\n    fn, ok := decl.(*ast.FuncDecl)\n    if !ok || fn.Body == nil || !hasDirective(fn) {\n        continue\n    }\n    lbrace := fset.Position(fn.Body.Lbrace)\n    // ... insert our statement right after lbrace.Offset ...\n}
\n

parser.ParseFile reads the file into an *ast.File, and ParseComments\nasks it to keep the comments, which we need because our marker is one. Then we\nloop over the file’s top-level declarations, keep the functions, and\nhasDirective checks each function’s doc comment for //demo:log. The\ntoken.FileSet is what turns a node back into a file, line and byte offset, so\nlbrace tells us exactly where the function’s opening brace is.

\n

Now we need to add our log statement. The textbook way is to build it as more\ntree: every call, identifier and literal becomes a struct, and we splice them\ninto the function body. It works, but it’s wordy, in the way tax forms are\nwordy. Here’s just start := time.Now() as AST nodes, from the\ninjector I wrote for a talk\n:

\n
startDecl := &ast.AssignStmt{\n    Lhs: []ast.Expr{ast.NewIdent("start")},\n    Tok: token.DEFINE,\n    Rhs: []ast.Expr{\n        &ast.CallExpr{\n            Fun: &ast.SelectorExpr{\n                X:   ast.NewIdent("time"),\n                Sel: ast.NewIdent("Now"),\n            },\n        },\n    },\n}
\n

That injector ended up at 306 lines for two log statements. Two. Log.\nStatements. 😩 Printing the tree\nback out has a catch too: go/ast comments “are stored by their byte offset\ninstead of attached to nodes, so re-arranging nodes breaks the output”. That’s\nstraight from the README of dst\n, a third-party Go\npackage (the name stands for Decorated Syntax Tree) built to fix exactly this\nproblem. It keeps comments attached to the nodes they belong to, which is why\nserious tools like otelc rewrite with it.

\n

Our toy takes a shortcut. It uses the tree only to find where each marked\nfunction’s body starts, and inserts the new statement as plain text right after\nthat brace, so every byte we didn’t touch stays where it was. When the compile\nfor our package comes through, toyhook writes the result into the action’s own\n$WORK directory, which it finds from the compiler’s -o flag, swaps the new\nfile into the argument list, and runs the real compiler. Let’s build it and use\nit the same way as the stopwatch:

\n
go build -o .bin/toyhook ./cmd/toyhook\nTOYHOOK_MODE=rewrite go build -o /tmp/app -toolexec=$PWD/.bin/toyhook ./app
\n

Here’s the difference between what we wrote and what the compiler actually gets:

\n
--- app/main.go\n+++ $WORK/b001/main.go\n@@ -21,6 +21,8 @@\n //\n //demo:log\n func countLines(path string) int {\n+    fmt.Fprintf(os.Stderr, "→ %s at %s\\n", "countLines", time.Now().Format(time.TimeOnly))\n+//line $REPO/app/main.go:24\n     data, err := os.ReadFile(path)\n     if err != nil {\n         fmt.Fprintln(os.Stderr, err)
\n

And the program now narrates itself:

\n
hello, toolchain\n→ countLines at 14:17:40\ngo.mod has 3 lines\ndone in 0s
\n

The second line we inserted, the //line comment, is easy to miss but\nimportant. It’s a\ncompiler directive\nthat resets the file name and line number, so compiler errors, panics and\nstack traces after our insertion still point at app/main.go:24, and not at a\ntemporary file that’s long gone by the time anyone reads the error. Without it,\nevery line below our insertion would be off by one, which is a great way to\nmake people distrust instrumentation.

\n

The wrapper is called for every package in the build, but it only rewrites one; for the other 58 compiles it gets out of the way and runs the compiler untouched. Each call is a fresh process that sees a single package, and it can only change the files of that package. That limitation is exactly what the next two experiments run into.

\n

Packages the build never asked for

\n

Our app already imported fmt, os and time, so the code we inserted only\nused packages the compiler knew about. Let’s get more ambitious and log with\nlog/slog, which our app never imports. Adding one import sounds simple\nenough, right? Well, with TOYHOOK_MODE=slog, toyhook adds the import and the\ncall, and the compiler says:

\n
$WORK/b001/main.go:3:8: could not import log/slog (open : no such file or directory)
\n

Remember the importcfg? The go command wrote it from the imports in the\noriginal file, before it ever called us. The compiler looks up log/slog in\nthat file, finds nothing, and tries to open an empty path. Told you it would\nbite.

\n

Every tool that adds imports hits this wall. Julio Guerra\nasked about it in 2019\n, and Ian\nLance Taylor’s answer\nwas that “the -toolexec option is not powerful enough to\nsupport arbitrary source code rewriting.” Julio\ndid it anyway\n,\nand so will we.

\n

The trick is to write the missing lines ourselves. The go command will happily\ntell us where the compiled archive of any package lives:

\n
go list -deps -export -f '{{if .Export}}packagefile {{.ImportPath}}={{.Export}}{{end}}' log/slog
\n

With TOYHOOK_IMPORTCFG=patch, toyhook adds the lines the compiler’s\nimportcfg is missing. Then, when the link command comes through at the end of\nthe build, it does the same for the linker’s file. That’s a separate process,\nso toyhook saves the go list answer in a small state directory the first\ntime and reads it back here. The linker needs those lines too, because it needs\nevery package that ends up in the binary, including all of log/slog’s own\ndependencies. The compiler’s file\ngrows from 7 to 79 lines, the linker’s from 60 to 80, and our program logs\nthrough slog:

\n
hello, toolchain\n2026/10/01 14:17:52 INFO enter func=countLines\ngo.mod has 3 lines\ndone in 2ms
\n

One thing to be careful about: that go list call runs in the middle of our\nbuild. If it inherits our -toolexec, through GOFLAGS for example, it goes\nthrough our wrapper too, and a wrapper that runs go list again from there\ncalls itself forever, which is a fun way to heat up your laptop. 🔥 toyhook clears GOFLAGS before calling it.

\n

Now that we can bring in any package we like, there’s one kind of code we still can’t reach: the code we didn’t write. Let’s go after it (politely, of course).

\n

Calling code you are not allowed to import

\n

Until now we’ve only touched our own package. Real instrumentation has to reach\ncode we didn’t write. Let’s make every call to os.ReadFile, in the\nstandard library, report to a function in our own module:

\n
package hooks\nfunc OnReadFile(name string) {\n    fmt.Fprintf(os.Stderr, "hooks.OnReadFile(%q)\\n", name)\n}
\n

Here’s the problem: package os can’t import hooks. hooks imports fmt,\nfmt imports os, and Go doesn’t allow import cycles. No amount of\nimportcfg patching gets us around that.

\n

The way out is a bit of sorcery called //go:linkname. It’s a\ncompiler directive\nthat tells the compiler “this name refers to a symbol defined somewhere else”,\nand leaves it to the linker to connect the two. That’s our way in. With\nTOYHOOK_MODE=linkname, toyhook adds one generated file to the compile of\nos:

\n
package os\nimport _ "unsafe"\n//go:linkname toyhookOnReadFile github.com/kakkoyun/hooking-into-the-go-toolchain/hooks.OnReadFile\nfunc toyhookOnReadFile(name string)
\n

This declares a function with no body, and the //go:linkname comment says its\nbody lives in our hooks package. Then toyhook inserts a call to it at the\ntop of ReadFile, the same way we did with countLines:

\n
 func ReadFile(name string) ([]byte, error) {\n+    toyhookOnReadFile(name)\n+//line $GOROOT/src/os/file.go:872\n     f, err := Open(name)
\n

And our app, which reads its own go.mod, now reports every read:

\n
hello, toolchain\nhooks.OnReadFile("go.mod")\ngo.mod has 3 lines\ndone in 0s
\n

The standard library just called into our module without importing it. Don’t\ntell anyone. 🤫 A\ncouple of things had to go right for that. First, os had to be compiled again,\nsince standard library packages come from the cache like everything else, so\nwe build with -a. Second, hooks had to end up in the binary at all.\nNothing imports it, so the linker has no reason to include it, and without help\nthe build fails:

\n
os.ReadFile: relocation target github.com/kakkoyun/hooking-into-the-go-toolchain/hooks.OnReadFile not defined
\n

The fix is a blank import, import _ ".../hooks", in a file of our main\npackage. This is why instrumentation tools generate a file for your main\npackage.

\n

If you’ve heard that Go 1.23\nlocked down //go:linkname\n, don’t worry:\nthat rule stops code from reaching into standard-library internals that\naren’t marked for it.\nWe’re going the other way, from the standard library out to a package we own,\nand the linker leaves that alone.

\n

Up to now, every trick has worked the first time we tried it. That’s about to change, because there’s one part of the build we’ve been ignoring all along: the cache. It has been quietly judging us the whole time.

\n

The cache will lie to you

\n

Let’s go back to the /usr/bin/time build from the beginning of the post, and\nrun a plain build afterwards, in the same cache and without -toolexec:

\n
$ go build -o /tmp/app ./app\n# internal/godebugs\n        0.01 real         0.00 user         0.00 sys\n# internal/coverage/rtcov\n        0.01 real         0.00 user         0.00 sys\n...
\n

It prints all 108 timing receipts again, even though nothing was timed this time.

\n

What happened? The go command saves a tool’s output together with its cache\nentry, and replays it whenever it reuses that entry. Cherry Mui\nreported exactly this case\nin\n2018, using -toolexec=/usr/bin/time, and the issue is still open. Our\nstopwatch only escapes it because it writes to a file.

\n

Replayed timings are just noise. Replayed object code is a real problem.\nRemember that the cache key contains the tool ID, but not the -toolexec flag.\nWhat happens, then, if our wrapper changes what the compiler produces, but answers\n-V=full exactly like the real compiler?

\n

Our demo has a small package, greet, shared by two programs, app and\nother. Let’s build app with toyhook rewriting greet, and then build\nother the normal way, without any wrapper, in the same cache:

\n
$ export GOCACHE=$(mktemp -d)   # keep the poisoned entries out of your real cache\n$ TOYHOOK_MODE=rewrite TOYHOOK_TARGET=github.com/kakkoyun/hooking-into-the-go-toolchain/greet \\\n    go build -o /tmp/app -toolexec=$PWD/.bin/toyhook ./app\n$ go build -o /tmp/other ./other && /tmp/other\n→ Hello at 14:18:08\nhello, other
\n

other was never built with -toolexec, and it’s instrumented anyway.\nSpooky action at a distance, build cache edition. 👻 Both\nbuilds computed the same cache key for greet, from the same sources, the same\nflags and the same tool ID, so the plain build happily reused our rewritten\nversion. Build them in the opposite order and it’s just as wrong, the other way\naround: the plain build fills the cache first, toyhook is never even called\nfor greet, and app ends up with no instrumentation at all. There’s the thought we held on\nto earlier: a wrapper only sees what the cache lets through. And the toyhook\nwe’ve been using all along has exactly this flaw.

\n

Daniel ran into this while building garble. In 2020 he\nproposed a way\nfor -toolexec\ntools to opt into caching, and Russ\nreplied\nthat\n“the tool that is altering the\nbehavior of the compiler should be responsible for altering the -V=full output\nas well”. Daniel withdrew the proposal, and that’s exactly what garble does. Its\nanswer to the question\nis the compiler’s own answer with a garble hash appended:

\n
fmt.Printf("%s +garble buildID=_/_/_/%s\\n", line, encodeBuildIDHash(contentID))
\n

toyhook can do the same thing. With TOYHOOK_MARK=1, it appends a shorter\nmarker, a hash of its own settings:

\n
compile version go1.27.1 toyhook@v1/0e25ba9b
\n

The answer is different, so the tool ID is different, so every cache key is\ndifferent, and other stays clean whichever order we build in. Every tool built\nthis way depends on that one line. Daniel later said that what garble does\nthere “is in\nundocumented territory\n”.

\n

Everything we’ve built so far is a toy, held together with environment variables and good intentions. Now that we know every trick, let’s see what it looks like when someone builds the real thing.

\n

From toy to tool: otelc

\n

Everything toyhook does, badly and for one package at a time, otelc does for\na whole program. It’s OpenTelemetry’s compile-time instrumentation tool for Go,\nbuilt by a special interest group that\nAlibaba, Datadog and Quesma started together\nin January 2025.

\n

Let’s point it at a small HTTP server that has no OpenTelemetry code at all. Its\n/hello handler calls /world on the same server, so one request makes two\nhops. We build it by putting otelc in front of the usual command:

\n
otelc go build -o hello .
\n

That one command does its work in two phases. First, otelc does a dry run of\nthe build with go build -a -x -n, which prints the same narration we read at\nthe start without running anything, so it can see which packages are going to\nbe compiled and match its rules against them. Then it runs the real build with\nitself as the -toolexec wrapper, rewriting the packages that matched.

\n

If we run the server and send it a single request, we get three spans, all in the same trace:

\n
GET /hello   server  trace f6411a6c…  span 26d028cb…  parent (root)\nGET          client  trace f6411a6c…  span 6db51783…  parent 26d028cb…\nGET /world   server  trace f6411a6c…  span edab8b01…  parent 6db51783…
\n

The incoming /hello request, the outgoing call to /world, and /world\nitself, each pointing at its parent. Nobody wrote a line of tracing code, and\nnobody had to sit through a meeting about it either. otelc\nkeeps its $WORK directory around, so we can open it and see what it did to\nnet/http:

\n
func (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {\n    //line <generated>:1\n    if hookContext4219161129, _ := OtelBeforeTrampoline_ServeHTTP4219161129(&sh, &rw, &req); false {\n    } else {\n        defer OtelAfterTrampoline_ServeHTTP4219161129(hookContext4219161129)\n    }\n    //line server.go:3405:2\n    handler := sh.srv.Handler\n    ...
\n

This is the method the standard library’s HTTP server runs for every request,\nand it now calls a “before” function on the way in and defers an “after”\nfunction for the way out. Those functions reach a hook package through the same\n//go:linkname trick we used for os.ReadFile, and the hooks themselves are\nordinary Go: the before hook starts a server span, and the after hook ends it.

\n

The call doesn’t go straight to the hook, though. It goes through a small generated function called a trampoline, which builds the hook’s context and catches any panic, so a broken hook can’t take the request down with it. One request through the instrumented method looks like this:

\n

That odd if …; false {} else { defer … } shape is deliberate too. In general\na hook can tell otelc to skip the original function entirely. When a hook\ndoesn’t need that, otelc rewrites the condition to false and leaves the rest\nto the compiler’s dead code elimination, one of the SSA passes Jesús walks\nthrough in his SSA post\n,\nwhich reduces the whole thing to a plain call and a defer.

\n

The rest of our toy’s tricks are in there too. otelc patches the importcfg\nfiles, and the trampolines it generates\ndon’t import anything at all\n,\nfor exactly the reason we ran into earlier. It answers the -V=full question\nwith its own marker, so instrumented builds never share cache entries with\nplain ones:

\n
compile version go1.27.1 otelc@v1.1.0/55ec54fb480c0c69
\n

The suffix is a hash of the rules that matched, so changing a rule changes\nevery cache key as well. And remember our //demo:log toy? In otelc, that\nwhole wrapper becomes a rule in a YAML file:

\n
demo_log:\n  target: main\n  where:\n    directive: "demo:log"\n  do:\n    - expand_directive:\n        template: |-\n          start := time.Now()\n          slog.Info("function entry", "func", "{{ .FuncName }}")\n          defer func() {\n            slog.Info("function exit", "func", "{{ .FuncName }}",\n              "duration", time.Since(start))\n          }()\n  imports:\n    slog: "log/slog"\n    time: "time"
\n

Look at the imports block at the bottom: that’s our importcfg problem,\nsolved with three lines of configuration. We build with\notelc --rules log.otelc.yml go build, call the handler, and get:

\n
2026/10/01 14:28:30 INFO function entry func=world\n2026/10/01 14:28:30 INFO function exit func=world duration=232.041µs
\n

My favourite rules reach into the runtime itself. One adds two fields to the\nruntime’s goroutine struct, and another copies them every time a new goroutine\nstarts. That way the trace context follows go statements even when nobody\npasses a context.Context along. Two small fields in a struct we were never\nmeant to touch, all from a wrapper sitting in front of the compiler. Purists,\nlook away. 🙈

\n

That’s a lot of machinery for a few free spans. All that sorcery must cost something, right? Luckily, we built just the tool to find out.

\n

Back to the stopwatch

\n

We started with a stopwatch, so let’s use it one last time. Here’s a full rebuild of our HTTP server, with no wrapper, with our stopwatch, and with otelc:

\n
plain, no wrapper:    real 5.97 s\nplain, stopwatch:     real 5.76 s\notelc --stats:        real 18.16 s
\n

It’s one run on one machine, so take the exact numbers with a pinch of salt; the stopwatch run even came out faster than the plain one. Instrumentation that speeds up your build: I’ll take it, but I won’t put it on a slide. Starting an extra process for every tool call is cheap next to running a compiler, so a wrapper costs us almost nothing.

\n

otelc takes about three times as long, but it isn’t building the same program.\nThe instrumented server pulls in the OpenTelemetry SDK, so the build compiles\n511 compiler runs instead of 188. And look at that --stats flag: it’s a hidden\notelc option that times every tool call from inside otelc’s own -toolexec\nwrapper. Our stopwatch, all grown up.

\n

Building on undocumented corners like these isn’t comfortable, and the people\nwho build these tools know it. The Orchestrion team at Datadog\nasked the Go team for better hooks\nin 2024, and the answer so far is that dedicated support for source rewriting\nwould add a lot of complexity to the go command. For\nnow, -toolexec is the interface, and everything in this post is how we live\nwith it.

\n

Enough watching me do it. Your turn.

\n

Try it yourself

\n

Every output above comes from a real run. The\ncompanion repository\nhas the stopwatch, toyhook, the small programs and the HTTP server, with one\nmake target per experiment:

\n
git clone https://github.com/kakkoyun/hooking-into-the-go-toolchain\ncd hooking-into-the-go-toolchain\nmake help\nmake step2         # the stopwatch\nmake step6-poison  # watch the cache lie\nmake otelc-install # otelc v1.1.0 into ./.bin\nmake step7         # otelc on the HTTP server
\n

Each target uses its own build cache and output path, so your real build cache stays clean (otelc’s modules still land in your module cache). You’ll need Go 1.25 or newer.

\n

The quickest experiment, though, is still the one we started with. Point a stopwatch at a project you work on, sort the log by milliseconds, and see which package you’ve been waiting for all along.

\n

And then go further. You now know where the compiler gets its files and how to\nhand it different ones. You know how to sneak packages into the importcfg,\nhow to make the standard library call your code, and how to keep the build\ncache honest. That’s the whole toolkit behind garble, Orchestrion and otelc.\nRewriting source like this isn’t something the Go team signed up to support,\nbut nobody took the flag away either. Go hack your toolchain. 🛠️ Build\nsomething awesome, something weird, something that makes your colleagues ask\n“wait, how?”. Just remember to change your -V=full answer. 🚀

","headings":[{"level":2,"text":"What go build actually runs","id":"what-go-build-actually-runs"},{"level":2,"text":"Building our own stopwatch","id":"building-our-own-stopwatch"},{"level":2,"text":"Rewriting code before the compiler sees it","id":"rewriting-code-before-the-compiler-sees-it"},{"level":2,"text":"Packages the build never asked for","id":"packages-the-build-never-asked-for"},{"level":2,"text":"Calling code you are not allowed to import","id":"calling-code-you-are-not-allowed-to-import"},{"level":2,"text":"The cache will lie to you","id":"the-cache-will-lie-to-you"},{"level":2,"text":"From toy to tool: otelc","id":"from-toy-to-tool-otelc"},{"level":2,"text":"Back to the stopwatch","id":"back-to-the-stopwatch"},{"level":2,"text":"Try it yourself","id":"try-it-yourself"}]}}