There's a Fedora box on my desk named stick. An app on it has been firing\nStatsD datagrams at 127.0.0.1:8125 ten times a second for about a week\nnow, the way apps have done since Etsy taught everyone to in 2011.
Nothing is listening on 8125.
\n$ ss -lun | grep 8125\n$Grafana has the request rate anyway.
\nThis is the first post in a series where we go down the list of exporters\neveryone runs next to Prometheus and replace them, one at a time, with a\nyeet script. The pitch for the series is short: you do\nnot need fifty exporters in six languages, each with its own port, its own\nconfig format, and its own opinion about vendoring. You need one binary\nthat can read the kernel. We're starting with statsd_exporter because it's\nthe one where the replacement stops being the same thing and turns into\nsomething kind of better.
statsd_exporter is a translator. Your app pushes StatsD lines at it over\nUDP, it holds them as Prometheus metrics, Prometheus pulls. It's maintained\nby the Prometheus project itself, so this is an official piece of the stack\nwe're pulling out.
We replaced the listener with an eBPF program that watches port 8125 at the\ntraffic control layer of every interface, and a JavaScript decoder that\nturns the lines into Prometheus families. The app keeps doing fire-and-forget\nUDP. Nothing changes on the app. Nothing gets installed on the host except\nyeet. And the metrics show up whether or not anyone is there to receive\nthe datagrams.
You'll learn:
\nyeet:telemetry turns a\nscript into a Prometheus target without the script ever touching a socket.\nThe whole thing is open source. To run it on your own boxes:
Less than you'd think. It binds a UDP socket, reads datagrams, splits them on newlines, and parses lines that look like this:
\napp.requests:1|c\napp.queue.depth:22|g\napp.request.time:143|ms\napp.users:20|s\napp.errors:1|c|@0.1|#service:api,region:usThen it keeps a counter, gauge, histogram or set per name, runs the names\nthrough a YAML mapping file so your dots turn into labels, and serves the\nwhole thing on :9102/metrics. That's it. That's the exporter. It's a\nperfectly good exporter and I've run it for years without thinking about it\nonce, which is about the nicest thing you can say about infrastructure.
But look at what it eats. UDP datagrams. On a well-known port. In a\nplaintext line format. Usually on the same host as the app. By the time\nthat socket gets the bytes, the kernel has already had them, looked at\nthem, and decided where they go. The listener is the second reader of\nevery packet.
\nSo we asked the first one.
\nThe kernel half is a tcx program clipped onto the ingress and egress hooks of every interface that's up, loopback included. It does three things: find the transport header, check if either port is one we care about, copy a bounded chunk of the payload into a ring buffer. Done.
\n/* The ports worth capturing, filled by the script. */\nstruct {\n __uint(type, BPF_MAP_TYPE_HASH);\n __uint(max_entries, 64);\n __type(key, __u16);\n __type(value, __u8);\n} ports SEC(".maps");\nstatic __always_inline int wanted(__u16 sport, __u16 dport)\n{\n return bpf_map_lookup_elem(&ports, &sport) != NULL\n || bpf_map_lookup_elem(&ports, &dport) != NULL;\n}\nstatic __always_inline int handle(struct __sk_buff *skb, __u8 dir)\n{\n /* ... walk Ethernet or raw IP, then IPv4 or IPv6, to the L4 header ... */\n __u16 sport = 0, dport = 0;\n bpf_skb_load_bytes(skb, l4, &sport, 2);\n bpf_skb_load_bytes(skb, l4 + 2, &dport, 2);\n sport = bpf_ntohs(sport);\n dport = bpf_ntohs(dport);\n if (!wanted(sport, dport))\n return TCX_NEXT;\n /* ... TCP data offset or the 8-byte UDP header, then the payload ... */\n struct wire_event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);\n if (!e)\n return TCX_NEXT;\n e->ts = bpf_ktime_get_ns();\n e->sport = sport;\n e->dport = dport;\n e->ifindex = skb->ifindex;\n e->dir = dir;\n e->total_len = plen;\n e->captured = cap;\n if (bpf_skb_load_bytes(skb, poff, e->data, cap) < 0) {\n bpf_ringbuf_discard(e, 0);\n return TCX_NEXT;\n }\n bpf_ringbuf_submit(e, 0);\n return TCX_NEXT;\n}That ifindex is there for a reason. A packet on loopback goes through\nthe egress hook on lo, and then, because the kernel just hands it right\nback to itself, through the ingress hook on lo. Same bytes, two events.\nTCP you can dedupe on the sequence number. UDP has no sequence number. So\non loopback the collector keeps the egress copy and drops the other one,\nand every datagram counts exactly once:
if (ev.dir === DIR_INGRESS && ev.ifindex === lo) return;Look at the ports map too. There's no port number anywhere in the C.\nThe script fills the map at startup from its arguments, and 8125 is contentionally\nwhat statsd used, but there's no reason why it couldn't be anything else.\nBecause the kernel side has no idea what StatsD is, the same mechanism covers anything else that speaks\na line protocol on a known port. It matches a port and copies up to 512\nbytes. Parsing is a JavaScript problem, and JavaScript is cheap. The\ncollector on stick already runs the Redis, memcached and Graphite\ndecoders off this one probe, each with its own flag, and those get their\nown posts. The next protocol costs a port number and a decoder.
Everything else stays in the kernel. ACKs, TLS, your Zoom call, none of it\never crosses into userspace. And the program returns TCX_NEXT on every\npath, so it never drops or rewrites a packet. It reads. That's the whole\njob.
yeet loads it from a script with\nyeet:bpf. The attach target is\nwhatever the system graph says is up:
import { BpfObject, HashMap, RingBuf } from "yeet:bpf";\nconst { data } = await yeet.graph.query(`{ network_interfaces { index is_up } }`);\nconst tcx = { kind: "tcx", ifindex: data.network_interfaces.filter((i) => i.is_up).map((i) => i.index) };\nconst control = await new BpfObject({ exe: "../bpf/bin/wire.bpf.o", base: import.meta.dirname })\n .bind("events", { kind: "ringbuf", btf_struct: "wire_event" })\n .bind("ports", { kind: "hash" })\n .attach("on_ingress", tcx)\n .attach("on_egress", tcx)\n .start();\nconst ports = new HashMap(control, "ports");\nfor (const port of [8125, 8126]) await ports.update(port, 1); // from yeet.args, really\nawait new RingBuf(control, "events").subscribe(onEvent, () => dropped.inc());btf_struct: "wire_event" is the entire deserializer. The daemon reads the\nstruct layout out of the object's BTF and hands onEvent a plain object\nwith sport, dport, data and friends already on it. Same trick in the\nother direction for the map: update(8125, 1) is a __u16 key and a\n__u8 value because the C said so. We did not write a byte parser. We're\nnever going to write a byte parser.
Here's the StatsD half with the plumbing cut out:
\nrequest(ev, bytes) {\n this.datagrams.inc();\n for (const line of latin1(bytes).split("\\n")) {\n const m = /^([^:]+):([^|]+)\\|([a-z]+)(?:\\|@([0-9.]+))?(?:\\|#(.*))?$/.exec(line.trim());\n if (!m) { this.lines.labels({ status: "invalid" }).inc(); continue; }\n const [, rawName, rawValue, type, rate, tags] = m;\n const name = rawName.replace(/\\./g, "_");\n const labels = parseTags(tags); // #service:api,region:us\n const sample = 1 / (Number(rate) || 1); // @0.1 counts for ten\n this.#apply(name, type, rawValue, Number(rawValue), labels, sample);\n }\n}#apply is a switch on the type letter. c bumps a counter by\nvalue * sample. g sets a gauge, or nudges it if the value came with a\nsign. ms, h and d observe a histogram, with milliseconds scaled to\nseconds. s throws the value in a set and publishes the size. That's the\nStatsD spec. All of it. It fits on a napkin.
Families get created the first time a name shows up, which is the part\nstatsd_exporter makes you do in YAML ahead of time:
#family(name, type, labels) {\n const key = `${type}:${name}`;\n let f = this.#series.get(key);\n if (!f) {\n const opts = { help: `StatsD ${type} ${name}, read off the wire.`, labels: labels.names };\n if (type === "histogram") opts.buckets = LATENCY_BUCKETS;\n f = this.telemetry[type](name, opts);\n this.#series.set(key, f);\n }\n return f;\n}this.telemetry is a\nTelemetry registry. The\nfirst app.errors line that shows up with #service:api,region:us on it\nlocks in the label set for app_errors_total. A later line with different\ntags gets counted under statsd_exporter_lines_total{status="rejected"}\ninstead of taking the collector down. Same rule the real exporter has, we\njust didn't make you write it in a config file.
Correct. The isolate a yeet script runs in cannot bind a port. It also\ncan't fetch, which we went on about last time\nand still think is the best thing about it. The only listeners anywhere in\nthis picture belong to the daemon, and the daemon speaks HTTP and\nWebSocket and that's the list.
So how does Prometheus scrape the thing?
\nTwo pieces. The collector runs as a SharedWorker and calls\ntelemetry.serve(), which answers snapshot requests over a message port.\nThe scrape endpoint is a separate script, and it's tiny:
import { render } from "yeet:telemetry";\nimport { emit } from "../../lib/emit.js";\nimport { pick, upAs } from "../../lib/snapshot.js";\n// Everything the wire collector holds that is not another protocol's.\nconst OTHERS = ["redis_", "memcached_", "graphite_"];\nemit(await render({ worker: "../../lib/collectors/wire.js" }, ({ worker: w }) => ({\n ...upAs(w, "statsd_exporter_up", "1 when the wire collector answered."),\n ...Object.fromEntries(Object.entries(w).filter(([k, f]) =>\n k !== "yeet_worker_up" && !OTHERS.some((p) => k.startsWith(p)) && !/^Graphite/.test(f?.help ?? ""))),\n})));render({ worker }) connects to the shared worker, asks for its registry,\nand hands the families to a function that reshapes them. This one throws\nout the series the other decoders on the same probe produced and keeps\nthe rest under statsd_exporter's name. What comes out gets printed to the\nconsole as Prometheus text format, and the encoder validates it on the way\nout so you can't ship a malformed document by accident.
Then yeet service does the bit where a console becomes an HTTP response:
yeet service unit add exporter-swap/gateway -W "http://0.0.0.0:9100"\nyeet service unit add exporter-swap/statsd_exporter -I exporters/statsd_exporter/metrics.js --lazy\nyeet service mount exporter-swap/gateway -L /statsd_exporter/metrics -t statsd_exporter -p console -T per-connectionA plain GET on /statsd_exporter/metrics spawns a fresh isolate, runs the\nscript above, streams its console into the response body, and exits. A\nscrape is a process that lives for a few milliseconds and never has\nnetwork access at any point in its life. Prometheus cannot tell the\ndifference:
scrape_configs:\n - job_name: statsd\n metrics_path: /statsd_exporter/metrics\n static_configs:\n - targets: ["stick:9100"]$ curl -s stick:9100/statsd_exporter/metrics\n# HELP statsd_exporter_up 1 when the wire collector answered.\n# TYPE statsd_exporter_up gauge\nstatsd_exporter_up 1\n# HELP statsd_exporter_udp_packets_total StatsD datagrams seen on the wire.\n# TYPE statsd_exporter_udp_packets_total counter\nstatsd_exporter_udp_packets_total 215702\n# HELP app_requests_total StatsD counter app_requests, read off the wire.\n# TYPE app_requests_total counter\napp_requests_total 129798\n# HELP app_errors_total StatsD counter app_errors, read off the wire.\n# TYPE app_errors_total counter\napp_errors_total{service="api",region="us"} 1296520\n# HELP app_queue_depth StatsD gauge app_queue_depth, read off the wire.\n# TYPE app_queue_depth gauge\napp_queue_depth 22\n# HELP app_request_time StatsD histogram app_request_time, read off the wire.\n# TYPE app_request_time histogram\napp_request_time_bucket{le="0.0005"} 0\napp_request_time_bucket{le="0.001"} 0\napp_request_time_bucket{le="0.002"} 462\n...\napp_request_time_sum 19534.094000000434\napp_request_time_count 129174app_errors_total is ten times app_requests_total because the app sends\nit with @0.1. The tags turned into labels. The timer turned into a\nhistogram in seconds. Nobody wrote a mapping file.
tl;dr: the datagram was always the metric. We just stopped needing a process to catch it.
\nNothing here is magic, so here's the receipt.
\nYou see what this host sends. A real listener sees what arrives. If\nyour app on host A ships StatsD to a collector on host B, this runs on A\nand counts at the source, and Prometheus's instance label tells you\nwhich A. I'd argue that's better. It is definitely different.
Names follow the default mapping. Dots become underscores and that's\nit. If you lean on statsd_exporter's mapping language to turn\napp.api.us.requests into app_requests{service="api", region="us"},\nyou'd write that in the decoder instead. Not a lengthy addition but\nwas outside the scope of this post. Feel free to submit a PR if you\nwant to add it.
Counters get _total. The registry appends the suffix the way every\nPrometheus client library does. statsd_exporter leaves it off unless you\nask. Fix your dashboards or fix the decoder, whichever annoys you less.
512 bytes per datagram. StatsD datagrams are\nsupposed to fit in an MTU and usually clients send batches way smaller, but\nDATA_MAX is a #define if yours don't.
It's plaintext. But so is StatsD. When this series\ngets to the HTTP exporters that stops being true and we'll talk about the\nTLS uprobe then.
\nThe listener was never the point. It was the thing that got the datagram out of the kernel and into a process that knew how to count. Now the counting happens right next to the kernel: a program the verifier has signed off on copies the line into a ring buffer, a decoder that fits on one screen does the math, and Prometheus pulls the answer out of a sandbox that has no sockets and gets scraped anyway.
\nNobody is listening on 8125. The numbers are fine.
\n*Next up: the nginx exporter you never installed, and why it knows more\nthan the one you did.*
","headings":[]}}