Previously, I was used to specifying behavior in terms of code: precise sets of instructions that compile and execute in a predictable way. And mentally, I was treating LLM output like code. But over time, I realized that LLMs play a fundamentally different role than code does in software engineering. LLMs, and their respective harnesses, are actors in a system. Code is just an output.

It took me years to get comfortable with non-deterministic code generation, yet looking back on it, it seems so obvious that programming was always just a means to an end. Let's use languages, always a hot topic of debate amongst programmers, as an example. There are infinite ways to reach the same result as expressed by code, and that's only using a single programming language! There are hundreds of widely used languages out there, and it's possible to write a solution to a particular problem in just about any of them.

For years, developers have been building perfectly functional applications using very different piles of technologies, languages, and frameworks. Choosing one tech stack over another may have made it easier for a human to complete a given task. But clearly, the means of getting to that result could be wildly different across implementations.

Now that we're working with probabilistic actors, it's our job as engineers, or managers (depending on how you look at it), to reduce the probability space to a small, useful area. This is where all of the common LLM tricks come into play: context engineering, MCP, skills, harnesses, graphs, and memories are all tools used to reduce the amount of randomness in model output. But randomness in some form will always be there, which is at odds with the fact that there is still a need for the correct answer, especially when working on mission-critical software.

So let's remove more randomness by turning non-deterministic output into deterministic tools, writing software for agents to use. Agents can call these tools to perform precise, tested, and authorized operations that have the same desired result on every execution. As an added bonus, depending on the tool's complexity, we get the same correct outcome using significantly fewer tokens by calling our new agent-built abstractions.

I've seen this pattern develop both in industry, with tools like https://swamp.club, and in my own work managing production-grade infrastructure. This pattern is the modern incarnation of the compounding automation that drew me into DevOps and software development in the first place. But instead of writing our own tools, we can now use AI to create its own abstractions and instruct it to use them as it builds the foundations of our new agentic software systems.

I find it fascinating that we can harness randomness today in a way that reduces randomness in the future. In the end, it's not much different from writing your own automations on top of earlier abstractions. It's just that the machine is calling them, processing output, and taking action faster than humans ever could.

Older posts

---

←  My Foray into Local LLMs

© 2026 Steven Sklar

:: Theme is based on Terminimal by pawroman