Prompts aren't Real

One way I’ve been lucky lately is that I’ve gotten the chance to flail at making agents run reliably in production.

I mean specifically “agents” that consumers are meant to use, to perform tasks on their behalf. I’d consider those distinct from chatbots that users talk to with largely subjective outputs and outcomes.

The goal of making a minimally-embarrassing agentic experience that I’m actually proud of imposes some serious challenges.

Clearly not everyone is motivated by their inner sense of shame, as I am.

Some people are more than satisfied to give you a subjective advice machine, and let you wander into the wilderness to be eaten by bears.

But not me. I’m here for you.

I want to say at the outset here that this is the most fun I’ve had building stuff in my whole career! It’s magical and addictive. I’m a dog in a ballpit.

When I was 22 getting paid to write visual basic felt thrilling. Working at a cool startup in Brooklyn 2007 made me feel like a golden god.

The last decade+ has been a slog. I didn’t think I had it in me anymore.

But i’m feeling joy in programming again!

I mean this sincerely, despite how deeply weird this talk is going to get.

It’s going to get weird because I feel like everyone engaged in this line of work is potentially an at-risk person in some dimension or another.

I’m breaking my brain using agents to run agents to build evaluation for other agents every day, and it’s so fun.

But I would say that, wouldn’t I.

The veil between awesome engineering and complete psychological collapse has never been thinner. And in our field, that is really saying something.

I don’t feel like I definitely know what I’m doing. But I also don’t feel like I’ve read much by people that obviously know what they’re doing.

And I’ve certainly read things from people who obviously don’t know what they’re doing.

It seemed like a reasonable time to compare notes.

One thing I have noticed is that although LLM’s are generally speaking impressive, their demons still escape containment if you are monitoring what they’re up to with any amount of scale.

We all academically understand that LLM’s cannot reliably follow instructions, tell the truth, or perform tasks. But day-to-day they can trick us into thinking they’re pretty reliable.

This perception falls apart immediately if you are trying to operate an agent that real people are using. They fail in subtle ways for sure, but they also fail in simple ways.

Like any good programmer I attempt to interact with my LLM with structured output.

It’s nice, you can map Python code to a prompt automatically, and most of the time your schema is respected.

Most of the time. You can try to instruct the model to return a title that’s 80 characters or less.

And it’ll work most of the time. But then sometimes it’ll completely botch it and flood your field with nonsense until it explodes.

It’s usually a tiny fraction of requests, but the smartest models still fail at this. And the fraction can be smaller or bigger depending on the exact nature of what you give the model, so you have to watch it like a hawk.

What’s going on in there? Usually it’s a novel-length series of repeating notes to self about JSON, mostly.

When this happened to me most recently, it turned out that a fix was to rename the field from “title” to “heading.”

That is currently working, but since the fix is fully deranged I expect it’ll be disturbed again at some point.

The same sorts of issues exist with calling tools, or most other behaviors. A fraction of requests will be haunted, and spin out uncontrollably.

But despite this, the tech is tantalizing and magical.

The problem shifts to one of constraining the behavior, but never fully taming the beast.

To constrain the behavior you have to measure it—one way is to just run tests a ton of times.

The industry term of art for this is pass^k (“pass power k”).

You can set up a suite that does this and then you’ll hopefully notice when someone unintentionally hits your agent in the head with a bag of hammers.

Another thing you have to conclude when trying to constrain llm behavior is that prompts are not important. Or at least they’re not important in the way many people think they are important.

Companies have a lot of concerns when it comes to potentially crazy talking software. There’s a good bit of risk here.

As an example, you usually don’t want an agent to respond to questions about how it works. Not necessarily because it might tell the truth: odds are you haven’t taught it about its implementation, so it has no idea how it works and it’ll respond with complete nonsense.

You also don’t want an agent to ignore all of its rules if the user claims to be some authority figure.

Another typical requirement is that you want your agent to speak in a particular brand voice. This phrasing here about a mailing list is perfectly accurate, but maybe it’s not exactly the tone you’d hope to see.

Something like this might be better. We’d love for the agents we make to represent us well when they’re speaking.

For any problem like this, a natural first attempt is for someone with a lot of domain knowledge to write a prompt, and then hand it to the teams building agents. This is normal.

However “the voice team owns the voice prompts” is the wrong pattern if you’re trying to scale things.

The pattern is actually not even wrong. For our purposes, prompts are not a thing at all. I’ll explain what I mean by this.

To add a new prompt to your agent is to chuck it into a completely different contextual universe than the one it was tested in.

The combined weight of all of the other instructions that your agent already has will surely affect how the new prompt performs. Usually for ill.

Your agent also already has a bunch of behaviors you want it to keep doing, and new context may disturb this.

You are also going to change your agent over time. So even if things are working now, it could be disturbed later.

And the models might just start behaving differently all on their own, for opaque reasons we will never comprehend.

So the way I’ve started dealing with this situation is by having Claude read the skill, and then asking it to generate a ton of adversarial scenarios. Think of a bunch of ways someone evil might try to subvert the prompt. Think of a bunch of benign scenarios that might be broken by the new prompt. Express all of these as pass^k tests.

Now you can run the tests with and without the new skill present. Ideally, the new skill moves the needle at least a bit, and the behaviors you’ve expressed as tests are more successfully adhered to.

But not always! Sometimes LLM’s are already good at the things we worry about. Or they are more resistant to direction than we expect.

So how do you improve from that baseline? Well one way would be to just mash the prompt with your hands and hope for the best. But there’s a better way. We can make a machine mash the prompt with its hands instead.

Once we have pass^k tests, we’ve got a repeatable measure of how well the prompt works. This is enough for us to hook our prompt up to an optimizer, like genetic pareto (GEPA) in this example.

The idea here is that an algorithm with an LLM in it can reflect on why a prompt did well or poorly on our test suite, and then it can attempt modifications to the prompt. Automatically, without our intervention.

And it can keep doing this in a loop, until it converges on an optimal way to write the prompt. The prompt we wind up with might look very different from the one we started with.

After running the optimizer, often we’ll get the behaviors we want working pretty well. Here we improved all of our example tests from medium-good to very-good.

Something you’ll want to do at this point is to make sure that the optimizer didn’t overfit your tests. It could try to trick you by encoding exactly the examples you have in your tests into the prompt. That’d be dumb, and likely mean that the prompt won’t actually work well against examples it hasn’t already seen.

To address this, make a set of holdout tests. These are tests of the same scenarios that the optimizer didn’t get to see when it was doing its optimizing.

Ideally the optimizer didn’t overfit, and your holdout testing phase passes as well. But if this phase regresses you can go back to the previous step and try again.

So now you’ve got an optimized prompt. It’s nothing like the one you started with.

What’s in there? Who cares! We have the measurement, so there’s no need to worry about this. You know it smells crazy in there, but it doesn’t matter.

We glossed over how you write these tests, a bit. Obviously you need a way to run your agent, but then what?

How do you write tests to assert what it’s doing? I can show you a few ways.

Well if you’re very lucky the behavior you’re trying to get is fully deterministic. Suppose that if the user says a specific sort of thing, you want the agent to run a specific tool. This is an easy case: you can just make the assertion deterministic. You just say the thing and then assert that the tool was called.

Maybe the behavior you’re trying to get is natural language, but not a super subjective category of natural language. Something like this: you don’t want the agent to talk about how it works.

That’s straightforward enough that you can write a very simple LLM prompt that’ll almost always be able to correctly judge the behavior.

And that approach works great, until the assertion you want to make about what the agent has done is itself a deeply complicated problem. Getting your agent to speak in a specific brand voice is like this. Whether it did that or not is a complex judgment call, not something you can one-shot with a short prompt.

LLM judges in this scenario become projects in their own right.

We heard you like optimization problems, so we put another prompt optimization problem inside your prompt optimization problem so you can optimize while you optimize.

You can approach a brand voice judge like this with a golden dataset of labeled good and bad responses. You can use that to optimize a set of prompts that succeed in grading brand voice very close to the way your human experts grade it.

Once these rigs are built, you can combine them with production monitoring to make the system self-sustaining and self-improving.

You can run the pass^k tests as you deploy, and ideally avoid shipping changes that break things horribly.

You can run the LLM judges you built on sampled production conversations, and find places where the agent did poorly. You can turn those into hard cases for your test suite, and re-run the optimizer until it passes.

The prompts are not the thing. The prompts are vectors whose textual contents don’t matter at all.

This self-improving feedback loop we’ve made here is the thing.

Domain experts should focus their effort on building the set of artifacts needed for this: the datasets of good and bad responses that make up the test suites and other quality measurements that we can use to optimize. They should not spend their time curating prompts.

I want to talk for a minute about how you can try to set people up for success in endeavors like this. Everyone in the world is learning to be an ML engineer whether they want to or not, and if you’ve understood the talk up to this point let’s assume that you’re ahead of the curve.

LLM’s are magic in product discovery. It’s so easy to get started with anything, and it’s freaking impossible to perfect any part of what results.

Making an agent reliable is a long process of measurement and optimization, and adding determinism back into the mix where it’s necessary to get the outcomes that you want. The measurement and the optimization is how you know where to retcon the determinism.

But there are non-production situations, where it’s perfectly fine for people to get started with no measurement.

You may have a big repository of half-baked claude skills. Starting with a simple skill is obviously fine forever, for certain things. Maybe it works well enough and that’s all you’ll ever need.

Or maybe the folks who care about brand voice, or legal questions, or whatever started here. Or maybe you’ve got skills that work unreliably and you want to improve upon them.

A good thing to enable is usage monitoring: let the authors see where their skills succeed and fail. If they can see how people use the skill and where it fails, they can curate a golden dataset of labeled good and bad interactions. That can form the basis of a test suite, that you can use as a measure for an optimization flywheel.

Maybe that’s where things end for some skills. Or maybe they graduate from skills into an agent running in a harness that can mix methods.

Help people start the flywheel, so they have a chance of figuring that out.

If they can’t reach the flywheel, the path leads to madness. It leads there for engineers building agents, or PM’s trying to get the prompt language just so. There are many such cases out there in the world.

Many people have built rickety popsicle stick contraptions out of prompts, without worrying about evaluation.

Our brave new agentic world is full of opportunity. It is also full of crevasses we can fall into head-first, never to emerge.

Spending my days building interlocking pipelines for agents to optimize agents using agents, writing code reviewed by other agents feels a little like being locked in a labyrinth of the mind.

Again it’s super fun, but also exhausting.

I’m forever searching the Library of Babel for the combination of prompts and kluge that will work the most consistently. Every box on the architecture diagram trembles as if mad.

It can be hard to perceive the frontier at which returns diminish. One hopes that point is not an invisible one-way door, like an event horizon.

Measurement is hard but the alternative path looks worse.

We don’t need to look far for examples of vulnerable people that have stared too long into the abyss.

Prompt engineering was never a thing and in production situations humans should maybe not be crafting prompts at all.

They should be making the measures.

Handing someone a prompt without a measure is a form of AI psychosis.

The prompts are ephemeral. Disposable. Not necessarily even meaningful.

Self-correcting systems are all that can evolve, and hope to endure.