Prompts deserve migrations, not vibes

You would never open a production database and run ALTER TABLE by hand, from memory, with no migration and no way back. We learned that one the hard way, a long time ago. So we built migrations: every schema change is a versioned, reviewable, reversible file with a timestamp and an author.

Then we turn around and edit the prompt that drives an LLM feature directly in a dashboard. Live. In production. No diff, no history, no test, no idea whether the last tweak helped or hurt.

We version the code around the model with real discipline. The prompt — the thing that actually decides what the model does — we treat like a sticky note.

The model is almost never what breaks

Here's the pattern I keep seeing across the last two years of LLM builds. The demo works. Production breaks. You trace the incident back, and the model was fine. What moved was the context around it, and most of the time, that means the prompt.

It rarely fails loudly. It fails as drift. A change here to fix one edge case. A line there to please a stakeholder. A "quick" instruction added the night before a launch. None of them reviewed against the others. Six weeks later the output is worse and nobody can point to the commit that did it, because there was no commit.

A prompt you can't diff is a prompt you can't debug.

The incident write-ups keep landing in the same place. It isn't model capability. It's the slow pile-up of unreviewed prompt edits and the drift that follows. Your model is the most stable thing in the stack. Your prompt is the part you change every week with the least rigor.

What changed when I treated prompts like code

Not a tool. Not a framework. A decision: a prompt is a software artifact, and it earns the same treatment as everything else in the repo. In practice, that was four things.

1. Versioning — every prompt lives in the repo

Prompts move out of the dashboard and into version control, next to the code that calls them. Every change is a commit with a message that says why. You can diff two versions. You can blame a line. And when a "small improvement" turns out not to be, you roll back to last Tuesday in one command.

prompts/
  extract_fields.v4.md      # was v3 — tightened the date-format rule
  classify_intent.v8.md     # was v7 — rolled back, v7 scored higher
  summarize_doc.v2.md

It sounds too simple to matter. It is the highest-leverage thing I have done for an LLM system's reliability.

2. Progressive retry — don't pay for the whole prompt to fix one field

When the model returns something that fails validation, the lazy fix is to retry the entire prompt and pray. That gets expensive, and at real traffic it compounds.

So retries got smaller. On a failure, the system retries with a targeted correction aimed at the field that actually broke, and only escalates if it has to. You stop re-sending the full context to repair one bad value.

~90% fewer tokens spent on validation retries.

That number isn't a benchmark trick. It's what happens when "retry" stops meaning "do the whole thing again."

3. Field-level sanitization — repair the input, not the universe

A cousin of the above. When one field comes back wrong, you sanitize and re-run that field, not the whole response. Most "the LLM is unreliable" complaints are really "we threw the entire request away because one corner of it was off." Scope the repair to the actual problem and it gets cheaper, faster, and easier to reason about.

4. Evals over vibes

Once prompts are versioned, you can finally answer the question intuition can't: did this change actually help? Every version runs against a set of evaluation cases before it ships. A tweak that fixes one example and quietly breaks five others never reaches production. Iteration stops being "this feels better" and becomes "this scores better on the cases we care about."

Where it broke first

I didn't arrive here from theory. I arrived from getting burned.

The first version of one of my pipelines had none of this. Prompts lived half in code, half in a config anyone could edit. One day the quality dropped on a specific kind of input. Not catastrophically — just enough that it took days to notice, and by then I couldn't tell which of a dozen recent changes was responsible. No history to bisect. No evals to say which variant was better. I read outputs and guessed.

That is the exact feeling migrations were built to kill. So I borrowed the whole playbook: version it, make changes reviewable, keep a way back, test before you ship. If I started over tomorrow, the first thing I'd build isn't a smarter model. It's the boring versioning layer, on day one, before a single prompt goes anywhere near a user.

---

Prompts are software

None of this is exotic. It's the discipline the rest of your stack already takes for granted, finally pointed at the layer we keep pretending is just text. Observable. Version-controlled. Costed. Deletable.

If a prompt is good enough to decide what your product says to a user, it's good enough to deserve a migration, not a vibe.

So: how does your team version its prompts today? And if the honest answer is "we don't," which of the four would you reach for first?

Next: versioning pays off twice. Once your prompts are under version control and eval, you have the scaffolding to harden them — to catch jailbreaks, fictional-framing, and the other ways people talk a model out of its job. That's where this goes next.