The standard advice for prompt quality is "set up evals." For most solo devs and small teams it is premature, and it skips the check that catches the most failures for the least effort. Prompt quality has three layers, and you almost certainly need the cheapest one working before you build the expensive ones. Here is how the three compare, a three-question test for which layer you actually need right now, and why we score every prompt before we build a single eval.
Three layers of prompt QA
They are not competitors. They catch different failures at different costs, and they stack.
Layer 1, structural scoring. Before you run the prompt, in seconds, you score its structure: is the task clear, is the context there, is the output format specified, does it use a role where the task needs one. It catches the failures that live in the prompt text itself, and it needs no data and no execution. This is the 6-criteria rubric and the Prompt Score model behind it. In practice it is the layer that stops you from shipping "summarize this" with no length, audience, or format, the kind of prompt no eval should ever have to see.
Layer 2, eval sets. You assemble a dataset of representative inputs with known-good outputs, pick a metric, and run the prompt against all of them. It catches behavioral regressions: did tightening the wording break case 14. It costs hours to build a dataset and, harder, a metric that actually correlates with what you care about. In practice it is what tells you that rewording the system prompt to fix one edge case quietly regressed three others, the kind of change you cannot eyeball across a whole dataset. This is the layer tools like Promptfoo, Braintrust, Langfuse, and LangSmith are built for.
Layer 3, A/B in production. You put two prompt versions on live traffic and measure the real outcome: resolution rate, conversion, thumbs-up. It catches what offline evals cannot, because real users do not match your test set. In practice it is the only layer that catches a prompt which scores well and passes every offline case yet still underperforms in the wild, because your test set was cleaner than reality. It costs real traffic volume and the patience to reach a readable signal.
Your prompts can improve. Promptimizer rewrites and auto-tests them for you.
The three layers of prompt QA: structural score, eval set, A/B in production, ordered by cost and what each catches
The three layers stack from cheapest to most expensive. Structural scoring runs before execution and needs nothing; eval sets need a labeled dataset; A/B needs live volume. Each catches failures the others miss.
Why "just set up evals" is premature for most
Evals are the right tool when you are shipping an LLM feature at scale. They need three things a small team often does not have yet: a labeled dataset that represents real inputs, a metric that correlates with the outcome you actually want, and enough iterations for the setup to pay back.
The eval platforms are genuinely good, and they are built for teams running LLM features in production with the volume to justify them. For a library of a few dozen prompts that you tweak by hand, the setup cost dwarfs the payoff. You spend a day wiring up a harness and labeling cases to catch regressions in prompts that are not yet stable enough to regress. We reach for evals when a prompt has earned them, not as the default first move.
The hardest part of an eval is not the dataset, it is the metric. "Did the model produce good output" is not something a script can check, so you reduce it to something it can, and a badly chosen metric gives you a green dashboard while quality slips underneath it. Exact-match punishes a correct answer for being phrased differently; an LLM-as-judge adds a second prompt that can be wrong in its own ways. Getting the metric to track what you actually care about takes iterations most small teams have not run yet. That is another reason the structural layer earns its place first: there, "good" is already defined by the rubric, so there is no metric to get wrong.
The three-question test
Three questions, in order. Stop at the first "no."
1. Will this prompt run reused, unattended, or in someone else's hands? If it is a one-off you run and read yourself, you are the eval: a structural score is enough, because you catch bad answers as they happen. If it will run without you watching, keep going.
2. Can you cheaply get twenty or so representative cases with a known-correct answer? If you cannot define what "correct" looks like on real inputs, you cannot build a meaningful eval yet. Score the prompt and add manual spot-checks. If you can, keep going.
The techniques you're reading about work. Test your prompts now with Prompt Score and see your score in real time.
3. Do you have enough live volume to read an A/B signal? If two versions would take months to separate from noise, an A/B test is theater. Build the offline eval set instead. If you do have the volume, run the eval set and the A/B together.
A three-question decision tree routing to structural score, score plus spot-checks, eval set, or eval plus A/B
The test routes you to the lightest layer that actually fits. Most solo-dev prompts stop at question one. Evals earn their place only past question two.
Where scoring wins
Structural scoring is the pre-execution filter, and most prompts that fail, fail structurally. They are vague, they are missing context, they never specified a format. Scoring catches those before you spend a token, let alone build a dataset. An eval set confirms behavior on the cases you thought to define; a structural score tells you the prompt was underspecified in the first place, which is a different and earlier problem.
Picture the common case: a prompt returns inconsistent output, so you reach for an eval to measure the inconsistency. Score it first and the diagnosis is usually immediate. No format was ever specified, so the model improvises a different shape every run. That is a one-line fix the rubric surfaces in seconds, not a dataset you needed to build. Working that fix criterion by criterion is its own loop, the one that takes a prompt from a 2 to a 4. Scoring turns a measurement problem back into a writing problem, and a writing problem is far cheaper to solve.
That is why scoring comes first even when you will eventually need evals. It is cheap enough to run on every prompt, every save, and it removes the whole class of failures that no eval should have to catch. By the time a prompt reaches your eval set, it should already score well structurally, or the eval is measuring the wrong thing.
A comparison of structural scoring versus eval sets across cost, when it runs, what it catches, and what it misses
Structural scoring and eval sets answer different questions. One asks whether the prompt is well built; the other asks whether it behaves on your cases. You want the cheap one first, and the expensive one only when it earns its keep.
When to graduate to evals
There are honest triggers. Build the eval set when you are shipping the prompt to users, when the same prompt runs thousands of times a day, or when correctness is measurable and the cost of getting it wrong is real: extraction, classification, routing, anything where a wrong answer propagates silently. At that point the labeled dataset pays for itself, and the eval platforms in the prompt management tools comparison are worth the setup.
Take an extraction prompt pulling fields from invoices into JSON. It runs unattended, thousands of times, and a wrong field lands in a database no one is watching. That prompt has clearly earned an eval set: representative invoices, known-correct JSON, an exact-field metric. But it should still pass a clean structural score first, because an extraction prompt that never specified its output format will fail the eval for a reason the rubric would have caught for free.
Even then, scoring stays the first gate. A prompt that scores well structurally and passes its eval set is in good shape. A prompt that fails its eval set but also scores badly structurally did not need the eval to tell you it was broken.
Score first
The order is the whole point. Score first, because it is free and catches the most common failures before execution. Build evals when a prompt has earned them with reuse, measurable correctness, and volume. Run A/B only when live traffic can actually tell two versions apart. Skipping straight to evals feels rigorous, but it is an expensive way to discover that a prompt was simply vague, and vague is exactly what scoring catches for nothing.
If you want the cheap layer in place today, Keep My Prompts scores every prompt you save on the six criteria, so the structural failures are caught before any of the expensive layers have to. Start there, and add evals the day a prompt makes you.
