Prompt Engineering Without Guesswork: Evals, Naive Prompts, and Reliable JSON

Ask an engineer how they know their code works and they will show you a test suite. Ask the same engineer how they know their prompt works and they will show you the one output they liked on Tuesday.

That gap is the whole story of unreliable AI features. Prompt engineering has real techniques, but without measurement you cannot tell whether a change helped, hurt, or did nothing. This post covers both halves: how to write better prompts, and how to know they are better. The worked examples and numbers come from Anthropic's Claude course, which walks this exact loop, and the approach matches how we evaluate prompts in production at Navero.

The Three Paths After Writing a Prompt

Every prompt author stands at the same fork:

1. Test it once, decide it is good enough. It will break in production on inputs you never imagined.
2. Test a few times, patch a corner case or two. Better, and still a trap: users are an inexhaustible source of inputs you did not consider.
3. Run it through an eval pipeline and iterate on a score. More work upfront, and the only path where "the prompt got better" is a fact instead of a feeling.

Paths 1 and 2 are where nearly everyone lives. Path 3 is the difference between guessing and knowing.

What Naive Prompts Look Like

Here is a real baseline from the course, a meal planner for athletes:

What should this person eat?

- Height: {height}
- Weight: {weight}
- Goal: {goal}
- Dietary restrictions: {restrictions}

It is polite, it is readable, and it scored 2.32 out of 10 against explicit criteria (accurate calories, macro breakdown, meal timing, portions in grams). The failures are predictable once you name them:

• It asks a question instead of giving an instruction. The model gets to decide what kind of answer this is. It will decide differently every run.
• It never states the task. "What should this person eat" could yield a paragraph of advice, a shopping list, or a lecture on nutrition.
• It specifies nothing about the output. No format, no required fields, no constraints. Whatever you get, you cannot call it wrong.

Two revisions, each measured. First, a clear and direct opening line: "Generate a one-day meal plan for an athlete that meets their dietary restrictions." Instruction, action verb, task named. Score: 2.32 to 3.92, from restructuring one line. Second, explicit quality guidelines (include daily calories, show macros, specify meal timing, portions in grams, respect restrictions). Score: 3.92 to 7.86, doubling quality.

Neither change is clever. Both are just specific. Being specific is not micromanaging the model; it is setting up the conditions where success is even defined.

The Eval Loop: How to Be Scientific About It

The pipeline that produced those numbers is small enough to build in an afternoon:

The dataset is a list of realistic inputs. Write a handful by hand, then have a cheap, fast model generate more; it is test data, not production output. Tens of cases is a fine start, hundreds is better.

Code graders are plain functions that return a score. Does the output parse as JSON? json.loads it: 10 if it parses, 0 if not. Valid Python? ast.parse. Valid regex? re.compile. Cheap, deterministic, and merciless.

Model graders are a second LLM call that judges what code cannot: did the answer actually address the task? The one technique that matters here: ask for strengths, weaknesses, and reasoning alongside the score. A grader forced to justify its number stops handing out a lazy 6 for everything.

Then change one thing. This is the discipline the whole loop exists to enforce. Change five things at once and a score improvement teaches you nothing. Change one, re-run, compare, keep or revert. Also: read the low-scoring cases and look for patterns across them. One bad output is noise; the same failure on eight cases is your next prompt change, found for you.

The absolute score does not matter much. A 2.32 baseline is not embarrassing, it is a starting line. What matters is that the number moves when you improve the prompt and drops when you break it, which is exactly what your gut cannot do.

The Best Practices, Distilled

Everything above generalizes into a short list, roughly in order of payoff:

1. Lead with a clear, direct instruction. The first line is the most important line. Action verbs, not questions: Write, Generate, Identify.
2. Add quality guidelines. A numbered list of attributes the output must have: lengths, fields, constraints, formats.
3. Give process steps for hard tasks. Force the model to brainstorm options, pick one, then produce, instead of jumping to the final answer.
4. Fence interpolated content with XML tags. <athlete_information>, <docs>, <my_code>. The model stops confusing your instructions with your data, and you can debug the prompt by reading it.
5. Show examples, especially of edge cases. A sarcastic tweet labeled Negative teaches sentiment analysis more than a paragraph about sarcasm ever will. Best source of examples: your own eval results. Take the highest scoring outputs and paste them in as ideal outputs, with a sentence on why they are ideal.

The eval loop also settles a question teams usually argue about instead: which model tier to run. You already have a scoring pipeline — run the same prompt on the tier below, compare scores, and downgrade whenever the score holds. Two rules of thumb fall out of it: match the tier to the blast radius of a bad output, not to how impressive the task sounds (eval-dataset generation belongs on the cheap fast tier, because a grader checks everything it touches), and let graders afford a stronger model than producers — grading runs once per eval, production runs forever, and a strong grader keeps a weak producer honest while the reverse is how bad outputs earn high scores.

Getting JSON You Can Actually Parse

Now the sharpest practical problem: you need structured output, and the model keeps wrapping it in friendliness.

The naive prompt says "Return the result as JSON." The model happily responds:

Sure! Here is the JSON you asked for:

{ "name": "web-api", "replicas": 3 }

Let me know if you would like me to explain any field!

json.loads throws, and the usual escalation begins: ONLY RETURN JSON in capital letters, threats in the system prompt, then a regex that fishes for the first { and hopes. Everyone has written that regex. It is a confession that the prompt lost.

The reliable fix uses two output controls together, and the interesting part is why it works:

• Prefilling: the API lets you write the beginning of the assistant's reply yourself. The model does not know you wrote it; it believes it already started answering that way and continues from exactly there.
• Stop sequences: the model halts the instant it generates any string you list, before it gets a chance to add commentary.

So you prefill the assistant message with an opening JSON code fence, and set the closing fence as the stop sequence:

messages = []
add_user_message(messages, "Generate a minimal EventBridge rule as JSON")
add_assistant_message(messages, "```json")      # prefill: the reply already began
 
text = chat(messages, stop_sequences=["```"])   # halt at the closing fence
data = json.loads(text)                         # parses, every time

The psychology is the point. You are not asking the model to skip the preamble; you have made the preamble impossible, because from the model's perspective the preamble window already passed. It thinks it wrote the opening fence itself, and the only coherent continuation is the JSON body. The closing fence then cuts generation before any "Let me know if" can exist. The same trick powers eval pipelines end to end: dataset generation and model graders both use prefill plus stop to return machine readable results.

Modern APIs also offer native structured output and tool calling with schemas, and you should use them where available. But prefilling is worth understanding anyway, because it teaches the deeper lesson: you get reliability by constraining what the model can emit, not by asking more politely in the instructions.

Closing the Loop

The meal planner went from 2.32 to 7.86 in two measured steps. Not because someone found magic words, but because every change was tested against the same dataset and the same graders, and only improvements survived.

That is the entire method. Write the naive prompt, accept the embarrassing baseline, build the small loop, and let the score tell you what your intuition cannot. I have applied the same discipline to higher stakes pipelines, like sizing an ensemble for LLM based CV scoring, and it holds at every scale: prompts are code, and code you do not test is code you do not trust.

Related Reading

• Production RAG: Three Retrieval Strategies and When Each One Wins — the same eval discipline, applied to deciding how much retrieval machinery you actually need