What a prompt lab actually validates

Automated prompt evaluation can prove only what its grader checks. This lab sends the same 20 synthetic review pages through three PromptFoo lessons, each using claude-haiku-4-5-20251001 through Anthropic’s Messages API.

The prompts grow from a two-field CSV row to a nested, season-grouped JSON object. The graders do not grow at the same rate. That difference is the most useful part of the implementation: it separates a prompt’s declared contract from the evidence produced by its tests.

Three additive lessons

Each lesson contains system_prompt.md, prompt.yaml, promptfooconfig.yaml, grader.py, and a local package.json. The prompt file is the variable under test; the page fixture arrives as the user message.

The shared fixtures contain explicit and missing ratings, decimal and zero values, irregular reviewer names, locations with commas and punctuation, sentiment boundaries, dates, review statistics, and previous-review content. Reusing them makes the added requirements visible instead of changing both the task and the test data at once.

Run one lesson

Each directory installs and runs independently.

cd 01_basic_table
npm install
export ANTHROPIC_API_KEY=sk-...
npm run eval
npm run view

npm run eval sets PROMPTFOO_PYTHON=python3 and invokes promptfoo eval. The repository therefore requires Node.js, npm, Python 3, an Anthropic API key, and network access to the configured model.

The prompt contract

The first lesson establishes rules that every later prompt carries forward:

• reject ratings inferred from stars, fill width, or icon count;
• preserve explicit decimals and 0;
• normalize the reviewer to FirstName LastName;
• use null when the rating is absent; and
• ignore previous-review content.

Its format contract requires a header followed by one quoted, two-field CSV row.

<format_contract>
- Output raw CSV only — no prose, markdown, fences, or extra columns.
- The very first line of your output MUST be the header: reviewer,rating
- Double-quote every field value in every data row (RFC 4180).
- After the header, output one data row per page with exactly 2 comma-separated fields.
- Use "null" for missing `rating`.
</format_contract>

Lesson 2 adds exact-source locations and five sentiment values: positive, negative, neutral, mixed, and unclear. The rules define neutral, mixed, and unclear positively instead of leaving the labels as names alone.

Lesson 3 changes the output to JSON. It requests all five season arrays and a review object containing reviewer, location, stats, rating, date, season, sentiment, current text, and an optional previous review.

The grader contract

The Python graders parse one model response at a time and return a PromptFoo pass/fail object. They normalize values before comparison so formatting noise does not obscure semantic errors.

The CSV graders normalize honorifics, Last, First names, missing-value synonyms, and numeric ratings. The extended grader preserves non-null location text exactly and lowercases sentiment before comparison.

The advanced grader accepts three structures:

1. a season-grouped object under reviews;
2. a flat list under reviews; or
3. a bare list.

It then flattens the records and compares only reviewer, rating, location, and sentiment.

Tighten the advanced evaluation

The smallest credible follow-up keeps the current fixtures and makes the assertions match the prompt:

1. Reject bare arrays and flat reviews arrays.
2. Require exactly the five declared season keys.
3. Check that the record appears in the array matching its season.
4. Compare date, stats, current review text, and previous-review fields.
5. Reject missing or extra object keys when schema conformance matters.
6. Commit a machine-readable result artifact with the model ID and PromptFoo version.

That change would turn the advanced lesson from a four-field extraction check into evidence for the schema it publishes.

What the lab demonstrates

The repository is a compact example of incremental prompt construction. One fixture set supports three levels of output complexity, and each lesson can be run without a shared build system.

Its stronger lesson is about evaluation design. Prompt text, parser behavior, normalization, fixture coverage, and saved results are separate parts of the system. Reliability claims are no stronger than the narrowest one.

Takeaways