Iterative prompt engineering for structured data extraction

Overview

This project uses a test-driven approach to prompt engineering. Given 20 HTML review pages, the goal is to write system prompts that extract reviewer data into precise formats, then verify correctness against a ground-truth fixture.

The fixture includes edge cases: missing ratings, zero ratings, decimals (3.5, 0.5, 4.5), unusual location formats, and boundary-line sentiments.

A recurring challenge was enforcing output format. LLMs tend to wrap output in markdown fences, invent extra columns, skip quoting, or normalize values. Each stage required increasingly explicit format rules to override these habits. Format enforcement turned out to be as important as the extraction logic itself.

Development Setup

The system forms a closed feedback loop: write the prompt, run it, validate the output, analyze failures, and refine.

Two nested feedback loops drive the work:

• Inner loop (prompt refinement) — edit, run, validate, analyze, refine. The analysis step is LLM-powered: Sonnet receives each failed page’s HTML alongside the expected and actual values, then suggests specific prompt edits.
• Outer loop (fixture calibration) — when both models consistently produce the same “wrong” answer, the ground truth itself is re-examined.

Prompt Evolution: Stage by Stage

The three prompts build on each other. Each stage inherits the previous stage’s rules and adds one layer of complexity.

Consistent core: <role> and <task>

The <role> stays nearly identical across all stages — a one-line persona anchoring the model as a data extraction expert. The <task> is the part that changes. Here are all three variants:

<role>
You are a data extraction expert extracting review data from HTML pages.
</role>

<task>
Extract the main current review from each page. Return raw CSV with columns `reviewer,rating`.
</task>

<role>
You are a data extraction expert extracting review data from HTML pages.
</role>

<task>
Extract the main current review from each page. Return raw CSV with columns `reviewer,rating,location,sentiment`.
</task>

<role>
You are a data extraction expert extracting review data from HTML pages into structured JSON.
</role>

<task>
Extract the main current review from each page. Return JSON grouped by season under a `reviews` key.
</task>

Rules and format contracts

Each prompt has two main working sections: <rules> (what to extract and how) and <format_contract> (how to shape the output). Rules are additive — each stage carries forward every rule from the previous stage, then adds new ones for the new fields. Format contracts, by contrast, are rewritten at each stage because the output shape changes.

The rule patterns that emerged follow a few recurring styles:

# Guarding against inference
- Do not infer [value] from [visual signal]

# Requiring explicit values
- [field]: explicit numeric value only; preserve decimals and 0

# Normalization
- Normalize [field] to [TargetFormat]

# Defining ambiguous labels with positive statements
- Use [label] when [specific observable condition]

# Scoping extraction
- Ignore [content type] / Include [content type] when present

For example, the star-icon rule follows the “do not infer” pattern, the rating rule follows “explicit values only,” and the name rule follows “normalize X to Y.” Sentiment labels follow the “positive definition” pattern — each label says what it is, not what it isn’t.

Format contracts serve a different purpose: they override the model’s default formatting habits. Representative rules include:

# Setting the output mode (first line of every format contract)
- Output raw [format] only — no prose, markdown, fences, or extra columns.

# Forcing the first token (prevents markdown fencing)
- Start directly with `.

# Mandating quoting (prevents comma-in-value breakage)
- Double-quote every field value in every data row (RFC 4180).

# Declaring the missing-value token
- Use "null" for missing [field].

The format contract turned out to be the section that required the most iteration. Quoting rules, anti-fencing instructions, and missing-value tokens each went through at least one revision cycle before stabilizing.

Rules: additive layering

Side-by-side prompt comparison

Format contract evolution

Prompt structure

Every prompt follows the same XML scaffold. The sections below show the general shape, with inherited content summarized rather than repeated.

<role>
  You are a data extraction expert extracting review data from HTML pages.
  ← Stage 3 adds: "into structured JSON"
</role>

<task>
  Extract the main current review from each page. Return [format] with [fields/structure].
</task>

<rules>
  # Foundation rules (Stage 1, inherited by all)
  - Do not infer [value] from [visual signal].
  - [field]: explicit numeric value only; preserve decimals and 0; use null when missing.
  - Normalize [field] to [TargetFormat].
  - One [row/object] per page.
  - Ignore previous-review content.

  # Sentiment rules (Stage 2+, inherited by Stage 3)
  - [field]: one of [label_1], [label_2], … [label_n].
  - Use [label] when [specific observable condition].

  # Season + previous review (Stage 3 only)
  - [field]: one of [value_1], … [value_n]. Infer from [source_1], then [source_2].
  - Include previous review when present; use null otherwise.
</rules>

<format_contract>
  - Output raw [format] only — no prose, markdown, fences.
  - [Format-specific quoting, header, or structural rules.]
  - Use [token] for missing values.
</format_contract>

<schema>              ← Stage 3 only
  { "reviews": { "Fall": [ { … } ], "Winter": [], … } }
</schema>

How Stage 3 reinforces earlier rules

• Season grouping checks date accuracy. A wrong date lands the review in the wrong season bucket, failing the season_grouping check.
• The schema constrains field types. Rating must be a number, stats must be integers, and the review object needs both current and previous keys.
• Previous-review extraction tests the scoping rule. Stages 1–2 say “ignore previous content.” Stage 3 says “include it in review.previous.” The model must distinguish current from previous data precisely.

Iteration Process

Each iteration follows a five-step cycle: change one thing, run it, measure the result, analyze failures, then decide what to do next.

Every run saves a timestamped bundle to runs/, so any two iterations can be compared directly.

Iteration Log: What Changed and Why

Model Comparison

The same prompts were tested on two Claude model tiers via Amazon Bedrock.

Results (after fixture calibration)

Best-run summary: both models reached 20/20 on Basic Table, Sonnet 4.6 reached 20/20 on Extended Table, and Haiku 4.5 reached 20/20 on JSON. The remaining drift was limited to 0-2 nondeterministic sentiment mismatches per run.

Both models achieve full marks on their best runs. The 0–2 remaining mismatches per run are nondeterministic — borderline sentiment cases that flip at temperature 0.3. This is the noise floor, and it shows which classifications are robust versus which sit on genuine semantic boundaries.

Model-specific behaviors

Ground-Truth Calibration

Final Results and Takeaways

Validation checks

Key takeaways