Skill development beyond intuition: authoring with plugin-dev and evaluating with skill-creator

Most skill writers work from intuition. They write a draft SKILL.md, run it against one prompt, see output that looks right, and declare the skill done. The output “looks correct” and that feels like enough.

It is not enough. Without evidence, a skill that feels helpful might be doing nothing — or worse, adding token overhead while degrading output quality. The agent might have performed identically without the skill. It might have performed better. There is no way to know, because there is no measurement.

Two tools from the Claude Code plugin ecosystem replace intuition with evidence. plugin-dev provides structured authoring and static validation — it catches structural errors before a skill ever runs. skill-creator provides runtime evaluation — it measures whether a skill actually improves agent outputs, and by how much. Together they cover the full lifecycle.

Where the fundamentals live

This article assumes familiarity with the SKILL.md format, progressive disclosure, implicit vs. explicit invocation, scope hierarchy, and plugin packaging. These fundamentals are covered in Building an agent skills ecosystem.

Where that post explains the what and why of skills, this one explains the how — the authoring workflow and the evaluation methodology that make skill development systematic rather than improvisational.

Plugin-dev is the authoring toolkit

plugin-dev is not a single tool. It is a complete authoring toolkit, itself distributed as a Claude Code plugin. It contains seven specialized skills, three autonomous agents, and one guided command — each covering a different aspect of plugin and skill development.

The seven skills are reference material: skill-development, agent-development, command-development, hook-development, plugin-structure, plugin-settings, and mcp-integration. Each loads through progressive disclosure — the metadata stays in context at all times, and the full body loads only when a question matches. These are consult-on-demand references, not active tools.

The active tools are the three agents and the single command.

/plugin-dev:create-plugin is an eight-phase guided workflow that takes a plugin from concept to documentation. It asks clarifying questions at each phase — discovery, component planning, detailed design, structure creation, component implementation, validation, testing, and documentation. It loads relevant skills automatically as needed and uses agents for AI-assisted generation. The critical phase is detailed design: before writing anything, it identifies underspecified aspects and forces explicit decisions about scope, interfaces, and behavior.

The three agents do work rather than just advise:

• agent-creator generates complete agent configurations from natural language. Describe what the agent should do — it extracts the core intent, designs an expert persona, architects a system prompt with edge-case handling, and writes the complete agent file with YAML frontmatter.
• plugin-validator performs static validation across ten areas: manifest syntax, directory structure, command files, agent files, skill files, hooks configuration, MCP configuration, file organization, security checks, and naming conventions. It produces a structured report with critical issues, warnings, and positive findings.
• skill-reviewer evaluates skill quality through structural analysis: description specificity (trigger phrases, concrete language), content organization (word count, writing style, imperative form), progressive disclosure implementation (whether detailed content lives in references/ rather than bloating SKILL.md), and supporting file completeness.

What static analysis cannot tell you

plugin-validator and skill-reviewer are thorough, but they share a fundamental limitation. Everything they check is visible in the files themselves. They examine structure, not behavior.

plugin-validator checks whether plugin.json parses as valid JSON. It does not check whether the plugin works when installed. It flags hardcoded credentials. It does not flag a skill whose instructions are syntactically valid but produce wrong answers. It verifies that all referenced files exist. It does not verify that those files contain correct information.

skill-reviewer checks whether a description uses specific trigger phrases and whether SKILL.md is under the recommended word count. It does not check whether those trigger phrases actually cause the skill to activate on real user prompts. It evaluates progressive disclosure structure. It does not evaluate whether the skill produces better outputs than having no skill at all.

A skill that passes every validation test can still fail in practice. The description might look specific on paper but never trigger in reality. The instructions might read clearly but produce worse outputs than the agent would generate on its own. The workflow might sound reasonable but lead the agent into time-wasting loops that a simpler approach would avoid.

Skill-creator fills the evidence gap

skill-creator is, fittingly, a skill for creating skills. Unlike plugin-dev’s agents — which read files and report on structure — skill-creator runs agents with and without the skill and measures the difference.

Internally, skill-creator contains three specialized sub-agents. grader evaluates agent outputs against predefined assertions and cites specific evidence for each PASS or FAIL. comparator performs blind A/B comparisons between two skill versions — it sees outputs labeled only as A and B, scores them against a rubric, and determines a winner without knowing which skill produced which output. analyzer extracts patterns from aggregate benchmark data: which assertions discriminate between configurations, where variance is highest across runs, and whether the token cost of the skill is justified by the quality gain.

Seven Python scripts automate the mechanical parts of the pipeline: spawning parallel runs, grading assertions, computing aggregate statistics, optimizing descriptions, and generating interactive review views. The scripts do the repetitive work; the sub-agents do the judgment work.

The methodology is borrowed from software testing. Define expected behavior. Run the system. Compare actual output to expectations. Measure the delta. Applied to skills, this means running identical prompts twice: once with the skill loaded, once without. The difference between the two runs is the skill’s actual contribution — the only number that matters.

The evaluation pipeline

The evaluation runs as a seven-step pipeline. Each step feeds into the next, and the full cycle repeats with each iteration of the skill.

Step 1: Design test cases

Test cases start simple: a realistic prompt and a human-readable description of the expected output. No assertions yet — at this stage the goal is to capture what real usage looks like. The prompts must mirror actual user behavior: file paths, column names, domain-specific jargon, occasional typos. Vague prompts (“process this data”) test nothing. The same prompt phrased three different ways (“Review this PR”, “Can you check my pull request?”, “Look over #342 for issues”) reveals whether the skill’s description is robust or brittle.

Start with two or three test cases. More is not better if the cases are artificial — three realistic cases catch more real failures than ten contrived ones.

Step 2: Spawn parallel runs

For every test case, two subagents launch in the same turn. One receives the skill path. One does not. Both receive the same prompt and input files. Both save outputs to parallel directories — with_skill/outputs/ and without_skill/outputs/.

Parallel execution keeps total iteration time manageable. Waiting for one run to finish before starting the next doubles the cycle. When runs execute concurrently, the wall-clock time is bounded by the slowest individual run, not the sum of all of them.

Step 3: Draft assertions while runs execute

Rather than waiting idly, the time while runs execute is used to write assertions — specific, verifiable checks that define success. Assertions should be programmatically verifiable: something a script or a grader subagent can determine as true or false by examining the output.

Good assertion: “The bar chart has labeled axes.” Weak assertion: “The output looks good.” Good assertion: “The response references at least two specific file paths.” Weak assertion: “The response is thorough.” Good assertions are falsifiable. If an assertion cannot fail, it is not measuring anything.

Assertions are added to evals/evals.json alongside the prompts. They are not frozen — if a run reveals that an assertion is non-discriminating (it passes in both configurations every time) or broken (it fails in both every time), it should be removed. Assertions that discriminate — that pass with the skill but fail without it — are the ones that prove the skill’s value.

Step 4: Capture timing data

When each subagent task completes in Claude Code, the task notification includes total_tokens and duration_ms. These values must be saved immediately — they are not persisted anywhere else and the notification is the only opportunity to capture them.

Timing data matters because skill quality is a cost-benefit calculation. A 50-point pass rate improvement that costs 5,000 extra tokens per run is a different proposition than the same improvement costing 500 tokens. Both numbers need to be visible to make that judgment.

Step 5: Grade and aggregate

A grader subagent reads each test case’s transcript and output files, then evaluates every assertion as PASS or FAIL with cited evidence:

{
  "expectations": [
    {
      "text": "The bar chart has labeled axes",
      "passed": false,
      "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label"
    },
    {
      "text": "The chart shows exactly three months of data",
      "passed": true,
      "evidence": "Chart displays bars for March, July, and November — three distinct months"
    }
  ],
  "summary": {
    "passed": 3,
    "failed": 1,
    "total": 4,
    "pass_rate": 0.75
  }
}

The grader also extracts claims from the output — factual assertions, process descriptions, quality statements — and verifies them independently. This catches issues that predefined assertions might miss: a factual error that no assertion anticipated, a skipped step in a documented procedure, a claim about a file that does not exist.

Once every run is graded, the aggregation script computes summary statistics across configurations:

{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83, "stddev": 0.06 },
      "time_seconds": { "mean": 45.0, "stddev": 12.0 },
      "tokens": { "mean": 3800, "stddev": 400 }
    },
    "without_skill": {
      "pass_rate": { "mean": 0.33, "stddev": 0.10 },
      "time_seconds": { "mean": 32.0, "stddev": 8.0 },
      "tokens": { "mean": 2100, "stddev": 300 }
    },
    "delta": {
      "pass_rate": 0.50,
      "time_seconds": 13.0,
      "tokens": 1700
    }
  }
}

The delta tells the story. This skill improves pass rate by 50 percentage points at a cost of 13 extra seconds and 1,700 extra tokens per run. Whether that tradeoff is acceptable depends on the task — for a PR review that catches a security vulnerability, it is cheap. For a routine formatting task, it warrants scrutiny. The benchmark does not make the decision. It provides the numbers to make it.

Step 6: Launch the viewer for human review

Assertion grades catch what was anticipated. Human review catches what was not — the output that passes every assertion but misses the point, the formatting that is technically correct but confusing to read, the approach that solves the narrow test case but would not generalize.

The eval viewer shows each test case side-by-side with its baseline, displays assertion grades inline, and provides a feedback textbox for each test case. The reviewer can navigate with arrow keys, compare outputs directly, and leave specific, actionable comments: “The chart is missing axis labels” rather than “looks bad.”

Step 7: Read feedback and iterate

Empty feedback means the test case passed human review. Specific feedback guides the next revision. The skill is improved based on feedback, the entire test suite reruns into a new iteration directory (iteration-2/), and the cycle repeats.

Stop when feedback is consistently empty or when successive iterations show plateauing benchmark scores. A skill that improves 50 points on the first iteration and 2 points on the fourth is done — further tweaking is polishing, not improvement.

Description optimization

Output quality and description accuracy are independent concerns. A skill can produce excellent outputs but never activate because the description is too vague. Or it can trigger reliably on every relevant prompt but produce results no better than the baseline. Both must be verified independently.

skill-creator includes a separate optimization loop, driven by run_loop.py, that tunes the description for triggering accuracy.

The process starts by generating 20 eval queries: 10 that should trigger the skill across different phrasings (“Review this PR”, “Can you check my pull request?”, “Look over #342 for issues”) and 10 that should not — near-misses from adjacent domains, ambiguous phrasing, tasks the agent should handle without the skill. The set is reviewed with the user through an HTML template that allows editing, toggling trigger expectations, and adding edge cases.

Once the eval set is locked, the optimization loop begins. The set is split 60/40 into train and test, stratified by should_trigger so both sets are representative. Each query runs three times to compute a reliable trigger rate — model behavior is nondeterministic, and a single run can be misleading. The loop evaluates the current description against the train set, identifies failures (should-trigger queries that did not activate, should-not-trigger queries that did), and calls improve_description.py — which invokes claude -p with the full skill content, failed cases, and history of previous attempts — to propose a better description.

This repeats for up to five iterations. The best description is selected by test set score, not train set score, to prevent overfitting. A description that achieves 95% accuracy on train but 60% on test has memorized the training queries; the selection rule catches this.

python -m scripts.run_loop \
  --eval-set path-to-trigger-eval.json \
  --skill-path path/to/skill \
  --model claude-sonnet-4-6 \
  --max-iterations 5 \
  --verbose

The output is an HTML report with per-iteration scores and a best_description field — a replacement for the skill’s description frontmatter that is backed by measured trigger rates rather than author instinct.

A broader industry pattern

The plugin-dev + skill-creator combination is not an isolated design. The same pattern — structured authoring paired with runtime evaluation — is emerging across the agent development ecosystem.

GitButler proposes a four-tier evaluation taxonomy for agent skills. Tier 1 is static analysis: does the file parse? Tier 2 is single-turn tool selection: does the agent reach for the right command? Tier 3 is multi-turn workflow with mocked execution: does the sequence of decisions hold together? Tier 4 is full integration against real repositories: does it work end-to-end? Their core insight: evaluations should be created before writing documentation, not after. Eval-driven development applies to skills the same way test-driven development applies to code. Their baseline pass rates dropped from 80% to 14–29% once stricter assertions were applied — a finding that mirrors the static-vs-runtime gap plugin-dev and skill-creator are designed to address.

LangChain’s engineering team reports that adding modular skills to their coding agent improved task completion rate from 9% to 82%. The mechanism: Docker sandboxes for reproducibility, plus independent A/B testing of skill variations — the same baseline-comparison pattern skill-creator uses. Their key finding was that skill consolidation matters as much as skill content. Around 12 well-scoped skills produced more reliable invocation than roughly 20 overlapping ones. The agent got confused by having too many near-identical descriptions to choose from.

SkillsBench introduces the “Normalized Gain” metric — measuring improvement as a fraction of the gap between baseline and perfect performance, across 84 tasks spanning engineering, cybersecurity, economics, and other domains. This is a more nuanced version of the delta computation in skill-creator’s benchmark.json. It rests on the same three-layer abstraction model the skills ecosystem uses: skills extend agent capabilities like applications extend an operating system.

OpenAI’s internal practices for Codex skill development follow the same contours. Define success criteria across four categories — outcome, process, style, and efficiency — before writing the skill. Use manual triggering to surface hidden assumptions about when a skill should activate. Employ deterministic graders against JSONL traces for reproducible evaluation. Add a model-assisted rubric pass for qualitative concerns that deterministic rules cannot capture.

The pattern holds across every example: authoring tools provide structure; evaluation tools provide evidence. Together they close the loop from “this feels right” to “the numbers show it works.” The tools are new. The methodology — define, measure, compare, iterate — is not.

Takeaways