25 hyhmrright

run-iteration-eval Skill

为一次迭代运行 Logic-Lens content-eval 流水线并生成带评分的 summary.json——用于衡量技能改动效果。封装 scripts/run-content-evals.sh(运行器,消耗 token)和 scripts/grade-iteration.py(评分器,免费,可重复运行)。务必先同步插件缓存。当用户想"跑评估"、"给这次迭代打分"、"衡量技能改动"、"完整运行前先冒烟测试"或"重新给已有输出评分"时使用。

安装方式:把技能目录放入 ~/.claude/skills/(Claude Code)或在 claude.ai 设置中启用;也可复制右侧安装命令一键添加。

查看源码

技能指令原文(SKILL.md)

run-iteration-eval

Measures a skill change by running the content cases in evals/content/v2/evals-v2.json through
claude -p and grading the outputs. Outputs land in skills-workspace/iteration-/.

The runner and grader are split on purpose: running calls Claude and costs tokens; grading
is pure regex Python and is free to re-run on outputs that already exist. Never re-run the runner
just to re-score — re-grade instead.

Steps

  1. Sync the cache first — non-negotiable. The runner loads the skill from the plugin cache,

not skills/. Run the sync-skill-cache skill (or its script directly). If you skip this, the
eval grades the previously-published skill and the entire run is wasted:

   bash .claude/skills/sync-skill-cache/scripts/sync-cache.sh
  1. Pick a scope. Full runs cost real tokens; scope down while iterating:
   SMOKE=1 bash scripts/run-content-evals.sh              # one case per mode (~$0.10) — fast sanity
   CASES="200 201 202" bash scripts/run-content-evals.sh  # only the cases a diagnosis flagged
   TAG=myfix bash scripts/run-content-evals.sh            # full run, named tag
   bash scripts/run-content-evals.sh                      # full run, tag = git short SHA

The runner is idempotent — a case with an existing output.md is skipped. Delete the
eval-/ dir to force a re-run of that case.

  1. Read summary.json in the iteration dir. It carries overall pass rate plus the per-mode and

per-subscore (logic vs format) breakdown. The logic subscore reflects reasoning quality;
format reflects Output-Skeleton compliance and is the historical bottleneck with high
single-run variance. Judge a change on the right subscore — a format wobble is not a reasoning
regression.

  1. Re-grade without re-running (free) after editing the grader or to recompute on existing

outputs:

   python3 scripts/grade-iteration.py skills-workspace/iteration-<TAG>
  1. Single-iteration grade without the full suite — when you already have outputs and only want

the score table, grade-iteration.py is the cheapest path (see scripts/README.md).

Variance caveat

logic-review single-run scores are variance-dominated (see project memory). One run is a signal,
not a verdict — for a decision near the margin, run the affected cases 2–3× or widen the case set
before concluding a change helped or hurt. Hand the result to iteration-guard for the
ship/rollback call rather than eyeballing a single number.