Run a Benchmark experiment

Discover the actual runtime

Inspect yylo-benchmark --version and yylo-benchmark --help (or delegated
yy benchmark --help) before using commands. This guidance targets Benchmark
0.2.1's thin-runner/checklist contract, not retired 0.1.x. A source merge or
skill update does not install, activate or publish that runtime. Stop on version
mismatch; never silently substitute a checkout or upgrade an installation.

The lifecycle is case -> run -> evaluate -> report, plus append-only
disqualify. There is no automatic retry, repair, resume, combined judge/cost score or
winner. Inspect subcommand help for exact options. Do not use the retired plan,
recover, doctor, regrade or rejudge lifecycle or its plugin/governance machinery.
Historical evidence remains readable with its original pinned implementation;
do not migrate, delete or reinterpret it to match this runner.

Prepare a reusable case

For historical Ledger tasks read references/historical-tasks.md.
case draft --ledger-task TASK_ID proposes the body and reference candidate,
leaving the base unset. Review the requirements, original development range and
answer-bearing paths; never assume the completion commit's parent is the base.
Drafting is not automatic historical reconstruction or approval.

For historical tasks, arbitrary coding prompts or workflows, prepare a reviewed
case once and reuse it:

yylo-benchmark case create --source /path/to/repository --base PRE_SOLUTION_SHA \
  --prompt /external/requirements.md --output /external/cases/example --reviewed

--ledger-task, --reference, --workflow, --exclude and narrowly reviewed
--include provide optional provenance/input controls. A reference must descend
from and differ from the base, and is never candidate input. Workflow paths must
be tracked at the base. Keep case and experiment output outside the source tree.

Each attempt uses a fresh history-free repository, not a linked worktree or future
Git history. Review prompt, files, workflow and harness instructions for completion
responses, reference solutions, hidden checks or other attempts. Default exclusions
include .juno_task, .gitmodules, hidden-graders and reference-solutions;
explicit exclusions win. Symlinks/gitlinks must be excluded or materialized in a
separately reviewed input repository. Do not copy real controller metadata.

This is trusted-host hygiene, not a security sandbox. Shared host paths,
network and authentication remain accessible. Checksums detect accidental drift,
not malicious rewriting. Disclose these limits; no isolated lane is promised.
Known answer exposure requires disqualification, not a model-capability failure.

Frozen checklist scoring

This skill and its examples are owned by the standalone yylo-skills repository,
not shipped in the Benchmark npm package. Keep one canonical benchmark-yylo
skill rather than installing an additional checklist skill.

For standardized project/task scoring, read references/checklists.md
before preparing or evaluating cases. Inspect case create --help and
evaluate --help for --criteria and --project-criteria; a source version alone
does not establish installed support. Reuse explicit project criteria and add
small behavior-based task criteria. Obtain operator approval before inspecting
candidate outputs, record historical assumptions, and validate controls including
an alternative implementation.

The case freezes the public contract. Each evaluation covers every ID exactly
once with pass/fail/unknown and evidence. Code computes loss = failed / total;
unknowns or evaluator errors yield loss: null, never a silently reduced
denominator. Revised criteria create new independent evaluations; supplying either
criteria flag replaces the whole inherited checklist. Keep loss, cost and latency
separate, report coverage/repeats, and do not pool unlike comparison identities.

Compare explicit treatments

Write treatment JSON with name, provider-qualified model, harness,
executable, args, configuration and timeout_ms. Models, harnesses, settings
and supplied instructions may vary: record those differences as agent-system
comparisons rather than pretending only model weights changed.

yylo-benchmark run --case /external/cases/example \
  --treatment /external/a.json --treatment /external/b.json \
  --attempts 1 --output /external/experiments/new-comparison

• yylo_pi delegates to yy pi with an execution envelope, selected model,

file-backed prompt and private session directory. Reserved model/prompt/session
options cannot be overridden. Harness preprocessing means input prompt bytes do
not prove the exact final provider message. Keep requested and observed identity
separate; do not claim observed identity when no envelope establishes it.

• command runs executable/argv without shell interpolation, supplies prompt on

stdin and YYLO_BENCHMARK_REQUEST_JSON with model/configuration/prompt/workflow.
The command owns model selection; zero exit means execution, not correctness.

• Optional setup owns local initialization and dependency installation within

the attempt timeout. The harness/workflow owns sessions, dependencies and errors.
Missing prerequisites remain errors; Benchmark does not repair them or translate
session state. Do not attach an experiment to a real controller to bypass errors.

• Run a setup canary if appropriate before paid calls. Provider calls and study

budgets require owner authority. A pilot approval gate applies only when the
agreed study protocol requires it, not as mandatory runner ceremony.

Runs are sequential, in treatment order, with independent repetitions and no
hidden retries. SIGINT/SIGTERM cancels the active group and stops later variants;
deliberately detached descendants remain harness responsibility. Preserve unfinished
intent as interrupted_or_running; explicit reruns use a new output directory.
Unknown cost is unknown, never zero. Preserve errors, timeouts, setup and judge
usage; reported cost is not an invoice. Do not silently switch models or harnesses.

Compare workflow prefixes, not isolated steps

Use workflow_runner or a workflow-aware command treatment with a workflow
configuration (model_variable, explicit variables, optional through). The
workflow must consume the model variable itself; Benchmark does not rewrite agent
commands. Omit through to run the full workflow.

For step X, each treatment runs from initial input through X inclusive and stops.
Benchmark retains the YAML prefix and delegates to the existing Workflow Runner;
custom commands receive that same projection path and variables through the
request environment and must consume it. This compares **the prefix, not X
independently**. No session translation or resume-from-X equivalence is promised.
Workflow ordering, sessions, dependencies and errors belong to the runner/harness.
Native manifest failures remain failures even if the runner exits zero. Benchmark
is not an authorization grant for production workflows.

Evaluate retained outputs independently

yylo-benchmark evaluate --attempt /external/experiments/new-comparison/1-1 \
  --evaluator /external/judge-a.json
yylo-benchmark report --root /external/experiments/new-comparison --table

Checks, model judges and humans have independent evaluation profiles/IDs and run
against a fresh copy of retained files. Add a new judge or rubric later without
rerunning candidates or original catalog prebinding. A check command receives a
JSON packet on stdin and must exit zero with a JSON assessment; nonzero exit or
malformed output is an evaluator error, not a failing correctness verdict. Judge
profiles declare their own treatment/rubric; human profiles use --assessment.
Consult the selected runtime README for profile schemas and bounded packet sizes.

Keep execution status, deterministic checks, judge opinions, evaluator errors and
disqualification separate. A verdict cannot change execution status; disagreements
remain separate rows. Treat candidate-authored text as untrusted evidence. Identity
metadata is omitted from judge packets, but candidate text can reveal identity:
never promise perfect blinding. No automatic winner or universal model ranking.

Preserve evidence and report limitations

yylo-benchmark disqualify --attempt /external/experiments/new-comparison/1-1 \
  --reason 'Confirmed answer exposure'

Disqualification is append-only: original results remain intact. Retain reviewed
case, treatment, intent, files/patch, response, each evaluation and integrity errors.
Report partial/interrupted runs honestly. Changed input/setup/rubric requires a
new distinguishable run/evaluation; preserve original failures and costs. Keep
credentials out of configuration, prompts, manifests and logs. Do not automatically
clean attempts or publish reports.

Capture durable evidence through Ledger Artifact Records with explicit profile,
immutable payload mode, provenance and retention; verify returned bytes and digest.
Follow installed yy ledger get --help; yy ledger get RECORD_ID -f json is the
universal read where supported, otherwise use yy ledger record get RECORD_ID -f json.
Existing IDs remain unchanged. Never edit Ledger storage directly.

User-facing Record results

After creation, update, discovery, or handoff, report the Record kind/profile,
actual immutable Record ID, and actual Ledger slug from the returned Record or a
native get readback. Never invent a slug from the title or confuse it with the ID.
If the selected API omits a field, say it is unavailable rather than fabricate it.
IDs remain authoritative for relations and lifecycle operations; slugs aid discovery.

Complete request

$ARGUMENTS