SOTA Agent#

Search intent: sota agent, state of the art benchmark scouting, cv benchmark campaign, gpu vm research workflow

Goal#

Turn a vague "beat the benchmark" request into a disciplined campaign:

• fixed target metric and split
• explicit literature and leaderboard snapshot
• bounded reproduction plan
• explicit handoff to the separate execution lane when runs need external tools
• evidence requirements that can be reviewed without relying on live session state
• ablations that answer one question at a time
• promotion only when the claim survives review

This skill is the frontier-planning and candidate-selection layer. For execution artifacts or promotion evidence, pair it with data-science-cv-repro-lab; this skill stays focused on planning and claim review.

Use This Skill When#

• the user wants a CV or DS system pushed toward state-of-the-art results
• the task involves reproducing or surpassing recent papers
• the workflow needs paper triage, leaderboard tracking, or claim review
• the workflow needs a clean handoff to an execution skill after the benchmark contract is frozen
• the user needs experiment management across local runs, notebooks, and long-running jobs
• the question is whether execution evidence supports a SOTA candidate
• the question is whether a candidate is a real SOTA step or only noise, leakage, or benchmark overfitting

If the campaign includes serious execution or release review, use this skill to choose and rank candidates, then use data-science-cv-repro-lab as the execution lane.

Quick Start#

1. Freeze the claim target before touching recipes.
   • Name the task, dataset, metric, split, and target score.
   • Name the current trusted baseline.
   • Name the claim threshold for "match", "beat", or "not enough".
2. Initialize the campaign records immediately.
   • Use python3 {baseDir}/scripts/init_sota_campaign.py --root <dir> --campaign-id <id> --title <title>.
   • Use python3 {baseDir}/scripts/init_sota_leaderboard_snapshot.py --out <json> --task <task> --dataset <dataset> --metric <metric> --split <split>.
   • Use python3 {baseDir}/scripts/init_sota_paper_triage.py --out <json> --campaign-id <id> --task <task>.
   • Use python3 {baseDir}/scripts/init_sota_program.py --out <json> --campaign-id <id> --task <task> --dataset <dataset> --metric <metric> --split <split> when you need one machine-readable benchmark, rerun, delegation, and auth plan.
   • Use python3 {baseDir}/scripts/init_sota_candidate_card.py --out <json> --candidate-id <id> --campaign-id <id> --objective <goal>.
   • If execution review depends on synced QA runs, runtime sweeps, or benchmark panels, store the paired data-science-cv-repro-lab review dashboard path in the program and candidate records before the claim review starts.
   • If external execution evidence exists, record the reviewed artifact manifest path in the program and candidate records instead of acting through a live session.
   • If the review surface needs manual or visual QA, use python3 {baseDir}/scripts/init_sota_validation_scorecard.py --out <json> --scorecard-id <id> --surface <surface>.
   • If an external export bundle matters, use python3 {baseDir}/scripts/init_sota_artifact_manifest.py --out <json> --bundle-root <dir>.
   • If a long execution run is involved, record only sanitized summaries and artifact references in the SOTA campaign files.
3. Separate the campaign roles even if one agent performs all of them.
   • Scout: papers, leaderboards, repos, and benchmark rules.
   • Reproducer: baseline and top-paper reproduction.
   • Ablator: controlled change sets and compute allocation.
   • Reviewer: contamination, metric drift, and claim integrity.
   • Promoter: final claim or hold decision.
   • Keep the benchmark definition and final claim wording fixed.
   • Use bounded scouting and review lanes for literature triage, repo inspection, per-paper extraction, and hard-case review.
   • For repeated audits, batch over a manifest or CSV instead of free-form context accumulation.
4. Pick the execution lane explicitly.
   • Execution handoff lane: use data-science-cv-repro-lab for external runs and artifact capture.
   • Local lane: cheap falsification, tiny reruns, and artifact review.
5. Keep file writes inside one campaign workspace.
   • Create one dedicated campaign root and keep every --out, --bundle-root, and --output-root path under it.
   • Do not point the bundled scripts at unrelated home-directory or system paths.
   • Treat scripts/sota_public_safety.py as the canonical public-redaction layer for URLs, refs, and paths.
6. Work the SOTA ladder in order.
   • Freeze the benchmark definition and auth rule before using more compute.
   • Reproduce the trusted baseline first.
   • Reproduce one relevant reference result or a close public checkpoint.
   • Build a hypothesis backlog from literature gaps, not vibes.
   • Run narrow ablations before broad recipe churn.
   • Stress the best candidate on the fixed review surfaces.
7. Claim only on full-surface wins.
   • Fixed benchmark score
   • Reproduced baseline delta
   • Compute or cost context
   • Browser or GUI evidence if that lane mattered
   • Failure-case review
   • Exact evidence bundle
   • Render the final review with python3 {baseDir}/scripts/render_sota_claim_summary.py --candidate-card <json> --out <md>.

Operating Rules#

Campaign rules#

• One campaign has one target benchmark contract.
• Do not let the target metric or split drift midstream.
• Keep a short hypothesis backlog and kill low-information ideas quickly.
• Record why each experiment exists before running it.

Codex multi-agent rules#

• Main thread owns the benchmark contract, stop conditions, and final claim decision.
• Subagents should do bounded work only: scout, reproduce, ablate, or review.
• Do not let one exploratory thread silently rewrite the campaign contract.
• For repeated claim checks or literature extraction, prefer manifest-driven fanout over conversational drift.

Literature rules#

• Read only the papers or repos that change the candidate plan.
• Extract the minimum useful fields: task, metric, split, data, compute, architecture, augmentations, training tricks, and caveats.
• Prefer a reproduced strong baseline over copying five tricks from five papers without control.
• Do not treat leaderboard rows as ground truth without checking task definition and split rules.

Ablation rules#

• Change one meaningful variable at a time when the goal is causal understanding.
• If several knobs move together, label the run as a package change, not an ablation.
• Keep one canonical baseline recipe alive for comparison.
• Require the first winning candidate to survive at least one rerun or adjacent-seed check before escalating the claim.

Compute rules#

• Spend cheap compute on reproduction and short falsification first.
• Do not push a long run unless the hypothesis would matter if it wins.
• Record training cost, wall time, and hardware for every serious candidate.
• Cut branches that cannot plausibly clear the target with the remaining budget.

Runtime and auth rules#

• This public skill does not require API keys, account tokens, live sessions, or account-bound credentials.
• Prefer local files, public URLs, and user-supplied artifacts over account-bound execution paths.
• Do not require or recommend OPENAI_API_KEY, other vendor API keys, or paid inference APIs as the default campaign runtime path.
• If a third-party framework only works through paid API keys, treat it as reference material unless it can run through local tools or public artifacts.

External execution rules#

• Execution rules live in data-science-cv-repro-lab.
• In this skill, record only the benchmark contract, candidate rationale, review status, and sanitized artifact references.

Claim safety rules#

• No SOTA claim without a fixed metric, split, and baseline.
• No SOTA claim on a contaminated benchmark or hidden train-on-test path.
• If the execution story depends on a dashboard or synced review surface, keep the dashboard path, source audit, and leakage audit in the claim packet.
• If a candidate wins only on one slice while regressing important surfaces, hold it.
• Report uncertainty honestly: "best internal result so far" is not the same as "new SOTA".
• Small deltas need rerun or adjacent-seed support before they become claim language.

References#

Read only the reference that matches the task:

• references/sota-campaign-playbook.md
  • Full campaign structure, role separation, and stop conditions.
• references/sota-program-rules.md
  • Rules for queues, stage discipline, ablations, and promotion gating.
• references/campaign-harness-stack.md
  • What to reuse from Codex subagents, harness engineering, OpenEvolve, Symphony, Paperclip, and OptiLLM under a local-first campaign rule.
• references/benchmark-discipline.md
  • How to avoid contamination, metric drift, and invalid comparisons.
• references/paper-triage.md
  • How to filter papers and extract only decision-relevant details.
• references/public-research-lane.md
  • How to review public literature and leaderboard pages without private sessions.
• references/external-evidence-handoff.md
  • How to record sanitized evidence from external notebook or UI runs without controlling a live session.
• references/execution-evidence-summary.md
  • How to summarize execution evidence that belongs in the paired execution skill.
• references/claim-safety.md
  • Review rules for whether a candidate deserves a SOTA claim at all.
• references/public-safety.md
  • Publication review rules for secrets, private refs, and raw notebook paths.

Bundled Scripts#

• scripts/sota_public_safety.py
  • Pure local helpers for path, URL, ref, env, and command redaction. No network I/O or subprocess execution.
• scripts/init_sota_campaign.py
  • Create a reusable campaign folder with benchmark, program, agent, research, leaderboard, plan, ablation, evidence, and claim files.
• scripts/init_sota_program.py
  • Create a machine-readable program record with the fixed benchmark, baselines, rerun policy, bounded subagent roles, and local-first runtime rules.
• scripts/init_sota_leaderboard_snapshot.py
  • Create a machine-readable snapshot of the target benchmark contract and current reference scores.
• scripts/init_sota_paper_triage.py
  • Create a machine-readable literature queue for paper screening and extraction.
• scripts/init_sota_browser_run_card.py
  • Create a sanitized external-evidence record for notebook or UI run artifacts.
• scripts/init_sota_validation_scorecard.py
  • Create a machine-readable GUI or notebook validation scorecard when visible state matters to the campaign.
• scripts/init_sota_artifact_manifest.py
  • Create a machine-readable export-bundle manifest for external artifacts with redacted public path metadata.
• scripts/init_sota_candidate_card.py
  • Create a machine-readable card for a serious candidate, its execution lane, auth mode, and claim state.
• scripts/init_sota_candidate.py
  • Create a machine-readable candidate record with change set, risks, and redacted public artifact refs.
• scripts/init_sota_ablation_queue.py
  • Create a focused ablation queue for one candidate family.
• scripts/init_sota_vm_bootstrap_manifest.py
  • Create a redacted long-run summary manifest for already-approved execution artifacts.
• scripts/update_sota_scoreboard.py
  • Refresh a ranked scoreboard for a fixed metric and goal direction.
• scripts/init_sota_review_packet.py
  • Join the core artifacts for a promotion, hold, or cut decision.
• scripts/render_sota_claim_summary.py
  • Render a concise markdown review from the machine-readable candidate card.
• scripts/render_sota_program_summary.py
  • Render a concise markdown summary from the program, candidate, scoreboard, and review packet.