openclaw-eval-skill#

Evaluation framework for any OpenClaw skill. No claude CLI dependency — all agent execution runs through sessions_spawn + sessions_history.

Scope: Works with CLI tool skills, conversational skills, and API integration skills.

---

Runtime Actions Disclosure#

This skill performs the following actions during evaluation:

ActionPurposeWhenRead ~/.openclaw/openclaw.jsonFind skill directories (extraDirs)Path resolutionWrite to eval-workspace/Store evaluation resultsEvery eval runCall sessions_spawnRun test queries in isolated sessionsTrigger & quality testsCall sessions_historyCollect conversation data for analysisAfter each spawnPersist cleanup="keep" sessionsRequired for trigger detectionTrigger rate tests

NOT performed automatically: Gateway restart, config modification, skill installation. These require manual user action (see "Bundled Test Skill" section).

---

Quick Eval#

Just say:

text

evaluate weather

The agent will:

1. Run scripts/resolve_paths.py weather to find all paths
2. Execute trigger rate + quality compare with detected evals
3. Output results to eval-workspace/weather/iter-N/

Options:

• evaluate weather trigger — trigger rate only
• evaluate weather quality — quality compare only
• evaluate github --mode all — explicit mode

What gets auto-detected:

• Skill path: from OpenClaw built-in skills or registered extraDirs
• Evals: from evals/{skill-name}/ or fallback to evals/example-*.json
• Output: next available iter-N directory

First step for agent: Run the resolver to get paths:

bash

python scripts/resolve_paths.py {skill-name} --mode {trigger|quality|all}

Use the JSON output to fill in paths for the workflows below.

---

Bundled Test Skill: fake-tool#

A test skill ( test-skills/fake-tool/) is included for validating trigger rate detection. It simulates a fictional "Zephyr API" that models cannot know from training.

Manual setup required: The agent will NOT automatically install fake-tool or restart your gateway. If you want to test with fake-tool:

1. Copy fake-tool to your skills directory: bash

   cp -r test-skills/fake-tool ~/.openclaw/workspace/skills/
   
   

2. Restart OpenClaw gateway (from terminal): bash

   openclaw gateway restart
   
   

3. Verify registration: bash

   python scripts/resolve_paths.py fake-tool
   
   

If step 3 returns a valid path, fake-tool is ready. If "not found", check that your ~/.openclaw/openclaw.json includes the skills directory in skills.load.extraDirs.

---

Evaluation Scenarios#

Tier 1: Core (Always Run)#

ScenarioWhat It TestsOutputTrigger RateDoes description trigger SKILL.md reads at the right times? Includes positive (should trigger) AND negative (should NOT trigger) cases.recall, specificity, precision, F1Quality CompareDoes skill improve output vs no-skill baseline?quality_score, assertion pass rateDescription DiagnosisWhy did triggers fail? Analyzes both false negatives AND false positives.gap analysis, recommendations

Tier 2: Optional (Run When Needed)#

ScenarioWhat It TestsWhen to UseModel ComparisonQuality + speed across haiku/sonnet/opusBefore deployment: which model is enough?Efficiency ProfileResponse time + retry patternsWhen skill feels slow: is agent walking wrong paths?

Tier 3: Future (Roadmap)#

ScenarioWhat It TestsStatusCross-skill ConflictTwo skills with overlapping descriptionsPlannedError RecoveryDoes agent recover when CLI fails?Planned

---

How This Skill Works#

Two-layer architecture:

text

Layer 1: Agent (main OpenClaw session) — YOU ARE HERE
  → Reads evals.json
  → Calls sessions_spawn to run subagents
  → Calls sessions_history to collect results
  → Writes raw data to workspace/

Layer 2: Python analysis scripts (run via exec)
  → Read the raw data from workspace/
  → Compute statistics
  → Generate reports

Python scripts ( analyze_*.py) are data processors — they cannot call sessions_spawn. The agent drives the workflow.

---

Usage#

Follow USAGE.md for all workflows.

Quick reference:

WorkflowWhat It TestsUSAGE.md SectionTrigger RateDoes description trigger SKILL.md reads at the right times?Workflow 1Quality CompareDoes skill improve output vs no-skill baseline?Workflow 2Model ComparisonQuality + Speed across haiku/sonnet/opusWorkflow 3Latency ProfileResponse time p50/p90Workflow 4

Each workflow follows the same pattern:

1. Agent spawns subagents using sessions_spawn
2. Agent collects histories using sessions_history
3. Agent writes raw data to workspace/{skill}/iter-{n}/raw/
4. Agent runs analysis script via exec

---

Core Principles#

1. Never modify the evaluated skill — observe only, give recommendations
2. Keep eval records in workspace — output goes to eval-workspace/<skill-name>/iteration-N/
3. Keep full records — save full_history.json (including tool_use + tool_result)

---

agents/ Reference#

FilePurposeWhen to Usegrader.mdCheck assertions, record behavior anomalies, give priority recommendationsRequired for every Quality Compare evalcomparator.mdBlind A/B comparison without assertionsWhen unbiased comparison is neededanalyzer.mdAnalyze cross-eval patterns after all evals completePost-analysis

---

Directory Structure#

text

eval-workspace/<skill-name>/
├── evals.json                    ← Eval definition (shared across iterations)
└── iteration-1/
    ├── raw/
    │   ├── histories/            ← Trigger test session histories
    │   └── transcripts/          ← Quality compare transcripts
    ├── trigger_results.json      ← analyze_triggers output
    ├── quality_results.json      ← analyze_quality output
    └── diagnostics/
        └── RECOMMENDATIONS.md

---

evals.json Format#

Quality Compare (prompt + assertions):

json

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": 1,
      "name": "onboarding-fresh",
      "prompt": "Check the weather in Tokyo",
      "context": "Clean machine, no prior setup. For grader only.",
      "expected_output": "Install → configure → verify profile",
      "assertions": [
        {
          "id": "a1-1",
          "description": "Install command executed",
          "type": "output_contains",
          "value": "pip install"
        },
        {
          "id": "a1-2",
          "description": "Profile verified after setup",
          "type": "output_contains",
          "value": "profile current",
          "priority": true
        }
      ]
    }
  ]
}

Trigger Rate (query + expected):

json

{
  "id": 1,
  "name": "direct-weather",
  "query": "What's the weather in Singapore?",
  "expected": true,
  "category": "positive"
}

---

Assertion Types#

TypeDetectionoutput_containsValue appears in conversation or tool outputoutput_not_containsValue does not appearoutput_count_maxOccurrences ≤ maxtool_calledSpecific tool called at least oncetool_not_calledSpecific tool not calledconversation_containsValue appears anywhere in conversationconversation_contains_anyAt least one value appears

Priority assertions ( "priority": true): any failure → overall=FAIL. Gap assertions ( "note": "Best practice..."): failure = skill design gap.

---

Issue Priority (grader output)#

text

🔴 P0 Critical  — Core functionality broken
🟠 P1 High      — Significantly impacts usability
🟡 P2 Medium    — Room for improvement
🟢 P3 Low       — Minor polish

---

Behavior Anomaly Tracking#

Grader records these signals beyond assertions:

FieldTriggerpath_correctionsWrong path then self-correctedretry_countSame command executed multiple timesmissing_file_readsAttempted to read non-existent filesskipped_stepsSteps required by skill were not executedhallucinationsFabricated non-existent commands/APIs

---

Key Constraints#

• sandbox="inherit" — subagents inherit skill registration environment
• cleanup="keep" — history must be retained for trigger detection
• Skill must be in a real directory under skills.load.extraDirs (symlinks rejected)