Experiment Lifecycle Governance#

Part of the Exo suite — literature ( hfpclawer) → experiments ( expflow-pde) → proofs ( omega-architect). Three independent CLIs that meet through files and CLI calls, never imports. Entry skill: exo-suite-linkage (wiring, cost tiers low → medium → high, degradation ladder). Install: uv tool install hfpclawer · uv tool install expflow-pde · uv tool install "omega-architect @ git+https://github.com/diamond2nv/omega-architect@v0.2.3"

Overview#

Governance layer for experiment workflows: protect destructive operations, standardize metrics, rank experiments with gating, and audit against competition rules.

Three sub-systems:

1. PIN Protection — 4-digit PIN guard for cancel/stop/delete operations
2. Metrics Registry — Standardized metric definitions with thresholds
3. Compare-Scores — Multi-model ranking with gating

Installation#

bash

pip install expflow-pde

Version floor: install expflow-pde>=0.7.0 (public/PyPI line). The development line is ahead (0.7.3): anything marked "development line" below is not in the PyPI 0.7.0 artifact.

bash

uv tool install expflow-pde                  # 1) isolated CLI, `expflow` on PATH (recommended)
# pin the version — `uvx`/`uv tool run` reuse an installed tool env (may run an older
# release), and an unpinned launch is a supply-chain (rug-pull) risk
uvx --from "expflow-pde==0.7.0" expflow --help        # 2) run it without installing anything
uv pip install expflow-pde                   # 3) inside an existing project / venv
pip install expflow-pde                      # 4) no uv — pip and pipx both work
pipx install expflow-pde                     #    (pipx needs a recent version; 1.0.0 cannot parse specs)
# extras ride along with any form, e.g.: uv tool install "expflow-pde[clearml]"

All five forms above were run and verified (uv tool install / uvx --from / uv pip / pip / pipx).

1. PIN Protection Pattern#

Architecture#

text

~/.expflow/pin.hash          # SHA-256 hash of 4-digit PIN (never plaintext)
~/.expflow/experiments.jsonl # Experiment registry (each line = JSON record)

Module Design#

python

# pin.py — 4 components:
# 1. init_pin(pin: str) -> hash          # Validate + hash + write
# 2. verify_pin(pin: str) -> bool         # Hash comparison
# 3. pin_is_set() -> bool                 # Check if PIN configured
# 4. guard(action_description) -> bool    # Interactive prompt

# sha256 hash — never store raw PIN
def _hash_pin(pin: str) -> str:
    return hashlib.sha256(pin.encode()).hexdigest()

# Validate exactly 4 digits
def _validate_pin(pin: str) -> None:
    if not pin.isdigit() or len(pin) != 4:
        raise ValueError("PIN must be exactly 4 digits (0-9)")

CLI Commands#

bash

expflow pin init 1234          # Set PIN (SHA-256 stored)
expflow pin check              # Interactive verify
expflow pin clear [--force]    # Remove PIN
expflow pin status             # Show if active

# Guarded commands (require PIN unless --force):
expflow run cancel <id>            # Interactive PIN prompt
expflow run cancel <id> --force    # Skip PIN

2. Standardized Metrics Registry#

Structure#

python

STANDARD_METRICS = {
    "seg_total": {
        "type": "scalar", "group": "Score",
        "higher_is_better": True,
        "description": "Total segment score (primary competition metric)",
    },
    "pde_mean": {
        "type": "scalar", "group": "PDE",
        "higher_is_better": False,
        "threshold": 18.09,  # Competition gate
    },
    "train_time_min": {
        "type": "scalar", "group": "Time",
        "higher_is_better": False,
        "threshold": 60,  # Competition limit
    },
    # ... 13 total metrics across Score/Loss/PDE/Time/Model/Training groups
}

report_standard()#

python

def report_standard(task: Any | None = None, **kwargs: float) -> dict[str, float]:
    reported = {}
    for name, value in kwargs.items():
        info = STANDARD_METRICS.get(name)
        if info is None:
            raise ValueError(f"Unknown metric '{name}'...")
        reported[name] = float(value)
        if task is not None:
            task.report_scalar(title=info["group"], series=name, value=float(value), iteration=0)
    return reported

3. Compare-Scores: Multi-Model Ranking#

CLI#

bash

expflow clearml compare-scores \
    --project PDEBench --tags task1 \
    --sort-by pde_mean --ascending \
    --gate pde_mean:lt:18.09 --gate train_time_min:lt:60

Gate Format#

Gates use metric:op:value triplets:

• pde_mean:lt:18.09 — PDE mean < 18.09
• train_time_min:le:60 — Training time ≤ 60 min
• seg_total:ge:50 — Score ≥ 50

Operators: lt, le, gt, ge.

4. Competition Rules Audit#

CLI#

bash

expflow audit validate exp-001 --competition-rules --task-id abc123

Python API#

python

from expflow_pde.audit import validate_competition_rules

result = validate_competition_rules(
    task_metrics={"seg_total": 57.09, "pde_mean": 15.0, "train_time_min": 45.5},
    task_params={"Args/--sub_step": "5"},
)
print(f"All pass: {result['all_pass']}")

Validation Checks#

CheckConditionDetailsseg_totalPrimary competition score (no gating)Reported, not gatedpde_meanMust be < 18.09Threshold from STANDARD_METRICStrain_time_minMust be < 60Threshold from STANDARD_METRICSsub_step parameterMust exist and be > 0Searches case-insensitive

Testing Patterns#

PIN Tests (36 tests)#

• Hash consistency: same input → same hash
• Validation rejects: wrong length, non-numeric, empty
• Init → file exists with correct hash
• Guard mock: correct → True, quit → False

Metrics Tests#

• Registry structure: each metric has type, group, higher_is_better
• report_standard: returns dict of reported metrics

Compare Tests#

• _apply_gate: all 4 operators (lt/le/gt/ge) with passing and failing cases

Experiment Loop Commands (0.7.0)#

GroupSub-commandsPurposeexpflow dispatch``status list tree stats audit-log archivelocal SQLite dispatch database (what was submitted, when, by whom)expflow iterate``runone-shot iteration: diagnose → suggest → submitexpflow repeat``diagnose resubmitrepair diagnosis and resubmission (Hermes /goal compatible)expflow hypothesis``open record list show rejected closehypothesis tracking, including negative results

Pair them: dispatch gives the provenance trail, iterate/ repeat close the loop, and hypothesis keeps the failed directions from being retried blindly.

Only in the development line (not in the 0.7.0 PyPI artifact yet)#

• expflow competition — init stop status merge mask bootstrap collect validate (competition logging sessions, parameterized scoring).
• Modules added after 0.7.0: noise-aware champion validation + agent arbiter ( validate.py), cross-session dead-end registry ( registry.py), stagnation detection and experiment reports ( monitor.py).

Hermes Agent Environment#

• MCP server — expflow mcp starts a FastMCP server (18+ tools) for agent integration: yaml

  # ~/.hermes/config.yaml
  mcp:
    servers:
      expflow:
        command: "expflow"
        args: ["mcp"]
  
  

• Config — expflow init writes the toolkit config; real credentials (ClearML API, Langfuse keys, dataset paths) belong in .env / a local config.yaml, never in tracked files.

• Environment overrides — EXPFLOW_HOME, EXPFLOW_PIN_HASH, EXPFLOW_COMPETITION_DEADLINE, EXPFLOW_SEMANTIC_URL.

• Cost profile (not zero-cost) — the orchestration layer itself makes no LLM call, but runs cost real money and hardware: ClearML workers/queue time, GPU hours for the submitted experiments, and storage for artifacts. Quote costs from recorded runs, never an estimate.

Pitfalls#

1. YAML/Env vs File Storage for PIN#

PIN hash must NOT go into config.yaml (risk of git commit). Use ~/.expflow/pin.hash. Precedence: pin.hash file > .env EXPFLOW_PIN_HASH > config.yaml pin.hash.

2. get_last_scalar_metrics() clearml API#

Returns nested dict: {"Score": {"seg_total": {"last": 57.09, ...}}, ...}. Flatten to {"seg_total": 57.09} for compare_scores.

3. --force Flag for Script Calls#

Always provide --force / -f on PIN-guarded commands for CI/automation.

4. Interactive getpass vs Non-Interactive#

getpass.getpass() works in terminals but fails in piped commands, CI, or subagent calls. Always provide --pin or --force as alternative paths.