Batch Executor#

Corpus-scale processing: classify → prioritize → spawn → checkpoint → reconcile.

Unlike task-extractor (for 3-12 inline tasks) or batch-cognition (for idea analysis), this skill EXECUTES at scale with sub-agent parallelism.

When to Use#

• Google Drive folder dump (mixed docs, notes, spreadsheets)
• ChatGPT conversation export (3K+ prompts)
• Apple Notes dump (years of ideas)
• Any input > 20 items or > 10K tokens of raw content
• File-based input (not inline chat messages — use task-extractor for those)

Architecture#

text

INPUT (any scale)
    ↓
PHASE 1: INGEST — save raw to disk, never lose
    ↓
PHASE 2: CLASSIFY — type each item, estimate effort
    ↓
PHASE 3: TRIAGE — score by value, group by dependency
    ↓
PHASE 4: EXECUTE — spawn sub-agents (max 3 concurrent)
    ↓
PHASE 5: RECONCILE — verify completions, retry failures
    ↓
PHASE 6: REPORT — value stack, patterns, action items

Phase 1: INGEST#

Save ALL raw input to systems/batch-executor/corpus/YYYY-MM-DD-SOURCE.md BEFORE any processing.

For file inputs:

• PDF → extract text via pdf tool
• CSV/JSON → parse, one item per row/object
• Markdown → split on ## headers or --- separators
• ChatGPT export → parse conversations.json, group by chain_id
• Google Drive → process each file, flatten into items

Create the manifest:

markdown

# Corpus Manifest: [source] [date]
# Total items: [N]
# Raw file: [path]
# Status: INGESTED

| # | First 80 chars | Type | Effort | Status |
|---|---------------|------|--------|--------|
| 1 | ... | ? | ? | INGESTED |

Phase 2: CLASSIFY#

For each item, assign:

TypeDescriptionActionTASKHas a clear action verb + deliverableEXECUTEIDEASpeculative, "what if", product conceptSCORE (ICE)REFERENCELink, citation, spec, documentationCATALOGDECISION"We decided X", "going with Y"RECORDHALF_THOUGHTFragment, incomplete, trails offCOMPLETE then re-classifyMODEL_OUTPUTAI-generated, assistant voiceEXTRACT core idea, discard wrapperDUPLICATESame as item #XMERGENOISETest, filler, meta-commentarySKIP

Effort per item:

• TRIVIAL (< 1 min): file rename, note capture, config change
• QUICK (1-5 min): web search, small edit, API call
• MEDIUM (5-30 min): build a page, write a doc, research topic
• HEAVY (30+ min): full app build, deep research, multi-step workflow
• BLOCKED: needs human input, credentials, or external dependency

Update manifest with Type + Effort columns.

Phase 3: TRIAGE#

Score each TASK and IDEA using quick ICE:

• I (Impact): 1-5 — how much does this move the needle?
• C (Cost): 1-5 — how cheap/fast to do? (inverted: 5 = trivial)
• E (Exploit): 1-5 — how quickly does this produce value?
• Score = I × C × E (max 125)

Sort by score descending. Group by dependency chains.

Create execution plan:

markdown

# Execution Plan

## Wave 1 (parallel, no dependencies)
- Item #14 (ICE: 100) — HEAVY → sub-agent
- Item #3 (ICE: 80) — MEDIUM → sub-agent
- Item #7 (ICE: 75) — QUICK → inline

## Wave 2 (depends on Wave 1)
- Item #9 (depends on #14) — MEDIUM → after #14 completes

## Skip (NOISE/DUPLICATE)
- Items #2, #5, #11 — reason: [...]

## Blocked (needs human)
- Item #8 — needs API key from Ryan

Phase 4: EXECUTE#

Rules:

1. Max 3 sub-agents concurrent. Wait for one to complete before spawning another.
2. QUICK items: execute inline (no sub-agent overhead for < 5 min tasks).
3. MEDIUM/HEAVY items: spawn sub-agent with clear task description + acceptance criteria.
4. Each sub-agent gets: the item content, relevant context from other items, and the target artifact path.
5. Track in manifest: status → EXECUTING, then ✅ DONE / ❌ FAILED / ⚠️ PARTIAL.

Sub-agent spawn template:

text

Task: [item summary]
Context: [relevant items from this corpus]
Deliverable: [specific file/artifact expected]
Acceptance: [how to verify it's done]
Workspace: [path]

Checkpoint every 5 completed items:

• Update manifest
• Report to user: "[X]/[N] done. [Y] in progress. Top findings so far: [...]"
• If user is idle (no response in 30s), continue
• Commit progress to git

Phase 5: RECONCILE#

After all waves complete (or all sub-agents return):

1. Re-read manifest
2. For each ❌ FAILED: log reason, decide retry or escalate
3. For each 🔄 sub-agent still running: check status, kill if stale (> 30 min no progress)
4. For each ⚠️ PARTIAL: note what's left
5. Retry failed items once (different approach if possible)

Phase 6: REPORT#

Generate final report at systems/batch-executor/reports/YYYY-MM-DD-SOURCE-report.md:

markdown

# Corpus Report: [source]
# Processed: [date]
# Total: [N] items
# Results: [done] ✅ | [failed] ❌ | [partial] ⚠️ | [skipped] ⏭️ | [blocked] 🔒

## Value Stack (top items by impact)
1. [item] — [outcome] — [next step]
2. ...

## Patterns Discovered
- [theme or connection across items]

## Action Items (immediate)
- [ ] [task from corpus that needs follow-up]

## Parked (valuable but not now)
- [item] — reason: [why later]

## Blocked (needs human)
- [item] — needs: [what]

## Statistics
- Items by type: TASK [x], IDEA [x], REFERENCE [x], NOISE [x]
- Items by effort: TRIVIAL [x], QUICK [x], MEDIUM [x], HEAVY [x]
- Sub-agents spawned: [x]
- Total execution time: [x min]

Append to systems/batch-cognition/value-stack.md (shared with batch-cognition skill). Log learnings to .learnings/LEARNINGS.md.

Commands#

status — show manifest progress pause — stop spawning, let running agents finish resume — continue from where we left off (re-read manifest) skip [#] — skip item number retry [#] — retry failed item block [#] [reason] — mark as blocked priority [#] — move item to top of queue done — trigger report even if items remain

Key Rules#

1. INGEST FIRST. Raw content hits disk before ANY processing.
2. Max 3 concurrent sub-agents. More = chaos, dropped results, context confusion.
3. Checkpoint every 5. Git commit progress. User update.
4. Never mark ✅ without artifact evidence. File exists, build passes, URL responds.
5. NOISE is not failure. Skipping noise is correct behavior. Report it transparently.
6. Corpus items cross-reference. Item #14 may be context for item #27. Pass relevant context to sub-agents.
7. Resume is first-class. If session dies, resume re-reads manifest and continues from last checkpoint.
8. ICE scoring is fast. 30 seconds per item max. Don't overthink triage — execute.

Integration with Other Skills#

• task-extractor: For inline chat messages (3-12 items). Batch-executor is for file/corpus scale (20+).
• batch-cognition: For idea analysis (THINK-heavy). Batch-executor is for execution (PLAY-heavy).
• orchestrator: Batch-executor can be invoked BY the orchestrator when it detects a corpus dump.
• recorder: After batch-executor completes, route to recorder to update STATUS.md.