Batch File Processor#

Process large numbers of files in parallel using sub-agents, avoiding main agent context overflow.

Workflow#

1. List files#

bash

find <directory> -type f -name "*.md" | sort

2. Group#

Split into batches of 2-4 files each (3 is optimal).

3. Dispatch sub-agents#

One sub-agent per batch. Task template:

text

Read the following files completely and generate a brief summary (under 50 words) for each.
1. /path/to/file1.md
2. /path/to/file2.md
3. /path/to/file3.md
Return ONLY a JSON array:
[{"file": "relative/path/file1.md", "summary": "..."},...]

Key parameters:

• mode: "run" (one-shot task)
• runTimeoutSeconds: 120 (increase to 180 for large files)
• label: descriptive label, e.g. idx-project-batch1

4. Collect results#

Sub-agents push results on completion. Use sessions_yield to wait and collect incrementally.

5. Compile output#

Once all results are in, the main agent compiles the final deliverable (index file, report, etc.).

Rules#

• 2-4 files per sub-agent — never let one sub-agent process an entire directory sequentially
• Read full file content — no head/tail truncation; partial reads produce incomplete summaries
• Standardize output format — JSON makes it easy for the main agent to parse and merge
• One spawn per turn — system limitation; use multiple spawn + yield cycles

Anti-patterns#

MistakeConsequencehead -20 to skim file headersPoor summary quality, key information missedOne sub-agent processes entire directoryContext overflow, timeout failureMain agent reads all files sequentiallyContext window exhausted, later files unreadableOne sub-agent per large directoryLarge directories timeout, small ones waste capacity

Benchmarks#

70 files → 25 sub-agents (3 files each) → parallel execution → completed in 5 minutes → high accuracy summaries

Task Template Variants#

File summarization (default)#

text

Generate a brief summary (under 50 words) for each file.

Information extraction#

text

Extract the following fields from each file: project name, budget, key contacts, risks.
Return JSON: [{"file": "...", "project": "...", "budget": "...", "contacts": [...], "risks": [...]}]

Content classification#

text

Classify each file by checking for these topics: security, compliance, migration.
Return JSON: [{"file": "...", "has_security": true/false, "has_compliance": true/false, "has_migration": true/false}]

Code analysis#

text

Analyze each source file: count lines, list imports/dependencies, identify main functions.
Return JSON: [{"file": "...", "lines": N, "imports": [...], "main_functions": [...]}]