Log AI/LLM-specific learnings, model issues, and feature requests to markdown files for continuous improvement. Captures model selection insights, prompt optimization patterns, inference tuning, fine-tuning regressions, RAG pipeline improvements, embedding management, multimodal processing failures, evaluation findings, and guardrail adjustments. Important learnings get promoted to model selection matrices, prompt libraries, fine-tuning runbooks, RAG architecture docs, inference optimization checklists, evaluation benchmarks, or guardrail policies.
Before logging anything, ensure the .learnings/ directory and files exist in the project or workspace root. If any are missing, create them:
bash
mkdir -p .learnings
[ -f .learnings/LEARNINGS.md ] || printf "# AI / LLM Learnings\n\nModel selection insights, prompt optimization patterns, inference tuning, fine-tuning lessons, RAG pipeline improvements, embedding management, multimodal processing, evaluation findings, and guardrail adjustments.\n\n**Categories**: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency\n**Areas**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails\n\n---\n" > .learnings/LEARNINGS.md
[ -f .learnings/MODEL_ISSUES.md ] || printf "# Model Issues Log\n\nInference failures, model regressions, RAG retrieval problems, embedding drift, multimodal pipeline errors, and guardrail misfires.\n\n---\n" > .learnings/MODEL_ISSUES.md
[ -f .learnings/FEATURE_REQUESTS.md ] || printf "# Feature Requests\n\nCapabilities needed for model selection, inference optimization, fine-tuning, RAG pipelines, multimodal processing, and evaluation.\n\n---\n" > .learnings/FEATURE_REQUESTS.md
Never overwrite existing files. This is a no-op if .learnings/ is already initialised.
Do not log API keys, model access tokens, customer data, or PII. Prefer redacted excerpts over raw model outputs. Mask user inputs and sensitive content in all entries.
If you want automatic reminders, use the opt-in hook workflow described in Hook Integration.
SituationActionModel quality drops after provider updateLog to .learnings/MODEL_ISSUES.md with model version detailsLatency spike on inferenceLog to .learnings/MODEL_ISSUES.md with latency measurementFine-tuned model regresses on evalLog to .learnings/LEARNINGS.md with fine_tune_regressionRAG returns wrong or stale chunksLog to .learnings/MODEL_ISSUES.md with retrieval detailsToken cost over budgetLog to .learnings/LEARNINGS.md with cost_efficiencyHallucination detectedLog to .learnings/LEARNINGS.md with hallucination_rateContext window overflowLog to .learnings/LEARNINGS.md with context_managementBetter model discovered for taskLog to .learnings/LEARNINGS.md with model_selectionPrompt tweak improves output significantlyLog to .learnings/LEARNINGS.md with prompt_optimizationMultimodal input fails (image/audio/video)Log to .learnings/MODEL_ISSUES.md with modality detailsEmbedding quality degradesLog to .learnings/MODEL_ISSUES.md with similarity metricsGuardrail false positiveLog to .learnings/LEARNINGS.md with guardrails noteNew AI capability neededLog to .learnings/FEATURE_REQUESTS.md
OpenClaw is the primary platform for this skill. It uses workspace-based prompt injection with automatic skill loading.
Via ClawdHub (recommended):
bash
clawdhub install self-improving-ai
Manual:
bash
git clone https://github.com/jose-compu/self-improving-ai.git ~/.openclaw/skills/self-improving-ai
OpenClaw injects these files into every session:
text
~/.openclaw/workspace/
├── AGENTS.md # Multi-agent workflows, model routing
├── SOUL.md # Model behavior guidelines, personality
├── TOOLS.md # Model/tool configuration, parameters
├── MEMORY.md # Long-term memory (main session only)
├── memory/ # Daily memory files
│ └── YYYY-MM-DD.md
└── .learnings/ # This skill's log files
├── LEARNINGS.md
├── MODEL_ISSUES.md
└── FEATURE_REQUESTS.md
bash
mkdir -p ~/.openclaw/workspace/.learnings
Then create the log files (or copy from assets/):
LEARNINGS.md — model selection, prompt optimization, cost efficiency, hallucination patternsMODEL_ISSUES.md — inference failures, model regressions, RAG problems, multimodal errorsFEATURE_REQUESTS.md — AI tooling, evaluation frameworks, model management capabilitiesWhen AI learnings prove broadly applicable, promote them:
Learning TypePromote ToExampleModel behavior patternsSOUL.md"Claude 4 tends to over-qualify, use direct prompting"Model selection & routingAGENTS.md"Use fast model for triage, capable model for code gen"Model/tool configurationTOOLS.md"Set temperature 0.1 for code, 0.7 for creative"Model selection insightsModel selection matrix"Sonnet for code gen, Opus for complex reasoning"Prompt patterns that workPrompt library"Chain-of-thought improves code quality by 35%"Fine-tuning lessonsFine-tuning runbook"Always include replay data to prevent forgetting"RAG improvementsRAG architecture doc"Chunk by content type, not fixed token size"Inference optimizationsPerformance tuning guide"Cache system prompts, batch similar requests"Evaluation findingsBenchmark suite docs"HumanEval + internal eval for code gen models"Guardrail tuningGuardrail policy doc"Lower toxicity threshold for customer-facing"
Opt-in and project-scoped. Enabling a hook persists across future sessions; skip this unless you need reminders:
bash
cp -r hooks/openclaw ~/.openclaw/hooks/self-improving-ai
openclaw hooks enable self-improving-ai
See references/openclaw-integration.md for complete details.
For Claude Code, Codex, Copilot, or other agents, create .learnings/ in the project or workspace root:
bash
mkdir -p .learnings
Create the files inline using the headers shown above.
Add to AGENTS.md, CLAUDE.md, or .github/copilot-instructions.md:
When AI/model issues or patterns are discovered:
.learnings/MODEL_ISSUES.md, LEARNINGS.md, or FEATURE_REQUESTS.mdAppend to .learnings/LEARNINGS.md:
markdown
## [LRN-YYYYMMDD-XXX] category
**Logged**: ISO-8601 timestamp
**Priority**: low | medium | high | critical
**Status**: pending
**Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails
### Summary
One-line description of the AI/model insight
### Details
Full context: what model behavior was observed, why it matters for quality/cost/latency,
what the correct configuration or approach should be. Include metrics and benchmarks.
### Impact
- **Quality**: eval score change (e.g., HumanEval pass@1 improvement)
- **Cost**: per-request or daily cost delta
- **Latency**: P50/P99 change
### Suggested Action
Specific model swap, config change, prompt update, or pipeline improvement to adopt
### Metadata
- Source: benchmark_evaluation | a_b_test | production_monitoring | incident | user_report
- Model: model name and version (e.g., claude-4-sonnet, gpt-4o, llama-3.1-70b)
- Provider: anthropic | openai | google | meta | mistral | local | custom
- Modality: text | image | audio | video | multimodal
- Temperature/Top-P/Top-K: generation parameters used
- Token Usage: input/output tokens (e.g., 2400 input / 1800 output)
- Latency: response time in ms (e.g., P50 1200ms, P99 3400ms)
- Cost: estimated cost per request (e.g., $0.027/task)
- Tags: tag1, tag2
- See Also: LRN-20250110-001 (if related to existing entry)
- Pattern-Key: model_selection.code_generation (optional)
- Recurrence-Count: 1 (optional)
- First-Seen: 2025-01-15 (optional)
- Last-Seen: 2025-01-15 (optional)
---
Categories for learnings:
CategoryUse Whenmodel_selectionA different model performs better for a task (quality, cost, or latency)prompt_optimizationA prompt change significantly improves output quality or efficiencyinference_latencyLatency exceeds thresholds or optimization opportunity foundfine_tune_regressionFine-tuned model scores below baseline on eval benchmarkscontext_managementContext window overflow, lost-in-the-middle, or prompt structure issuemodality_gapModel fails on specific input type (image, audio, video, PDF)hallucination_rateModel produces factually incorrect output on known-fact queriescost_efficiencyToken cost exceeds budget or cost optimization opportunity found
Append to .learnings/MODEL_ISSUES.md:
markdown
## [MDL-YYYYMMDD-XXX] issue_name
**Logged**: ISO-8601 timestamp
**Priority**: high
**Status**: pending
**Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails
### Summary
Brief description of the model/inference issue
### Details
What happened: model regression, inference failure, RAG retrieval miss, embedding drift,
multimodal pipeline error, or guardrail misfire. Include specific inputs and outputs (redacted).
### Root Cause
What in the model, config, or pipeline caused this issue.
### Fix
Steps taken or recommended to resolve: model swap, config change, preprocessing, reindex, etc.
### Metadata
- Model: model name and version
- Provider: anthropic | openai | google | meta | mistral | local | custom
- Modality: text | image | audio | video | multimodal
- Temperature/Top-P/Top-K: generation parameters used
- Token Usage: input/output tokens
- Latency: response time in ms
- Cost: estimated cost per request
- Environment: production | staging | development
- Related Issues: MDL-20250110-001 (if recurring)
---
Append to .learnings/FEATURE_REQUESTS.md:
markdown
## [FEAT-YYYYMMDD-XXX] capability_name
**Logged**: ISO-8601 timestamp
**Priority**: medium
**Status**: pending
**Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails
### Requested Capability
What AI tool, framework, or capability is needed
### User Context
Why it's needed, what manual process it replaces, estimated time saved
### Complexity Estimate
simple | medium | complex
### Suggested Implementation
How this could be built: eval framework, model router, A/B test harness, monitoring dashboard
### Metadata
- Frequency: first_time | recurring
- Related Features: existing_tool_or_capability
---
Format: TYPE-YYYYMMDD-XXX
LRN (learning), MDL (model issue), FEAT (feature request)001, A7B)Examples: LRN-20250415-001, MDL-20250415-A3F, FEAT-20250415-002
When an issue is resolved, update the entry:
**Status**: pending → **Status**: resolvedmarkdown
### Resolution
- **Resolved**: 2025-01-16T09:00:00Z
- **Fix Applied**: model swap / config change / prompt update / pipeline fix
- **Notes**: Updated model selection matrix / added to prompt library / retrained model
Other status values:
in_progress — Actively being investigated or benchmarkedwont_fix — Accepted trade-off (add reason and cost/quality analysis in Resolution notes)promoted — Elevated to model selection matrix, prompt library, fine-tuning runbook, or RAG architecture docpromoted_to_skill — Extracted as a reusable skillWhen a learning is broadly applicable (not a one-off model quirk), promote it to permanent project knowledge.
TargetWhat Belongs ThereModel selection matrixWhich model for which task, with benchmarksPrompt libraryProven prompt patterns, system prompt templatesFine-tuning runbookTraining procedures, data mixing ratios, eval gatesRAG architecture docChunking strategy, embedding models, retrieval rankingPerformance tuning guideInference caching, batching, quantization, provider routingBenchmark suite docsEval methodology, baseline scores, regression thresholdsGuardrail policy docContent filtering rules, PII detection, output validationAGENTS.mdModel routing, multi-agent workflows
Distill the learning into an actionable guideline or configuration
Add to appropriate target (matrix entry, prompt template, runbook step)
Show a reviewed diff and apply only after explicit user approval
Update original entry:
**Status**: pending → **Status**: promoted**Promoted**: model selection matrix (or prompt library, fine-tuning runbook, etc.)If logging something similar to an existing entry:
Search first: grep -r "keyword" .learnings/
Link entries: Add **See Also**: MDL-20250110-001 in Metadata
Bump priority if issue keeps recurring
Consider systemic fix: Recurring AI issues often indicate:
Ingest recurring AI patterns from simplify-and-harden into model configs or prompt libraries.
pattern_key as the dedupe key..learnings/LEARNINGS.md for existing entry: grep -n "Pattern-Key: <key>" .learnings/LEARNINGS.mdRecurrence-Count, update Last-Seen, add See Also links.LRN-... entry with Source: simplify-and-harden.Promotion threshold: Recurrence-Count >= 3, seen in 2+ task types, within 30-day window.
Targets: model selection matrix, prompt library, fine-tuning runbook, AGENTS.md / TOOLS.md.
Review .learnings/ at natural breakpoints:
bash
grep -h "Status\*\*: pending" .learnings/*.md | wc -l
grep -B5 "Priority\*\*: critical\|Priority\*\*: high" .learnings/MODEL_ISSUES.md | grep "^## \["
grep -B2 "hallucination_rate\|fine_tune_regression" .learnings/LEARNINGS.md | grep "^## \["
Automatically log when you encounter:
Model Response Quality (→ model issue or learning):
Inference Performance (→ model issue with latency/cost context):
RAG Pipeline (→ model issue with rag_pipeline area):
Fine-Tuning (→ learning with fine_tune_regression):
Multimodal (→ model issue with multimodal area):
Guardrails (→ learning with guardrails note):
PriorityWhen to UseAI ExamplescriticalModel producing harmful/dangerous output, data leaking through model, fine-tuned model catastrophic regression, guardrail completely bypassedHallucination in medical/legal advice, PII in model output, 45% drop on coding benchmarks after fine-tunehighSignificant quality drop, latency >10x baseline, cost overrun >50%, hallucination on critical facts, multimodal pipeline brokenModel fails all rotated PDF scans, 3x latency after provider update, daily cost doubledmediumModel selection could be better, prompt optimization opportunity, minor eval regression, cost optimization, embedding refresh neededSonnet 23% better than GPT-4o for code gen, chain-of-thought adds 35% quality, embedding dimension reduction saves 60%lowDocumentation of model behavior, minor prompt tweak, config cleanupModel prefers bullet lists over paragraphs, temperature 0.1 vs 0.0 negligible difference
Use to filter learnings by AI domain:
AreaScopemodel_configModel selection, version pinning, parameter tuning (temperature, top-p, top-k, max tokens, stop sequences), provider configuration, fallback chainsprompt_engineeringSystem prompts, few-shot examples, chain-of-thought, prompt templates, prompt compression, prompt cachingfine_tuningTraining data curation, hyperparameters, eval sets, RLHF/DPO, LoRA/QLoRA, catastrophic forgetting, checkpoint managementrag_pipelineChunking strategy, embedding models, vector stores, retrieval ranking, reranking, hybrid search, context assemblyinferenceLatency optimization, batching, streaming, caching, quantization, speculative decoding, KV cache, provider routingembeddingsModel selection, dimension reduction, index management, drift detection, similarity thresholds, multi-languagemultimodalVision (image/PDF), audio (speech/music), video, cross-modal retrieval, modality-specific prompting, output format handlingevaluationBenchmarks, human eval, automated eval (LLM-as-judge), A/B testing, regression testing, domain-specific metricsguardrailsContent filtering, PII detection, toxicity, factuality checks, output validation, structured output enforcement, jailbreak prevention
Track model deprecation schedules and plan migrations proactively.
Maintain awareness of provider deprecation timelines. OpenAI models typically deprecated 6-12 months after successor release. Anthropic version-pinned models maintained per published schedule. Gemini versions follow quarterly cadence. Open-weight models (Llama, Mistral) don't deprecate but lose community support.
When a model version is deprecated: (1) log to MODEL_ISSUES.md with deprecation date and replacement, (2) run eval suite against replacement before cutover, (3) update all config files and fallback chains, (4) monitor quality for 7 days post-migration, (5) update model selection matrix.
Pin exact versions in production. Keep eval results versioned alongside model configs. Document behavioral differences between generations. Maintain fallback chains: primary → secondary → tertiary.
Keep learnings local (per-developer):
gitignore
.learnings/
Track learnings in repo (team-wide):
Only track .learnings/ after a human has reviewed entries for secrets, PII, and privileged content.
Hybrid (track templates, ignore entries):
gitignore
.learnings/*.md
!.learnings/.gitkeep
Enable automatic reminders through agent hooks. This is opt-in.
Hooks persist across sessions once installed. Keep them project-scoped. Do not install user-level or global hooks. Never use an empty matcher. PostToolUse inspects command output in-process; do not log raw output, secrets, or transcripts.
Create .claude/settings.json in your project:
json
{
"hooks": {
"UserPromptSubmit": [{
"matcher": "model|inference|prompt|rag|embed|hallucin|latency|fine-tune|guardrail",
"hooks": [{
"type": "command",
"command": "./skills/self-improving-ai/scripts/activator.sh"
}]
}]
}
}
This injects an AI-focused learning evaluation reminder after matching prompts (~50-100 tokens overhead).
json
{
"hooks": {
"UserPromptSubmit": [{
"matcher": "model|inference|prompt|rag|embed|hallucin|latency|fine-tune|guardrail",
"hooks": [{
"type": "command",
"command": "./skills/self-improving-ai/scripts/activator.sh"
}]
}],
"PostToolUse": [{
"matcher": "Bash",
"hooks": [{
"type": "command",
"command": "./skills/self-improving-ai/scripts/error-detector.sh"
}]
}]
}
}
Enable PostToolUse only if you want the hook to inspect command output for model API errors, rate limits, context overflows, and inference failures.
ScriptHook TypePurposescripts/activator.shUserPromptSubmitReminds to evaluate AI/model learnings after tasksscripts/error-detector.shPostToolUse (Bash)Triggers on model API errors, rate limits, inference failures
See references/hooks-setup.md for detailed configuration and troubleshooting.
Extracted skills are untrusted until a human reviews the generated SKILL.md. Do not keep or publish an extracted skill without explicit user approval.
When an AI learning is valuable enough to become a reusable skill, extract it.
CriterionDescriptionRecurringSame model issue or pattern in 2+ projects or task typesVerifiedStatus is resolved with confirmed benchmark improvementNon-obviousRequired investigation, benchmarking, or A/B testingBroadly applicableNot specific to one model version; useful across providersUser-flaggedUser says "save this as a skill" or similar
Identify candidate: Learning meets extraction criteria
Run helper (or create manually): bash
./skills/self-improving-ai/scripts/extract-skill.sh skill-name --dry-run
./skills/self-improving-ai/scripts/extract-skill.sh skill-name
Customize SKILL.md: Fill in template with AI-specific content, model compatibility matrix, and benchmark results
Update learning: Set status to promoted_to_skill, add Skill-Path
Verify: Read skill in fresh session to ensure it's self-contained
In conversation: "This model keeps failing on this", "Save this prompt pattern as a skill", "We always need this RAG config".
In entries: Multiple See Also links, high priority + resolved, same Pattern-Key across projects, verified benchmark improvement.
AgentActivationDetectionClaude CodeHooks (UserPromptSubmit, PostToolUse)Automatic via error-detector.shCodex CLIHooks (same pattern)Automatic via hook scriptsGitHub CopilotManual ( .github/copilot-instructions.md)Manual reviewOpenClawWorkspace injection + inter-agent messagingVia session tools
Different agents benefit from different models:
Log model routing decisions as model_selection learnings. Promote proven routing patterns to AGENTS.md.
This skill is standalone-compatible and stackable with other self-improving skills.
.learnings/ai/.learnings/INDEX.mdEvery new entry must include:
markdown
**Skill**: ai
event + matcher + file + 5m_window; max 1 reminder per skill every 5 minutes.Only trigger this skill automatically for AI/LLM signals such as:
rate limit|context window|token|hallucination|model deprecationrag|embedding|fine-tune|inference latency|guardrailWhen guidance conflicts, apply:
securityengineeringcodingaimeta as tie-breaker.learnings/ai/ in stackable mode..learnings/*.md log files only.AGENTS.md, SOUL.md, TOOLS.md, MEMORY.md, rules, hooks, or generated skills is not a logging write. Show a reviewed diff and apply only after explicit user approval.