Memory Engine 🧠⚡#

Persistent memory that survives session restarts — with three layers of protection against amnesia.

The Problem#

AI agents wake up blank every session. Existing solutions either:

• Need embedding APIs (cost money, need keys)
• Only search (useless if nothing was written)
• Rely on the AI "remembering" to save (it won't)

Memory Engine solves all three.

v5.0 — Memory Guardian Architecture#

v5.0 redefines Memory Engine's role: instead of a "full-stack memory system", it's now a memory guardian — handling everything OpenClaw native memory doesn't do.

FeatureNative memorySearchMemory EngineSemantic vector search✅ (embedding API)❌ (deferred to native)BM25 keyword search✅ (hybrid mode)✅ (FTS5 fallback)Auto-write daily logs❌✅ memory-write.jsSession reset detection❌✅ memory-watcher.sh (<30s)Active session extraction❌✅ --active modeHealth check + GC❌✅ memory-maintain.jsSession resume context❌✅ memory-resume.jsGitHub backup/restore❌✅ memory-backup.shZero-dep writing❌ (needs embedding API)✅ (pure Node.js)

Upgrade from v3/v4 (zero data loss)#

bash

node scripts/memory-migrate.js              # preview changes
node scripts/memory-migrate.js --apply      # apply (backs up config first)
node scripts/memory-migrate.js --rollback --apply   # revert if needed

The migrate tool:

1. Enables native memorySearch with hybrid search + temporal decay
2. Enables session-memory hook (saves session on /new and /reset)
3. Updates cron to 1h if still at 6h
4. Preserves ALL existing data — your memory files, index, daily logs untouched
5. FTS5 search remains as fallback if native isn't configured

No breaking changes: all v3/v4 commands still work. memory-search.js auto-detects native mode and adapts. memory-boot.js skips FTS5 indexing when native is active (saves time).

Three-Layer Anti-Amnesia Architecture#

text

Layer 1: SYSTEM (runs without AI)     Layer 2: PLATFORM (OpenClaw built-in)
┌─────────────────────────┐           ┌──────────────────────────┐
│ cron job (every 1h)     │           │ memory-flush             │
│ • rebuild search index  │           │ • auto-triggers before   │
│ • health check          │           │   context compaction     │
│ • auto-create daily log │           │ • forces AI to write     │
│   if missing            │           │   memory/YYYY-MM-DD.md   │
│ • active session extract│           │ • user-invisible         │
│ • ensure watcher alive  │           │                          │
│                         │           │ ⚙️ Built into OpenClaw,  │
│ 🔴 watcher daemon       │           │   just needs config      │
│ • polls every 30s       │           └──────────────────────────┘
│ • detects session reset │
│ • extracts memory <30s  │
│ • auto-started by cron  │
│                         │
│ ⚙️ Pure shell/node,     │
│   no AI involvement     │
└─────────────────────────┘
Layer 3: AGENT (AI calls these)
┌──────────────────────────────────────────────────┐
│ memory-write.js    → append daily log / MEMORY.md│
│ memory-search.js   → FTS5 search (native fallback)│
│ memory-index.js    → build/update FTS5 index     │
│ memory-maintain.js → stats / rebuild / prune     │
│ memory-migrate.js  → upgrade/rollback helper  NEW│
│                                                  │
│ 📋 Governed by AGENTS.md rules:                  │
│   Session start → health check + read MEMORY.md  │
│   During conversation → log key events           │
│   Session end → summarize + promote to MEMORY.md │
└──────────────────────────────────────────────────┘

Why three layers? Any single layer can fail:

• AI forgets to write → cron + memory-flush still capture data
• Cron stops → AI + memory-flush still work
• Memory-flush doesn't trigger (short session) → AI + cron still work

All three failing simultaneously = near zero probability.

Setup (2 minutes)#

Quick Setup (v5.0 — recommended for OpenClaw ≥ 2026.4)#

bash

# 1. Install cron (watcher + maintenance)
(crontab -l 2>/dev/null; echo "0 * * * * $(pwd)/scripts/memory-cron.sh") | crontab -

# 2. Auto-configure native search + hooks
node scripts/memory-migrate.js --apply

# 3. Restart gateway
openclaw gateway restart

That's it! Native memorySearch handles search, we handle everything else.

Classic Setup (FTS5 mode — for OpenClaw < 2026.4 or no embedding API)#

bash

# 1. Install FTS5 dependency
npm install -g better-sqlite3

# 2. Install cron job
(crontab -l 2>/dev/null; echo "0 * * * * $(pwd)/scripts/memory-cron.sh") | crontab -

Step 3: Enable memory-flush (add to openclaw.json)#

json

{
  "agents": {
    "defaults": {
      "compaction": {
        "memoryFlush": {
          "enabled": true,
          "softThresholdTokens": 4000,
          "prompt": "Pre-compaction memory flush. Store durable memories in memory/YYYY-MM-DD.md (create memory/ if needed). APPEND only, do not overwrite. Include: key decisions, task outcomes, user preferences, project status changes. Format each entry as: - HH:MM [tag] description. If nothing worth storing, reply with NO_REPLY."
        },
        "reserveTokensFloor": 20000,
        "postIndexSync": "async"
      },
      "heartbeat": {
        "every": "30m"
      }
    }
  }
}

⚠️ Important: Do NOT instruct memory-flush to write MEMORY.md or run exec commands. During flush, OpenClaw restricts writes to memory/YYYY-MM-DD.md only — MEMORY.md, SOUL.md, AGENTS.md are read-only. Exec/shell commands may also be restricted. The cron job handles index rebuilding independently.

Step 4: Add rules to AGENTS.md#

markdown

## Session Startup (MANDATORY)
1. Run: node skills/memory-engine/scripts/memory-boot.js
   (single command: health check + index update + MEMORY.md content)
2. Use memory-search.js for specific recall (NOT full file reads)

## During Conversation (MANDATORY)
- Key decisions: node skills/memory-engine/scripts/memory-write.js --today "decision" --tag decision
- Task completion: node skills/memory-engine/scripts/memory-write.js --today "done" --tag done

## Session End (MANDATORY)
1. node skills/memory-engine/scripts/memory-write.js --today "Session summary"
2. node skills/memory-engine/scripts/memory-write.js --core "durable fact" --section category
3. node skills/memory-engine/scripts/memory-index.js

Commands#

Migrate (upgrade/rollback) — NEW in v5.0#

bash

node scripts/memory-migrate.js              # preview: what would change
node scripts/memory-migrate.js --apply      # apply changes (backs up config)
node scripts/memory-migrate.js --rollback --apply   # revert to FTS5-only mode
node scripts/memory-migrate.js --status     # show current mode (JSON)

Enables native memorySearch, session-memory hook, optimal cron frequency. Config backup created automatically before changes. Zero data loss guaranteed.

Boot (session startup)#

bash

node scripts/memory-boot.js                     # health + index + MEMORY.md output

Replaces 3 separate calls ( --status + memory-index.js + read MEMORY.md) with one command. Auto-recovers from corrupted database. v5.0: auto-detects native mode, skips FTS5 when native active.

Write (most important!)#

bash

# Daily log (auto-creates memory/YYYY-MM-DD.md)
node scripts/memory-write.js --today "Deployed v2.0 to production"
node scripts/memory-write.js --today "User prefers dark mode" --tag preference

# Long-term memory (appends to MEMORY.md, capacity-managed — 5000 char soft cap)
node scripts/memory-write.js --core "Stack: Next.js + PostgreSQL + Redis"
node scripts/memory-write.js --core "Deploy via Docker Compose" --section infrastructure

# User profile (appends to USER.md — 1500 char soft cap)
node scripts/memory-write.js --user "Prefers concise answers, dislikes verbose explanations"

# Snapshot MEMORY.md (before risky operations)
node scripts/memory-write.js --snapshot

# Health check (now includes capacity % for MEMORY.md and USER.md)
node scripts/memory-write.js --status

Health output:

json

{
  "hasTodayLog": true,
  "hasCoreMemory": true,
  "coreMemory": { "chars": 2651, "cap": 5000, "usage": "53%" },
  "userProfile": { "chars": 380, "cap": 1500, "usage": "25%" },
  "snapshots": 3,
  "gapCount": 2,
  "healthScore": 71,
  "warnings": ["2 gaps in last 14 days"]
}

Resume (zero-latency session recovery) — Enhanced in v4.0#

bash

node scripts/memory-resume.js                    # generate recovery summary (<2000 tokens)
node scripts/memory-resume.js --max-chars 1000   # shorter for tight budgets

Use in AGENTS.md as the first session startup step. Outputs:

• Last session's final topic & status
• Recent 3 days' key entries (priority-sorted by tag)
• Active MEMORY.md sections summary
• NEW v4.0: Auto-detects unextracted reset sessions and extracts them before resuming (eliminates the "resume misses recent work" gap)

Search (0 tokens, ~300 token results)#

bash

node scripts/memory-search.js "deployment plan"          # top 3, 200 chars
node scripts/memory-search.js "API重构" --json --max 5   # CJK support
node scripts/memory-search.js "query" --max 1 --max-chars 100  # ultra-minimal

# Date filtering (auto-detects "4月2日" / "April 2" in queries)
node scripts/memory-search.js "4月2日完成了什么"          # auto date filter
node scripts/memory-search.js "tasks" --date 2026-04-02  # exact date
node scripts/memory-search.js "progress" --recent 7      # last 7 days
node scripts/memory-search.js "bugs" --after 2026-04-01 --before 2026-04-08

# Last N entries (no query needed, time-ordered) — NEW in v3.0
node scripts/memory-search.js --last 5                   # last 5 entries across all logs
node scripts/memory-search.js --last 10 --today          # today's last 10
node scripts/memory-search.js --last 3 --tag done        # last 3 [done] entries
node scripts/memory-search.js --last 5 --tag decision    # last 5 decisions

Index#

bash

node scripts/memory-index.js          # incremental (skips unchanged)
node scripts/memory-index.js --force  # full rebuild

Maintain#

bash

node scripts/memory-maintain.js              # show stats
node scripts/memory-maintain.js --reindex    # force rebuild
node scripts/memory-maintain.js --prune-days 90  # trim old entries

# GC: detect & remove stale MEMORY.md entries — NEW in v3.0
node scripts/memory-maintain.js --gc             # preview stale entries
node scripts/memory-maintain.js --gc --apply     # remove them

GC detects: old version references (>14d), completed TODOs, entries >60d old, duplicates.

Consolidate (capacity management) — NEW in v6.0#

bash

node scripts/memory-maintain.js --consolidate          # analyze merge candidates
node scripts/memory-maintain.js --consolidate --apply  # remove duplicates (snapshots first)

When MEMORY.md exceeds 70% of the 5000-char soft cap, use this to find and merge duplicate entries. Auto-snapshots MEMORY.md before any destructive operation. For deeper consolidation (merging semantically similar entries), manually edit MEMORY.md.

Auto-Extract (session transcript mining) — Enhanced in v4.0#

bash

node scripts/memory-auto-extract.js                # extract from active session (full scan)
node scripts/memory-auto-extract.js <path.jsonl>   # extract from specific file
node scripts/memory-auto-extract.js --scan          # scan all unprocessed reset sessions
node scripts/memory-auto-extract.js --active        # incremental extract from active sessions (NEW v4.0)
node scripts/memory-auto-extract.js --dry-run       # preview without writing

Extracts: git commits, file writes, deployments, PM2 restarts, user instructions, test results, API responses, config changes, version releases, bug fixes (NEW v4.0). --active mode reads only new bytes since last extraction (offset-based), safe for frequent runs. Integrated into cron: auto-scans reset sessions + active sessions every 1h.

Watcher (real-time session reset detection) — NEW in v4.0#

bash

bash scripts/memory-watcher.sh          # start daemon (polls every 30s)
bash scripts/memory-watcher.sh --once   # single check, no loop

• Detects new .reset. files within 30 seconds
• Immediately extracts memory + rebuilds index
• Auto-started by cron (ensures watcher is always running)
• PID managed via .memory/watcher.pid
• Solves the #1 failure mode: session reset → instant amnesia

Compact (compress old logs)#

bash

node scripts/memory-compact.js --stats                  # show candidates
node scripts/memory-compact.js --older-than 30 --dry-run  # preview
node scripts/memory-compact.js --older-than 30          # execute (originals → memory/archive/)

Compaction extracts headings + key bullets (✅/🔴/重要), saves originals to memory/archive/. Typical savings: 60-70%.

How Search Works#

1. Chunking: ~300 chars per chunk, 60 char overlap, split at headings
2. Indexing: SQLite FTS5 full-text index + file metadata
3. Search: BM25 + phrase matching for English; TF-density bigram for CJK
4. Date filtering: Auto-detects dates in queries (4月2日, April 2, 2026-04-02)
5. Ranking: TF_density × coverage × core_boost(1.5×) × temporal_decay(30-day half-life)
6. Output: truncated snippets with file#line citations

Token Budget#

OperationTokensSearch (SQLite)0Results (3 × 200 chars)300Read full daily file2000Read 4 daily files~8000Savings****~95%

Backup & Disaster Recovery#

Auto-backup (recommended)#

Memory-cron automatically pushes to GitHub every 6 hours (if git remote is configured):

bash

# One-time setup:
cd ~/.openclaw/workspace
git init && git branch -M main
git remote add origin https://<token>@github.com/<user>/openclaw-workspace.git
bash skills/memory-engine/scripts/memory-backup.sh   # first push
# After this, cron handles it automatically every 6h

Restore after reinstall#

bash

# After fresh OpenClaw install:
bash memory-restore.sh https://<token>@github.com/<user>/openclaw-workspace.git
# Restores: memories, config, skills, crontab, rebuilds search index

What's backed up#

• MEMORY.md + memory/*.md (all memories — irreplaceable)
• SOUL.md, USER.md, IDENTITY.md (personality)
• AGENTS.md, TOOLS.md, HEARTBEAT.md (behavior rules)
• skills/ (all installed skills)
• openclaw.json → backup copy
• crontab → backup copy

What's NOT backed up (rebuilt automatically)#

• .memory/index.sqlite (rebuilt from md files by restore script)
• .memory/cron.log (transient)

Requirements#

v5.0 Native mode (recommended)#

• Node.js 18+
• OpenClaw ≥ 2026.4 (for native memorySearch + session-memory hook)
• better-sqlite3 optional (only needed for FTS5 fallback)
• No API keys required (OpenClaw auto-detects embedding providers)

Classic FTS5 mode#

• Node.js 18+
• better-sqlite3 (for search/index)
• Write + health check: zero dependencies
• No API keys, no cloud, no Docker