AI DevOps Toolkit — Observability for Local AI Fleets#

DevOps tooling for running local LLM inference at production quality. This DevOps skill provides the observability, tracing, and health monitoring layer for an Ollama Herd fleet. Every DevOps workflow — from request tracing to capacity planning — runs through a single SQLite-backed observability stack.

DevOps Prerequisites#

bash

pip install ollama-herd
herd              # start the DevOps router (exposes all DevOps observability endpoints)
herd-node         # start on each DevOps-monitored node

Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd

DevOps Scope#

This DevOps toolkit assumes you have an Ollama Herd router running at http://localhost:11435 with one or more node agents reporting in. It focuses on the DevOps operational side: are requests succeeding? what's slow? which apps consume the most tokens? are nodes healthy? is capacity adequate?

DevOps Observability Stack#

Everything in this DevOps observability layer is backed by SQLite at ~/.fleet-manager/latency.db. No external databases, no time-series infrastructure. Query DevOps traces with standard sqlite3.

text

~/.fleet-manager/
├── latency.db          # DevOps traces, latency history, usage stats
└── logs/
    └── herd.jsonl      # DevOps structured logs, daily rotation, 30-day retention

DevOps Health Checks#

Automated DevOps fleet health analysis#

bash

devops_health=$(curl -s http://localhost:11435/dashboard/api/health)
echo "$devops_health" | python3 -m json.tool

Fifteen DevOps checks, each returning a severity (info/warning/critical) and recommendation:

DevOps CheckWhat it detectsOffline nodesNodes that stopped sending heartbeatsDegraded nodesNodes reporting errors or high memory pressureMemory pressureNodes approaching memory limitsUnderutilized nodesHealthy nodes not receiving trafficVRAM fallbacksRequests rerouted to loaded alternatives to avoid cold loadsVersion mismatchNodes running different versions than the routerContext protectionnum_ctx values stripped or models upgraded to prevent reloadsZombie reaperStuck in-flight requests cleaned upModel thrashingModels loading/unloading frequently (memory contention)Request timeoutsRequests exceeding expected DevOps latency thresholdsError ratesElevated failure rates per model or per node

DevOps node-level status#

bash

devops_fleet_status=$(curl -s http://localhost:11435/fleet/status)
echo "$devops_fleet_status" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"DevOps Fleet: {d['fleet']['nodes_online']}/{d['fleet']['nodes_total']} online, {d['fleet']['requests_active']} active requests\")
for n in d['nodes']:
    mem = n.get('memory', {})
    cpu = n.get('cpu', {})
    print(f\"  {n['node_id']:20s} {n['status']:10s} CPU={cpu.get('utilization_pct',0):.0f}% MEM={mem.get('used_gb',0):.0f}/{mem.get('total_gb',0):.0f}GB pressure={mem.get('pressure','?')}\")
"

DevOps Request Tracing#

Every DevOps routing decision is recorded with full observability context.

Recent DevOps traces#

bash

devops_traces=$(curl -s "http://localhost:11435/dashboard/api/traces?limit=20")
echo "$devops_traces" | python3 -m json.tool

Each DevOps trace includes: request_id, model, original_model (before fallback), node_id, score, scores_breakdown (all 7 signals), status, latency_ms, time_to_first_token_ms, prompt_tokens, completion_tokens, retry_count, fallback_used, tags.

DevOps failure investigation#

bash

# Recent DevOps failures with error details
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, error_message, latency_ms/1000.0 as secs, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='failed' ORDER BY timestamp DESC LIMIT 20"

# DevOps retry frequency — which nodes need attention?
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, SUM(retry_count) as retries, COUNT(*) as total, ROUND(100.0 * SUM(CASE WHEN status='failed' THEN 1 ELSE 0 END) / COUNT(*), 1) as fail_pct FROM request_traces GROUP BY node_id ORDER BY fail_pct DESC"

# DevOps fallback frequency — which models are unreliable?
sqlite3 ~/.fleet-manager/latency.db "SELECT original_model, model as fell_back_to, COUNT(*) as n FROM request_traces WHERE fallback_used=1 GROUP BY original_model, model ORDER BY n DESC"

DevOps Latency Analysis#

bash

# DevOps P50/P75/P99 latency by model
sqlite3 ~/.fleet-manager/latency.db "
WITH ranked AS (
  SELECT model, latency_ms,
    PERCENT_RANK() OVER (PARTITION BY model ORDER BY latency_ms) as pct
  FROM request_traces WHERE status='completed'
)
SELECT model,
  ROUND(MIN(CASE WHEN pct >= 0.5 THEN latency_ms END)/1000.0, 1) as p50_s,
  ROUND(MIN(CASE WHEN pct >= 0.75 THEN latency_ms END)/1000.0, 1) as p75_s,
  ROUND(MIN(CASE WHEN pct >= 0.99 THEN latency_ms END)/1000.0, 1) as p99_s,
  COUNT(*) as n
FROM ranked GROUP BY model HAVING n > 10 ORDER BY p75_s DESC
"

# DevOps time-to-first-token observability (cold load detection)
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, model, ROUND(AVG(time_to_first_token_ms), 0) as avg_ttft_ms, ROUND(MAX(time_to_first_token_ms), 0) as max_ttft_ms, COUNT(*) as n FROM request_traces WHERE time_to_first_token_ms IS NOT NULL GROUP BY node_id, model HAVING n > 5 ORDER BY avg_ttft_ms DESC"

# DevOps outlier detection — slowest requests
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, ROUND(latency_ms/1000.0, 1) as secs, prompt_tokens, completion_tokens, retry_count, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='completed' ORDER BY latency_ms DESC LIMIT 10"

DevOps Per-Application Analytics#

Tag requests to track DevOps usage per application, team, or environment.

DevOps request tagging#

bash

# DevOps tag via request body
curl -s http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}],"metadata":{"tags":["devops-prod","devops-code-review"]}}'

# DevOps tag via header
curl -s -H "X-Herd-Tags: devops-prod, devops-code-review" \
  http://localhost:11435/v1/chat/completions \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}]}'

DevOps per-tag dashboards#

bash

curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool
curl -s http://localhost:11435/dashboard/api/apps/daily | python3 -m json.tool

DevOps token consumption by tag#

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT j.value as devops_tag, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)) as prompt_tok, SUM(COALESCE(completion_tokens,0)) as completion_tok, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as total_tok FROM request_traces, json_each(tags) j WHERE tags IS NOT NULL GROUP BY j.value ORDER BY total_tok DESC"

DevOps Traffic Patterns#

bash

# DevOps requests per hour (find peak load times)
sqlite3 ~/.fleet-manager/latency.db "SELECT CAST((timestamp % 86400) / 3600 AS INTEGER) as hour_utc, COUNT(*) as requests, ROUND(AVG(latency_ms)/1000.0, 1) as avg_secs FROM request_traces GROUP BY hour_utc ORDER BY hour_utc"

# DevOps daily request volume
sqlite3 ~/.fleet-manager/latency.db "SELECT date(timestamp, 'unixepoch') as day, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as tokens FROM request_traces GROUP BY day ORDER BY day DESC LIMIT 14"

DevOps Capacity Planning#

DevOps model recommendations per node#

bash

devops_recommendations=$(curl -s http://localhost:11435/dashboard/api/recommendations)
echo "$devops_recommendations" | python3 -m json.tool

Returns DevOps recommendations based on hardware capabilities, current usage, and curated benchmark data. Use for DevOps capacity planning: which models fit on which machines, and what's the optimal mix.

DevOps usage statistics#

bash

curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool

DevOps Configuration#

bash

# View all DevOps settings
curl -s http://localhost:11435/dashboard/api/settings | python3 -m json.tool

# Toggle DevOps runtime settings
curl -s -X POST http://localhost:11435/dashboard/api/settings \
  -H "Content-Type: application/json" \
  -d '{"auto_pull": false}'

DevOps Log Analysis#

Structured JSONL logs at ~/.fleet-manager/logs/herd.jsonl — the DevOps log layer:

bash

# Recent DevOps errors
grep '"level": "ERROR"' ~/.fleet-manager/logs/herd.jsonl | tail -10 | python3 -m json.tool

# DevOps context protection events
grep "Context protection" ~/.fleet-manager/logs/herd.jsonl | tail -10

# DevOps stream errors
grep "Stream error" ~/.fleet-manager/logs/herd.jsonl | tail -10

DevOps Dashboard#

Web dashboard at http://localhost:11435/dashboard. Key DevOps tabs:

• Trends — DevOps requests/hour, latency, token throughput over 24h–7d
• Apps — DevOps per-tag analytics with daily breakdowns
• Health — automated DevOps health checks with severity and recommendations
• Model Insights — per-model DevOps latency and throughput comparison

Guardrails#

• Never restart DevOps services without explicit user confirmation.
• Never delete or modify ~/.fleet-manager/ contents.
• Do not pull or delete models without user confirmation.
• Report DevOps issues to the user rather than attempting automated fixes.
• If the router isn't running, suggest herd or uv run herd.