GPU Cluster Manager#

You are managing a GPU cluster that combines multiple machines into one inference endpoint for running local LLMs via Ollama. The GPU cluster routes every request to the best available device automatically.

What this GPU cluster solves#

Your desktop, laptop, and maybe an old Linux box all have GPUs sitting idle most of the time. You want one GPU cluster URL that uses all of them — without Kubernetes, without Docker, without editing config files. Just point your AI apps at the GPU cluster endpoint and let the cluster figure out which machine should handle each request.

This GPU cluster manager does exactly that. Install it, run two commands, and your GPU cluster machines discover each other automatically. The GPU cluster learns when your devices are free, pauses during video calls, and picks the best GPU cluster node for every request based on real-time conditions.

Getting started with the GPU cluster#

bash

pip install ollama-herd    # GPU cluster manager from PyPI

On your main GPU cluster machine (the router):

bash

herd    # starts GPU cluster router

On each other GPU cluster machine:

bash

herd-node    # joins the GPU cluster automatically

That's it. The GPU cluster nodes find the router via mDNS. No config files. Your GPU cluster is running.

If mDNS doesn't work on your GPU cluster network: herd-node --router-url http://router-ip:11435

GPU Cluster Endpoint#

Your GPU cluster runs at http://localhost:11435. Point any AI app at the GPU cluster:

python

from openai import OpenAI
# GPU cluster client
gpu_cluster_client = OpenAI(base_url="http://localhost:11435/v1", api_key="not-needed")
gpu_cluster_response = gpu_cluster_client.chat.completions.create(
    model="llama3.3:70b",
    messages=[{"role": "user", "content": "Explain GPU cluster routing for AI inference"}]
)

Works with: LangChain, CrewAI, AutoGen, LlamaIndex, Aider, Cline, Continue.dev, and any OpenAI-compatible client pointing at the GPU cluster.

GPU Cluster Smart Features#

• GPU cluster auto-discovery — machines find each other via mDNS, no config
• 7-signal GPU cluster scoring — picks the best machine based on loaded models, memory, queue depth, latency, and more
• GPU cluster meeting detection — pauses inference when your camera/mic is active (macOS)
• GPU cluster capacity learning — learns your weekly patterns (168-hour behavioral model)
• GPU cluster context protection — prevents models from reloading when apps send different context sizes
• GPU cluster auto-pull — if you request a model that doesn't exist, it downloads to the best GPU cluster node
• GPU cluster auto-retry — if a machine hiccups, retries on the next-best GPU cluster node

Check your GPU cluster#

GPU cluster status — all machines#

bash

curl -s http://localhost:11435/fleet/status | python3 -m json.tool

What models are available on the GPU cluster?#

bash

curl -s http://localhost:11435/api/tags | python3 -m json.tool

What's loaded in GPU cluster memory right now?#

bash

curl -s http://localhost:11435/api/ps | python3 -m json.tool

How healthy is the GPU cluster?#

bash

curl -s http://localhost:11435/dashboard/api/health | python3 -m json.tool

GPU cluster model recommendations#

bash

curl -s http://localhost:11435/dashboard/api/recommendations | python3 -m json.tool

Returns GPU cluster recommendations based on your hardware — which models fit, which are too big, and the optimal GPU cluster mix.

GPU cluster recent activity#

bash

curl -s "http://localhost:11435/dashboard/api/traces?limit=10" | python3 -m json.tool

GPU cluster usage stats#

bash

curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool

GPU cluster settings#

bash

curl -s http://localhost:11435/dashboard/api/settings | python3 -m json.tool

curl -s -X POST http://localhost:11435/dashboard/api/settings \
  -H "Content-Type: application/json" \
  -d '{"auto_pull": false}'

Manage GPU cluster models#

bash

# What's on each GPU cluster node
curl -s http://localhost:11435/dashboard/api/model-management | python3 -m json.tool

# Download a model to a specific GPU cluster node
curl -s -X POST http://localhost:11435/dashboard/api/pull \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.3:70b", "node_id": "gpu-cluster-studio"}'

# Remove a model from a GPU cluster node
curl -s -X POST http://localhost:11435/dashboard/api/delete \
  -H "Content-Type: application/json" \
  -d '{"model": "old-model:7b", "node_id": "gpu-cluster-studio"}'

GPU cluster per-app tracking#

bash

curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool

Tag your GPU cluster requests to see which apps use the most time:

bash

curl -s http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Summarize GPU cluster utilization"}],"metadata":{"tags":["gpu-cluster-app"]}}'

GPU Cluster Dashboard#

Open http://localhost:11435/dashboard for a visual GPU cluster overview. Eight tabs: Fleet Overview (live GPU cluster node cards), Trends (charts), Model Insights (performance comparison), Apps (per-app usage), Benchmarks, Health (automated GPU cluster checks), Recommendations (what models to run), Settings.

Try the GPU cluster#

bash

# Quick GPU cluster test
curl -s http://localhost:11435/api/chat \
  -d '{"model":"llama3.2:3b","messages":[{"role":"user","content":"Hello from the GPU cluster!"}],"stream":false}'

GPU Cluster Troubleshooting#

Check what's slow in the GPU cluster#

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT model, node_id, AVG(latency_ms)/1000.0 as avg_secs, COUNT(*) as n FROM request_traces WHERE status='completed' GROUP BY node_id, model HAVING n > 5 ORDER BY avg_secs DESC LIMIT 10"

See GPU cluster failures#

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, status, error_message, latency_ms/1000.0 as secs FROM request_traces WHERE status='failed' ORDER BY timestamp DESC LIMIT 10"

GPU Cluster Guardrails#

• Never restart or stop the GPU cluster without explicit user confirmation.
• Never delete or modify files in ~/.fleet-manager/ (contains all your GPU cluster data and logs).
• Do not pull or delete models on the GPU cluster without user confirmation — downloads can be 10-100+ GB.
• If a GPU cluster machine shows as offline, report it rather than attempting to SSH into it.

GPU Cluster Failure Handling#

• Connection refused → GPU cluster router may not be running, suggest herd or uv run herd
• 0 nodes online → suggest starting herd-node on GPU cluster devices
• mDNS discovery fails → use --router-url http://router-ip:11435
• GPU cluster requests hang → check for num_ctx in client requests; context protection handles it
• GPU cluster errors → check ~/.fleet-manager/logs/herd.jsonl