Ubuntu AI — Build a Local AI Platform on Ubuntu#

Build a complete local AI platform on Ubuntu. LLM inference, image generation, and embeddings across desktops, servers, Raspberry Pis, and Jetson boards. Ubuntu AI supports x86-64 and ARM in the same cluster — heterogeneous hardware, one endpoint. No cloud APIs, no subscriptions.

Why Ubuntu AI#

• Largest Linux ecosystem — Ubuntu is the #1 Linux distribution for AI/ML workloads
• apt-get native — install dependencies with the Ubuntu package manager you know
• systemd integration — Ubuntu AI starts on boot, restarts on failure
• NVIDIA CUDA on Ubuntu — best GPU driver support on any Linux distribution
• ARM + x86 — Ubuntu AI runs on x86-64 servers and ARM devices (Raspberry Pi, Jetson)
• Free forever — Ubuntu is free, Ollama is free, Ollama Herd is free

Ubuntu AI quick start#

bash

# Ubuntu prerequisites
sudo apt update
sudo apt install python3-pip curl

# Install Ollama on Ubuntu
curl -fsSL https://ollama.ai/install.sh | sh

# Install Ubuntu AI router
pip install ollama-herd

# Start Ubuntu AI
herd          # Ubuntu AI router on port 11435
herd-node     # register this Ubuntu machine

On other Ubuntu/Debian machines:

bash

pip install ollama-herd
herd-node     # auto-discovers the Ubuntu AI router

Ubuntu AI with NVIDIA CUDA#

bash

# Install NVIDIA drivers on Ubuntu
sudo apt install nvidia-driver-550
sudo reboot

# Verify Ubuntu NVIDIA CUDA
nvidia-smi

# Ollama on Ubuntu automatically detects NVIDIA CUDA
ollama ps    # should show GPU acceleration

Ubuntu AI systemd services#

bash

# Ubuntu AI router service
sudo tee /etc/systemd/system/herd-router.service << 'EOF'
[Unit]
Description=Ubuntu AI Router
After=network.target ollama.service

[Service]
Type=simple
ExecStart=/usr/local/bin/herd
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now herd-router

bash

# Ubuntu AI node service
sudo tee /etc/systemd/system/herd-node.service << 'EOF'
[Unit]
Description=Ubuntu AI Node
After=network.target ollama.service

[Service]
Type=simple
ExecStart=/usr/local/bin/herd-node
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now herd-node

Use Ubuntu AI#

OpenAI SDK#

python

from openai import OpenAI

# Your Ubuntu AI endpoint
client = OpenAI(base_url="http://localhost:11435/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="qwen3.5:32b",
    messages=[{"role": "user", "content": "Write an Ubuntu systemd timer for daily backups"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

curl#

bash

# Ubuntu AI inference
curl http://localhost:11435/api/chat -d '{
  "model": "llama3.3:70b",
  "messages": [{"role": "user", "content": "Explain Ubuntu package management"}],
  "stream": false
}'

Ubuntu AI hardware guide#

Ubuntu DeviceGPUBest Ubuntu AI modelsUbuntu desktop (RTX 4090)24GBllama3.3:70b, qwen3.5:32bUbuntu desktop (RTX 4080)16GBphi4, codestral, qwen3.5:14bUbuntu Server (A100)80GBdeepseek-v3, qwen3.5:72bUbuntu Server (no GPU)CPUphi4-mini, gemma3:4bRaspberry Pi 5 (Ubuntu)CPUgemma3:1b, phi4-mini — edge Ubuntu AIJetson Orin Nano (Ubuntu)8GB sharedphi4, llama3.2:3b — ARM Ubuntu AI

Ubuntu AI supports x86-64 and ARM architectures. Heterogeneous Ubuntu AI clusters work automatically.

Ubuntu AI environment#

bash

# Optimize Ollama on Ubuntu
sudo systemctl edit ollama
# Add under [Service]:
#   Environment="OLLAMA_KEEP_ALIVE=-1"
#   Environment="OLLAMA_MAX_LOADED_MODELS=-1"
sudo systemctl restart ollama

Ubuntu AI firewall#

bash

# Ubuntu UFW
sudo ufw allow 11435/tcp
sudo ufw reload

Monitor Ubuntu AI#

bash

# Ubuntu AI fleet status
curl -s http://localhost:11435/fleet/status | python3 -m json.tool

# Ubuntu AI health — 15 automated checks
curl -s http://localhost:11435/dashboard/api/health | python3 -m json.tool

# Ubuntu AI logs
journalctl -u herd-router -f

Dashboard at http://localhost:11435/dashboard — live Ubuntu AI monitoring.

Also available on Ubuntu AI#

Image generation#

bash

curl http://localhost:11435/api/generate-image \
  -d '{"model": "z-image-turbo", "prompt": "Ubuntu penguin mascot", "width": 1024, "height": 1024}'

Embeddings#

bash

curl http://localhost:11435/api/embed \
  -d '{"model": "nomic-embed-text", "input": "Ubuntu AI local inference Debian"}'

Full documentation#

• Agent Setup Guide
• API Reference

Contribute#

Ollama Herd is open source (MIT). Ubuntu AI users welcome:

• Star on GitHub
• Open an issue

Guardrails#

• Ubuntu AI model downloads require explicit user confirmation.
• Ubuntu AI model deletion requires explicit user confirmation.
• Never delete or modify files in ~/.fleet-manager/.
• No models are downloaded automatically — all pulls are user-initiated or require opt-in.