Pixeltable

What is Pixeltable?#

Open-source Python library for declarative multimodal AI data infrastructure — tables, computed columns, embedding indexes, and 25+ AI providers in one interface.

pip install pixeltable (Python >= 3.10) | Docs | GitHub

STOP — Do NOT Use These With Pixeltable#

Pixeltable replaces the entire imperative stack you're about to reach for. If you find yourself importing any of these, you are off-path:

1. Do NOT use LangChain / LlamaIndex / Haystack / LangGraph — Pixeltable has built-in document chunking ( document_splitter), embedding indexes ( add_embedding_index), retrieval ( .similarity()), and tool-calling ( pxt.tools() + invoke_tools()). There is zero reason to add a framework on top.
2. Do NOT use pandas as a working data store — Pixeltable tables ARE the store. Use .collect().to_pandas() only for final export/visualization, never as an intermediate.
3. Do NOT write for row in ...: loops calling AI models — wrap the call in a computed column. Pixeltable handles batching, rate limiting, caching, and retry automatically.
4. Do NOT install a separate vector database (Pinecone, Chroma, FAISS, Qdrant, Weaviate, pgvector) — call t.add_embedding_index(col, embedding=fn) and search with .similarity(). The index lives inside the table.
5. Do NOT write while not done: agent loops — model the agent as a table where inserting a row triggers the entire computed-column chain (LLM → tool selection → tool execution → final answer) declaratively.

See anti-patterns.md for the full 15-bias reference with wrong/right code examples.

---

Task Router#

Jump to the right section based on what you're building:

If the user wants to...ReadCreate tables, insert data, queryCore Concepts (below) and core-api.mdAdd AI-powered columns (summarize, classify, embed)Computed Columns (below) and providers.mdChunk documents, extract video frames, split audioViews and Iterators (below) and core-api.md → ViewsBuild semantic search / embedding indexesEmbedding Indexes (below) and core-api.md → Embedding IndexesBuild a RAG pipelineworkflows.md → RAG PipelineBuild a tool-calling agentTool-Calling Agent Pipeline (below) and workflows.md → Tool-Calling AgentBuild an agent with persistent memoryagents-memory-mcp.md — chat history, knowledge bank, user scopingUse MCP tools with an agentagents-memory-mcp.md → Adding MCP ToolsUse invoke_tools() with OpenAI, Groq, Gemini, Bedrockagents-memory-mcp.md → Multi-ProviderBuild a video RAG agent (video + search + agent)video-rag-agents.md — dedicated combined recipeProcess video (frames, transcription, visual search)workflows.md → Video Analysis PipelineProcess images (classify, tag, search)workflows.md → Image Classification and SearchProcess audio (transcribe, summarize)workflows.md → Audio TranscriptionWrangle data for ML training (label, version, export)ml-data-pipeline.md — ingest, enrich, snapshot, PyTorch exportExport to PyTorch, Parquet, or pandasml-data-pipeline.md → Export for TrainingLook up structured data with retrieval_udfml-data-pipeline.md → Retrieval UDFsRetry failed computed columnsError Handling (below) — recompute_columns()Use agentic patterns (chaining, routing, parallelization, eval-optimize)agentic-patterns.md — 6 patterns + 2 reasoning strategiesRun batch processing (ingest, compute, export, exit)workflows.md → Batch ProcessingConfigure rate limits, media storage, API keyscore-api.md → ConfigurationExport to CSV, JSON, Parquet, LanceDBcore-api.md → ExportExport to SQL databases (Postgres, Snowflake, SQLite)core-api.md → Export to SQLShare tables across teams ( publish, replicate)core-api.md → Data SharingCompare multiple AI providersworkflows.md → Multi-Provider ComparisonBuild a FastAPI web app (hand-written endpoints)workflows.md → FastAPI App PatternServe tables/queries via FastAPIRouter (v0.6+)workflows.md → FastAPIRouter and core-api.md → ServingServe via CLI ( pxt serve + TOML config)core-api.md → pxt serveStore media in Pixeltable Cloud ( pxtfs://)core-api.md → Media DestinationsWrite UDFs or query functionsUDFs / Query Functions (below) and core-api.md → UDFsUse pxt.tools() and invoke_tools() for agentsTool-Calling Agent Pipeline (below) and core-api.md → Tools and AgentsAvoid common mistakes (wrong imports, broken schemas, serialization)Common Pitfalls (below) and core-api.md → Common PitfallsUnderstand what NOT to use with Pixeltable (LangChain, pandas, vector DBs)anti-patterns.md — 15 training-distribution biases with wrong/right codeLook up a specific provider's import and output shapeproviders.md → Quick Reference

Critical Warnings — Read Before Writing Code#

1. openai.vision does not exist — use openai.chat_completions with image_url content blocks
2. Cast to pxt.String before embedding — use .text.astype(pxt.String) on AI function outputs before add_embedding_index
3. if_exists='ignore' won't fix bugs — if a computed column has wrong logic, you must drop_column() then recreate; re-running is a silent no-op
4. Import frame_iterator as a function — from pixeltable.functions.video import frame_iterator, NOT from pixeltable.iterators import FrameIterator
5. Use string= keyword in similarity — always t.col.similarity(string=query), not positional

See Common Pitfalls below for full details and code examples.

Starting a New Project#

Scaffold a complete Pixeltable project from the Starter Kit in one command:

bash

# Application templates (each builds on a structural pattern)
uvx pixeltable-new --template knowledge-base my-kb             # web UI + API
uvx pixeltable-new --template chat-agent my-agent              # web UI + API
uvx pixeltable-new --template audio-transcription my-podcast   # web UI + API
uvx pixeltable-new --template full-stack-showcase my-sitewatch # web UI + API (complete reference app)
uvx pixeltable-new --template video-search my-video-app        # API only
uvx pixeltable-new --template media-indexing my-pipe           # API + batch
uvx pixeltable-new --template image-dataset my-dataset         # API + batch

# Structural patterns (API/pipeline scaffolds)
uvx pixeltable-new myapp                # default: declarative serving pattern
uvx pixeltable-new myapp --backend      # FastAPI API scaffold (headless)
uvx pixeltable-new myapp --batch        # batch processing script with export_sql

# Discovery
uvx pixeltable-new --list               # show all patterns + templates

Each template builds on one of the three structural patterns (serving, backend, batch), so you already know how to run and deploy it.

Core Concepts#

Tables and Column Types#

python

import pixeltable as pxt

pxt.create_dir('my_project', if_exists='ignore')

t = pxt.create_table('my_project.documents', {
    'title': pxt.String,
    'content': pxt.String,
    'image': pxt.Image,
    'video': pxt.Video,
    'audio': pxt.Audio,
    'doc': pxt.Document,
    'metadata': pxt.Json,
    'score': pxt.Float,
    'count': pxt.Int,
    'is_active': pxt.Bool,
    'created_at': pxt.Timestamp,
}, if_exists='ignore')

Available types: String, Int, Float, Bool, Image, Video, Audio, Document, Json, Array, Timestamp, Date, UUID, Binary. Use pxt.Required[pxt.String] for non-nullable.

Tables with Auto-Generated Keys#

Use uuid7() for auto-generated primary keys (recommended for production):

python

from pixeltable.functions.uuid import uuid7

t = pxt.create_table('my_project.items', {
    'content': pxt.String,
    'uuid': uuid7(),           # auto-generated on insert
    'timestamp': pxt.Timestamp,
}, primary_key=['uuid'], if_exists='ignore')

Inserting Data#

python

t.insert([{'title': 'Doc 1', 'content': 'Hello world', 'score': 0.95}])  # list of dicts
t.insert(title='Doc 2', content='Single row', score=0.75)                 # keyword syntax
t.insert(source='path/to/data.csv')                                       # from file

Computed Columns#

Auto-run on insert. Chain AI providers, UDFs, or expressions:

python

from pixeltable.functions.openai import chat_completions

t.add_computed_column(
    summary=chat_completions(
        messages=[{'role': 'user', 'content': t.content}],
        model='gpt-4o-mini'
    ).choices[0].message.content,
    if_exists='ignore'
)

t.add_computed_column(upper_title=t.title.upper(), if_exists='ignore')

Querying#

python

results = t.select(t.title, t.score).collect()
results = t.where(t.score > 0.8).select(t.title, t.content).collect()
results = t.order_by(t.score, asc=False).limit(10).select(t.title).collect()
count = t.count()
df = t.select(t.title, t.score).collect().to_pandas()
items = list(t.select(title=t.title, score=t.score).collect().to_pydantic(MyModel))

Views and Iterators#

Split rows into sub-rows (chunking, frame extraction, audio splitting):

python

from pixeltable.functions.document import document_splitter
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.string import string_splitter
from pixeltable.functions.audio import audio_splitter

# Chunk documents into 300-token pieces (requires: pip install tiktoken)
chunks = pxt.create_view(
    'my_project.doc_chunks', t,
    iterator=document_splitter(t.doc, separators='token_limit', limit=300),
    if_exists='ignore'
)

# Extract video frames at 1 fps
frames = pxt.create_view(
    'my_project.video_frames', t,
    iterator=frame_iterator(t.video, fps=1.0),
    if_exists='ignore'
)

# Split text into sentences
sentences = pxt.create_view(
    'my_project.sentences', t,
    iterator=string_splitter(t.content, separators='sentence'),
    if_exists='ignore'
)

# Split audio into 30-second chunks
audio_chunks = pxt.create_view(
    'my_project.audio_chunks', t,
    iterator=audio_splitter(audio=t.audio, duration=30.0),
    if_exists='ignore'
)

# Filtered view (no iterator needed)
active = pxt.create_view(
    'my_project.active', t.where(t.is_active == True),
    if_exists='ignore'
)

Embedding Indexes and Similarity Search#

python

from pixeltable.functions.huggingface import clip, sentence_transformer

embed_fn = clip.using(model_id='openai/clip-vit-base-patch32')
t.add_embedding_index('content', embedding=embed_fn, if_exists='ignore')

# Search
sim = t.content.similarity(string='search query')
results = t.order_by(sim, asc=False).limit(5).select(t.title, t.content, sim).collect()

# Image search with text (multimodal CLIP)
sim = t.image.similarity(string='a photo of a cat')
results = t.order_by(sim, asc=False).limit(5).select(t.image, sim).collect()

Built-in Image and Video Functions#

python

from pixeltable.functions import image as pxt_image
from pixeltable.functions.video import extract_audio

# Image thumbnails and encoding
t.add_computed_column(
    thumbnail=pxt_image.b64_encode(
        pxt_image.thumbnail(t.image, size=(320, 320))
    ),
    if_exists='ignore'
)

# Extract audio from video
t.add_computed_column(
    audio=extract_audio(t.video, format='mp3'),
    if_exists='ignore'
)

User-Defined Functions (UDFs)#

python

@pxt.udf
def clean_text(text: str) -> str:
    return text.strip().lower()

@pxt.udf
def safe_length(text: str | None) -> str:
    return 0 if text is None else len(text)

t.add_computed_column(cleaned=clean_text(t.content), if_exists='ignore')

Query Functions (also usable as agent tools)#

python

@pxt.query
def search_documents(query_text: str, limit: int = 10):
    sim = t.content.similarity(string=query_text)
    return t.order_by(sim, asc=False).limit(limit).select(t.title, t.content, sim)

results = search_documents('machine learning').collect()

Tool-Calling Agent Pipeline#

Inserting a row triggers the entire computed column chain automatically.

python

import pixeltable as pxt
from pixeltable.functions.anthropic import messages, invoke_tools
from datetime import datetime

tools = pxt.tools(web_search, search_documents)  # @pxt.udf + @pxt.query

@pxt.udf
def assemble_context(question: str, tool_outputs: list | None, doc_context: list | None) -> str:
    tool_str = str(tool_outputs) if tool_outputs else 'N/A'
    doc_str = '\n'.join(
        f"- {item.get('text', '')}" for item in (doc_context or []) if isinstance(item, dict)
    ) or 'N/A'
    return (f"QUESTION: {question}\n\n"
            f"<tool_results>\n{tool_str}\n</tool_results>\n\n"
            f"<retrieved_documents>\n{doc_str}\n</retrieved_documents>")

agent = pxt.create_table('my_project.agent', {
    'prompt': pxt.String, 'timestamp': pxt.Timestamp,
    'system_prompt': pxt.String, 'max_tokens': pxt.Int, 'temperature': pxt.Float,
}, if_exists='ignore')

# LLM selects tools → execute tools → RAG retrieval → assemble → final answer
agent.add_computed_column(initial_response=messages(
    model='claude-sonnet-4-20250514',
    messages=[{'role': 'user', 'content': [{'type': 'text', 'text': agent.prompt}]}],
    tools=tools, tool_choice=tools.choice(required=True),
    max_tokens=agent.max_tokens,
    model_kwargs={'system': agent.system_prompt, 'temperature': agent.temperature},
), if_exists='ignore')

agent.add_computed_column(tool_output=invoke_tools(tools, agent.initial_response), if_exists='ignore')
agent.add_computed_column(doc_context=search_documents(agent.prompt), if_exists='ignore')
agent.add_computed_column(context=assemble_context(agent.prompt, agent.tool_output, agent.doc_context), if_exists='ignore')

agent.add_computed_column(final_response=messages(
    model='claude-sonnet-4-20250514',
    messages=[{'role': 'user', 'content': [{'type': 'text', 'text': agent.context}]}],
    max_tokens=agent.max_tokens,
    model_kwargs={'system': agent.system_prompt, 'temperature': agent.temperature},
), if_exists='ignore')

agent.add_computed_column(answer=agent.final_response.content[0].text, if_exists='ignore')

# Usage
agent.insert([{'prompt': 'What is quantum computing?', 'timestamp': datetime.now(),
               'system_prompt': 'You are a helpful assistant.', 'max_tokens': 1024}])
result = agent.where(agent.prompt == 'What is quantum computing?').select(agent.answer).collect()

AI Provider Integrations#

Built-in functions for 25+ providers in pixeltable.functions.*:

ProviderModuleKey FunctionsOpenAIopenai``chat_completions (supports multimodal/vision via messages), embeddings, image_generations, speech, transcriptionsAnthropicanthropic``messages, invoke_toolsGeminigemini``generate_content, invoke_toolsHugging Facehuggingface``clip, sentence_transformer, detr_for_object_detectionTogethertogether``chat_completions, embeddings, image_generationsFireworksfireworks``chat_completions, embeddingsOllamaollama``chat_completions, embeddingsMistralmistralai``chat_completions, embeddingsGroqgroq``chat_completions, invoke_toolsDeepSeekdeepseek``chat_completionsReplicatereplicate``runVoyage AIvoyageai``embedBedrockbedrock``converse, invoke_toolsOpenRouteropenrouter``chat_completionsWhisperwhisper``transcribe (local transcription)WhisperXwhisperx``transcribe (local, with speaker diarization)Twelve Labstwelvelabs``embed (video understanding)Jina AIjina``embeddings, rerankBFL FLUXbfl``generate, edit, expand, fill (image generation/editing)RunwayMLrunwayml``text_to_video, image_to_video, text_to_image, video_to_videofal.aifal``run (execute any fal.ai model)Revereve``create, edit, remix (image generation)Microsoft Fabricfabric``chat_completions, embeddings (Azure OpenAI via Fabric)llama.cppllama_cpp``create_chat_completion (local GGUF models)YOLOXyolox``yolox (object detection)

Import/Export#

python

# From CSV / Parquet
t = pxt.create_table('dir.from_csv', source='data.csv')
t = pxt.create_table('dir.from_parquet', source='data.parquet')

# With schema overrides (remap columns to media types)
t = pxt.create_table('dir.data', source='data.csv',
    schema_overrides={'image_col': pxt.Image, 'doc_col': pxt.Document})

# From Hugging Face
from pixeltable.io import import_huggingface_dataset
import datasets
ds = datasets.load_dataset('squad', split='train[:1000]')
t = import_huggingface_dataset('dir.squad', ds)

# From pandas
from pixeltable.io import import_pandas
t = import_pandas('dir.from_df', df)

# Export
from pixeltable.io import export_parquet
export_parquet(t, 'output/')

Idempotent Operations and Error Handling#

CRITICAL: Always use if_exists='ignore' on every create_* and add_* call.

python

# Fault-tolerant inserts
status = t.insert(rows, on_error='ignore')
# Inspect errors
t.where(t.summary.errortype != None).select(t.title, t.summary.errormsg).collect()
# Retry failed columns
t.recompute_columns(columns=['summary'], where=t.summary.errortype != None)

Common Pitfalls#

#WrongCorrect1openai.vision(prompt=..., image=t.image)``openai.chat_completions(messages=[{'role':'user','content':[{'type':'text','text':'...'}, {'type':'image_url','image_url':{'url':t.image}}]}], model='gpt-4o-mini').choices[0].message.content2from pixeltable.iterators import FrameIterator``from pixeltable.functions.video import frame_iterator3t.add_embedding_index('transcript', ...) on Json colExtract .text.astype(pxt.String) first, then index4Fix code + re-run with if_exists='ignore'Must t.drop_column('col') then recreate — re-run is a no-op5{'type':'image', 'data': t.image} in messagesUse {'type':'image_url', 'image_url':{'url': t.image}}6t.content.similarity(query) (positional)t.content.similarity(string=query) (keyword)7Schema corruption ( IntegrityError)pip install -U pixeltable && rm -rf ~/.pixeltable8.collect() or pxt.get_table() inside @pxt.query``@pxt.query compiles the body at decoration time with expression placeholders — don't call .collect(), insert(), or reference tables that may not exist. Use a plain def for imperative logic9'id': pxt.String as primary keyPK columns must be non-nullable. Use pxt.Required[pxt.String] or uuid7() as a computed default10Module-level Table object used in FastAPI endpointTable objects are thread-bound. Call pxt.get_table() inside each endpoint function, not at module level

Full examples in core-api.md → Common Pitfalls.

Table Management#

python

pxt.list_tables()
t = pxt.get_table('my_project.my_table')
pxt.drop_table('my_project.my_table')
pxt.drop_dir('my_project', force=True)
t.describe()
t.columns()

# Snapshots (point-in-time copy)
snap = pxt.create_snapshot('my_project.snapshot_v1', t, if_exists='ignore')

# Update and delete
t.update({'score': 1.0}, where=t.category == 'important')
t.delete(where=t.is_active == False)

Building Apps with Pixeltable#

• Pixeltable IS the data layer — no ORM, no SQLAlchemy
• Prefer FastAPIRouter (v0.6+) over hand-written endpoints — add_insert_route, add_query_route, add_delete_route generate endpoints from tables and @pxt.query functions
• Use background=True on add_insert_route for long-running inserts (returns a job handle, client polls for completion)
• FastAPI endpoints: use def not async def (Pixeltable is synchronous)
• Business logic in @pxt.udf / @pxt.query, not in endpoint handlers
• Schema in one file, queries co-located with routes in each router file
• Insert a row → entire computed column chain runs automatically

python

from pixeltable.serving import FastAPIRouter
import pixeltable as pxt

router = FastAPIRouter(prefix="/api/data", tags=["data"])
docs = pxt.get_table("app.documents")

router.add_insert_route(docs, path="/upload", uploadfile_inputs=["document"],
                        inputs=["timestamp"], outputs=["uuid"], background=True)
router.add_delete_route(docs, path="/delete")

@pxt.query
def list_docs():
    return docs.select(uuid=docs.uuid, name=docs.document).order_by(docs.timestamp, asc=False)

router.add_query_route(path="/list", query=list_docs, method="get")

Reference: Pixeltable Starter Kit | workflows.md → FastAPIRouter | core-api.md → Serving

Resources#

• Starter Kit — 3 structural patterns + 7 application templates:

  • Patterns: backend/ (FastAPI + React), batch/ (no HTTP server), serving/ ( pxt serve + TOML)
  • app.py templates (have UI, run python app.py): knowledge-base, chat-agent, audio-transcription, full-stack-showcase
  • pxt-serve templates (API only, run python schema.py then pxt serve <name>): video-search, media-indexing, image-dataset
  • All app.py templates include port auto-detection (probes upward from 8000; override with PORT env var)
  • Scaffold with pixeltable-new: uvx pixeltable-new --template <name> my-app

• MCP Server — Explore Pixeltable tables via MCP

• LLM Docs — Complete documentation as plain text | llms.txt

Reference Files#

FileCoveragecore-api.mdTables, querying, views, embeddings, UDFs, tools, serving (FastAPIRouter), B-tree indexes, recompute, config, data sharing, SQL exportproviders.mdQuick-reference table + full examples for all 25+ AI providersworkflows.mdRAG, video analysis, image classification, audio, multi-provider, agent, batch processing, FastAPI, FastAPIRouter, exportvideo-rag-agents.mdVideo + transcript/frame retrieval + tool-calling agentagents-memory-mcp.mdAgent with persistent memory, MCP integration, multi-provider invoke_toolsml-data-pipeline.mdIngest, enrich, version, export to PyTorch/Parquet/pandasagentic-patterns.md6 architectural patterns + 2 reasoning strategiesanti-patterns.md15 training-distribution biases LLMs bring; wrong/right code for each