talkies#

Self-hosted speech service — ASR and TTS, one container. OpenAI-compatible wire shape on both endpoints; point an OpenAI client at it, change the model slug, done.

ASR ( POST /v1/audio/transcriptions): fourteen bundled slugs — whisper-large-v3, whisper-large-v3-turbo, parakeet-tdt-0.6b-v3, nemotron-3.5-asr-0.6b, canary-180m-flash, canary-1b-flash, canary-qwen-2.5b, four selectable English Sherpa Zipformer variants, vosk-small-en-us-0.15, and two phoneme recognizers, wav2vec2-xlsr-53-espeak and zipa-ipa, that return the IPA phones spoken instead of words.

TTS ( POST /v1/audio/speech): 3 engines / 4 backends across 8 slugs — kokoro-82m (PyTorch) and kokoro-82m-nvidia (ONNX/ORT) with 41 baked voices across en/es/fr/hi/it/pt, plus the CUDA-only Qwen3-TTS family: qwen3-tts-0.6b / qwen3-tts-1.7b (voice cloning from reference clips), qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom (9 preset speakers), qwen3-tts-1.7b-design (voice from an NL description), plus the CUDA-only chatterbox-turbo (English only; 19 inline emotion tags; clones from a reference .wav with no transcript). Discover voices via GET /v1/audio/voices.

Extras: live PCM ASR over WebSocket, stereo diarization on transcription, URL file_path fetching, server-side file staging, MCP endpoint with 6 ASR-side tools, optional bearer-token auth.

For installation, configuration, and container setup, see references/setup.md.

Security & safety#

This skill is not low-risk to orchestrate blindly — it issues local shell commands ( curl, ffmpeg, docker in the typical deployment/setup path) and outbound HTTP requests to whatever $TALKIES_URL points at:

• Outbound HTTP to an operator-chosen host — data leaves your host. Every command in this skill, including TTS input text and voice-cloning reference samples, is sent to whatever $TALKIES_URL points at — a server the skill does not control or vet. Point it only at an instance you run or explicitly trust; prefer HTTPS.
• file_path URL fetches happen server-side, not client-side. Passing a URL causes the talkies server to download it — see URL file_path (Download + Cache) below.
• Staged files persist server-side until explicitly deleted, and are enumerable by anyone who can reach the API — see Server-Side File Staging below.
• Shared file staging — staged files ( /v1/files) live in one namespace with no per-caller ownership: any caller can see or remove any staged file. Only remove paths you staged in the current task, and treat file management as admin-only on a shared instance — see Server-Side File Staging below.
• No auth by default — TALKIES_AUTH_TOKEN is opt-in; an unconfigured server is wide open to anyone who can reach the port. Require the token and least-privilege network exposure in any non-trusted environment.
• Voice cloning requires consent. Only clone or synthesize a voice you have explicit authorization/consent to use — synthesized speech of a real person can be used for impersonation, fraud, or deception. See Qwen3-TTS Custom Voices.
• Local shell execution. references/setup.md and workflow examples run docker, curl, and ffmpeg directly on the local host — review commands before running them against unfamiliar hosts/images.

When To Use#

• Transcribe audio files (any format ffmpeg decodes — WAV, MP3, M4A, FLAC, OGG, WebM, Opus, MP4 audio).
• Generate SRT/VTT subtitles for video.
• Transcribe podcasts, lectures, interviews, voicemails, calls.
• Stream 16 kHz mono PCM from a microphone or decoded live-audio source through WS /v1/audio/transcriptions/stream.
• Stereo two-mic recordings → per-speaker diarized output ( L: / R: channel tagging).
• German/French/Spanish ↔ English speech-to-text translation via Canary-1B-Flash.
• Synthesize speech from text via Kokoro-82M — English (American + British), Spanish, French, Hindi, Italian, Portuguese.
• Voice-clone speech via Qwen3-TTS-0.6B from a reference .wav you provide — drop into /data/custom-voices/, immediately appears under GET /v1/audio/voices with origin=custom. Only clone voices you're authorized to use, with the speaker's consent — see Qwen3-TTS Custom Voices.
• Voice-clone via chatterbox-turbo from the same /data/custom-voices/ drop — no sibling transcript needed, but the clip must be longer than 5 seconds. Same consent requirement applies.
• Emotive English delivery via chatterbox-turbo — write tags inline in input, e.g. Oh no [sigh] not again. The tokenizer defines exactly 19: [angry] [fear] [surprised] [whispering] [advertisement] [dramatic] [narration] [crying] [happy] [sarcastic] [clear throat] [sigh] [shush] [cough] [groan] [sniff] [gasp] [chuckle] [laugh]. Anything else is read as literal text.
• Drop-in replacement for api.openai.com/v1/audio/transcriptions and api.openai.com/v1/audio/speech in existing client code.

When NOT To Use#

• OpenAI-compatible live ASR — /v1/audio/transcriptions remains request/response only. Use talkies-specific WS /v1/audio/transcriptions/stream for raw PCM. (TTS has one streaming exception: qwen3-tts-* + response_format=pcm streams chunked PCM — see Streaming PCM (Qwen3-TTS).)
• Speaker identification from voice (only stereo-channel diarization is supported, not voice clustering).
• Per-request prompt / temperature on /v1/audio/transcriptions — accepted for OpenAI compat, ignored. ( instructions on /v1/audio/speech is different: Kokoro ignores it, but Qwen3-TTS honors it in most modes — see Request Body.)
• Japanese / Chinese TTS — Kokoro upstream supports them but talkies filters those voices out (they need the misaki[ja] / misaki[zh] extras).
• Kokoro on OpenAI aliases ( alloy, echo, fable, onyx, nova, shimmer) — Kokoro exposes its native voice names only ( af_*, bm_*, etc.). Map client-side. (Qwen3-TTS does ship alloy / echo / fable as builtin voice slugs, but they're voice-cloned samples, not OpenAI's voices — there's no audio compatibility.)
• qwen3-tts-0.6b on CPU — voice cloning hard-fails without CUDA at load time. The faster_qwen3_tts upstream raises ValueError on non-CUDA devices; talkies surfaces this as a load failure on the first request.
• qwen3-tts-0.6b speed parameter — Qwen3-TTS has no playback-rate control. Field is accepted for OpenAI compat but ignored (only Kokoro honors speed; chatterbox-turbo ignores it too).
• chatterbox-turbo for anything but English — it is an English-only checkpoint. Use Kokoro or Qwen3-TTS for other languages.
• chatterbox-turbo on CPU — the slug is registered in the CUDA image only. The model does run on CPU but measures roughly 5-10x slower than realtime, so it is not offered as a CPU slug.
• chatterbox-turbo with default settings where output must be unwatermarked. Every waveform carries Resemble AI's PerTh neural watermark, which the upstream package applies unconditionally. The server can turn it off: set TALKIES_CHATTERBOX_WATERMARK=false on the container. That is an operator-side setting, not a per-request one, so a caller cannot change it through the API.
• chatterbox-turbo reference clips of 5 seconds or shorter — rejected with a 400. Supply a longer clip.
• arm64 hosts — linux/amd64 only.

Setup#

The container should already be running. Set the base URL:

bash

export TALKIES_URL=http://localhost:8000

If the server has TALKIES_AUTH_TOKEN set, export it too:

bash

export TALKIES_AUTH_TOKEN=<your-token>
# every request below needs: -H "Authorization: Bearer $TALKIES_AUTH_TOKEN"

Verify: curl $TALKIES_URL/healthz returns {"ok": true, "device": "...", "models": [...]}.

For install / configuration / env vars / CPU vs CUDA images / custom model registry, see references/setup.md.

Quick Start#

bash

# Discover what's available.
curl -s $TALKIES_URL/v1/models | jq

# Simplest transcribe — file upload, JSON response.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Same call, but the audio lives at a URL — talkies downloads + caches it.
# WARNING: the URL is fetched by the talkies SERVER, not this client, and the
# download is cached persistently on disk (see "URL file_path" below). Don't
# pass URLs to private/sensitive media unless you trust the server operator.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Full Whisper-shape JSON with per-segment + per-word timestamps.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" \
  -F "response_format=verbose_json" | jq

# SRT subtitles.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@lecture.mp3" \
  -F "model=whisper-large-v3" \
  -F "response_format=srt" > lecture.srt

# Discover TTS voices, then synthesize an MP3.
curl -s $TALKIES_URL/v1/audio/voices | jq
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "Hello from talkies.",
        "voice": "af_heart",
        "response_format": "mp3"
      }' \
  --output hello.mp3

Live streaming ASR#

WS /v1/audio/transcriptions/stream is talkies-specific rather than an OpenAI endpoint. It accepts headerless 16 kHz, mono, signed 16-bit little-endian PCM. Send one JSON start message, wait for ready, stream binary PCM frames, then send {"type":"end"} for final and stats (or {"type":"cancel"} to discard the stream). Use a configured ASR slug with streaming support; native sessions are available through parakeet.cpp, Sherpa-ONNX, and Vosk, while faster-whisper uses a bounded rolling window. When authentication is enabled, send the bearer token in the WebSocket upgrade header, never in the URL.

See the repository's docs/streaming.md for the exact start/event shapes, a Python client, limits, and custom Sherpa or Vosk model-registry entries.

Supported Models#

ASR#

SlugFamilyCPUCUDALanguagesStrengthwhisper-large-v3faster-whisperyesyes99 auto-detectbest accuracy, slowestwhisper-large-v3-turbofaster-whisperyesyes99 auto-detectsweet spot — fast, accurateparakeet-tdt-0.6b-v3NeMo TDTnoyesEnglish onlyvery fast on GPUnemotron-3.5-asr-0.6bparakeet.cpp / ggmlyesyes40+ locales, auto-detectCPU or CUDA multilingual; pin via language=``canary-180m-flashNeMo CanaryyesyesEnglish only (small)smallest, runs anywherecanary-1b-flashNeMo Canarynoyesen/de/fr/es + translationmultilingual, translationcanary-qwen-2.5bNeMo SALMnoyesEnglish onlybest English accuracy (no timestamps)sherpa-zipformer-en-left-64Sherpa-ONNX ZipformeryesyesEnglish onlynative live ASR, lower left contextsherpa-zipformer-en-left-128Sherpa-ONNX ZipformeryesyesEnglish onlynative live ASR, higher left contextsherpa-zipformer-en-int8-left-64Sherpa-ONNX Zipformer INT8yesyesEnglish onlysmaller native live ASR variantsherpa-zipformer-en-int8-left-128Sherpa-ONNX Zipformer INT8yesyesEnglish onlysmaller native live ASR variantvosk-small-en-us-0.15VoskyesyesEnglish onlynative live ASR; CPU decoderwav2vec2-xlsr-53-espeakwav2vec2 CTC phonemeyesyesmultilingualIPA phones, not words; eSpeak alphabetzipa-ipaZIPA (Zipformer CTC phoneme)yesyesmultilingualIPA phones, not words; 71 MB, very fast

Pick by use case:

• General-purpose: whisper-large-v3-turbo.
• English-only, max accuracy on GPU: canary-qwen-2.5b (but no per-segment timestamps).
• Translation EN↔DE/FR/ES: canary-1b-flash (requires custom model registry — see Translation).
• Low-overhead live English ASR: one of the Sherpa Zipformer variants or vosk-small-en-us-0.15; Sherpa uses CUDA in the CUDA image, while Vosk remains CPU-decoded.
• Phones, not words (pronunciation checks, phonetics): wav2vec2-xlsr-53-espeak or zipa-ipa. Both return a space-separated IPA stream with per-phone timestamps and run no language model, so mispronunciations are not corrected away. zipa-ipa is the small fast pick; wav2vec2-xlsr-53-espeak is the larger transformers-native one.

TTS#

3 engines / 4 backends across 8 slugs. Kokoro ships in two runtimes ( kokoro-82m PyTorch, kokoro-82m-nvidia ONNX/ORT) — same weights, same voice catalog, same wire format. Qwen3-TTS ships 5 CUDA-only slugs across three modes (base cloning / custom_voice preset speakers / voice_design). Mode is implicit in the slug — see Qwen3-TTS Modes. chatterbox-turbo (CUDA-only, English) rounds out the set — 19 inline emotion tags, transcript-free voice cloning; see When To Use above.

SlugFamilyModeCPUCUDALanguagesVoiceskokoro-82mKokoro (PyTorch in-process, 24 kHz)—yesyesen (US + UK), es, fr, hi, it, pt41 baked (discover via GET /v1/audio/voices)kokoro-82m-nvidiaKokoro (ONNX via ORT, 24 kHz)—yesyesen (US + UK), es, fr, hi, it, pt41 baked (same catalog as kokoro-82m)qwen3-tts-0.6bQwen3-TTS (24 kHz)basenoyes17 (en, zh, ja, ko, fr, de, es, it, pt, ru, vi, th, id, ar, tr, pl, nl)3 builtin samples + any .wav under /data/custom-voices/``qwen3-tts-1.7bQwen3-TTS (24 kHz)basenoyes10 (en, zh, ja, ko, fr, de, es, it, pt, ru)3 builtin samples + any .wav under /data/custom-voices/``qwen3-tts-0.6b-customQwen3-TTS (24 kHz)custom_voicenoyesen, zh, ja, ko9 preset speakers ( instructions dropped — 0.6B limitation)qwen3-tts-1.7b-customQwen3-TTS (24 kHz)custom_voicenoyesen, zh, ja, ko9 preset speakers + emotion via instructions``qwen3-tts-1.7b-designQwen3-TTS (24 kHz)voice_designnoyesen, zh, ja, kovoice synthesized from NL description in instructions (required)chatterbox-turboChatterbox Turbo (24 kHz, buffered only)—noyesEnglish onlybuiltin speaker, or any .wav (>5s) under /data/custom-voices/

Pick by use case:

• General-purpose multi-voice TTS: kokoro-82m — fast, 41 baked voices, runs on CPU. Use kokoro-82m-nvidia for the ONNX/ORT execution path (CUDA EP on the CUDA image, CPU EP otherwise).
• Voice cloning from a reference clip: qwen3-tts-0.6b / qwen3-tts-1.7b — drop a .wav into /data/custom-voices/, immediately usable. CUDA required.
• Preset speakers (no reference WAV): qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom — 9 baked speakers; the 1.7B honours instructions for emotion. CUDA required.
• Invent a voice from a description: qwen3-tts-1.7b-design — the NL description goes in instructions. CUDA required.
• Expressive English delivery, transcript-free cloning: chatterbox-turbo — 19 inline emotion tags in input, clones from a bare .wav (no sibling transcript needed, clip must be longer than 5 seconds). English only, CUDA required, output always carries a neural watermark.

canary-qwen-2.5b produces no segment/word timestamps — verbose_json.segments and .words come back empty, srt/ vtt collapse to a single full-duration cue. Transcription itself is whole-file. Use a Whisper or Canary multitask slug if you need timing.

API — POST /v1/audio/transcriptions#

Multipart form. Same field names as OpenAI's transcription endpoint where they overlap.

Request Fields#

FieldRequiredDefaultNotesfileone of file/ file_path—Audio file. Capped at TALKIES_MAX_UPLOAD_BYTES (default 100 MB).file_pathone of file/ file_path—Either a path under the staging area ( /v1/files) or an http(s):// URL (downloaded + cached server-side). Not subject to the 100 MB upload cap; URL downloads capped by TALKIES_MAX_DOWNLOAD_BYTES (default 1 GiB).modelyes—One of the configured slugs (see GET /v1/models). Unknown → 404.languagenomodel defaultISO-639-1 code. Whisper auto-detects when omitted; Canary uses its default_source_lang.response_formatnojson``json / text / verbose_json / srt / vtt.timestamp_granularities[]no—Accepted for OpenAI compat; ignored — verbose_json always emits both segment + word.promptno—Accepted, ignored.temperatureno—Accepted, ignored.diarizationnofalseStereo-channel diarization. Requires 2-channel input — mono returns 400.

Exactly one of file or file_path must be set — passing both or neither returns 400.

Response Formats#

response_formatContent-TypeShapejson (default)application/json``{"text": "..."} — just the transcript.text``text/plainThe transcript as plain text.verbose_json``application/jsonFull Whisper shape — task, language, duration, text, segments[], words[].srt``application/x-subripSubRip subtitle file, one cue per VAD-segmented chunk.vtt``text/vttWebVTT subtitle file, one cue per VAD-segmented chunk.

json shape:

json

{ "text": " full transcript as a single string" }

verbose_json shape — segments and words are always present (empty arrays for backends with no alignment output):

json

{
  "task": "transcribe",
  "language": "en",
  "duration": 6.42,
  "text": " full transcript",
  "segments": [{ "id": 0, "start": 0.0, "end": 2.31, "text": " ...", "tokens": [], "temperature": 0.0, "avg_logprob": null, "compression_ratio": null, "no_speech_prob": null }],
  "words": [{ "word": " the", "start": 0.0, "end": 0.12 }]
}

Whisper-only confidence fields ( avg_logprob, compression_ratio, no_speech_prob) are emitted as null regardless of backend so clients reading them don't crash. tokens is always [].

The sherpa and vosk executors add a per-word confidence in the 0–1 range to each entry in words — Vosk reports the decoder's own score, Sherpa derives one from the model's per-token acoustic log-probabilities. No other backend emits it, so treat the field as optional.

Stereo Diarization#

Pass diarization=true and upload a 2-channel file. Left channel = speaker L, right channel = speaker R. Each channel is transcribed independently, the two timelines are merged chronologically by segment start time.

bash

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@interview-stereo.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "diarization=true" \
  -F "response_format=verbose_json" | jq

What changes:

• verbose_json — every segment/word gets "channel": "L" or "R". Segments re-numbered after merge.
• text / response_format=text — rebuilt as alternating turn lines: L: ...\nR: ...\n.... Consecutive same-channel segments collapsed into one line per turn.
• srt / vtt — each cue prefixed with L: / R:.

Caveats:

• Exactly 2 channels required. Mono → 400. >2 channels → 400.
• Latency ~2× the mono case (model runs sequentially on each channel).
• The technique is exact for true two-mic setups (interview rigs, podcast splits). It does NOT magically separate speakers from a single-mic recording that's been rendered to stereo.

Translation#

Canary multitask models can translate speech → text in a non-source language. canary-1b-flash covers en↔de, en↔fr, en↔es. The task is baked into the model slug, not passed per-request — you add a translation-specific slug via custom models.json (see Customizing the model registry):

json

{
  "models": {
    "canary-1b-flash-de2en": {
      "repo": "nvidia/canary-1b-flash",
      "executor": "canary_multitask",
      "default_source_lang": "de",
      "default_target_lang": "en",
      "default_task": "s2t_translation",
      "languages": ["de"]
    }
  }
}

Then call it normally — text carries the English translation:

bash

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@german-clip.wav" \
  -F "model=canary-1b-flash-de2en" | jq

canary-180m-flash is English-ASR-only — don't point a translation slug at it. canary-qwen-2.5b is English ASR only too.

Long Files + VAD Chunking#

Audio longer than 30 s ( TALKIES_VAD_CHUNK_THRESHOLD) gets sliced through Silero VAD into ≤28 s speech regions before being handed to the backend. Timestamps are re-assembled by offsetting each chunk's segment/word timings — you get one continuous segments list spanning the whole file.

No client-side change. Long files just work. Verify by checking duration in verbose_json.

Error Contract#

StatusShapeWhen200per response_formatsuccess400{"detail": "..."}bad audio, mono+diarization, >2 ch+diarization, both/neither of file/ file_path, invalid file_path, URL download failure (DNS, HTTP error, size exceeded, SSRF blocked)401{"detail": "..."}only when TALKIES_AUTH_TOKEN is set: missing/wrong bearer. Includes WWW-Authenticate: Bearer.404{"detail": "..."}unknown model slug, file_path references missing file, model-evict on an unloaded model, file op on a missing /v1/files path413{"detail": "..."}upload exceeded TALKIES_MAX_UPLOAD_BYTES (multipart file and PUT /v1/files/{path} only — not file_path URL)422{"detail": [...]}Pydantic validation (missing fields, wrong types)500{"detail": "..."}unhandled backend failure

API — POST /v1/audio/speech (TTS)#

JSON body (not multipart). Returns the encoded audio bytes in the body with the matching Content-Type — no JSON envelope.

Every call sends your input text (and, for voice-cloning slugs, the referenced voice sample) to whatever $TALKIES_URL points at — that data leaves your host. Point $TALKIES_URL only at a talkies instance you run or explicitly trust; prefer HTTPS when it's not localhost/LAN. Don't synthesize sensitive or confidential text through a server you don't control.

bash

curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "The quick brown fox jumps over the lazy dog.",
        "voice": "af_heart",
        "response_format": "mp3",
        "speed": 1.0
      }' \
  --output fox.mp3

Request Body#

FieldRequiredDefaultNotesmodelyes—TTS model slug. Kokoro: kokoro-82m, kokoro-82m-nvidia. Qwen3-TTS: qwen3-tts-0.6b, qwen3-tts-1.7b (base/cloning), qwen3-tts-0.6b-custom, qwen3-tts-1.7b-custom (preset speakers), qwen3-tts-1.7b-design (voice from NL description). Chatterbox: chatterbox-turbo (English only). Unknown → 404. ASR slug → 400.inputyes—Text to synthesize. Empty / whitespace-only → 400. No fixed length cap; for very long inputs split client-side.voicenomodel default_voiceSemantics shift per Qwen3 mode (see Qwen3-TTS Modes). Kokoro: voice name (default af_heart). Qwen3 base: path of a reference WAV (default alloy). Qwen3 custom_voice: one of the 9 preset speakers (default Vivian). Qwen3 voice_design: ignored — sentinel "design". chatterbox-turbo: builtin (default) or the name of a .wav under /data/custom-voices/, longer than 5 seconds — shorter clips → 400. Unknown → 400 with catalog listed.response_formatnomp3``mp3 / opus / aac / flac / wav / pcm.speedno1.0Playback rate, Kokoro only. Clamped to [0.25, 4.0]. Ignored by every Qwen3-TTS slug (no speed control in Qwen3-TTS) and by chatterbox-turbo.instructionsno—Free-form style prompt. Required for qwen3-tts-1.7b-design (the NL voice description; empty → 400). Honoured by Qwen3-TTS base mode and qwen3-tts-1.7b-custom (threaded as instruct). Dropped by qwen3-tts-0.6b-custom (0.6B CustomVoice checkpoint limitation — logs a WARNING) and both Kokoro slugs (no instruction input). Accepted on every slug for OpenAI parity.languagenomodel default_language ( English)Non-OpenAI extra field (send via extra_body={"language": "..."} on official SDKs). Selects the spoken language for Qwen3 custom_voice / voice_design; base mode reads it from the voice's sibling .lang file. Silently ignored by Kokoro.temperatureno0.9Non-OpenAI extra, Qwen3-TTS only ( extra_body). Sampler temperature, [0.0, 2.0]. Ignored by Kokoro.top_kno50Non-OpenAI extra, Qwen3-TTS only. Top-k truncation, [1, 1000]. Ignored by Kokoro.top_pno1.0Non-OpenAI extra, Qwen3-TTS only. Nucleus sampling, [0.0, 1.0]. Ignored by Kokoro.repetition_penaltyno1.05Non-OpenAI extra, Qwen3-TTS only. Penalizes codec-token repeats, [0.5, 2.0]. Ignored by Kokoro.max_new_tokensno2048 (model max)Non-OpenAI extra, Qwen3-TTS only. Codec-step cap, [1, 2048]. Ignored by Kokoro.do_samplenotrueNon-OpenAI extra, Qwen3-TTS only. false = greedy decode. Ignored by Kokoro.

Out-of-range sampling values → 422 (Pydantic validation).

Qwen3-TTS Modes#

The Qwen3-TTS mode is implicit in the model slug — the OpenAI wire format stays pure ( model / voice / instructions / input), with voice and instructions carrying mode-specific semantics. No new endpoints.

ModeSlugsWhat voice meansWhat instructions meansbase (voice cloning)qwen3-tts-0.6b, qwen3-tts-1.7bPath of a reference .wav under the voices dirs ( .wav stripped)Optional style hint (passed as instruct)custom_voice (preset speakers)qwen3-tts-0.6b-custom, qwen3-tts-1.7b-customOne of 9 preset speaker namesEmotion / style cue — 1.7B honours it; 0.6B drops it (checkpoint limitation, logs a WARNING)voice_design (NL description)qwen3-tts-1.7b-designIgnored — sentinel "design"Required. NL description of the voice (e.g. "A warm, friendly young female voice"). Empty → 400.

The 9 custom_voice preset speakers (also returned by GET /v1/audio/voices for the chosen slug): Vivian, Serena, Uncle_Fu, Dylan, Eric (Chinese), Ryan, Aiden (English), Ono_Anna (Japanese), Sohee (Korean).

Output Formats#

response_format picks the encoder applied to Kokoro's raw 24 kHz mono PCM. ffmpeg does the conversion in-process; no temp files.

response_formatContent-TypeCodec / containerNotesmp3 (default)audio/mpeglibmp3lame, 128 kbps CBRMost universal.opus``audio/ogglibopus, 64 kbps VBR, Ogg containerBest quality-per-byte for speech.aac``audio/aacAAC-LC, 128 kbps, ADTSiOS-friendly.flac``audio/flacFLACLossless.wav``audio/wavPCM s16le, 24 kHz mono, RIFF headerLossless, largest.pcm``application/octet-streamRaw PCM s16le, 24 kHz mono — no container, no headerReal-time chaining. Caller must know sample rate / format.

Streaming PCM (Qwen3-TTS)#

qwen3-tts-* slugs stream response_format=pcm requests as chunked int16 PCM instead of buffering the whole utterance — first-audio latency drops from seconds to sub-second. Kokoro and every non- pcm format are unaffected (buffered as normal). Response carries an X-Sample-Rate header; chunk size (codec steps per yielded chunk) is tunable via TALKIES_QWEN3_STREAM_CHUNK_SIZE (default 8) — see references/setup.md.

bash

curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-tts-0.6b","input":"Streaming test.","response_format":"pcm"}' \
  --output stream.pcm

Voices#

bash

curl -s $TALKIES_URL/v1/audio/voices | jq

Returns {"voices": [{"voice", "model", "default", "origin"}]}. The origin field is only present for engines that distinguish baked-in vs user-supplied voices (currently qwen3-tts-0.6b — "builtin" for image-baked samples, "custom" for /data/custom-voices/ mounts). Kokoro entries omit origin.

Kokoro voices encode <lang_code><gender>_<name>:

PrefixLanguageaf_ / am_American English (female / male)bf_ / bm_British English (female / male)ef_ / em_Spanishff_Frenchhf_ / hm_Hindiif_ / im_Italianpf_ / pm_Portuguese (Brazilian)

41 voices ship in the image. Japanese ( jf_* / jm_*) and Chinese ( zf_* / zm_*) are filtered out because they need the optional misaki[ja] / misaki[zh] extras (MeCab + pypinyin chains).

Qwen3-TTS voices come from two on-disk dirs merged into one catalog:

• /opt/talkies/qwen3-voices/ — baked into the CUDA image. Ships three curated samples ( alloy, echo, fable) so voice cloning works out-of-the-box. origin=builtin.
• /data/custom-voices/ — host-mounted via the data volume. Drop foo/bar/me.wav and voice foo/bar/me immediately appears in GET /v1/audio/voices (catalog is rescanned per request — no restart). origin=custom.

Voice names are the wav's path relative to its parent dir with .wav stripped — nested subdirs are preserved. custom-voices/team-a/jane.wav → voice team-a/jane. Custom voices shadow builtin voices with the same name; dropping a custom-voices/alloy.wav overrides the builtin alloy sample (its origin flips to custom).

Optional sibling metadata next to each <name>.wav:

• <name>.txt — reference transcript for the clip (ICL voice cloning works without it, but clone fidelity is noticeably better with a faithful transcript).
• <name>.lang — language label string (defaults to English).

Path-traversal guard: hostile symlinks whose resolve() escapes the voices dir are skipped (the wav can't be used to read arbitrary host files as a voice prompt).

bash

# Add a custom clone voice (server picks it up on next request — no restart).
mkdir -p ~/talkies-data/custom-voices/team-a
cp jane-reading.wav ~/talkies-data/custom-voices/team-a/jane.wav
echo "And the silken sad uncertain rustling of each purple curtain." \
  > ~/talkies-data/custom-voices/team-a/jane.txt

# Use it.
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "qwen3-tts-0.6b",
        "input": "Hello from a cloned voice.",
        "voice": "team-a/jane",
        "response_format": "wav"
      }' \
  --output cloned.wav

First synth is slow on Qwen3-TTS — the predictor + talker CUDA graphs are captured on first call (~30-60 s on a mid-range GPU). Subsequent generations are sub-second. The model and graphs stay resident until evicted by sibling load or the idle sweeper.

Error Contract (TTS)#

StatusWhen200success (audio bytes in body)400empty input, unknown voice, unsupported response_format, model isn't TTS (e.g. POSTing whisper-large-v3 here)401TALKIES_AUTH_TOKEN set, missing / wrong bearer404unknown model slug422Pydantic validation (missing required fields, wrong types)500unhandled ffmpeg or kokoro internal failure503TTS snapshot files missing under ${TALKIES_DATA_DIR}/models/<slug>/ (slug excluded from TALKIES_ENABLED_MODELS but still being called); or qwen3-tts-0.6b requested on a non-CUDA device (the backend hard-fails at load time)

Resource-Management Endpoints (Ollama-Style)#

talkies mirrors a subset of speaches / Ollama, so a LiteLLM proxy can drive both.

EndpointBehaviorGET /healthzUnauthenticated liveness. Returns {ok, device, models}.GET /v1/modelsOpenAI-style list of configured slugs. Each entry includes a modality field ( asr or tts) so clients can filter.GET /api/psCurrently-loaded models with per-model idle_seconds.DELETE /api/ps/{model_id}Evict one model from memory. Slug can be URL-encoded ( / → %2F). 404 if not loaded.POST /unloadEvict every loaded model. Returns the list actually unloaded.

Model eviction ( DELETE /api/ps/..., POST /unload) forces a cold-load for anyone mid-request, so only do it for explicit maintenance the user asked for (e.g. "free up VRAM"). Auth is only enforced when TALKIES_AUTH_TOKEN is set — require it on shared deployments and don't expose these routes on an unauthenticated network.

Behind these: an idle sweeper runs every TALKIES_SWEEPER_INTERVAL s (default 60) and unloads anything not used in TALKIES_MODEL_TTL s (default 600). Set TALKIES_MODEL_TTL=0 to disable.

There's also sibling eviction at request time — every transcribe or speech request evicts other loaded models so VRAM doesn't get split. ASR and TTS share the same pool; loading Kokoro evicts a resident Whisper and vice versa. One model resident at a time, per container. If you need two models simultaneously, run two containers.

bash

# Which models are loaded right now.
curl -s $TALKIES_URL/api/ps | jq

# Free VRAM after a job — evict one model.
curl -s -X DELETE "$TALKIES_URL/api/ps/whisper-large-v3-turbo"

# Or evict everything.
curl -s -X POST $TALKIES_URL/unload | jq

Server-Side File Staging ( /v1/files)#

For repeated transcribes of the same file (different response_format, different model, iterating on params), stage the file once and reference it by path. Files land under ${TALKIES_DATA_DIR}/files/<path>.

Staged files persist until explicitly removed — nothing auto-expires them — and GET /v1/files enumerates every staged path to anyone who can reach the API. Don't stage sensitive/private media on a server without auth enabled ( TALKIES_AUTH_TOKEN); clean up staged files when done.

Guardrail — this is a shared, unisolated bucket, not a private workspace. There's no per-caller ownership: any path any caller staged is listable and readable by any other caller with API access. An agent must:

• only read or delete paths it staged itself in the current, user-approved workflow;
• never call GET /v1/files to browse/enumerate what other callers have staged, and never delete a path it didn't create;
• clean up what it staged once the workflow is done, since nothing expires automatically.

At the deployment level: require TALKIES_AUTH_TOKEN by default, treat per-caller isolation and retention limits as the operator's responsibility (talkies itself provides neither).

EndpointBehaviorGET /v1/filesList every staged file. Returns {"files": [{"path", "size", "modified"}]}. Enumerable by anyone with API access — no per-file ownership/isolation.PUT /v1/files/{path}Upload raw bytes ( --data-binary @local-file). Capped at TALKIES_MAX_UPLOAD_BYTES. Atomic write ( .part → rename).GET /v1/files/{path}Streams file back. Content-Type guessed by extension. 404 if missing.DELETE /v1/files/{path}Removes a staged file and prunes empty parent dirs (404 if missing). Files don't self-expire, so call this to clean up when done. On a shared bucket every path is listable by any caller, so only remove paths you staged yourself.bash

# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
  -H "Content-Type: audio/mpeg" \
  $TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3

# Reuse across multiple transcribe calls.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=lectures/2026-03-15/lecture.mp3" \
  -F "model=whisper-large-v3-turbo" \
  -F "response_format=verbose_json" | jq

# Cleanup.
curl -X DELETE $TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3

Path safety: null bytes, backslashes, . / .. segments and double slashes are rejected (400). Symlinks pointing outside the root are refused. Leading / is stripped — /foo/bar.mp3 and foo/bar.mp3 resolve identically.

URL file_path (Download + Cache)#

file_path also accepts http:// / https:// URLs. First request downloads to ${TALKIES_DATA_DIR}/files/downloads/<sha256(url)[:16]>-<basename>, subsequent requests with the same URL hit the cache.

The download happens server-side, and the result is cached persistently on the talkies server's disk — not a transient client-side fetch. Anyone who can reach the API can later list/read that cached copy via GET /v1/files (see Server-Side File Staging above). Don't pass URLs to private/sensitive media unless the talkies server itself is trusted and access-controlled ( TALKIES_AUTH_TOKEN); invalidate the cache entry when done.

bash

# First call: downloads, transcribes off the cached copy.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Second call: same URL → cache hit, no re-download.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=canary-1b-flash" \
  -F "response_format=srt" > ep-042.srt

Downloads appear in GET /v1/files listings under downloads/. Invalidate a single cached URL by removing it from /v1/files/downloads/.

Constraints applied during download:

• Size capped by TALKIES_MAX_DOWNLOAD_BYTES (default 1 GiB).
• 5 redirect hops max; SSRF guard re-applied at every hop.
• 10 s connect, 300 s per-chunk read timeout.
• SSRF off by default. Set TALKIES_BLOCK_PRIVATE_DOWNLOADS=true to reject URLs whose hostname resolves to private/loopback/link-local/multicast/reserved IPs.

MCP Endpoint ( /v1/mcp)#

talkies exposes a Model Context Protocol server over Streamable HTTP at /v1/mcp. Same FastAPI process, same BACKENDS / REGISTRY, same auth middleware — a model loaded by the MCP transcribe tool is the same instance the HTTP endpoint sees.

MCP exposes the ASR surface only. TTS ( /v1/audio/speech) is HTTP-only — generated audio bytes don't round-trip through JSON-RPC cleanly. list_models filters out TTS slugs so transcribe only ever sees ASR backends.

ToolWhat it doeslist_modelsDiscover ASR slugs (TTS slugs are filtered out). Returns [{slug, executor, default_source_lang, default_target_lang, default_task, loaded}].transcribeRun ASR on a file_path (URL or staged path). Args: model, language?, response_format? ( json/ verbose_json/ text/ srt/ vtt), diarization?. JSON formats return a JSON-encoded string; text/srt/vtt return raw.list_filesSame payload as GET /v1/files.put_fileUpload to staging. Body is base64 ( content_base64). Decoded size capped at TALKIES_MAX_UPLOAD_BYTES. For big files, prefer PUT /v1/files/{path} over HTTP — JSON-RPC + base64 chews token budget.get_fileRead a staged file as base64. Same size cap. Same advice — for big bytes, hit GET /v1/files/{path} over HTTP.delete_fileRemove a staged file, prune empty parents.

The transport requires Accept: application/json, text/event-stream. Wire it into Claude Code:

bash

claude mcp add --transport http talkies $TALKIES_URL/v1/mcp

With auth:

bash

claude mcp add --transport http talkies $TALKIES_URL/v1/mcp \
  --header "Authorization: Bearer $TALKIES_AUTH_TOKEN"

Note: the canonical mount path is /v1/mcp/ (trailing slash). Bare /v1/mcp is rewritten internally to /v1/mcp/ so clients that don't follow Starlette's 307 redirect work too.

Raw JSON-RPC#

For debugging or non-MCP-aware callers, hit it as JSON-RPC over HTTP POST:

bash

# tools/list
curl -s $TALKIES_URL/v1/mcp/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

# tools/call
curl -s $TALKIES_URL/v1/mcp/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0", "id": 2, "method": "tools/call",
    "params": {
      "name": "transcribe",
      "arguments": {
        "file_path": "https://example.com/clip.mp3",
        "model": "whisper-large-v3-turbo",
        "response_format": "json"
      }
    }
  }'

Bearer-Token Auth#

If TALKIES_AUTH_TOKEN is set on the server, every route except /healthz and CORS preflight ( OPTIONS) requires Authorization: Bearer <token>. Wrong/missing token returns 401 with WWW-Authenticate: Bearer. Compared with hmac.compare_digest (constant-time).

bash

curl -H "Authorization: Bearer $TALKIES_AUTH_TOKEN" $TALKIES_URL/v1/models

Empty / unset token = wide open. For untrusted networks, combine the token with a reverse proxy doing TLS + rate limiting.

Typical Workflows#

Quick one-off transcribe#

bash

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" | jq -r .text

Generate subtitles for a video#

bash

ffmpeg -i video.mp4 -vn -acodec libmp3lame audio.mp3
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3" \
  -F "response_format=srt" > video.srt
# burn in:  ffmpeg -i video.mp4 -vf subtitles=video.srt -c:a copy video-subbed.mp4

Iterate on the same file with different settings#

bash

# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
  -H "Content-Type: audio/mpeg" \
  $TALKIES_URL/v1/files/work/lecture.mp3

# Try different models / formats without re-uploading.
for fmt in json verbose_json srt; do
  curl -s $TALKIES_URL/v1/audio/transcriptions \
    -F "file_path=work/lecture.mp3" \
    -F "model=whisper-large-v3-turbo" \
    -F "response_format=$fmt" > "lecture.$fmt"
done

# Clean up the staged file when done (see the /v1/files reference above).

Diarized interview transcript#

bash

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@interview-stereo.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "diarization=true" \
  -F "response_format=text"
# stdout:
#   L: hi how's it going
#   R: not bad you
#   L: cool man

Synthesize speech from text#

bash

# Default voice, MP3 output.
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"kokoro-82m","input":"Greetings, human."}' \
  --output greetings.mp3

# Pick a voice from GET /v1/audio/voices, choose a format.
curl -s $TALKIES_URL/v1/audio/voices | jq -r '.voices[].voice'
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "Buongiorno, mondo.",
        "voice": "if_sara",
        "response_format": "opus"
      }' \
  --output ciao.opus

Free VRAM after a job#

bash

curl -s -X POST $TALKIES_URL/unload | jq

Bulk transcribe from URLs#

bash

for url in $(cat urls.txt); do
  curl -s $TALKIES_URL/v1/audio/transcriptions \
    -F "file_path=$url" \
    -F "model=whisper-large-v3-turbo" \
    -F "response_format=text"
  echo "---"
done

The first hit on each URL downloads + caches; re-running the loop is free. For local files, replace the file_path form field with file=@path/to/audio in the same request shape.

Tips#

1. Use whisper-large-v3-turbo as your default — it's the speed/quality sweet spot for general-purpose ASR. Switch to whisper-large-v3 only when you need the last few % of accuracy on hard audio.
2. URL file_path over multipart upload — if the audio is already at a URL, send the URL. Saves bandwidth (the file isn't going up and then back down), gets cached server-side, no upload size cap.
3. Stage repeated files via PUT /v1/files/{path} and call with file_path= to avoid re-uploading on every retry/iteration.
4. response_format=text for the "just give me the string" case — no jq -r .text needed, content-type is text/plain.
5. One active model family at a time — every transcribe request evicts other loaded models. Multiple live streams may share one pinned model up to TALKIES_STREAM_MAX_CONNECTIONS; a request or stream for a different model receives a conflict until the streams end. Use two containers if you need concurrent models.
6. POST /unload after a job — explicit eviction frees VRAM/RAM faster than waiting for the 10-min idle sweeper. Useful in CI / batch scripts.
7. canary-qwen-2.5b has no timestamps — verbose_json.segments / .words come back empty, srt/ vtt collapse to one cue. Use a Whisper or Canary multitask slug if you need timing data.
8. Diarization requires true stereo — if your "stereo" file is the same mono signal copied to both channels, diarization won't separate speakers. The technique is exact for two-mic setups, useless otherwise.
9. Long files just work — VAD chunking happens transparently. Don't pre-split. Send the whole file.
10. ASR's prompt / temperature are ignored even though the request schema accepts them. TTS's instructions is different — Kokoro ignores it, but Qwen3-TTS honors it in base mode and *-custom (except the 0.6B checkpoint) and requires it for *-design.
11. Watch /api/ps to see what's resident. A request that hangs at "loading model" is doing the first cold load — subsequent calls are fast.
12. Customizing the model registry for translation slugs or to restrict the served set — see references/setup.md.
13. Kokoro uses native voice names — no OpenAI aliases. Hit GET /v1/audio/voices once to discover what's shipped; pass the voice field accordingly. The 41 voices cover en (US + UK), es, fr, hi, it, pt; ja/zh are filtered out.
14. Voice cloning is qwen3-tts-0.6b — drop a .wav (10-30 s of clean speech is plenty) into /data/custom-voices/<anywhere>.wav. Optionally drop a sibling .txt with a faithful transcript for higher clone fidelity. The voice appears in GET /v1/audio/voices on the next request — no restart. CUDA required.
15. Qwen3-TTS first synth is slow — CUDA graph capture runs once after model load (~30-60 s). Subsequent synths are sub-second. If you're benchmarking, throw away the first call.
16. Qwen3-TTS ignores speed — the model has no playback-rate control. Pass it for OpenAI compat; nothing happens. Only Kokoro honors speed.
17. Both TTS engines emit 24 kHz mono PCM — Kokoro and Qwen3-TTS both output int16 24 kHz mono. ffmpeg re-encodes into your chosen response_format; pcm (raw, no container) hands back that same rate directly — check the X-Sample-Rate response header on Qwen3-TTS streaming responses if you need it confirmed per-request.
18. TTS response_format=pcm is for chaining — raw int16 mono PCM, no container, no header. Use it when piping into another encoder or a real-time playback path. Otherwise stick with mp3 (default) or opus for size.
19. TTS evicts loaded ASR and vice versa — they share the same one-model-resident pool. Synthesizing with Kokoro after a transcribe burst incurs Kokoro's cold load. Same applies to Qwen3-TTS (plus the CUDA-graph capture re-runs on cold reload).