OmniVoice#

Ten operations across four capabilities: identify (认) · manage (存) · transcribe (听) · clone (说).

Dependencies#

ComponentInstallPurposeWhisperpip install openai-whisperSpeech-to-textSpeaker IDpip install transformers librosaSpeaker identification (UniSpeech-SAT)CosyVoice2SiliconFlow API ( SF_API_KEY)Voice cloningffmpegSystem packageAudio conversion

Voice references are stored in voice-refs/ at workspace root. Metadata lives in TOOLS.md under a "Voice Library" section. See references/voice-library-format.md for format spec.

Operations#

Op 1 · Speaker Identification (声纹查询)#

Input: audio → Output: who is speaking (or "unknown")

bash

python3 scripts/voice_identify.py <audio_file> [--threshold 0.75]

Compares audio against all voice-refs/*-ref*.* using UniSpeech-SAT x-vector embeddings. First run downloads model (~360MB) to /tmp/hf_models/.

Accuracy: Reliably separates male/female voices. Same-gender speakers need ≥5s audio for best results. Threshold 0.75 is default; raise to 0.85 for stricter matching.

Op 2 · Add Voice to Library (声音入库)#

Input: audio + speaker name → stores in voice library

1. Copy audio to voice-refs/<name>-ref1.<ext>
2. Transcribe to get reference text: whisper <audio> --model small --output_format txt --output_dir /tmp
3. Add entry to TOOLS.md (see format in references/)
4. Register speaker in voice_identify.py SPEAKER_MAP

Good reference audio: 10-15s clear speech, minimal noise, natural pace. 5s minimum.

Op 3 · Voice Library CRUD (声音库管理)#

• List: Check TOOLS.md voice library section + ls voice-refs/
• Add: See Op 2
• Update: Replace file in voice-refs/, update TOOLS.md entry
• Delete: Remove file from voice-refs/, remove TOOLS.md entry, remove from SPEAKER_MAP

Op 4 · Voice Clone (声音克隆)#

Input: text + library speaker → Output: audio in that speaker's voice

bash

set -a; source <env_file_with_SF_API_KEY>; set +a

python3 scripts/cosyvoice_clone.py \
  --text "Text to speak" \
  --ref voice-refs/<speaker>-ref1.<ext> \
  --ref-text "What is said in reference audio" \
  --output /tmp/clone_output.wav

Long reference (>15s): truncate first with ffmpeg -y -i <ref> -t 15 -ar 24000 -ac 1 /tmp/ref_trimmed.wav.

Op 5 · Transcribe (纯转文字)#

Input: audio → Output: text

bash

whisper <audio_file> --model small --output_format txt --output_dir /tmp --language <lang>

Languages: zh (Chinese), en (English), ja (Japanese). Omit for auto-detect.

Op 6 · Transcribe + Identify (转文字+识别)#

Input: audio → Output: who said what

Run Op 5 and Op 1 in parallel, report both results together.

Op 7 · Speaker Verification (声纹验证)#

Input: two audio files → Output: same person or not

bash

python3 scripts/voice_identify.py <audio_1> --threshold 0.75
python3 scripts/voice_identify.py <audio_2> --threshold 0.75

Compare the top-ranked speaker from both runs. If they match → same person. For direct pairwise comparison without a library, extract embeddings and compute cosine similarity (see voice_identify.py internals).

Op 8 · Voice Swap (声音换皮)#

Input: audio + library speaker → Output: same words, different voice

1. Transcribe input audio (Op 5)
2. Clone with target speaker's voice (Op 4), using transcribed text

Op 9 · Persona Voice Reply — from Audio (人格化语音回复·语音版)#

Input: audio question + library speaker → Output: AI answer in that speaker's voice

1. Transcribe the question (Op 5)
2. Generate answer text via LLM
3. Clone answer with target speaker's voice (Op 4)

Op 10 · Persona Voice Reply — from Text (人格化语音回复·文字版)#

Input: text question + library speaker → Output: AI answer in that speaker's voice

1. Generate answer text via LLM
2. Clone answer with target speaker's voice (Op 4)

Send Audio (Feishu)#

bash

set -a; source <env_file>; set +a
bash scripts/feishu_send_audio.sh <wav_file> <receive_id>

Converts wav → opus, uploads, sends as voice message. Requires FEISHU_APP_ID + FEISHU_APP_SECRET env vars.

Extract Audio from Video#

bash

ffmpeg -y -i <video_file> -vn -ar 24000 -ac 1 /tmp/extracted_audio.wav