ControlFoley Audio Generator#

A multi-functional audio generation tool powered by the ControlFoley model, integrating video sound effect (SFX) generation, video background music composition, text-to-audio and other functions to realize diversified creative audio generation.

This tool supports four modes: Video-to-Audio (V2A), Text-Controlled Video-to-Audio (TC-V2A), Audio-Controlled Video-to-Audio (AC-V2A), and Text-to-Audio (T2A).

Basic Info#

FieldValueService OperatorXiaomi LLM Plus TeamAPI Endpointhttps://controlfoley.ai.xiaomi.comOpen Source Repohttps://github.com/xiaomi-research/controlfoleyProject Pagehttps://yjx-research.github.io/ControlFoley_web_page/Online Demohttps://yjx-research.github.io/ControlFoley_web_page/#try-genModel Weightshttps://huggingface.co/YJX-Xiaomi/ControlFoley/API KeyNot requiredScript Pathscripts/foley.py

Prerequisites#

bash

python3 --version   # Python 3.x
curl --version      # curl for API submission
ffmpeg -version     # optional, for audio format conversion

Modes#

ModeCommandInputOutputDescriptionV2Av2a video.mp4Video file.mp4 + .flacGenerate audio matching the video contentTC-V2Av2a video.mp4 --prompt "text"Video + text.mp4 + .flacGenerate audio aligned with text prompts while staying synchronized with the videoAC-V2Av2a video.mp4 --ref-audio ref.wavVideo + reference audio.mp4 + .flacGenerate audio with timbre matching reference audio while staying synchronized with the videoT2At2a "prompt"Text description.flacGenerate audio from text descriptions

Usage (CLI version)#

1. Text-to-Audio (T2A, default 8s)#

bash

python3 scripts/foley.py t2a "dog barking loudly in a park"

2. Video-to-Audio (V2A)#

bash

python3 scripts/foley.py v2a input.mp4

3. Text-Controlled Video-to-Audio (TC-V2A)#

bash

python3 scripts/foley.py v2a input.mp4 --prompt "footsteps on gravel with birds chirping"

4. Audio-Controlled Video-to-Audio (AC-V2A)#

bash

python3 scripts/foley.py v2a input.mp4 --ref-audio reference.wav

5. Specify duration#

bash

python3 scripts/foley.py t2a "A mountain stream murmurs, its gentle current lapping against the pebbles." --duration 15

6. Generate multiple candidates#

bash

python3 scripts/foley.py t2a "cat purring softly" --count 3

7. Fixed seed (reproducible results)#

bash

python3 scripts/foley.py t2a "rain on a tin roof" --seed 42

8. List available models#

bash

python3 scripts/foley.py models

Usage (API version)#

POST#

bash

curl -X POST "https://controlfoley.ai.xiaomi.com/api/v1/v2a/submit" -F "file=@video_path" -F "prompt=footsteps on gravel with birds chirping"

return

json

{"taskId": "xxx", "message": "Task submitted successfully"}

GET#

1. Available Models

bash

curl -X GET "https://controlfoley.ai.xiaomi.com/api/v1/v2a/models"

return

json

{"models":[{"name":"ControlFoley","enabled":true}]}

2. Status Inquiry

bash

curl -X GET "https://controlfoley.ai.xiaomi.com/api/v1/v2a/status/{taskId}"

return

1. success：

json

{"urls":["{Domain name}/ControlFoley_output/{taskId}/{filename}"],"status":"success","done":true}

1. processing：

json

{"status":"processing","done":false}

1. pending:

json

{"status":"pending","queue_pos":1,"queue_position":1,"total_queue":2,"done":false}

3. Result Download

bash

curl -X GET "https://controlfoley.ai.xiaomi.com/ControlFoley_output/{taskId}/{filename}" --output ./output.flac

4. Status Inquiry & Result Download

bash

curl -X GET "https://controlfoley.ai.xiaomi.com/api/v1/v2a/status_download/{taskId}" --output-dir ./output --output audio.zip

Parameters#

T2A (Text-to-Audio)#

ParameterDescriptionDefaultExamplepromptAudio description text (required)—"dog barking in park"``--modelModel IDControlFoley``--model ControlFoley``--durationAudio length in seconds (max 30)8``--duration 15``--negativeNegative prompt to exclude unwanted sounds—--negative "noise, human voice"``--cfgCFG strength — higher = stricter prompt adherence4.5``--cfg 6.0``--countNumber of variants to generate (1–5)1``--count 3``--seedFixed random seed for reproducibility—--seed 42``-o/--outdirOutput directory./output``-o ./my_audio

V2A (Video-to-Audio)#

ParameterDescriptionDefaultExamplevideoInput video path (required)—input.mp4``--modelModel IDControlFoley``--model ControlFoley``--promptText prompt to guide audio generation (TC-V2A)—--prompt "keyboard tapping"``--negativeNegative prompt to exclude unwanted sounds—--negative "music, noise"``--ref-audioReference audio file for timbre control (AC-V2A)—--ref-audio reference.wav``--cfgCFG strength4.5``--cfg 7.0``--countNumber of variants to generate (1–5)1``--count 2``--seedFixed random seed (not forwarded to API currently)—--seed 42``-o/--outdirOutput directory./output``-o ./results

Prompt Tips#

• Be specific: "cat footsteps on wooden floor" beats "cat sound"
• Use negative prompts: --negative "human voice, music, noise" to filter unwanted audio
• CFG tuning: high CFG (6.0–7.5) for precise control, low CFG (3.0–4.5) for creative freedom

Output & Post-Processing#

• Audio: .flac (44100 Hz, lossless)
• Video: .mp4 (original video + generated audio track)
• Results saved to --outdir, paths printed to stdout

Convert to MP3 for sharing:

bash

ffmpeg -i output.flac -codec:a libmp3lame -qscale:a 2 output.mp3

Error Handling#

IssueCauseFixInternal URL inaccessibleResult URL uses .xiaomi.srv internal domainScript auto-falls back to /api/v1/v2a/ControlFoley_output/{task_id}/{filename}Queue busyTask is waitingScript auto-polls up to ~5 min; check load via curl $API_BASE/healthModel unavailableModel not enabledRun foley.py models to see available modelsTask timeoutService overloadedResubmit the task

API Reference#

See ./references/api-reference.md for full endpoint documentation.

⚠️ Privacy & Security#

• Service Operator: Cloud processing is operated by the Xiaomi LLM Plus Team at https://controlfoley.ai.xiaomi.com
• Data Upload: V2A/TC-V2A/AC-V2A modes upload the full video file to the remote service for processing. Do not upload videos containing sensitive personal or identifiable information
• Data Processing: Uploaded videos and audio are used solely for audio generation. Results are returned via URL. Refer to the Xiaomi LLM Plus Team's terms of service for data retention and access control policies
• No API Key Required: The service requires no authentication — please use it responsibly to avoid unnecessary load
• Recommendation: Before first use, validate with a small, non-sensitive test clip