| 1 | # F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching |
| 2 | |
| 3 | [](https://github.com/SWivid/F5-TTS) |
| 4 | [](https://arxiv.org/abs/2410.06885) |
| 5 | [](https://swivid.github.io/F5-TTS/) |
| 6 | [](https://huggingface.co/spaces/mrfakename/E2-F5-TTS) |
| 7 | [](https://modelscope.cn/studios/AI-ModelScope/E2-F5-TTS) |
| 8 | [](https://x-lance.sjtu.edu.cn/) |
| 9 | [](https://www.sii.edu.cn/) |
| 10 | [](https://www.pcl.ac.cn) |
| 11 | <!-- <img src="https://github.com/user-attachments/assets/12d7749c-071a-427c-81bf-b87b91def670" alt="Watermark" style="width: 40px; height: auto"> --> |
| 12 | |
| 13 | **F5-TTS**: Diffusion Transformer with ConvNeXt V2, faster trained and inference. |
| 14 | |
| 15 | **E2 TTS**: Flat-UNet Transformer, closest reproduction from [paper](https://arxiv.org/abs/2406.18009). |
| 16 | |
| 17 | **Sway Sampling**: Inference-time flow step sampling strategy, greatly improves performance |
| 18 | |
| 19 | ### Thanks to all the contributors ! |
| 20 | |
| 21 | ## News |
| 22 | - **2025/03/12**: 🔥 F5-TTS v1 base model with better training and inference performance. [Few demo](https://swivid.github.io/F5-TTS_updates). |
| 23 | - **2024/10/08**: F5-TTS & E2 TTS base models on [🤗 Hugging Face](https://huggingface.co/SWivid/F5-TTS), [🤖 Model Scope](https://www.modelscope.cn/models/SWivid/F5-TTS_Emilia-ZH-EN), [🟣 Wisemodel](https://wisemodel.cn/models/SJTU_X-LANCE/F5-TTS_Emilia-ZH-EN). |
| 24 | |
| 25 | ## Installation |
| 26 | |
| 27 | ### Create a separate environment if needed |
| 28 | |
| 29 | ```bash |
| 30 | # Create a conda env with python_version>=3.10 (you could also use virtualenv) |
| 31 | conda create -n f5-tts python=3.11 |
| 32 | conda activate f5-tts |
| 33 | |
| 34 | # Install FFmpeg if you haven't yet |
| 35 | conda install ffmpeg |
| 36 | ``` |
| 37 | |
| 38 | ### Install PyTorch with matched device |
| 39 | |
| 40 | <details> |
| 41 | <summary>NVIDIA GPU</summary> |
| 42 | |
| 43 | > ```bash |
| 44 | > # Install pytorch with your CUDA version, e.g. |
| 45 | > pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128 |
| 46 | > |
| 47 | > # And also possible previous versions, e.g. |
| 48 | > pip install torch==2.4.0+cu124 torchaudio==2.4.0+cu124 --extra-index-url https://download.pytorch.org/whl/cu124 |
| 49 | > # etc. |
| 50 | > ``` |
| 51 | |
| 52 | </details> |
| 53 | |
| 54 | <details> |
| 55 | <summary>AMD GPU</summary> |
| 56 | |
| 57 | > ```bash |
| 58 | > # Install pytorch with your ROCm version (Linux only), e.g. |
| 59 | > pip install torch==2.9.1+rocm7.2 torchaudio==2.9.1+rocm7.2 --extra-index-url https://download.pytorch.org/whl/rocm7.2 |
| 60 | > |
| 61 | > # For older GPUs (RDNA1/2/3 only): |
| 62 | > # pip install torch==2.5.1+rocm6.2 torchaudio==2.5.1+rocm6.2 --extra-index-url https://download.pytorch.org/whl/rocm6.2 |
| 63 | > ``` |
| 64 | > |
| 65 | > **Note:** RDNA 3.5 and RDNA 4 GPUs (Radeon 8050S/8060S, RX 9060/9070 series) require |
| 66 | > ROCm 7.x — these architectures (gfx1151/gfx1201) are not included in ROCm 6.x |
| 67 | > ([6.2 compatibility matrix](https://rocm.docs.amd.com/en/docs-6.2.4/compatibility/compatibility-matrix.html) vs |
| 68 | > [7.2 compatibility matrix](https://rocm.docs.amd.com/en/docs-7.2.3/compatibility/compatibility-matrix.html)). |
| 69 | > Using ROCm 6.x on these GPUs causes `HIP error: invalid device function` ([#1236](https://github.com/SWivid/F5-TTS/issues/1236)). |
| 70 | |
| 71 | </details> |
| 72 | |
| 73 | <details> |
| 74 | <summary>Intel GPU</summary> |
| 75 | |
| 76 | > ```bash |
| 77 | > # Install pytorch with your XPU version, e.g. |
| 78 | > # Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit must be installed |
| 79 | > pip install torch torchaudio --index-url https://download.pytorch.org/whl/test/xpu |
| 80 | > |
| 81 | > # Intel GPU support is also available through IPEX (Intel® Extension for PyTorch) |
| 82 | > # IPEX does not require the Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit |
| 83 | > # See: https://pytorch-extension.intel.com/installation?request=platform |
| 84 | > ``` |
| 85 | |
| 86 | </details> |
| 87 | |
| 88 | <details> |
| 89 | <summary>Apple Silicon</summary> |
| 90 | |
| 91 | > ```bash |
| 92 | > # Install the stable pytorch, e.g. |
| 93 | > pip install torch torchaudio |
| 94 | > ``` |
| 95 | |
| 96 | </details> |
| 97 | |
| 98 | ### Then you can choose one from below: |
| 99 | |
| 100 | > ### 1. As a pip package (if just for inference) |
| 101 | > |
| 102 | > ```bash |
| 103 | > pip install f5-tts |
| 104 | > ``` |
| 105 | > |
| 106 | > ### 2. Local editable (if also do training, finetuning) |
| 107 | > |
| 108 | > ```bash |
| 109 | > git clone https://github.com/SWivid/F5-TTS.git |
| 110 | > cd F5-TTS |
| 111 | > # git submodule update --init --recursive # (optional, if use bigvgan as vocoder) |
| 112 | > pip install -e . |
| 113 | > ``` |
| 114 | |
| 115 | ### Docker usage also available |
| 116 | ```bash |
| 117 | # Build from Dockerfile |
| 118 | docker build -t f5tts:v1 . |
| 119 | |
| 120 | # Run from GitHub Container Registry |
| 121 | docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main |
| 122 | |
| 123 | # Quickstart if you want to just run the web interface (not CLI) |
| 124 | docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main f5-tts_infer-gradio --host 0.0.0.0 |
| 125 | ``` |
| 126 | |
| 127 | ### Runtime |
| 128 | |
| 129 | Deployment solution with Triton and TensorRT-LLM. |
| 130 | |
| 131 | #### Benchmark Results |
| 132 | Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE. |
| 133 | |
| 134 | | Model | Concurrency | Avg Latency | RTF | Mode | |
| 135 | |---------------------|----------------|-------------|--------|-----------------| |
| 136 | | F5-TTS Base (Vocos) | 2 | 253 ms | 0.0394 | Client-Server | |
| 137 | | F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.0402 | Offline TRT-LLM | |
| 138 | | F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.1467 | Offline Pytorch | |
| 139 | |
| 140 | See [detailed instructions](src/f5_tts/runtime/triton_trtllm/README.md) for more information. |
| 141 | |
| 142 | |
| 143 | ## Inference |
| 144 | |
| 145 | - In order to achieve desired performance, take a moment to read [detailed guidance](src/f5_tts/infer). |
| 146 | - By properly searching the keywords of problem encountered, [issues](https://github.com/SWivid/F5-TTS/issues?q=is%3Aissue) are very helpful. |
| 147 | |
| 148 | ### 1. Gradio App |
| 149 | |
| 150 | Currently supported features: |
| 151 | |
| 152 | - Basic TTS with Chunk Inference |
| 153 | - Multi-Style / Multi-Speaker Generation |
| 154 | - Voice Chat powered by Qwen2.5-3B-Instruct |
| 155 | - [Custom inference with more language support](src/f5_tts/infer/SHARED.md) |
| 156 | |
| 157 | ```bash |
| 158 | # Launch a Gradio app (web interface) |
| 159 | f5-tts_infer-gradio |
| 160 | |
| 161 | # Specify the port/host |
| 162 | f5-tts_infer-gradio --port 7860 --host 0.0.0.0 |
| 163 | |
| 164 | # Launch a share link |
| 165 | f5-tts_infer-gradio --share |
| 166 | ``` |
| 167 | |
| 168 | <details> |
| 169 | <summary>NVIDIA device docker compose file example</summary> |
| 170 | |
| 171 | ```yaml |
| 172 | services: |
| 173 | f5-tts: |
| 174 | image: ghcr.io/swivid/f5-tts:main |
| 175 | ports: |
| 176 | - "7860:7860" |
| 177 | environment: |
| 178 | GRADIO_SERVER_PORT: 7860 |
| 179 | entrypoint: ["f5-tts_infer-gradio", "--port", "7860", "--host", "0.0.0.0"] |
| 180 | deploy: |
| 181 | resources: |
| 182 | reservations: |
| 183 | devices: |
| 184 | - driver: nvidia |
| 185 | count: 1 |
| 186 | capabilities: [gpu] |
| 187 | |
| 188 | volumes: |
| 189 | f5-tts: |
| 190 | driver: local |
| 191 | ``` |
| 192 | |
| 193 | </details> |
| 194 | |
| 195 | ### 2. CLI Inference |
| 196 | |
| 197 | ```bash |
| 198 | # Run with flags |
| 199 | # Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage) |
| 200 | f5-tts_infer-cli --model F5TTS_v1_Base \ |
| 201 | --ref_audio "provide_prompt_wav_path_here.wav" \ |
| 202 | --ref_text "The content, subtitle or transcription of reference audio." \ |
| 203 | --gen_text "Some text you want TTS model generate for you." |
| 204 | |
| 205 | # Run with default setting. src/f5_tts/infer/examples/basic/basic.toml |
| 206 | f5-tts_infer-cli |
| 207 | # Or with your own .toml file |
| 208 | f5-tts_infer-cli -c custom.toml |
| 209 | |
| 210 | # Multi voice. See src/f5_tts/infer/README.md |
| 211 | f5-tts_infer-cli -c src/f5_tts/infer/examples/multi/story.toml |
| 212 | ``` |
| 213 | |
| 214 | |
| 215 | ## Training |
| 216 | |
| 217 | ### 1. With Hugging Face Accelerate |
| 218 | |
| 219 | Refer to [training & finetuning guidance](src/f5_tts/train) for best practice. |
| 220 | |
| 221 | ### 2. With Gradio App |
| 222 | |
| 223 | ```bash |
| 224 | # Quick start with Gradio web interface |
| 225 | f5-tts_finetune-gradio |
| 226 | ``` |
| 227 | |
| 228 | Read [training & finetuning guidance](src/f5_tts/train) for more instructions. |
| 229 | |
| 230 | |
| 231 | ## [Evaluation](src/f5_tts/eval) |
| 232 | |
| 233 | |
| 234 | ## Development |
| 235 | |
| 236 | Use pre-commit to ensure code quality (will run linters and formatters automatically): |
| 237 | |
| 238 | ```bash |
| 239 | pip install pre-commit |
| 240 | pre-commit install |
| 241 | ``` |
| 242 | |
| 243 | When making a pull request, before each commit, run: |
| 244 | |
| 245 | ```bash |
| 246 | pre-commit run --all-files |
| 247 | ``` |
| 248 | |
| 249 | Note: Some model components have linting exceptions for E722 to accommodate tensor notation. |
| 250 | |
| 251 | |
| 252 | ## Acknowledgements |
| 253 | |
| 254 | - [E2-TTS](https://arxiv.org/abs/2406.18009) brilliant work, simple and effective |
| 255 | - [Emilia](https://arxiv.org/abs/2407.05361), [WenetSpeech4TTS](https://arxiv.org/abs/2406.05763), [LibriTTS](https://arxiv.org/abs/1904.02882), [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) valuable datasets |
| 256 | - [lucidrains](https://github.com/lucidrains) initial CFM structure with also [bfs18](https://github.com/bfs18) for discussion |
| 257 | - [SD3](https://arxiv.org/abs/2403.03206) & [Hugging Face diffusers](https://github.com/huggingface/diffusers) DiT and MMDiT code structure |
| 258 | - [torchdiffeq](https://github.com/rtqichen/torchdiffeq) as ODE solver, [Vocos](https://huggingface.co/charactr/vocos-mel-24khz) and [BigVGAN](https://github.com/NVIDIA/BigVGAN) as vocoder |
| 259 | - [FunASR](https://github.com/modelscope/FunASR), [faster-whisper](https://github.com/SYSTRAN/faster-whisper), [UniSpeech](https://github.com/microsoft/UniSpeech), [SpeechMOS](https://github.com/tarepan/SpeechMOS) for evaluation tools |
| 260 | - [ctc-forced-aligner](https://github.com/MahmoudAshraf97/ctc-forced-aligner) for speech edit test |
| 261 | - [mrfakename](https://x.com/realmrfakename) huggingface space demo ~ |
| 262 | - [f5-tts-mlx](https://github.com/lucasnewman/f5-tts-mlx/tree/main) Implementation with MLX framework by [Lucas Newman](https://github.com/lucasnewman) |
| 263 | - [F5-TTS-ONNX](https://github.com/DakeQQ/F5-TTS-ONNX) ONNX Runtime version by [DakeQQ](https://github.com/DakeQQ) |
| 264 | - [Yuekai Zhang](https://github.com/yuekaizhang) Triton and TensorRT-LLM support ~ |
| 265 | |
| 266 | ## Citation |
| 267 | If our work and codebase is useful for you, please cite as: |
| 268 | ``` |
| 269 | @article{chen-etal-2024-f5tts, |
| 270 | title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, |
| 271 | author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen}, |
| 272 | journal={arXiv preprint arXiv:2410.06885}, |
| 273 | year={2024}, |
| 274 | } |
| 275 | ``` |
| 276 | ## License |
| 277 | |
| 278 | Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia, which is an in-the-wild dataset. Sorry for any inconvenience this may cause. |
| 279 |