Fun-CosyVoice 3.0已经发布了

亮点🔥

Fun-CosyVoice 3.0是基于大语言模型（LLM）的先进文本转语音（TTS）系统，在内容一致性、说话人相似度和韵律自然度上全面超越前代（CosyVoice 2.0）。该系统专为开放场景下的零样本多语言语音合成而设计。

核心特性

• 语言覆盖：支持9种常用语言（中、英、日、韩、德、西、法、意、俄）及18+种汉语方言/口音（广东话、闽南语、四川话、东北话、陕西话、山西话、上海话、天津话、山东话、宁夏话、甘肃话等），同时支持多语言/跨语言零样本音色克隆。
• 内容一致性&自然度：在文本还原度、音色相似度和韵律流畅性方面达到业界领先水平。
• 发音修复：支持中文拼音和英文CMU音素的发音校正，提供更强可控性，满足生产级需求。
• 文本归一化：无需传统前端模块即可正确朗读数字、特殊符号及各类文本格式。
• 双流式处理：同时支持文本输入流与音频输出流，在保持高质量音频输出的情况下实现最低150ms的延迟。
• 指令控制：支持语言、方言、情感、语速、音量等多种调节指令。

路线图

• 2025年12月

  • 发布Fun-CosyVoice3-0.5B-2512基础模型、强化学习模型及其训练/推理脚本
  • 发布Fun-CosyVoice3-0.5B modelscope gradio空间

• 2025年8月

  • 感谢NVIDIA张悦铠的贡献，新增了triton trtllm运行时支持以及cosyvoice2 grpo训练支持

• 2025年7月

  • 发布Fun-CosyVoice 3.0评估集

• 2025年5月

  • 添加CosyVoice2-0.5B vllm支持

• 2024年12月

  • 发布25hz CosyVoice2-0.5B版本

• 2024年9月

  • 25hz CosyVoice-300M基础模型
  • 25hz CosyVoice-300M语音转换功能

• 2024年8月

  • 采用重复感知采样(RAS)推理提升大语言模型稳定性
  • 支持流式推理模式，包括用于实时率优化的kv缓存和sdpa技术

• 2024年7月

  • 支持流匹配训练
  • 当ttsfrd不可用时支持WeTextProcessing
  • Fastapi服务端与客户端

评估

安装

克隆与安装

• 克隆仓库

  git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git # If you failed to clone the submodule due to network failures, please run the following command until success cd CosyVoice git submodule update --init --recursive

• 安装 Conda：请参阅 https://docs.conda.io/en/latest/miniconda.html

• 创建 Conda 环境：

  conda create -n cosyvoice -y python=3.10 conda activate cosyvoice pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com # If you encounter sox compatibility issues # ubuntu sudo apt-get install sox libsox-dev # centos sudo yum install sox sox-devel

模型下载

fromhuggingface_hubimportsnapshot_download snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',local_dir='pretrained_models/Fun-CosyVoice3-0.5B')snapshot_download('FunAudioLLM/CosyVoice-ttsfrd',local_dir='pretrained_models/CosyVoice-ttsfrd')

可选地，您可以解压ttsfrd资源并安装ttsfrd包以获得更好的文本规范化性能。

请注意此步骤并非必需。若不安装ttsfrd包，我们将默认使用wetext。

cd pretrained_models/CosyVoice-ttsfrd/ unzip resource.zip -d . pip install ttsfrd_dependency-0.1-py3-none-any.whl pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl

基本用法

importsys sys.path.append('third_party/Matcha-TTS')fromcosyvoice.cli.cosyvoiceimportAutoModelimporttorchaudio""" CosyVoice3 Usage, check https://funaudiollm.github.io/cosyvoice3/ for more details """cosyvoice=AutoModel(model_dir='pretrained_models/Fun-CosyVoice3-0.5B')# en zero_shot usagefori,jinenumerate(cosyvoice.inference_zero_shot('CosyVoice is undergoing a comprehensive upgrade, providing more accurate, stable, faster, and better voice generation capabilities.','You are a helpful assistant.<|endofprompt|>希望你以后能够做的比我还好呦。','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('zero_shot_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)# zh zero_shot usagefori,jinenumerate(cosyvoice.inference_zero_shot('八百标兵奔北坡，北坡炮兵并排跑，炮兵怕把标兵碰，标兵怕碰炮兵炮。','You are a helpful assistant.<|endofprompt|>希望你以后能够做的比我还好呦。','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('zero_shot_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)# fine grained control, for supported control, check cosyvoice/tokenizer/tokenizer.py#L280fori,jinenumerate(cosyvoice.inference_cross_lingual('You are a helpful assistant.<|endofprompt|>[breath]因为他们那一辈人[breath]在乡里面住的要习惯一点，[breath]邻居都很活络，[breath]嗯，都很熟悉。[breath]','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('fine_grained_control_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)# instruct usage, for supported control, check cosyvoice/utils/common.py#L28fori,jinenumerate(cosyvoice.inference_instruct2('好少咯，一般系放嗰啲国庆啊，中秋嗰啲可能会咯。','You are a helpful assistant. 请用广东话表达。<|endofprompt|>','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('instruct_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)fori,jinenumerate(cosyvoice.inference_instruct2('收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。','You are a helpful assistant. 请用尽可能快地语速说一句话。<|endofprompt|>','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('instruct_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)# hotfix usagefori,jinenumerate(cosyvoice.inference_zero_shot('高管也通过电话、短信、微信等方式对报道[j][ǐ]予好评。','You are a helpful assistant.<|endofprompt|>希望你以后能够做的比我还好呦。','./asset/zero_shot_prompt.wav',stream=False)):torchaudio.save('hotfix_{}.wav'.format(i),j['tts_speech'],cosyvoice.sample_rate)

致谢

1. 我们借鉴了大量来自FunASR的代码。
2. 我们借鉴了大量来自FunCodec的代码。
3. 我们借鉴了大量来自Matcha-TTS的代码。
4. 我们借鉴了大量来自AcademiCodec的代码。
5. 我们借鉴了大量来自WeNet的代码。

引用文献

@article{du2024cosyvoice, title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens}, author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and Yang, Yexin and Hu, Hangrui and Zheng, Siqi and Gu, Yue and Ma, Ziyang and others}, journal={arXiv preprint arXiv:2407.05407}, year={2024} } @article{du2024cosyvoice, title={Cosyvoice 2: Scalable streaming speech synthesis with large language models}, author={Du, Zhihao and Wang, Yuxuan and Chen, Qian and Shi, Xian and Lv, Xiang and Zhao, Tianyu and Gao, Zhifu and Yang, Yexin and Gao, Changfeng and Wang, Hui and others}, journal={arXiv preprint arXiv:2412.10117}, year={2024} } @article{du2025cosyvoice, title={CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training}, author={Du, Zhihao and Gao, Changfeng and Wang, Yuxuan and Yu, Fan and Zhao, Tianyu and Wang, Hao and Lv, Xiang and Wang, Hui and Shi, Xian and An, Keyu and others}, journal={arXiv preprint arXiv:2505.17589}, year={2025} } @inproceedings{lyu2025build, title={Build LLM-Based Zero-Shot Streaming TTS System with Cosyvoice}, author={Lyu, Xiang and Wang, Yuxuan and Zhao, Tianyu and Wang, Hao and Liu, Huadai and Du, Zhihao}, booktitle={ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={1--2}, year={2025}, organization={IEEE} }