File size: 9,921 Bytes
fa99f66 0c97490 fa99f66 0c97490 2674843 0c97490 fa99f66 0c97490 fa99f66 0c97490 fa99f66 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | ---
license: cc-by-nc-nd-4.0
datasets:
- amphion/Emilia-Dataset
language:
- en
- zh
- ja
- ko
- de
- fr
tags:
- tts
- vc
- svs
- svc
- music
---
# Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation
[](https://arxiv.org/abs/2508.16332)
[](https://huggingface.co/RMSnow/Vevo2)
[](https://versasinger.github.io/)
[](https://github.com/open-mmlab/Amphion/blob/main/models/svc/vevo2/README.md)
We present **Vevo2**, a unified and controllable framework for speech and singing voice generation. Vevo2 bridges controllable speech and singing voice generation via unified prosody learning, and supports a comprehensive set of generation tasks, including:
1. Zero-shot Text-to-Speech (TTS), Text-to-Singing, and Singing Voice Synthesis (SVS)
2. Style-preserved Voice/Singing Voice Conversion (VC/SVC)
3. Style-converted Voice/Singing Voice Conversion (VC/SVC)
4. Speech/Singing Voice Editing
5. Singing Style Conversion
6. Humming-to-Singing and Instrument-to-Singing
## Pre-trained Models
We have included the following pre-trained models at [🤗 RMSnow/Vevo2](https://huggingface.co/RMSnow/Vevo2):
| Model | Description | Pre-trained Data and Checkpoint |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Prosody Tokenizer** | Converting speech/singing waveform to **coarse-grained prosody tokens** (which can also be interpreted as *melody contour* from a musical perspective). It is a single codebook VQ-VAE with a vocabulary size of 512. The frame rate is 6.25 Hz. (i.e., **56.25 bps**) | [🤗 Emilia-101k, SingNet-7k](https://huggingface.co/RMSnow/Vevo2/tree/main/tokenizer/prosody_fvq512_6.25hz) |
| **Content-Style Tokenizer** | Converting speech/singing waveform to **fine-grained content-style tokens**. It is a single codebook VQ-VAE with a vocabulary size of 16384. The frame rate is 12.5 Hz. (i.e., **175 bps**) | [🤗 Emilia-101k, SingNet-7k](https://huggingface.co/RMSnow/Vevo2/tree/main/tokenizer/contentstyle_fvq16384_12.5hz) |
| **AR Model** | A Qwen-based (Qwen2.5-0.5B) large language model post-trained to predict content-style tokens from text tokens and optionally prosody tokens, with unified prosody learning across speech and singing. | [🤗 Emilia-101k, SingNet-7k](https://huggingface.co/RMSnow/Vevo2/tree/main/contentstyle_modeling/posttrained) |
| **Flow-matching Transformer** | Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (350M). | [🤗 Emilia-101k, SingNet-7k](https://huggingface.co/RMSnow/Vevo2/tree/main/acoustic_modeling/fm_emilia101k_singnet7k_repa) |
| **Vocoder** | Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M). | [🤗 Emilia-101k, SingNet-7k](https://huggingface.co/RMSnow/Vevo2/tree/main/vocoder) |
The training data includes:
- **Emilia-101k**: about 101k hours of speech data
- **SingNet-7k**: about 7,000 hours of internal singing voice data, preprocessed using the [SingNet pipeline](https://openreview.net/pdf?id=X6ffdf6nh3).
## Usage
You can refer to our [recipe](https://github.com/open-mmlab/Amphion/blob/main/models/svc/vevo2/README.md) at GitHub for more usage details. For example, to use the speech/singing voice editing, after you clone the Amphion github repository, you can use the script like:
```python
import os
import torch
from huggingface_hub import snapshot_download
from models.svc.vevo2.vevo2_utils import *
def vevo2_editing(
tgt_text,
raw_wav_path,
raw_text=None,
output_path=None,
):
gen_audio = inference_pipeline.inference_ar_and_fm(
target_text=tgt_text,
prosody_wav_path=raw_wav_path,
style_ref_wav_path=raw_wav_path,
style_ref_wav_text=raw_text,
timbre_ref_wav_path=raw_wav_path,
use_prosody_code=True,
)
assert output_path is not None
save_audio(gen_audio, output_path=output_path)
def load_inference_pipeline():
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
local_dir = snapshot_download(
repo_id="RMSnow/Vevo2",
repo_type="model",
local_dir="./ckpts/Vevo2",
resume_download=True,
)
content_style_tokenizer_ckpt_path = os.path.join(
local_dir, "tokenizer/contentstyle_fvq16384_12.5hz"
)
prosody_tokenizer_ckpt_path = os.path.join(
local_dir, "tokenizer/prosody_fvq512_6.25hz"
)
ar_cfg_path = os.path.join(
local_dir, "contentstyle_modeling/posttrained/amphion_config.json"
)
ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/posttrained")
fmt_cfg_path = os.path.join(
local_dir, "acoustic_modeling/fm_emilia101k_singnet7k_repa/config.json"
)
fmt_ckpt_path = os.path.join(
local_dir, "acoustic_modeling/fm_emilia101k_singnet7k_repa"
)
vocoder_cfg_path = os.path.join(local_dir, "vocoder/config.json")
vocoder_ckpt_path = os.path.join(local_dir, "vocoder")
inference_pipeline = Vevo2InferencePipeline(
prosody_tokenizer_ckpt_path=prosody_tokenizer_ckpt_path,
content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path,
ar_cfg_path=ar_cfg_path,
ar_ckpt_path=ar_ckpt_path,
fmt_cfg_path=fmt_cfg_path,
fmt_ckpt_path=fmt_ckpt_path,
vocoder_cfg_path=vocoder_cfg_path,
vocoder_ckpt_path=vocoder_ckpt_path,
device=device,
)
return inference_pipeline
if __name__ == "__main__":
inference_pipeline = load_inference_pipeline()
output_dir = "./models/svc/vevo2/output"
os.makedirs(output_dir, exist_ok=True)
### Zero-shot Singing Editing ###
adele_path = "./models/svc/vevosing/wav/adele.wav"
adele_text = "Never mind, I'll find someone like you. I wish nothing but."
vevo2_editing(
tgt_text="Never mind, you'll find anyone like me. You wish nothing but.",
raw_wav_path=adele_path,
raw_text=adele_text, # "Never mind, I'll find someone like you. I wish nothing but."
output_path=os.path.join(output_dir, "editing_adele.wav"),
)
vevo2_editing(
tgt_text="对你的人生如果你有太多的期盼，跌倒了就不该低头认输，为什么啊，人要这么的彷徨堕",
raw_wav_path=jaychou_path,
raw_text=jaychou_text, # "对这个世界如果你有太多的抱怨，跌倒了就不该继续往前走，为什么，人要这么的脆弱堕"
output_path=os.path.join(output_dir, "editing_jaychou.wav"),
)
```
## Citations
If you find this work useful for your research, please cite our paper:
```bibtex
@article{vevo2,
title={Vevo2: Bridging Controllable Speech and Singing Voice Generation via Unified Prosody Learning},
author={Zhang, Xueyao and Zhang, Junan and Wang, Yuancheng and Wang, Chaoren and Chen, Yuanzhe and Jia, Dongya and Chen, Zhuo and Wu, Zhizheng},
journal={arXiv preprint arXiv:2508.16332},
year={2025}
}
@inproceedings{vevo,
author = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
title = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
booktitle = {{ICLR}},
publisher = {OpenReview.net},
year = {2025}
}
```
If you use the Vevo2 pre-trained models or training recipe of Amphion, please also cite:
```bibtex
@article{amphion2,
title = {Overview of the Amphion Toolkit (v0.2)},
author = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
year = {2025},
journal = {arXiv preprint arXiv:2501.15442},
}
@inproceedings{amphion,
author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
year={2024}
}
``` |