Transformers documentation

Pocket TTS

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Model ini disumbangkan ke Hugging Face Transformers pada 2026-10-20.

Pocket TTS

Ikhtisar

Pocket TTS adalah model text-to-speech 100M parameter dari Kyutai, sebuah Continuous Audio Language Model yang berjalan lebih cepat dari waktu nyata pada CPU. Sebuah transformer kausal mengeluarkan satu latent audio kontinu per frame (12.5 Hz) melalui satu langkah flow head, dan sebuah end-of-speech head menentukan kapan harus berhenti.

Processor menyimpan Pocket TTS audio tokenizer, sebuah codec berbasis Mimi yang mengenkripsi prompt suara dan mendekode latens yang dihasilkan menjadi audio 24 kHz. Suara yang sudah didefinisikan sebelumnya disertakan sebagai KV cache (voices/).

Dua flow head dirilis: Lagrangian self-distillation (flow_type="lsd", default) dan drifting (flow_type="drifting", english_drifting_26-09). Finetuning didukung, lihat Training.

Model ini disumbangkan oleh Eustache Le Bihan. Kode aslinya adalah kyutai-labs/pocket-tts.

Tips penggunaan

Siapkan input dengan apply_chat_template() (satu pesan user per teks, add_generation_prompt=True). Normalizer dari tokenizer menyiapkan teks seperti aslinya, dengan pengaturan bahasa checkpoint.
generate mengembalikan audio_latents dan audio_latents_mask; dekodekan dengan PocketTTSProcessor.decode().
max_new_tokens secara default menggunakan anggaran berdasarkan panjang teks, dan generasi berhenti segera setelah logit akhir ucapan melampaui eos_threshold.
do_sample=False sesuai dengan kondisi asli pada suhu 0. Dengan sampling, temperature (default 0.3) adalah varians dari noise awal aliran.
Pisahkan teks panjang dengan PocketTTSProcessor.split_text(), satu percakapan per chunk.

Penggunaan

Suara

Suara dapat diberikan dengan tiga cara.

1. Mengkloning rekaman: item audio dalam percakapan.

from transformers import AutoModelForTextToWaveform, AutoProcessor

model_id = "eustlb/pocket-tts"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")

url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/en-Alice_woman.wav"
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "url": url},
            {"type": "text", "text": "Hello world. I am Kyutai's Pocket TTS."},
        ],
    }
]
inputs = processor.apply_chat_template(
    conversation, tokenize=True, return_dict=True, add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "cloned.wav")

2. Laten yang sudah dihitung: item audio kosong, dengan laten yang diteruskan ke prosesor.

from transformers import AutoModelForTextToWaveform, AutoProcessor
from transformers.audio_utils import load_audio

model_id = "eustlb/pocket-tts"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")

# e.g. a recording encoded once by the processor's audio tokenizer,
# to reuse without re-encoding it
url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/en-Alice_woman.wav"
features = processor.feature_extractor(
    load_audio(url, sampling_rate=24000), sampling_rate=24000
).to(processor.audio_tokenizer.device)
# (num_frames, latent_dim)
voice = processor.audio_tokenizer.encode(**features).latents[0]

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio"},
            {"type": "text", "text": "Hello world. I am Kyutai's Pocket TTS."},
        ],
    }
]
inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
    add_generation_prompt=True,
    processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)

outputs = model.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "latents.wav")

3. Konteks yang di-cache: cache KV dari prompt suara, yang diteruskan sebagai past_key_values. Suara bawaan (voices/) menggunakan bentuk ini.

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForTextToWaveform, AutoProcessor, DynamicCache

model_id = "eustlb/pocket-tts"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")

voice = load_file(hf_hub_download(model_id, "voices/alba.safetensors"))
layers = []
for i, layer in enumerate(model.model.layers):
    # each layer's cache on that layer's device
    device = layer.self_attn.k_proj.weight.device
    layers.append(
        (voice[f"layers.{i}.keys"].to(device), voice[f"layers.{i}.values"].to(device))
    )
voice_cache = DynamicCache(layers, config=model.config)

conversation = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "Hello world. I am Kyutai's Pocket TTS."}],
    }
]
inputs = processor.apply_chat_template(
    conversation, tokenize=True, return_dict=True, add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs, past_key_values=voice_cache)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "cached.wav")

Batch dan teks panjang

Batch beberapa teks dan suara, dengan teks panjang dipecah menjadi chunk:

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForTextToWaveform, AutoProcessor, DynamicCache

model_id = "eustlb/pocket-tts"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")


def load_voices(names):
    """The shipped voices, one per conversation, as one cache, left-padded to the
    longest voice, and the mask of their positions.

    `generate` extends the cache: load it for every call.
    """
    voices = [
        load_file(hf_hub_download(model_id, f"voices/{name}.safetensors"))
        for name in names
    ]
    lengths = [voice["layers.0.keys"].shape[2] for voice in voices]
    length = max(lengths)
    layers = []
    for i, layer in enumerate(model.model.layers):
        device = layer.self_attn.k_proj.weight.device
        keys, values = (
            torch.cat(
                [
                    torch.nn.functional.pad(voice[f"layers.{i}.{kind}"], (0, 0, length - n, 0))
                    for voice, n in zip(voices, lengths)
                ]
            ).to(device)
            for kind in ("keys", "values")
        )
        layers.append((keys, values))
    mask = torch.tensor([[0] * (length - n) + [1] * n for n in lengths])
    return DynamicCache(layers, config=model.config), mask


text = (
    "A first sentence that is long enough to be worth splitting. "
    "Then a second one, and a third one for good measure."
)
chunks = processor.split_text(text)
conversations = [
    [{"role": "user", "content": [{"type": "text", "text": chunk}]}]
    for chunk in chunks + ["Hello world"]
]
inputs = processor.apply_chat_template(
    conversations, tokenize=True, return_dict=True, add_generation_prompt=True
).to(model.device)

voice_cache, voice_mask = load_voices(["alba"] * len(chunks) + ["jean"])
# the mask covers the cached voices too
inputs["attention_mask"] = torch.cat(
    [voice_mask.to(inputs["attention_mask"]), inputs["attention_mask"]], dim=-1
)
outputs = model.generate(**inputs, past_key_values=voice_cache)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(
    [torch.cat(audio[: len(chunks)]), audio[-1]], ["long.wav", "hello.wav"]
)

Checkpoint yang bergeser

Penggunaan yang sama, dengan suara milik checkpoint itu sendiri:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForTextToWaveform, AutoProcessor, DynamicCache

model_id = "eustlb/pocket-tts-english-drifting"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")

voice = load_file(hf_hub_download(model_id, "voices/alba.safetensors"))
layers = []
for i, layer in enumerate(model.model.layers):
    device = layer.self_attn.k_proj.weight.device
    layers.append(
        (voice[f"layers.{i}.keys"].to(device), voice[f"layers.{i}.values"].to(device))
    )

conversation = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "Hello world. I am Kyutai's Pocket TTS."}],
    }
]
inputs = processor.apply_chat_template(
    conversation, tokenize=True, return_dict=True, add_generation_prompt=True
).to(model.device)

outputs = model.generate(
    **inputs, past_key_values=DynamicCache(layers, config=model.config)
)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "drifting.wav")

Pelatihan

Letakkan audio target pada giliran assistant. Prosesor mengembalikan latennya sebagai labels, dan model mengembalikan loss dari audio asli (loss aliran LSD ditambah config.eos_loss_weight kali loss akhir ucapan). Prompt suara biasanya merupakan bagian awal dari ucapan yang sama.

import torch
from transformers import AutoModelForTextToWaveform, AutoProcessor

model_id = "eustlb/pocket-tts"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")
model.train()

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": voice_prompt},
            {"type": "text", "text": target_text},
        ],
    },
    {"role": "assistant", "content": [{"type": "audio", "audio": target_audio}]},
]
inputs = processor.apply_chat_template(
    conversation, tokenize=True, return_dict=True
).to(model.device)
loss = model(**inputs).loss
loss.backward()

Beri suara sebagai rekaman atau latents, bukan cache.
Untuk classifier-free guidance, hilangkan item audio atau text dari giliran user (aslinya menghilangkan masing-masing dengan probabilitas 0,2).
Latents yang sudah dihitung sebelumnya dimasukkan ke dalam processor_kwargs={"audio_prompt_latents": [...], "labels": [...]}, dengan item {"type": "audio"} kosong sebagai placeholder.
Resep asli: AdamW, learning_rate=2e-5 (2e-4 untuk bahasa baru), adam_beta2=0.95, weight_decay=0.1, max_grad_norm=1.0, 1000 langkah warmup lalu konstan, EMA dari bobot (decay 0.999).

Transformers

Pocket TTS

Ikhtisar

Tips penggunaan

Penggunaan

Suara

Batch dan teks panjang

Checkpoint yang bergeser

Pelatihan

PocketTTSConfig

kelas transformers.PocketTTSConfig

PocketTTSFlowHeadConfig

kelas transformers.PocketTTSFlowHeadConfig

PocketTTSFeatureExtractor

kelas transformers.PocketTTSFeatureExtractor

__call__

PocketTTSProcessor

kelas transformers.PocketTTSProcessor

__call__

split_text

decode

save_audio

PocketTTSModel

kelas transformers.PocketTTSModel

forward

PocketTTSForConditionalGeneration

kelas transformers.PocketTTSForConditionalGeneration

forward

generate

sample_latent

denormalize_latents

PocketTTSGenerateOutput

kelas transformers.models.pocket_tts.generation_pocket_tts.PocketTTSGenerateOutput

call

call