Transformers documentation
聊天消息模式
聊天消息模式
聊天模型期望以字典列表的形式接收对话。每个字典使用 role 和 content 键。content 键保存传递给模型的用户消息。大型语言模型接受文本和工具，而多模态模型则将文本与图像、视频和音频相结合。
Transformers 采用统一格式，其中每种模态类型都需显式指定，因此可以在单条消息中轻松混合搭配不同的输入。
本指南涵盖了每种模态的消息格式模式、工具、批量推理以及多轮对话。
文本
文本是最基本的内容类型，也是所有其他模式的基础。请将您的消息作为字符串传递给 "content"。
message = [
{
"role": "user",
"content": "Explain the French Bread Law."
}
]您也可以使用显式的 "type": "text" 格式，以便在后续添加图像、视频或音频时保持代码的一致性。
message = [
{
"role": "user",
"content": [{"type": "text", "text": "Explain the French Bread Law."}]
}
]工具
工具是聊天模型可以调用的函数，例如获取实时天气数据，而不是自行生成。
assistant 角色负责处理工具请求。在 "tool_calls" 键中设置 "type": "function"，并将你的工具提供给 "function" 键。将助手的工具请求追加到你的消息中。
weather = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
message.append(
{
"role": "assistant",
"tool_calls": [{"type": "function", "function": weather}]
}
)tool 角色用于处理结果。将其追加到 "content" 中。该值始终应为字符串。
message.append({"role": "tool", "content": "22"})多模态
多模态模型扩展了此格式，以处理图像、视频和音频。每个输入都通过 "type" 指定其类型，并使用 "url" 或 "path" 提供媒体。
图像
设置 "type": "image"，对于链接使用 "url"，对于本地文件使用 "path"。
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What pastry is shown in the image?"}
]
}
]视频
设置 "type": "video"，对于链接使用 "url"，对于本地文件使用 "path"。
message = [
{
"role": "user",
"content": [
{"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
{"type": "text", "text": "What is shown in this video?"}
]
}
]音频
设置 "type": "audio"，对于链接使用 "url"，对于本地文件使用 "path"。
message = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"},
{"type": "text", "text": "Transcribe the speech."}
]
}
]混合多种
content 列表接受任意类型的组合。模型会同时处理所有输入，从而实现比较或跨模态推理。
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
{"type": "text", "text": "What does the image and video share in common?"},
],
},
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
{"type": "text", "text": "What type of pastries are these?"},
],
}
]批量
批量推理在单次前向传播中处理多个对话，以提升吞吐量和效率。请将每个对话分别包裹在各自的列表中，然后将它们作为一个列表的列表一起传入。
messages = [
[
{"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What type of pastry is this?"}
]
},
],
[
{"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
{"type": "text", "text": "What type of pastry is this?"}
]
},
],
]多轮
对话包含多轮交互，在 "user" 和 "assistant" 角色之间交替进行。每一轮都会向列表中添加一条新消息，使模型能够访问完整的对话历史。这种上下文有助于模型生成更恰当的回复。
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What pastry is shown in the image?"}
]
},
{
"role": "assistant",
"content": [{"type": "text", "text": "This is kouign amann, a laminated dough pastry (i.e., dough folded with layers of butter) that also incorporates sugar between layers so that during baking the sugar caramelizes."}]
},
{
"role": "user",
"content": [
{"type": "image", "url": "https://static01.nyt.com/images/2023/07/21/multimedia/21baguettesrex-hbkc/21baguettesrex-hbkc-videoSixteenByNineJumbo1600.jpg"},
{"type": "text", "text": "Compare it to this image now."}
]
}
]