Accelerate documentation

Accelerate

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Accelerate

Accelerate 是一个库,只需添加四行代码,即可让相同的 PyTorch 代码在任何分布式配置上运行!简而言之,它让大规模训练和推理变得简单、高效且易于适配。

+ from accelerate import Accelerator
+ accelerator = Accelerator()

+ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+     model, optimizer, training_dataloader, scheduler
+ )

  for batch in training_dataloader:
      optimizer.zero_grad()
      inputs, targets = batch
      inputs = inputs.to(device)
      targets = targets.to(device)
      outputs = model(inputs)
      loss = loss_function(outputs, targets)
+     accelerator.backward(loss)
      optimizer.step()
      scheduler.step()

Accelerate 基于 torch_xla 和 torch.distributed 构建,负责处理繁重的工作,因此您无需编写任何自定义代码来适配这些平台。 将现有代码库转换为使用 DeepSpeed,执行 fully sharded data parallelism,并自动支持混合精度训练!

为了更好地了解这一过程,请务必查看教程!

然后,可以通过 Accelerate 的 CLI 接口在任意系统上启动此代码:

accelerate launch {my_script.py}
Update on GitHub