# Performing gradient accumulation with Accelerate

**URL:** <https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836>\
**Category:** 🤗Accelerate\
**Created:** [February 12, 2024, 2:05pm UTC](https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836 "2024-02-12T14:05:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon40544052](https://avatars.discourse-cdn.com/v4/letter/a/82dd89/32.png) [@anon40544052](https://discuss.huggingface.co/u/anon40544052)\
**Post date:** [February 12, 2024, 2:05pm UTC](https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/1 "2024-02-12T14:05:20Z")

</div>

It seems that there is no effect of gradient accumulation = 2 at all.

My code :

```auto
for epoch in range(init_epoch, args.num_epoch + 1):
    for iteration, (x, y) in enumerate(data_loader):
        x_0 = x.to(device, dtype=dtype, non_blocking=True)
        y = None if not use_label else y.to(device, non_blocking=True)
        model.zero_grad()
        if is_latent_data:
            z_0 = x_0 * args.scale_factor
        else:
            z_0 = first_stage_model.encode(x_0).latent_dist.sample().mul_(args.scale_factor)
        # sample t
        t = torch.rand((z_0.size(0),), dtype=dtype, device=device)
        t = t.view(-1, 1, 1, 1)
        z_1 = torch.randn_like(z_0)
        # 1 is real noise, 0 is real data
        z_t = (1 - t) * z_0 + (1e-5 + (1 - 1e-5) * t) * z_1
        u = (1 - 1e-5) * z_1 - z_0
        # estimate velocity
        v = model(t.squeeze(), z_t, y)
        loss = loss.mean()            
        accelerator.backward(loss)

        #change below
        if (iteration + 1) % args.gradient_accumulation_steps == 0:
            optimizer.step()
            scheduler.step()
            
            # clear gradient
            model.zero_grad() 
            global_step += 1
            log_steps += 1
        #change above from here

        if iteration % 100 == 0:
            if accelerator.is_main_process:
                # Measure training speed:
                end_time = time()
                steps_per_sec = log_steps / (end_time - start_time)
                accelerator.print(
                    "epoch {} iteration{}, Loss: {}, Train Steps/Sec: {:.2f}".format(
                        epoch, iteration, loss.item(), steps_per_sec
                    )
                )
                # Reset monitoring variables:
                log_steps = 0
                start_time = time()

```

code ref : [LFM/train\_flow\_latent.py at main · VinAIResearch/LFM · GitHub](https://github.com/VinAIResearch/LFM/blob/main/train_flow_latent.py)

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [February 12, 2024, 6:04pm UTC](https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/2 "2024-02-12T18:04:12Z")

</div>

You need to also use the gradient accumulation wrapper as detailed/shown here: [accelerate/examples/by\_feature/gradient\_accumulation.py at main · huggingface/accelerate · GitHub](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/gradient_accumulation.py#L168-L177)

---

<div class="post-metadata">

**Author:** ![anon40544052](https://avatars.discourse-cdn.com/v4/letter/a/82dd89/32.png) [@anon40544052](https://discuss.huggingface.co/u/anon40544052)\
**Post date:** [March 4, 2024, 4:07am UTC](https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/3 "2024-03-04T04:07:10Z")

</div>

```auto
            accelerator.accumulate(model)
            loss = loss.mean()
            # change above from here
            accelerator.backward(loss)
            optimizer.step()
            scheduler.step()
            model.zero_grad()
            global_step += 1
            log_steps += 1
            optimizer.zero_grad()
            # change above from here

```

is this correctly? Thank you in advance

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [March 4, 2024, 2:26pm UTC](https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/4 "2024-03-04T14:26:53Z")

</div>

Yep! 🙂 Note there are examples for doing so in our repo and the docs: [accelerate/examples/by\_feature/gradient\_accumulation.py at main · huggingface/accelerate · GitHub](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/gradient_accumulation.py)