# OPT Memory problem

**URL:** <https://discuss.huggingface.co/t/opt-memory-problem/18450>\
**Category:** Beginners\
**Created:** [May 28, 2022, 4:05am UTC](https://discuss.huggingface.co/t/opt-memory-problem/18450 "2022-05-28T04:05:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lexipalmer13](https://avatars.discourse-cdn.com/v4/letter/l/df788c/32.png) [@lexipalmer13](https://discuss.huggingface.co/u/lexipalmer13)\
**Post date:** [May 28, 2022, 4:05am UTC](https://discuss.huggingface.co/t/opt-memory-problem/18450/1 "2022-05-28T04:05:51Z")

</div>

Hi!

I’m trying to replicate the basic OPT examples from the documentation and I keep getting a CUDA is out of memory error. I tried using low\_cpu\_mem\_usage=True since that has been a solution on other models, but it doesn’t make a difference.

Code is basic:

from transformers import AutoModelForCausalLM, AutoTokenizer  
import torch  
model = AutoModelForCausalLM.from\_pretrained(“facebook/opt-30b”, torch\_dtype=torch.float16).cuda()

And an example error is:

 ![Screen Shot 2022-05-28 at 12.00.27 AM](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f968512381bb0a46162e57d600c41b65a2d2a45e.png)

I know it’s a large model, but pytorch reserving 43 GiB seems high. All of the solutions I can find on outside forums wouldn’t seem to work with this model type (running in smaller batches, clearing memory mid run, or using koila wrappers). Any help much appreciated!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [May 31, 2022, 2:47pm UTC](https://discuss.huggingface.co/t/opt-memory-problem/18450/2 "2022-05-31T14:47:28Z")

</div>

The `facebook/opt-30b` model takes 60GB of memory in FP16 precision.

---

<div class="post-metadata">

**Author:** ![cog](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@cog](https://discuss.huggingface.co/u/cog)\
**Post date:** [June 2, 2022, 2:51am UTC](https://discuss.huggingface.co/t/opt-memory-problem/18450/3 "2022-06-02T02:51:12Z")

</div>

hi.

if your machine install multiple GPUs, load model with Data Parallel or Distributed Data parallel shall be help.

regards.