# BLOOM models don't run on my GPU using Transformers

**URL:** <https://discuss.huggingface.co/t/bloom-models-dont-run-on-my-gpu-using-transformers/23315>\
**Category:** 🤗Transformers\
**Created:** [September 18, 2022, 6:12pm UTC](https://discuss.huggingface.co/t/bloom-models-dont-run-on-my-gpu-using-transformers/23315 "2022-09-18T18:12:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TornButter](https://avatars.discourse-cdn.com/v4/letter/t/ac8455/32.png) [@TornButter](https://discuss.huggingface.co/u/TornButter)\
**Post date:** [September 18, 2022, 6:12pm UTC](https://discuss.huggingface.co/t/bloom-models-dont-run-on-my-gpu-using-transformers/23315/1 "2022-09-18T18:12:39Z")

</div>

The following code successfully runs on my CPU, maxing out a few cores while 3090’s usage remains at 0%:  
import torch  
from transformers import BloomTokenizerFast, BloomForCausalLM  
tokenizer = BloomTokenizerFast.from\_pretrained(“bigscience/bloom-560m”)  
model = BloomForCausalLM.from\_pretrained(“bigscience/bloom-560m”)  
prompt = “Dave picked up the baseball and”  
result\_length = 100  
inputs = tokenizer(prompt, return\_tensors=“pt”)  
raw = model.generate(inputs[“input\_ids”],max\_length=result\_length)[0]  
print(tokenizer.decode(raw))

However, I want to use my GPU. I have tried using different models like 1b7, but the same result. When using accelerate and device\_map=“auto”, torch\_dtype=“auto”, the models run on my GPU, but when trying to decode, I get an error that says “RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!”. When I add device = torch.device(“cuda:0”) and .to(device) to the end of the model line, I get the same runtime error. Same with .cuda(). I made sure to configure accelerate to not use my CPU. What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![TornButter](https://avatars.discourse-cdn.com/v4/letter/t/ac8455/32.png) [@TornButter](https://discuss.huggingface.co/u/TornButter)\
**Post date:** [September 18, 2022, 6:29pm UTC](https://discuss.huggingface.co/t/bloom-models-dont-run-on-my-gpu-using-transformers/23315/2 "2022-09-18T18:29:16Z")

</div>

I found the solution. I added these 2 lines before raw, then made raw run on inputs2

device = torch.device(“cuda:0”)  
inputs2 = inputs.to(device)

Here is the complete working code:  
import torch  
from transformers import BloomTokenizerFast, BloomForCausalLM  
tokenizer = BloomTokenizerFast.from\_pretrained(“bigscience/bloom-560m”)  
model = BloomForCausalLM.from\_pretrained(“bigscience/bloom-560m”).cuda()  
prompt = “Dave picked up the baseball and”  
result\_length = 100  
inputs = tokenizer(prompt, return\_tensors=“pt”)  
device = torch.device(“cuda:0”)  
inputs2 = inputs.to(device)  
raw = model.generate(inputs2[“input\_ids”],max\_length=result\_length)[0]  
print(tokenizer.decode(raw))