# Train Roberta from scratch for custom dataset

**URL:** <https://discuss.huggingface.co/t/train-roberta-from-scratch-for-custom-dataset/4331>\
**Category:** Intermediate\
**Created:** [March 11, 2021, 11:30pm UTC](https://discuss.huggingface.co/t/train-roberta-from-scratch-for-custom-dataset/4331 "2021-03-11T23:30:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![clementine](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@clementine](https://discuss.huggingface.co/u/clementine)\
**Post date:** [March 11, 2021, 11:30pm UTC](https://discuss.huggingface.co/t/train-roberta-from-scratch-for-custom-dataset/4331/1 "2021-03-11T23:30:56Z")

</div>

Hey there, I am training Roberta from scratch for protein sequences. To this end, I build a tokenizer for protein sequences, which is very much like a character-level tokenizer. After that, I stored my tokenizer and used it in the Roberta model followed this tutorial: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train).

The code runs fine on CPU but failed on GPU, and the error message is as follows:  
RuntimeError: CUDA error: CUBLAS\_STATUS\_NOT\_INITIALIZED when calling `cublasCreate(handle)`

Thanks in advance for any thoughts on this issue!

---

<div class="post-metadata">

**Author:** ![LidorPrototype](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@LidorPrototype](https://discuss.huggingface.co/u/LidorPrototype)\
**Post date:** [May 2, 2023, 4:08am UTC](https://discuss.huggingface.co/t/train-roberta-from-scratch-for-custom-dataset/4331/2 "2023-05-02T04:08:08Z")

</div>

Hello, have you found any solution for that?