# Data type error while trying to fine tune Deberta v3 Large

**URL:** <https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604>\
**Category:** Models\
**Created:** [November 11, 2021, 5:34pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604 "2021-11-11T17:34:26Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 11, 2021, 5:34pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/1 "2021-11-11T17:34:27Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/719a63ed7a50e561738dd99a0cb81b82e4929bc3.png)  
I was trying this - [microsoft/deberta-v3-large · Hugging Face](https://huggingface.co/microsoft/deberta-v3-large#fine-tuning-with-hf-transformers)  
Command ran :-

```auto
python3 run_glue.py --model_name_or_path microsoft/deberta-v3-large --task_name mnli --do_train --do_eval --evaluation_strategy steps --max_seq_length 256 --warmup_steps 50 --learning_rate 6e-5 --num_train_epochs 3 --output_dir outputv3 --overwrite_output_dir --logging_steps 10000 --logging_dir outputv3/

```

while in this folder-- /transformers/examples/pytorch/text-classification$

for the fuller sequence of issues that occured and got solved- [quote=“nielsr, post:9, topic:11486, full:true”]  
I’ve created a notebook for you: [Google Colab](https://colab.research.google.com/drive/1K_FGbcPG1hPltM-BYcx3IB10T-Hp2Lzy?usp=sharing)  
[/quote]

Thanks for reading.

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 15, 2021, 3:25pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/3 "2021-11-15T15:25:49Z")

</div>

Hi @NDugar,

You get the error below because the dataset you use for fine-tuning does not have a validation split. As you can see on the [imdb dataset card](https://huggingface.co/datasets/imdb#data-splits) this dataset has only the `train`, `unsupervised` and `test` splits available natively. You can either modify the dataset yourself to extract your own evaluation dataset or use the test split by replacing `--do_eval ` with `--do_predict` in your script.

I hope this helped you!

Error currently discussed:

```bash
Traceback (most recent call last):
  File "run_glue.py", line 568, in <module>
    main()
  File "run_glue.py", line 422, in main
    raise ValueError("--do_eval requires a validation dataset")
ValueError: --do_eval requires a validation dataset

```

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 15, 2021, 3:59pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/4 "2021-11-15T15:59:24Z")

</div>

In the command I ran the problem seems to be regarding data types and I am trying to figure out what do I change in it. Command is written below:-

```auto
python3 run_glue.py --model_name_or_path microsoft/deberta-v3-large --task_name mnli --do_train --do_eval --evaluation_strategy steps --max_seq_length 256 --warmup_steps 50 --learning_rate 6e-5 --num_train_epochs 3 --output_dir outputv3 --overwrite_output_dir --logging_steps 10000 --logging_dir outputv3/

```

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 15, 2021, 4:06pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/5 "2021-11-15T16:06:50Z")

</div>

@NDugar , I’m a little confused. The command you just shared with me does not match the one in the google colab you shared in your initial post.

If I understand correctly, I should ignore your notebook colab? The command you executed is :

```bash
python3 run_glue.py \ 
    --model_name_or_path microsoft/deberta-v3-large \
    --task_name mnli \
    --do_train \
    --do_eval \ 
    --evaluation_strategy steps \
    --max_seq_length 256 \
    --warmup_steps 50 \
    --learning_rate 6e-5 \
    --num_train_epochs 3 \
    --output_dir outputv3 \
    --overwrite_output_dir \
    --logging_steps 10000 \
    --logging_dir outputv3/

```

and the error you get is:

```bash
inputGrad = _softmax_backward_data(grad_output, output, self.dim, output)
TypeError: _softmax_backward_data(): argument 'input_dtype' (position 4) must be torch.dtype, not Tensor
0%|

```

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 15, 2021, 4:14pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/6 "2021-11-15T16:14:31Z")

</div>

yes correct. sorry for the confusion.

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 15, 2021, 4:57pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/7 "2021-11-15T16:57:05Z")

</div>

I couldn’t reproduce the error you having with `microsoft/deberta-v3-small` (because the large version doesn’t fit in google colab).

Can you confirm that it is the same problem as [this issue](https://github.com/huggingface/transformers/issues/14375)?

If so, do you know how long ago you cloned the `transformers` library for your test? There have been [changes](https://github.com/huggingface/transformers/pull/14180) to the file `transformers/models/deberta_v2/modeling_deberta_v2.py"` 17 days ago that might have solved your problem.

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 15, 2021, 5:01pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/8 "2021-11-15T17:01:04Z")

</div>

> [@NDugar](#):
>
> yes correct. sorry for the confusion.

No problem 😉

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 15, 2021, 5:10pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/9 "2021-11-15T17:10:48Z")

</div>

yes, same issue. I cloned 5 days back so I think I am on the latest version.

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 15, 2021, 8:06pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/10 "2021-11-15T20:06:12Z")

</div>

Thank you for the additional information.

As I can’t reproduce errors with the small version, I would have to try with the large version which I won’t have time to do today. I must admit that I have no idea where it could come from. I’ll get back to you as soon as possible.

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 15, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/11 "2021-11-15T20:12:59Z")

</div>

Thank you. I was facing the same issue with deberta v2. so I don’t think the problem lies with the model but rather how they both were made.

---

<div class="post-metadata">

**Author:** ![SaulLu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saullu/32/2637_2.png) [@SaulLu](https://discuss.huggingface.co/u/SaulLu)\
**Post date:** [November 17, 2021, 5:41pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/12 "2021-11-17T17:41:48Z")

</div>

I just tested with `microsoft/deberta-v3-large` and I don’t get any error 😟: the training starts well for me.

I just added 2 more arguments `--per_device_train_batch_size 2 --per_device_eval_batch_size 2` because otherwise I didn’t have enough VRAM.

I don’t have exactly the same setting as you, though. Would it be possible for you to try again with the same settings as me (first trying with the last transformers version on `master`, then with Python version: 3.8.10 and finally with PyTorch 1.9.0)?

My setting:

```auto
- `transformers` version: 4.13.0.dev0
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0+cu102 (True)

```

Can’t wait to see how it turns out on your end!

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 17, 2021, 5:44pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/13 "2021-11-17T17:44:00Z")

</div>

ok will try this and reply.

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 18, 2021, 4:12pm UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/14 "2021-11-18T16:12:36Z")

</div>

I have the same transformers version. but I am on python 3.6.9, and torch - 1.10.0+cu102.

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [November 19, 2021, 8:16am UTC](https://discuss.huggingface.co/t/data-type-error-while-trying-to-fine-tune-deberta-v3-large/11604/15 "2021-11-19T08:16:41Z")

</div>

I realised that my issue is that of CUDA. So I am closing this. Thank you for your help.