# Finetuning Transformers for Text Classification Issue

**URL:** <https://discuss.huggingface.co/t/finetuning-transformers-for-text-classification-issue/39337>\
**Category:** 🤗Transformers\
**Created:** [May 9, 2023, 10:41pm UTC](https://discuss.huggingface.co/t/finetuning-transformers-for-text-classification-issue/39337 "2023-05-09T22:41:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![smiraldr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smiraldr/32/16020_2.png) [@smiraldr](https://discuss.huggingface.co/u/smiraldr)\
**Post date:** [May 9, 2023, 10:41pm UTC](https://discuss.huggingface.co/t/finetuning-transformers-for-text-classification-issue/39337/1 "2023-05-09T22:41:32Z")

</div>

```auto
import torch
from datasets import load_dataset, load_metric
from transformers import AutoTokenizer, AutoModel, TrainingArguments, Trainer

# Load the CoLA dataset
cola_dataset = load_dataset("glue", "cola")

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased", num_labels=2 )

# Set the device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Define the training arguments
training_args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy='steps',
    eval_steps=100,
    save_total_limit=1,
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    push_to_hub=False,
)

# Define the trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=cola_dataset['train'],
    eval_dataset=cola_dataset['validation'],
    tokenizer=tokenizer,
    compute_metrics=load_metric('glue', 'cola'),
)

# Fine-tune on the CoLA dataset
trainer.train()

```

In the code above I am getting

```auto
ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['label']

```

I was checking out the official notebook for text classification tutorial which uses yelp review dataset , and even though i’m using the trainer way and not pytorch training loop I tried renaming label to labels as mentioned. On the following final code : [Google Colab](https://colab.research.google.com/drive/17cWbzRMovHnl6WuPBZfnOJwMksWIImMT?usp=sharing) (check modified code output error in the final cell)

`ValueError: The model did not return a loss from the inputs, only the following keys: last_hidden_state,pooler_output. For reference, the inputs it received are input_ids,token_type_ids,attention_mask.`

Can someone help with what exactly am i doing wrong here? Thanks!

---

<div class="post-metadata">

**Author:** ![gp06aug](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gp06aug/32/15965_2.png) [@gp06aug](https://discuss.huggingface.co/u/gp06aug)\
**Post date:** [May 10, 2023, 1:21am UTC](https://discuss.huggingface.co/t/finetuning-transformers-for-text-classification-issue/39337/2 "2023-05-10T01:21:43Z")

</div>

You will have to tokenize your input before passing them to the Trainer API. This includes both the text and the target.

Something like this : [Fine-tuning a model with the Trainer API - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter3/3?fw=pt)

---

<div class="post-metadata">

**Author:** ![smiraldr-legacy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smiraldr-legacy/32/4317_2.png) [@smiraldr-legacy](https://discuss.huggingface.co/u/smiraldr-legacy)\
**Post date:** [May 11, 2023, 8:31am UTC](https://discuss.huggingface.co/t/finetuning-transformers-for-text-classification-issue/39337/3 "2023-05-11T08:31:39Z")

</div>

Hey thanks for helping, but even after adding that I have the same error popping up. I’ve updated the colab notebook , please take a look.