# Accuracy metric throws during evaluation on sequence classification task

**URL:** <https://discuss.huggingface.co/t/accuracy-metric-throws-during-evaluation-on-sequence-classification-task/12103>\
**Category:** Beginners\
**Created:** [November 20, 2021, 6:19pm UTC](https://discuss.huggingface.co/t/accuracy-metric-throws-during-evaluation-on-sequence-classification-task/12103 "2021-11-20T18:19:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgreenbe](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mgreenbe/32/2827_2.png) [@mgreenbe](https://discuss.huggingface.co/u/mgreenbe)\
**Post date:** [November 20, 2021, 6:19pm UTC](https://discuss.huggingface.co/t/accuracy-metric-throws-during-evaluation-on-sequence-classification-task/12103/1 "2021-11-20T18:19:23Z")

</div>

I’m fine-tuning a BertFor SequenceClassification model and I want to compute the accuracy on the evaluation set after each training epoch. However, the evaluation step fails with:

```nohighlight
TypeError: 'list' object is not callable" when calling evaluate

```

Here’s a minimal example showing the error (see also [this Colab notebook](https://colab.research.google.com/drive/1YO4NiuLwBAr7RERsggMzJ6FDxOgM4dlK?usp=sharing)):

Standard tokenizer, toy model:

```auto
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
config = BertConfig(num_hidden_layers=0)
model = BertForSequenceClassification(config)

```

Grab the IMDb dataset, make training and evaluation sets:

```auto
imdb = load_dataset("imdb")

def mapper(x):
  return tokenizer(x["text"], max_length=384, truncation=True, padding="max_length")

eval_dataset = imdb["train"].select(range(100)).map(mapper, remove_columns=["text"])

```

Set up the trainer, including `compute_metrics=[accuracy]`:

```auto
accuracy = load_metric("accuracy")

args = TrainingArguments(
    "imdb",
    num_train_epochs=2,
    report_to="none",
    evaluation_strategy="epoch"
)

trainer = Trainer(
    model,
    args,
    eval_dataset=eval_dataset,
    compute_metrics=[accuracy],
    tokenizer=tokenizer,
)

```

Evaluate:

```auto
trainer.evaluate()

```

Raises:

```nohighlight
TypeError: 'list' object is not callable

```

Isn’t this something that should “just work”?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [November 21, 2021, 2:12pm UTC](https://discuss.huggingface.co/t/accuracy-metric-throws-during-evaluation-on-sequence-classification-task/12103/2 "2021-11-21T14:12:44Z")

</div>

`compute_metrics` should be a function that takes a namedtuple (of type `EvalPredictions`) and returns a dictionary metric nane/metric value.  
Look at the [text classificatione example](https://github.com/huggingface/transformers/blob/a59e7c1ed4ef1568ca0ba4140c9af641b17fa37e/examples/pytorch/text-classification/run_glue.py#L447) or the [course section on the Trainer](https://huggingface.co/course/chapter3/3?fw=pt).