# Add metrics to object detection example

**URL:** <https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213>\
**Category:** 🤗Transformers\
**Created:** [February 6, 2023, 8:52pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213 "2023-02-06T20:52:30Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrishrb](https://avatars.discourse-cdn.com/v4/letter/c/c77e96/32.png) [@chrishrb](https://discuss.huggingface.co/u/chrishrb)\
**Post date:** [February 6, 2023, 8:52pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/1 "2023-02-06T20:52:31Z")

</div>

Hello everybody,

im new with huggingface and wanted to try out the object detection. So i ran the transformers object detection example from the huggingface docs (this one here: [Object detection](https://huggingface.co/docs/transformers/main/tasks/object_detection)) and wanted to add some metrics while training the model. Everytime i get the following error:

TypeError: Can’t pad the values of type \<class ‘transformers.image\_processing\_utils.BatchFeature’\>, only of nested list/tuple/dicts of tensors.

I create a eval\_dataset like this:

```auto
cppe5["test"] = cppe5["test"].with_transform(transform_aug_ann)

```

and add it to the trainer:

```auto
metric = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    print("here")
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=splitted_ds_encoded["train"],
    eval_dataset=splitted_ds_encoded['test'],
    tokenizer=image_processor,
    compute_metrics=compute_metrics,
)

```

Maybe someone here can help me?

Thanks in advance. Greetings Christoph

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [February 7, 2023, 4:15pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/2 "2023-02-07T16:15:30Z")

</div>

cc @MariaK

---

<div class="post-metadata">

**Author:** ![MariaK](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariak/32/13014_2.png) [@MariaK](https://discuss.huggingface.co/u/MariaK)\
**Post date:** [February 17, 2023, 1:17pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/3 "2023-02-17T13:17:34Z")

</div>

Hi Christoph! Thanks for the question, and sorry about the delay. In the guide, we didn’t use the evaluation in the Trainer, because accuracy usually isn’t used for object detection. Here, we evaluated separately on standard COCO metrics. If you want to measure accuracy during training, you’ll need to check how the labels are batched in collate\_fn(batch), (e.g. use batch[“labels”] = [dict(label) for label in labels] instead). You may also need to modify your compute\_metrics function.  
If this doesn’t work, @ybelkada may be able to help with the evaluation.

---

<div class="post-metadata">

**Author:** ![macklenc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/macklenc/32/14547_2.png) [@macklenc](https://discuss.huggingface.co/u/macklenc)\
**Post date:** [March 6, 2023, 11:06pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/4 "2023-03-06T23:06:57Z")

</div>

Hey! Sorry for reviving this, and I may be kicking a dead horse here.

Some context: super new to both hugging face and AI/ML. Played around with the TF Object Detection API about a year ago (which was a horrible experience in comparison), and trying to update my “framework” to use huggingface tools.

Personally, I like the idea of having some kind of eval during training without having to write files to disk. I saw that there’s a metric for mean IoU [Mean IoU - a Hugging Face Space by evaluate-metric](https://huggingface.co/spaces/evaluate-metric/mean_iou), which I _think_ should be a good metric for this, but I can’t for the life of me get that to work. Do you have any tips assuming this is even a good thing to do?

---

<div class="post-metadata">

**Author:** ![dguo2](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@dguo2](https://discuss.huggingface.co/u/dguo2)\
**Post date:** [April 26, 2023, 10:10pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/5 "2023-04-26T22:10:39Z")

</div>

Hi @macklenc were you able to find a solution? I’ve been getting into object detection as well and I recently tried TFOD. I was looking for a way to do continuous evaluation with the coco metrics and I haven’t found a way to be able to implement that.

---

<div class="post-metadata">

**Author:** ![macklenc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/macklenc/32/14547_2.png) [@macklenc](https://discuss.huggingface.co/u/macklenc)\
**Post date:** [April 26, 2023, 11:38pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/6 "2023-04-26T23:38:12Z")

</div>

I did get something working, and sure hope it’s right. I decided to use torchmetrics’ MAP implementation `from torchmetrics.detection import MeanAveragePrecision`. It turns out that in the evaluation step, the model returns an extra dictionary element called `loss_dict` that wasn’t being accounted for properly in the returned logits, so I was able to disable that when I created my model, e.g.:

```auto
model = AutoModelForObjectDetection.from_pretrained(
    model_dest,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
    num_queries=3,
    keys_to_ignore_at_inference=["loss_dict"],
    )

```

And I structured my compute\_metrics as:

```auto
def compute_metrics(eval_pred: EvalPrediction, map: MeanAveragePrecision):
    (scores, pred_boxes, last_hidden_state, encoder_last_hidden_state), labels = eval_pred
    # scores shape: (batch_size, number of detected anchors, num_classes + 1) last class is the no-object class
    # pred_boxes shape: (batch_size, number of detected anchors, 4)
    # https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/detr-resnet50/README.md
    predictions = []
    for score, box in zip(scores, pred_boxes):
        # Extract the bounding boxes, labels, and scores from the model's output
        pred_scores = torch.from_numpy(score[:, :-1]) # Exclude the no-object class
        pred_boxes = torch.from_numpy(box)
        pred_labels = torch.argmax(pred_scores, dim=-1)

        # Get the scores corresponding to the predicted labels
        pred_scores_for_labels = torch.gather(pred_scores, 1, pred_labels.unsqueeze(-1)).squeeze(-1)
        predictions.append(
            {
                "boxes": pred_boxes,
                "scores": pred_scores_for_labels,
                "labels": pred_labels,
            }
        )
    target = [
        {
            "boxes": torch.from_numpy(labels[i]["boxes"]),
            "labels": torch.from_numpy(labels[i]["class_labels"]),
        }
        for i in range(len(labels))
    ]
    map.update(preds=predictions, target=target)
    results = map.compute()
    # Convert tensors to scalars/lists, MLFlow doesn't really like tensors
    results = {k: v.tolist() if isinstance(v, torch.Tensor) else v for k, v in results.items()}
    return results

```

When setting up the trainer I used the following:

```auto
mAP = MeanAveragePrecision(box_format="cxcywh", class_metrics=True)
metrics = functools.partial(compute_metrics, map=mAP, image_processor=image_processor)
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=lambda batch: collate_fn(batch, image_processor),
    train_dataset=train_ds,
    tokenizer=image_processor,
    eval_dataset=test_ds,
    compute_metrics=metrics,
    # preprocess_logits_for_metrics=lambda logits, labels: logits[1:], # This is another way to remove the extra dict
)

```

It’s not the cleanest solution, but seems to be working for me. If you come up with a cleaner solution (e.g. using the image processor to decode the tensors since using a different backend like YOLOS breaks the `compute_metrics` function) I’d definitely appreciate it if you could share.

---

<div class="post-metadata">

**Author:** ![harpergrieve](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@harpergrieve](https://discuss.huggingface.co/u/harpergrieve)\
**Post date:** [May 26, 2023, 5:48pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/7 "2023-05-26T17:48:32Z")

</div>

Hi @macklenc, could you please share your data loader/ anything done to the eval\_dataset as well as your collate\_fn() function? Im still running into a issue with my batched test set.  
Thanks

---

<div class="post-metadata">

**Author:** ![macklenc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/macklenc/32/14547_2.png) [@macklenc](https://discuss.huggingface.co/u/macklenc)\
**Post date:** [May 26, 2023, 8:29pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/8 "2023-05-26T20:29:53Z")

</div>

@harpergrieve Here’s my `collate_fn()`, if I recall I didn’t change any of the functionality there:

```python
def collate_fn(batch, image_processor):
    pixel_values = [item["pixel_values"] for item in batch]
    encoding = image_processor.pad(pixel_values, return_tensors="pt")
    labels = [dict(item["labels"]) for item in batch]
    batch = {"pixel_values": encoding["pixel_values"], "pixel_mask": encoding["pixel_mask"], "labels": labels}
    # batch = {"pixel_values": encoding["pixel_values"], "labels": labels} # For YOLOS backend
    return batch

```

My train/test datasets are created with:

```python
    train_ds = dataset["train"].with_transform(
        lambda examples: transform_aug_ann(examples, image_processor, train_transform)
    )
    test_ds = dataset["test"].with_transform(
        lambda examples: transform_aug_ann(examples, image_processor, test_transform)
    )

```

where `transform_aug_ann` is

```python
def transform_aug_ann(examples, image_processor, transform):
    image_ids = examples["image_id"]
    images, bboxes, area, categories = [], [], [], []
    for image, objects in zip(examples["image"], examples["objects"]):
        image = np.array(image.convert("RGB"))[:, :, ::-1]
        out = transform(image=image, bboxes=objects["bbox"], category=objects["category"])
        area.append(objects["area"])
        images.append(out["image"])
        bboxes.append(out["bboxes"])
        categories.append(out["category"])
    targets = [
        {"image_id": id_, "annotations": format_annotations(id_, cat_, ar_, box_)}
        for id_, cat_, ar_, box_ in zip(image_ids, categories, area, bboxes)
    ]
    return image_processor(images=images, annotations=targets, return_tensors="pt")

```

and

```python
def create_transform(width, height):
    train_transform = albumentations.Compose(
        [
            albumentations.SmallestMaxSize(max_size=1000, p=1),
            albumentations.RandomScale(p=0.5, scale_limit=(-0.4, 0.8)),
        ],
        bbox_params=albumentations.BboxParams(
            format="coco", label_fields=["category"], min_area=400, min_visibility=0.7
        ),
    )

    test_transform = albumentations.Compose(
        [
            albumentations.Resize(width, height),
        ],
        bbox_params=albumentations.BboxParams(format="coco", label_fields=["category"]),
    )
    return train_transform, test_transform

```

Unfortunately I haven’t run this against the CPPE5 (or equivalent) dataset, I ended up creating a custom one for a personal project, but it mimicked the structure of CPPE5 and my data loader is just `load_dataset(str(module_directory / "my_dataset"), name=config)`. If you can share your project I can see if I can find the source of the problem.

---

<div class="post-metadata">

**Author:** ![harpergrieve](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@harpergrieve](https://discuss.huggingface.co/u/harpergrieve)\
**Post date:** [May 26, 2023, 9:22pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/9 "2023-05-26T21:22:57Z")

</div>

> [@macklenc](#):
>
> `e`

> [@macklenc](#):
>
> `e`

Thanks! figured it out, now im getting a weird cuda out of mem error while validation is happening…

---

<div class="post-metadata">

**Author:** ![Roboserg](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/roboserg/32/18313_2.png) [@Roboserg](https://discuss.huggingface.co/u/Roboserg)\
**Post date:** [August 7, 2023, 12:54am UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/10 "2023-08-07T00:54:33Z")

</div>

Is there a mAP COCO eval metric for instance segmentation?

---

<div class="post-metadata">

**Author:** ![rimahajou](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@rimahajou](https://discuss.huggingface.co/u/rimahajou)\
**Post date:** [January 31, 2024, 2:29pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/11 "2024-01-31T14:29:11Z")

</div>

Hello @macklenc ,

I am trying to implement your solution. There is some bug with the return of EvalPredictions.  
The input to compute\_metrics (EvalPredictions) have some inconsistent lengths of **labels** and **scores**. The **scores** length is equal to the length of evaluation dataset, while the **labels** length is equal to the eval\_batch size, I searched a bit and debugged the Trainer class of huggingface, and I noticed that labels are concatenated together using the nested\_detach function. This will concatenate the two lists differently thant list1+list2, it will concatenate item1 of the list1 with item2 with list2 and so on. this will aggregate multiple images into the same item.

Do you have any idea how to resolve this? Someone talked about it in this issue ([here](https://github.com/huggingface/transformers/issues/25939)) on huggingface but never mentioned a solution/

---

<div class="post-metadata">

**Author:** ![zakmatt](https://avatars.discourse-cdn.com/v4/letter/z/c0e974/32.png) [@zakmatt](https://discuss.huggingface.co/u/zakmatt)\
**Post date:** [February 3, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/12 "2024-02-03T18:39:09Z")

</div>

The reason why (I also had this bug) is because trainer processes the label\_ids attribute and thus make it unintelligible. If you look into it, you’ll see that you’ve got a batch of image ids together and boxes also together. Anyways, Huggingface seems to care a bit less about CV 😉

Here’s the ugly solution which works (I did it).

1. Update transformers to the newest version
2. Opent the trainer.py file in the transformers library “wherever\_you\_have\_your\_python/lib/python3.11/site-packages/transformers/trainer.py”
3. go all the way to the line `3253` and in the next line add `all_eval_labels = []`
4. Then go to the for loop 11 lines down and `WITHIN THAT LOOP` add `all_eval_labels.extend(inputs["labels"])`
5. Go further to the line 3371, and just before `if self.compute_metrics is not None and all_preds is not None and all_labels is not None:` add `all_labels = all_eval_labels`  
It should look line this:

```auto
...
all_labels = all_eval_labels
# Metrics!
if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
...

```

Voilà! The first part is done. Now the compute metrics method.

```auto
def compute_metrics(eval_pred: EvalPrediction):
    """Compute detection metrics"""

    _, scores, pred_boxes, last_hidden_state, encoder_last_hidden_state = eval_pred.predictions

    # scores shape: (number of samples, number of detected anchors, num_classes + 1) last class is the no-object class
    # pred_boxes shape: (number of samples, number of detected anchors, 4)
    # https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/detr-resnet50/README.md
    predictions = []
    for score, box in zip(scores, pred_boxes):
        # Extract the bounding boxes, labels, and scores from the model's output
        pred_scores = torch.from_numpy(score[:, :-1]) # Exclude the no-object class
        pred_boxes = torch.from_numpy(box)
        pred_labels = torch.argmax(pred_scores, dim=-1)

        # Get the scores corresponding to the predicted labels
        pred_scores_for_labels = torch.gather(pred_scores, 1, pred_labels.unsqueeze(-1)).squeeze(-1)
        predictions.append(
            {
                "boxes": pred_boxes,
                "scores": pred_scores_for_labels,
                "labels": pred_labels,
            }
        )

    target = [
        {
            "boxes": eval_pred.label_ids[i]["boxes"].detach().cpu(),
            "labels": eval_pred.label_ids[i]["class_labels"].detach().cpu(),
        }
        for i in range(len(eval_pred.label_ids))
    ]
    map = MeanAveragePrecision(box_format="xywh")
    map.update(preds=predictions, target=target)
    results = map.compute()
    results = {k: v.tolist() if isinstance(v, torch.Tensor) else v for k, v in results.items()}
    return results

```

Then in my trainer I just add this method as `compute_metrics` parameter

```auto
trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=lambda batch: collate_fn(batch, is_yolo),
        train_dataset=train_dataset,
        eval_dataset=val_dataset,
        compute_metrics=compute_metrics,
        tokenizer=image_processor
    )

```

It will work but it’s not a perfect solution and you should remember to remove it if you decide to tackle other tasks like NLP.  
Cheers!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 8, 2024, 10:55am UTC](https://discuss.huggingface.co/t/add-metrics-to-object-detection-example/31213/13 "2024-05-08T10:55:29Z")

</div>

We finally have updated our object detection scripts which include computing metrics. Both a script with the Trainer and without (using Accelerate) are supported: [transformers/examples/pytorch/object-detection at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection)