# Trainer API object detection **URL:** \ **Category:** 🤗Transformers\ **Created:** [December 29, 2024, 7:23am UTC](https://discuss.huggingface.co/t/trainer-api-object-detection/133414 "2024-12-29T07:23:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![risper7](https://avatars.discourse-cdn.com/v4/letter/r/e274bd/32.png) [@risper7](https://discuss.huggingface.co/u/risper7)\ **Post date:** [December 29, 2024, 7:23am UTC](https://discuss.huggingface.co/t/trainer-api-object-detection/133414/1 "2024-12-29T07:23:05Z")
Hi, I’ve been getting a bug in my objectdetection for the longest time. My dataset is in the right format. If I change the batch size, I will still get the error saying the same thing. I’m unable to find the solution. How do I fix it? training\_args = TrainingArguments( output\_dir=“./model”, num\_train\_epochs=2, per\_device\_train\_batch\_size=5, per\_device\_eval\_batch\_size=5, # dataloader\_num\_workers=4, weight\_decay=1e-4, logging\_dir=‘./logs’, logging\_steps=70, gradient\_accumulation\_steps=4, metric\_for\_best\_model=“eval\_map”, greater\_is\_better=True, load\_best\_model\_at\_end=True, eval\_strategy=“epoch”, save\_strategy=‘epoch’, save\_total\_limit=2, # eval\_do\_concat\_batches=False, learning\_rate=5e-3, lr\_scheduler\_type=‘cosine’, fp16=True, max\_grad\_norm=0.01, report\_to=“none” ) trainer = Trainer( model=model, args=training\_args, data\_collator=lambda batch: collate\_fn(batch, image\_processor), tokenizer=image\_processor, train\_dataset=train\_dataset, eval\_dataset=val\_dataset, compute\_metrics=eval\_metrics ) The error: IndexError Traceback (most recent call last) Cell In[67], line 1 ----\> 1 trainer.train() File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:2164, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs) 2162 hf\_hub\_utils.enable\_progress\_bars() 2163 else: → 2164 return inner\_training\_loop( 2165 args=args, 2166 resume\_from\_checkpoint=resume\_from\_checkpoint, 2167 trial=trial, 2168 ignore\_keys\_for\_eval=ignore\_keys\_for\_eval, 2169 ) File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:2524, in Trainer.\_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval) 2517 context = ( 2518 functools.partial(self.accelerator.no\_sync, model=model) 2519 if i != len(batch\_samples) - 1 2520 and self.accelerator.distributed\_type != DistributedType.DEEPSPEED 2521 else contextlib.nullcontext 2522 ) 2523 with context(): → 2524 tr\_loss\_step = self.training\_step(model, inputs, num\_items\_in\_batch) 2526 if ( 2527 args.logging\_nan\_inf\_filter 2528 and not is\_torch\_xla\_available() 2529 and (torch.isnan(tr\_loss\_step) or torch.isinf(tr\_loss\_step)) 2530 ): 2531 # if loss is nan or inf simply add the average of previous logged losses 2532 tr\_loss = tr\_loss + tr\_loss / (1 + self.state.global\_step - self.\_globalstep\_last\_logged) File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:3654, in Trainer.training\_step(self, model, inputs, num\_items\_in\_batch) 3651 return loss\_mb.reduce\_mean().detach().to(self.args.device) 3653 with self.compute\_loss\_context\_manager(): → 3654 loss = self.compute\_loss(model, inputs, num\_items\_in\_batch=num\_items\_in\_batch) 3656 del inputs 3657 if ( 3658 self.args.torch\_empty\_cache\_steps is not None 3659 and self.state.global\_step % self.args.torch\_empty\_cache\_steps == 0 3660 ): File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:3708, in Trainer.compute\_loss(self, model, inputs, return\_outputs, num\_items\_in\_batch) 3706 loss\_kwargs[“num\_items\_in\_batch”] = num\_items\_in\_batch 3707 inputs = {\*\*inputs, \*\*loss\_kwargs} → 3708 outputs = model(\*\*inputs) 3709 # Save past state if it exists 3710 # TODO: this needs to be fixed and made cleaner later. 3711 if self.args.past\_index \>= 0: File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1553, in Module.\_wrapped\_call\_impl(self, \*args, \*\*kwargs) 1551 return self.\_compiled\_call\_impl(\*args, \*\*kwargs) # type: ignore[misc] 1552 else: → 1553 return self.\_call\_impl(\*args, \*\*kwargs) File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1562, in Module.\_call\_impl(self, \*args, \*\*kwargs) 1557 # If we don’t have any hooks, we want to skip the rest of the logic in 1558 # this function, and just call forward. 1559 if not (self.\_backward\_hooks or self.\_backward\_pre\_hooks or self.\_forward\_hooks or self.\_forward\_pre\_hooks 1560 or \_global\_backward\_pre\_hooks or \_global\_backward\_hooks 1561 or \_global\_forward\_hooks or \_global\_forward\_pre\_hooks): → 1562 return forward\_call(\*args, \*\*kwargs) 1564 try: 1565 result = None File /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data\_parallel.py:186, in DataParallel.forward(self, \*inputs, \*\*kwargs) 184 return self.module(\*inputs[0], \*\*module\_kwargs[0]) 185 replicas = self.replicate(self.module, self.device\_ids[:len(inputs)]) → 186 outputs = self.parallel\_apply(replicas, inputs, module\_kwargs) 187 return self.gather(outputs, self.output\_device) File /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data\_parallel.py:201, in DataParallel.parallel\_apply(self, replicas, inputs, kwargs) 200 def parallel\_apply(self, replicas: Sequence[T], inputs: Sequence[Any], kwargs: Any) → List[Any]: → 201 return parallel\_apply(replicas, inputs, kwargs, self.device\_ids[:len(replicas)]) File /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel\_apply.py:108, in parallel\_apply(modules, inputs, kwargs\_tup, devices) 106 output = results[i] 107 if isinstance(output, ExceptionWrapper): → 108 output.reraise() 109 outputs.append(output) 110 return outputs File /opt/conda/lib/python3.10/site-packages/torch/\_utils.py:706, in ExceptionWrapper.reraise(self) 702 except TypeError: 703 # If the exception takes multiple arguments, don’t try to 704 # instantiate since we don’t know how to 705 raise RuntimeError(msg) from None → 706 raise exception IndexError: Caught IndexError in replica 0 on device 0. Original Traceback (most recent call last): File “/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel\_apply.py”, line 83, in \_worker output = module(\*input, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/transformers/models/detr/modeling\_detr.py”, line 1466, in forward loss, loss\_dict, auxiliary\_outputs = self.loss\_function( File “/opt/conda/lib/python3.10/site-packages/transformers/loss/loss\_for\_object\_detection.py”, line 552, in ForObjectDetectionLoss loss\_dict = criterion(outputs\_loss, labels) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/transformers/loss/loss\_for\_object\_detection.py”, line 253, in forward indices = self.matcher(outputs\_without\_aux, targets) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/torch/utils/\_contextlib.py”, line 116, in decorate\_context return func(\*args, \*\*kwargs) File “/opt/conda/lib/python3.10/site-packages/transformers/loss/loss\_for\_object\_detection.py”, line 360, in forward indices = [linear\_sum\_assignment(c[i]) for i, c in enumerate(cost\_matrix.split(sizes, -1))] File “/opt/conda/lib/python3.10/site-packages/transformers/loss/loss\_for\_object\_detection.py”, line 360, in indices = [linear\_sum\_assignment(c[i]) for i, c in enumerate(cost\_matrix.split(sizes, -1))] IndexError: index 5 is out of bounds for dimension 0 with size 5 ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 29, 2024, 9:08am UTC](https://discuss.huggingface.co/t/trainer-api-object-detection/133414/2 "2024-12-29T09:08:06Z")
Perhaps this issue? > > > \### System Info > > \`\`\` > root@fb9fa1e6d8d8:/mnt/nas2/users/sbchoi/transformers/ex…amples/pytorch/object-detection# transformers-cli env > > Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points. > > \- \`transformers\` version: 4.42.0.dev0 > \- Platform: Linux-5.4.0-167-generic-x86\_64-with-glibc2.35 > \- Python version: 3.10.14 > \- Huggingface\_hub version: 0.23.4 > \- Safetensors version: 0.4.3 > \- Accelerate version: 0.30.1 > \- Accelerate config: not found > \- PyTorch version (GPU?): 2.2.2 (True) > \- Tensorflow version (GPU?): not installed (NA) > \- Flax version (CPU?/GPU?/TPU?): not installed (NA) > \- Jax version: not installed > \- JaxLib version: not installed > \- Using distributed or parallel set-up in script?: \ > \- Using GPU in script?: \ > \- GPU type: NVIDIA TITAN RTX > \`\`\` > > \`\`\` > Traceback (most recent call last): > File "/mnt/nas2/users/sbchoi/transformers/examples/pytorch/object-detection/run\_object\_detection.py", line 521, in \ > main() > File "/mnt/nas2/users/sbchoi/transformers/examples/pytorch/object-detection/run\_object\_detection.py", line 496, in main > train\_result = trainer.train(resume\_from\_checkpoint=checkpoint) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/trainer.py", line 1903, in train > return inner\_training\_loop( > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/trainer.py", line 2248, in \_inner\_training\_loop > tr\_loss\_step = self.training\_step(model, inputs) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/trainer.py", line 3275, in training\_step > loss = self.compute\_loss(model, inputs) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/trainer.py", line 3307, in compute\_loss > outputs = model(\*\*inputs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl > return self.\_call\_impl(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl > return forward\_call(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data\_parallel.py", line 185, in forward > outputs = self.parallel\_apply(replicas, inputs, module\_kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data\_parallel.py", line 200, in parallel\_apply > return parallel\_apply(replicas, inputs, kwargs, self.device\_ids\[:len(replicas)\]) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel\_apply.py", line 108, in parallel\_apply > output.reraise() > File "/opt/conda/lib/python3.10/site-packages/torch/\_utils.py", line 722, in reraise > raise exception > IndexError: Caught IndexError in replica 0 on device 0. > Original Traceback (most recent call last): > File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel\_apply.py", line 83, in \_worker > output = module(\*input, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl > return self.\_call\_impl(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl > return forward\_call(\*args, \*\*kwargs) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/models/detr/modeling\_detr.py", line 1485, in forward > loss\_dict = criterion(outputs\_loss, labels) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl > return self.\_call\_impl(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl > return forward\_call(\*args, \*\*kwargs) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/models/detr/modeling\_detr.py", line 2084, in forward > indices = self.matcher(outputs\_without\_aux, targets) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl > return self.\_call\_impl(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl > return forward\_call(\*args, \*\*kwargs) > File "/opt/conda/lib/python3.10/site-packages/torch/utils/\_contextlib.py", line 115, in decorate\_context > return func(\*args, \*\*kwargs) > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/models/detr/modeling\_detr.py", line 2213, in forward > indices = \[linear\_sum\_assignment(c\[i\]) for i, c in enumerate(cost\_matrix.split(sizes, -1))\] > File "/mnt/nas2/users/sbchoi/transformers/src/transformers/models/detr/modeling\_detr.py", line 2213, in \ > indices = \[linear\_sum\_assignment(c\[i\]) for i, c in enumerate(cost\_matrix.split(sizes, -1))\] > IndexError: index 8 is out of bounds for dimension 0 with size 8 > \`\`\` > @qubvel > > \### Who can help? > > \_No response\_ > > \### Information > > \- \[X\] The official example scripts > \- \[\] My own modified scripts > > \### Tasks > > \- \[X\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...) > \- \[\] My own task or dataset (give details below) > > \### Reproduction > > \`\`\`bash > python run\_object\_detection.py \\ > --model\_name\_or\_path facebook/detr-resnet-50 \\ > --dataset\_name cppe-5 \\ > --do\_train true \\ > --do\_eval true \\ > --output\_dir detr-finetuned-cppe-5-10k-steps \\ > --num\_train\_epochs 100 \\ > --image\_square\_size 600 \\ > --fp16 true \\ > --learning\_rate 5e-5 \\ > --weight\_decay 1e-4 \\ > --dataloader\_num\_workers 4 \\ > --dataloader\_prefetch\_factor 2 \\ > --per\_device\_train\_batch\_size 8 \\ > --gradient\_accumulation\_steps 1 \\ > --remove\_unused\_columns false \\ > --eval\_do\_concat\_batches false \\ > --ignore\_mismatched\_sizes true \\ > --metric\_for\_best\_model eval\_map \\ > --greater\_is\_better true \\ > --load\_best\_model\_at\_end true \\ > --logging\_strategy epoch \\ > --evaluation\_strategy epoch \\ > --save\_strategy epoch \\ > --save\_total\_limit 2 \\ > --push\_to\_hub true \\ > --push\_to\_hub\_model\_id detr-finetuned-cppe-5-10k-steps \\ > --hub\_strategy end \\ > --seed 1337 > \`\`\` > > \### Expected behavior > > I think this might be the cause number of GPU or hyperparameter. ---
**Author:** ![risper7](https://avatars.discourse-cdn.com/v4/letter/r/e274bd/32.png) [@risper7](https://discuss.huggingface.co/u/risper7)\ **Post date:** [December 29, 2024, 3:13pm UTC](https://discuss.huggingface.co/t/trainer-api-object-detection/133414/3 "2024-12-29T15:13:28Z")
Thank you! I’ve solved it