# Text Classification tokenizer problems on inference

**URL:** <https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648>\
**Category:** Intermediate\
**Created:** [May 7, 2022, 6:49pm UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648 "2022-05-07T18:49:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lfcc](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@lfcc](https://discuss.huggingface.co/u/lfcc)\
**Post date:** [May 7, 2022, 6:49pm UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648/1 "2022-05-07T18:49:58Z")

</div>

I just finned tunned a BERT model for text classification (NER), it works great!

However, I found some nuances associated with the tokenizer, i.e., when I use my model for inference, the resultant entities come with “#” due to the subword tokenizer I suppose.

In order to fix this, I used the `"aggregation_strategy"` parameter. This parameter can receive 5 different values: none, simple, first, average and max. When I use “None” or “simple” the code runs well but I still get the result as subwords with “#”. If i try to use " first, average or max" options, I get the following error:

```auto
 word_entities.append(self.aggregate_word(word_group, aggregation_strategy))
  Python\Python39\lib\site-packages\transformers\pipelines\token_classification.py", line 336, in aggregate_word
    word = self.tokenizer.convert_tokens_to_string([entity["word"] for entity in entities])
TypeError: 'NoneType' object is not iterable

```

Any idea how to fix this?

thank you

---

<div class="post-metadata">

**Author:** ![ibrahimishag](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ibrahimishag/32/23840_2.png) [@ibrahimishag](https://discuss.huggingface.co/u/ibrahimishag)\
**Post date:** [October 12, 2022, 4:07am UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648/2 "2022-10-12T04:07:05Z")

</div>

Hello, I ran into the same error!. I hope you found a way around it!..

---

<div class="post-metadata">

**Author:** ![lfcc](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@lfcc](https://discuss.huggingface.co/u/lfcc)\
**Post date:** [October 12, 2022, 7:05am UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648/3 "2022-10-12T07:05:10Z")

</div>

No, I did not.

---

<div class="post-metadata">

**Author:** ![ibrahimishag](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ibrahimishag/32/23840_2.png) [@ibrahimishag](https://discuss.huggingface.co/u/ibrahimishag)\
**Post date:** [October 12, 2022, 7:57am UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648/4 "2022-10-12T07:57:24Z")

</div>

Luckily for me, I found that one of the input sentences was none. Therefoere, I have preprocessed the input sentence by replacing the empty lines with dummy text and it worked!.  
Thanks for raising the issue and for your immediate respone!.

---

<div class="post-metadata">

**Author:** ![lfcc](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@lfcc](https://discuss.huggingface.co/u/lfcc)\
**Post date:** [October 12, 2022, 8:38am UTC](https://discuss.huggingface.co/t/text-classification-tokenizer-problems-on-inference/17648/5 "2022-10-12T08:38:58Z")

</div>

Ho yeah!  
I remember now, that was the problem. I had some NONE inputs as well, you are right.  
Removing them was the solution.