when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

There is some discussion in this therad and this . Perhaps it helps?

where is the wrong That’s a new one. I haven’t seen that expression before. LOL

Issue with tokenizer.tokenize

facehugger2020 November 16, 2020, 7:21pm 3

See also this post in the forum.

Topic		Replies	Views
RobertaTokenizer decode and tokenize do not have the same output 🤗Tokenizers	0	268	October 24, 2023
Punctuation and Spaces in RoBERTa Tokenizer for NER with Pre-tokenized Data 🤗Transformers	0	618	January 16, 2022
Using roberta for token-classification, strange characters Models	0	278	July 10, 2023
All my sequences get tokenized the same 🤗Tokenizers	2	650	February 12, 2022
Build a RoBERTa tokenizer from scratch 🤗Tokenizers	5	3447	December 12, 2020