# Tokenizer is splitting special token

**URL:** <https://discuss.huggingface.co/t/tokenizer-is-splitting-special-token/161147>\
**Category:** 🤗Tokenizers\
**Created:** [June 30, 2025, 2:43am UTC](https://discuss.huggingface.co/t/tokenizer-is-splitting-special-token/161147 "2025-06-30T02:43:18Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 30, 2025, 5:29am UTC](https://discuss.huggingface.co/t/tokenizer-is-splitting-special-token/161147/4 "2025-06-30T05:29:37Z")

</div>

Hmm… How about `save_pretrained` with `from_pretrained` way?

> [@Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library](https://discuss.huggingface.co/t/cannot-create-an-identical-pretrainedtokenizerfast-object-from-a-tokenizer-created-by-tokenizers-library/9317):
>
> I can’t seem to create a “PreTrainedTokenizerFast” object from my original tokenizers tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a [PAD] token, and does not have any special tokens at all. tokenizer = ByteLevelBPETokenizer() tokenizer.preprocessor = pre\_tokenizers.BertPreTokenizer() tokenizer.normalizer = normalizers.BertNormalizer() tokenizer.train\_from\_iterator(doc…

> <https://github.com/huggingface/tokenizers/issues/777>
>
> I can't seem to create a "PreTrainedTokenizerFast" object from my original \`toke…nizers\` tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a \[PAD\] token, and does not have any special tokens at all. 
> \`\`\` 
> tokenizer = ByteLevelBPETokenizer()
> tokenizer.preprocessor = pre\_tokenizers.BertPreTokenizer()
> tokenizer.normalizer = normalizers.BertNormalizer()
> tokenizer.train\_from\_iterator(docs, vocab\_size=16\_000, min\_frequency=15, special\_tokens = \["\[UNK\]", "\[CLS\]", "\[SEP\]", "\[PAD\]", "\[MASK\]"\])
> tokenizer.\_tokenizer.post\_processor = processors.BertProcessing(
> ("\[SEP\]", tokenizer.token\_to\_id("\[SEP\]")),
> ("\[CLS\]", tokenizer.token\_to\_id("\[CLS\]")),
> )
> tokenizer.enable\_truncation(max\_length=256)
> tokenizer.enable\_padding(pad\_id=3, pad\_token="\[PAD\]")
> fast\_tokenizer = PreTrainedTokenizerFast(tokenizer\_object=tokenizer)
> \`\`\`
> 
> The result of printing the fast\_tokenizer is:
> \`\`\`
> PreTrainedTokenizerFast(name\_or\_path='', vocab\_size=16000, model\_max\_len=1000000000000000019884624838656, is\_fast=True, padding\_side='right', special\_tokens={})
> \`\`\`
> Which \`model\_max\_len\` and \`special\_tokens\` are wrong in it. Also, there is no pad\_token and pad\_token\_id in the fast\_tokenizer object. (the warning for pad\_token for example: \`Using pad\_token, but it is not set yet.\`) Have I done anything wrong, or is this not supposed to happen?
> 
> The versions of libraries I'm using:
> \`\`\`
> \['tokenizers 0.10.3',
> 'transformers 4.10.0.dev0'\]
> \`\`\`

---

_[View the full topic](https://discuss.huggingface.co/t/tokenizer-is-splitting-special-token/161147)._
