# Two approaches to training a tokenizer

**URL:** <https://discuss.huggingface.co/t/two-approaches-to-training-a-tokenizer/33168>\
**Category:** Beginners\
**Created:** [March 6, 2023, 3:28pm UTC](https://discuss.huggingface.co/t/two-approaches-to-training-a-tokenizer/33168 "2023-03-06T15:28:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcgusty1g](https://avatars.discourse-cdn.com/v4/letter/m/a87d85/32.png) [@mcgusty1g](https://discuss.huggingface.co/u/mcgusty1g)\
**Post date:** [March 6, 2023, 3:28pm UTC](https://discuss.huggingface.co/t/two-approaches-to-training-a-tokenizer/33168/1 "2023-03-06T15:28:34Z")

</div>

I see these two approaches for training a tokenizer in HuggingFace:

### Approach 1

Ref: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)

```auto
from tokenizers.implementations import ByteLevelBPETokenizer

tokenizer = ByteLevelBPETokenizer()
paths = ['wikitext-2.txt']
tokenizer.train(files=paths)
tokenizer.save_model('../data/hf_bpe')

encoding = tokenizer.encode('hello world')
print (encoding.ids)

```

### Approach 2

Ref: [Building a tokenizer, block by block - Hugging Face LLM Course](https://huggingface.co/course/chapter6/8)

```auto
from tokenizers import models, trainers, Tokenizer

tokenizer = Tokenizer(model=models.WordPiece(unk_token="[UNK]"))

special_tokens = ["[UNK]", "[PAD]", "[CLS]", "[SEP]", "[MASK]"]
trainer = trainers.WordPieceTrainer(vocab_size=25000, special_tokens=special_tokens)

tokenizer.train(["wikitext-2.txt"], trainer=trainer)

encoding = tokenizer.encode("Let's test this tokenizer...", "on a pair of sentences.")
print(encoding.ids)

```

My question is what is the difference between two approaches and when should I use which approach?

If I understand correctly, in the latter approach, a model represents the tokenization algorithm. In that case, what does the trainer do? Does it represent the vocabulary and add new tokens to it?

Also, in approach 1, the tokenizer implicitly contains the model (the `.model` attribute). But it’s `train` method does not contain an argument for Trainer. Why?
