# Modeling\_bert use next-token prediction?

**URL:** <https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369>\
**Category:** 🤗Transformers\
**Created:** [September 5, 2024, 4:21am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369 "2024-09-05T04:21:51Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 9, 2024, 10:29am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/2 "2024-09-09T10:29:32Z")

</div>

Hi,

Yes for any LLM which you can train in the Transformers library, the model will internally shift the labels one position so that it learns to predict the next token. The convenience of this is that users can just copy the labels from the inputs, i.e. `labels = input_ids.clone()` - although users then typically also replace tokens which the models shouldn’t learn to predict (like padding tokens) by -100.

Visually (taken from my explanation [here](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/2)):

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/b/2befd4b8d668ac6d84b0ab15b785f7e3867c1f43.jpeg)

As can be seen, the labels (top row) are equal to the inputs (bottom row), just shifted one position to the left, and with tokens which the model shouldn’t learn to predict (like the special \<|begin\_of\_text|\> in the figure above) replaced by -100.

---

_[View the full topic](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369)._