# Modeling\_bert use next-token prediction?

**URL:** <https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369>\
**Category:** 🤗Transformers\
**Created:** [September 5, 2024, 4:21am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369 "2024-09-05T04:21:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![TianlaiChen](https://avatars.discourse-cdn.com/v4/letter/t/ee59a6/32.png) [@TianlaiChen](https://discuss.huggingface.co/u/TianlaiChen)\
**Post date:** [September 5, 2024, 4:21am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/1 "2024-09-05T04:21:51Z")

</div>

In modeling\_bert.py file class `BertLMHeadModel(BertPreTrainedModel)`,

> <https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L1372>

It has following codes for loss implementation:

```auto
        sequence_output = outputs[0]
        prediction_scores = self.cls(sequence_output)

        lm_loss = None
        if labels is not None:
            # we are doing next-token prediction; shift prediction scores and input ids by one
            shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
            labels = labels[:, 1:].contiguous()
            loss_fct = CrossEntropyLoss()
            lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

```

Why does BERT do next token prediction here with label shift?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 9, 2024, 10:29am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/2 "2024-09-09T10:29:32Z")

</div>

Hi,

Yes for any LLM which you can train in the Transformers library, the model will internally shift the labels one position so that it learns to predict the next token. The convenience of this is that users can just copy the labels from the inputs, i.e. `labels = input_ids.clone()` - although users then typically also replace tokens which the models shouldn’t learn to predict (like padding tokens) by -100.

Visually (taken from my explanation [here](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/2)):

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/b/2befd4b8d668ac6d84b0ab15b785f7e3867c1f43.jpeg)

As can be seen, the labels (top row) are equal to the inputs (bottom row), just shifted one position to the left, and with tokens which the model shouldn’t learn to predict (like the special \<|begin\_of\_text|\> in the figure above) replaced by -100.

---

<div class="post-metadata">

**Author:** ![TianlaiChen](https://avatars.discourse-cdn.com/v4/letter/t/ee59a6/32.png) [@TianlaiChen](https://discuss.huggingface.co/u/TianlaiChen)\
**Post date:** [September 9, 2024, 6:36pm UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/3 "2024-09-09T18:36:16Z")

</div>

Thanks a lot @nielsr !

Yes, I understand next token prediction and label shift. But BERT here is not a CLM model, so I am confused why it has label shift. Given its a MLM, I assume it should just do corss entropy over masked tokens and there is no need for shift?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 10, 2024, 7:30am UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/4 "2024-09-10T07:30:34Z")

</div>

That’s because there were some people interested in initializing decoder-only LLMs with the weights of BERT. This was mainly for the EncoderDecoderModel class, where the weights of the encoder and decoder were both initialized from a pre-trained BERT. See [Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models](https://huggingface.co/blog/warm-starting-encoder-decoder).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [September 16, 2024, 7:12pm UTC](https://discuss.huggingface.co/t/modeling-bert-use-next-token-prediction/105369/5 "2024-09-16T19:12:48Z")

</div>

Closed on September 16, 2024, 7:12pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.