# How to do sequence fine tuning?

**URL:** <https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419>\
**Category:** Beginners\
**Created:** [July 21, 2020, 6:59pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419 "2020-07-21T18:59:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\
**Post date:** [July 21, 2020, 6:59pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/1 "2020-07-21T18:59:39Z")

</div>

I have been training a BERT model on a large unsupervised dataset, and now I wish to fine tune the model on a small labelled dataset, but I can’t quite grasp how to do this conceptually, and I’m hoping some of you can help me out.

When doing the unsupervised training/self-training, everything seems fine, and I think I understand it.  
In this case, my network is a standard BERT, with a linear layer on top that takes the standard 768 hidden features in BERT down to 30, which is my vocab\_size. (I’m training on gene sequences, so basically one sample in my dataset, looks like this:

```auto
ASDGDFASGDFSGSDASFASDAUYRES

```

where I do the standard thing of masking out some of the letters and trying to predict them.  
So for standard training, my setup looks like this:

```auto
    predicted_sequence=bert(input_sequence,masked_input_sequence)
    loss = crossentropy(predicted_sequence,masked_input_sequence)

```

However when I now want to switch to fine-tuning I’m not really sure what to do. In this case my dataset now both consist of a gene sequence and a label sequence:

```auto
DFASDGFTHGFDDFSDASFDASF , 00000001111111100000000022222

```

How do I change my network such that I can now fine-train it to predict these new labels?, do I still use bert(input,masked\_input\_sequence)? Do I remove the linear layer on top of BERT? or what is the conceptual idea here?

---

<div class="post-metadata">

**Author:** ![swayson](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@swayson](https://discuss.huggingface.co/u/swayson)\
**Post date:** [July 21, 2020, 7:26pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/2 "2020-07-21T19:26:37Z")

</div>

I found this explanation by one of the original authors great to get a conceptual understanding.

[![](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0c6714edb11e2f4f138486922442aa2d7f74395c.jpeg "Day 4: Focused Lecture - Transformers with Lukasz Kaiser") ](https://www.youtube.com/watch?v=law_X0zAMkE)

---

<div class="post-metadata">

**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\
**Post date:** [July 21, 2020, 7:28pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/3 "2020-07-21T19:28:29Z")

</div>

If I understand correctly you need token classification. Example here: [https://github.com/huggingface/transformers/tree/master/examples/token-classification](https://github.com/huggingface/transformers/tree/master/examples/token-classification)

You can assign a label to each token you have. I think you will need to consider each “letter” a token.

---

<div class="post-metadata">

**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\
**Post date:** [July 21, 2020, 9:14pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/4 "2020-07-21T21:14:05Z")

</div>

Thank you for the link, that was the kind of conceptual talk I was looking for.

---

<div class="post-metadata">

**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\
**Post date:** [July 21, 2020, 9:15pm UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/5 "2020-07-21T21:15:06Z")

</div>

The examples there are a bit over my head for now, but I got some ideas for how to proceed now, and if I later get stuck again I might try to look at them more closely to see how they do it.

---

<div class="post-metadata">

**Author:** ![swayson](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@swayson](https://discuss.huggingface.co/u/swayson)\
**Post date:** [July 22, 2020, 6:12am UTC](https://discuss.huggingface.co/t/how-to-do-sequence-fine-tuning/419/6 "2020-07-22T06:12:39Z")

</div>

Found this talk from Alex Graves also to be complementary, he also speaks about transformers more generally with advantages/disadvantages.

The Q/A at the end of these videos I found to be helpful.

[![](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7f7e8f7ec788bf3268d3eb8a697a78bcf991ef35.jpeg "Day 6: Lecture - Attention & Memory with Alex Graves") ](https://www.youtube.com/watch?v=POtAGvwnB5s)