# Encoder decoder model

**URL:** <https://discuss.huggingface.co/t/encoder-decoder-model/28328>\
**Category:** 🤗Transformers\
**Created:** [December 23, 2022, 10:27am UTC](https://discuss.huggingface.co/t/encoder-decoder-model/28328 "2022-12-23T10:27:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JoAmps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joamps/32/13066_2.png) [@JoAmps](https://discuss.huggingface.co/u/JoAmps)\
**Post date:** [December 23, 2022, 10:27am UTC](https://discuss.huggingface.co/t/encoder-decoder-model/28328/1 "2022-12-23T10:27:39Z")

</div>

I want to use an encoder-decoder model, but I want to perform masked language modeling and casual language modeling (unsupervised learning) on the encoder-decoder so it understands the context I am using, before fine-tuning it. How best to go about this?