# Introducing BasicFormer, a backbone network written by me from scratch

**URL:** <https://discuss.huggingface.co/t/introducing-basicformer-a-backbone-network-written-by-me-from-scratch/168142>\
**Category:** Beginners\
**Created:** [September 5, 2025, 7:10pm UTC](https://discuss.huggingface.co/t/introducing-basicformer-a-backbone-network-written-by-me-from-scratch/168142 "2025-09-05T19:10:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![automated-perm-acid](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@automated-perm-acid](https://discuss.huggingface.co/u/automated-perm-acid)\
**Post date:** [September 5, 2025, 7:10pm UTC](https://discuss.huggingface.co/t/introducing-basicformer-a-backbone-network-written-by-me-from-scratch/168142/1 "2025-09-05T19:10:06Z")

</div>

repo: [GitHub - icy-obelisk/basicformer: A Basic Transformer without positional encoding, 100% huggingface compatible, created as a skeleton transformer for newcomers to integrate new thoughts/algorithms into huggingface ecosystem directly.](https://github.com/icy-obelisk/basicformer)

I’m pretty new to NLP field. During my study, I knew that huggingface transformers library has the biggest ecosystem in this field. However, writing a model from scratch to meet the huggingface standard like Seq2SeqLMOutput/GenerationMixin is quite difficult and tiresome. I did a lot of research and created a huggingface-compatible model from scratch in 20 days. Its source code is tiny and super user-friendly, with the following features:

- Encoder-only Transformer
- Decoder-only Transformer
- Encoder-Decoder Transformer with Cross Attention Mask
- Self/Cross Key/Value Cache
- Auto mask handling
- `GenerationMixin` compatible with drop-in `generate()` support.

`BasicFormer` is created to serve as a backbone Transformer for enthusiasts who loves to customize the model from scratch, it can be also used as a tutorial for beginners who really wants to know the inner working of Transformers and how Huggingface transformers library deal with these structures.  
It’s a complete pytorch model that doesn’t contain tokenizer, so the test case in the repo have to use other tokenizers. It also have some shortcomings:

- No Positional Encoding
- No tied weights between the input embedding layer and language model head.
- No sliding window cache support(That’s quite difficult to code I must say).

But I believe these can be customized and overcame. So take my model code and happy coding🤗

P.S. English isn’t my mother language, so forgive me if you find grammar/expression errors.