# Sequence masking

**URL:** <https://discuss.huggingface.co/t/sequence-masking/17154>\
**Category:** 🤗Transformers\
**Created:** [April 25, 2022, 12:10pm UTC](https://discuss.huggingface.co/t/sequence-masking/17154 "2022-04-25T12:10:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitvit](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vitvit](https://discuss.huggingface.co/u/vitvit)\
**Post date:** [April 25, 2022, 12:10pm UTC](https://discuss.huggingface.co/t/sequence-masking/17154/1 "2022-04-25T12:10:20Z")

</div>

Hello,  
I’m using a custom tokenizer with a Roberta model.  
I want to pre-train the model using the masking task.  
Is there a way to mask a span of tokens instead of randomly selecting which to mask?  
I know that there is an option for whole word masking collator, but it is incompatible with my tokenizer and I also want more flexibility in the sequence length selection.

Thanks!