MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Paper • 2602.21379 • Published • 1
Aitana-intellectual-property-mb-encoder (Spanish & English)
A ModernBERT-base model continually pretrained on Intellectual Property (IP) data in Spanish and English. This specialized encoder model is optimized for understanding IP-related texts.
Table of Contents
- Model Description
- Training Data
- Training Configuration
- Training Results
- Intended Uses
- Evaluation
- How to Use
- Additional Information
Model Description
| Attribute | Value |
|---|---|
| Base Model | BSC-LT/MrBERT |
| Architecture | FlexBERT (22 layers, 768 hidden, 12 heads) |
| Parameters | ~149M |
| Vocabulary Size | 256,000 tokens |
| Max Sequence Length | 8,192 tokens |
| Languages | Spanish (es), English (en) |
| Domain | Intellectual Property |
Training Data
This model was trained on the gplsi/alia_intellectual_property dataset (Spanish and English subsets) and on gplsi/discriminative_counterfeit_es and gplsi/discriminative_counterfeit_en filtered to samples labeled "not fake". This work has also been partially supported by Project HEART-NLP (PID2024-156263OB-C22).
Data Processing Pipeline
- Filtering: Selected only samples with language ∈ {"es", "en"} from gplsi/alia_intellectual_property and only samples labeled "not fake" from the counterfeit datasets.
- Tokenization: BPE tokenization with MrBERT tokenizer (256k vocab)
- Chunking: Packed into 8,192-token sequences
- Split: 90% train / 10% validation
Training Configuration
| Parameter | Value |
|---|---|
| Training Epochs | 3 |
| Sequence Length | 8,192 |
| MLM Probability | 30% (train), 15% (eval) |
| Batch Size | 32 |
| Learning Rate | 5e-5 (cosine decay to 5e-6) |
| Warmup | 101 batches (1%) |
| Optimizer | StableAdamW |
| Precision | bfloat16 |
| Hardware | 2× NVIDIA Blackwell 6000 Pro |
Training Results
| Epoch | Training Loss |
|---|---|
| 1 | 5.30 |
| 2 | 4.50 |
| Final | 2.53 |
Intended Uses
Primary Use Cases
- IP-focused NLP: NER, text classification, and sentiment analysis for intellectual property documents
- Semantic Search: Retrieval and similarity search for brand and regulatory queries
- Information Extraction: Extraction of brand-, regulation-, and IP-related entities
- Multilingual Support: Processing Spanish and English IP texts
Out-of-Scope Uses
- General-purpose language understanding outside IP domain
- Languages other than Spanish and English
- Text generation (this is an encoder-only model)
Users should evaluate the model's outputs for fairness and bias in their specific applications.
How to Use
Transformers
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
tokenizer = AutoTokenizer.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
# Fill-mask example
text = "El acceso a este [MASK] no implica cesión ni transmisión de derechos de explotación sobre el mismo.."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# Get predictions
import torch
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_token_id = outputs.logits[0, mask_token_index].argmax(axis=-1)
print(tokenizer.decode(predicted_token_id))
For Embeddings
from transformers import AutoModel, AutoTokenizer
import torch
model = AutoModel.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
tokenizer = AutoTokenizer.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
text = "El acceso a este material no implica cesión ni transmisión de derechos de explotación sobre el mismo."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
outputs = model(**inputs)
embeddings = outputs.last_hidden_state.mean(dim=1) # Mean pooling
Evaluation
This model was evaluated using the GLUE and SuperGLUE benchmarks.
| Suite | Task/Metric | Seeds | Scores by seed | Mean +/- std |
|---|---|---|---|---|
| GLUE | CoLA MCC | 4 | 19: 7.27, 8364: 13.42, 717: 11.28, 10536: 17.63 | 12.40 +/- 4.32 |
| GLUE | MNLI acc | 1 | 19: 69.56 | 69.56 |
| GLUE | MNLI-mm acc | 1 | 19: 70.51 | 70.51 |
| GLUE | MRPC F1 | 5 | 19: 69.50, 8364: 71.27, 717: 73.02, 10536: 71.22, 90166: 70.30 | 71.06 +/- 1.32 |
| GLUE | QNLI acc | 1 | 19: 70.58 | 70.58 |
| GLUE | QQP F1 | 1 | 19: 80.90 | 80.90 |
| GLUE | RTE acc | 5 | 19: 50.18, 8364: 50.54, 717: 48.74, 10536: 50.54, 90166: 48.74 | 49.75 +/- 0.93 |
| GLUE | SST-2 acc | 3 | 19: 83.72, 8364: 84.40, 717: 84.40 | 84.17 +/- 0.39 |
| GLUE | STS-B Spearman | 5 | 19: 43.16, 8364: 36.60, 717: 43.35, 10536: 41.89, 90166: 40.68 | 41.14 +/- 2.75 |
| SuperGLUE partial | MNLI acc | 1 | 19: 70.31 | 70.31 |
| SuperGLUE partial | MNLI-mm acc | 1 | 19: 70.36 | 70.36 |
| SuperGLUE partial | RTE acc | 5 | 19: 51.26, 8364: 50.54, 717: 48.74, 10536: 50.54, 90166: 48.38 | 49.89 +/- 1.26 |
| SuperGLUE partial | BoolQ acc | 3 | 23: 65.63, 42: 65.60, 6033: 64.62 | 65.28 +/- 0.57 |
| SuperGLUE partial | CB acc | 3 | 23: 67.86, 42: 66.07, 6033: 67.86 | 67.26 +/- 1.03 |
| SuperGLUE partial | CB F1 | 3 | 23: 57.63, 42: 55.20, 6033: 47.21 | 53.35 +/- 5.45 |
| SuperGLUE partial | COPA acc | 5 | 23: 56.00, 42: 51.00, 6033: 51.00, 1337: 56.00, 24: 55.00 | 53.80 +/- 2.59 |
| SuperGLUE partial | SWAG acc | 1 | 19: 33.47 | 33.47 |
| SuperGLUE partial | WiC acc | 3 | 23: 56.27, 42: 57.99, 6033: 58.93 | 57.73 +/- 1.35 |
Additional Information
Author
The model has been developed by the Language and Information Systems Group (GPLSI) and the Centro de Inteligencia Digital (CENID), both part of the University of Alicante (UA), as part of their ongoing research in Natural Language Processing (NLP).
Funding
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública, co-financed by the EU – NextGenerationEU, within the framework of the project Desarrollo de Modelos ALIA.
Acknowledgments
We would like to express our gratitude to all individuals and institutions that have contributed to the development of this work.
Special thanks to:
- Language Technologies Laboratory at Barcelona Supercomputing Center
- Centro Vasco de Tecnología de la Lengua (HiTZ)
- Centro Singular de Investigación en Tecnologías Inteligentes (CiTIUS)
- Sistemas Inteligentes de Acceso a la Información (SINAI)
- Instituto Universitario de Investigación Informática (IUII)
- Leonardo HPC System
- European supercomputing ecosystem (EUROHPC)
- MrBERT for the original model
We also acknowledge the financial, technical, and scientific support of the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA, whose contribution has been essential to the completion of this research.
License
This model is released under the Apache License 2.0.
Disclaimer
This model is intended for general purposes and is available under a permissive Apache License 2.0. Be aware that the model may have biases and/or undesirable outputs. Users deploying systems based on this model are responsible for mitigating risks and complying with applicable AI regulations.
Reference
If you use this model, please cite:
@misc{modernbert-ip-2026-encoder,
author = {Sepúlveda-Torres, Robiert and Estevanell-Valladares, Ernesto L. and Galeano, Santiago and Martínez-Murillo, Iván and Grande, Eduardo and Canal-Esteve, Miquel and Miró Maestre, María and Bonora, Mar and Gutierrez, Yoan and Abreu Salas, José Ignacio and Consuegra-Ayala, Juan Pablo and Lloret, Elena and Montoyo, Andrés and Muñoz-Guillena and Palomar, Manuel},
title = {Aitana IP Encoder: Domain-Adapted Language Model for Spanish and English Intellectual Property},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/gplsi/Aitana-intellectual-property-mb-encoder}}
}
Copyright © 2026 Language and Information Systems Group (GPLSI) and Centro de Inteligencia Digital (CENID), University of Alicante (UA). Distributed under the Apache License 2.0.
- Downloads last month
- 57