|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- code-search-net/code_search_net |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: fill-mask |
|
|
tags: |
|
|
- code |
|
|
- python |
|
|
- java |
|
|
- javascript |
|
|
- go |
|
|
- ruby |
|
|
- php |
|
|
--- |
|
|
|
|
|
# CodeModernBERT-Finch |
|
|
|
|
|
This model is a code-specific pretrained model created solely using the CodeSearchNet dataset. It supports six languages included in CodeSearchNet.\ |
|
|
For a version fine-tuned specifically for code search tasks, please refer to [Shuu12121/CodeSearch-ModernBERT-Finch](https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Finch). |
|
|
|
|
|
## Architecture |
|
|
|
|
|
* Base: ModernBERT-style encoder |
|
|
* Hidden size: 512 |
|
|
* Layers: 6 |
|
|
* Attention heads: 6 |
|
|
* Parameters: \~50M |
|
|
* Pretraining: Masked Language Modeling (MLM) |
|
|
* Fine-tuning: Domain-specific code tasks |
|
|
|
|
|
The results below were obtained by randomly sampling 10,000 examples per language from the CodeSearchNet dataset, training them in a Sentence-BERT fashion, and evaluating on the MTEB CodeSearchNetRetrieval benchmark. |
|
|
All models listed in the table below were fine-tuned using the same approach. Those marked with 200 and the Finch models were trained with a Multiple Negatives Ranking Loss batch size of 200. Others were trained with a batch size of 40 (because larger batches could not fit into memory).\ |
|
|
Finch-SmallBatch was trained with a smaller batch size of 40 to create a comparison model against the standard Finch models trained with batch size 200. |
|
|
|
|
|
| Model | go | java | javascript | php | python | ruby | |
|
|
| ---------------------------------- | ----- | ----- | ---------- | ----- | ------ | ----- | |
|
|
| Finch(40M) | 0.934 | 0.784 | 0.728 | 0.835 | 0.865 | 0.756 | |
|
|
| Finch-Pre(40M) | 0.937 | 0.705 | 0.685 | 0.828 | 0.843 | 0.725 | |
|
|
| Finch-SmallBatch(40M) | 0.930 | 0.765 | 0.707 | 0.825 | 0.859 | 0.748 | |
|
|
| ModernBERT-base-Finetuned(149M) | 0.933 | 0.779 | 0.748 | 0.839 | 0.885 | 0.794 | |
|
|
| Owl-4.1-Small-Fine-tuned(151M) | 0.942 | 0.780 | 0.729 | 0.843 | 0.893 | 0.772 | |
|
|
| Owl-4.1-Small-Fine-tuned-200(151M) | 0.943 | 0.850 | 0.747 | 0.858 | 0.894 | 0.802 | |
|
|
| CodeBERT-Fine-tuned(125M) | 0.932 | 0.708 | 0.709 | 0.828 | 0.870 | 0.772 | |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
|
