add RTEB, CoIR results
Browse files
README.md
CHANGED
|
@@ -5,7 +5,6 @@ tags:
|
|
| 5 |
- feature-extraction
|
| 6 |
- dense
|
| 7 |
- generated_from_trainer
|
| 8 |
-
- dataset_size:1880853
|
| 9 |
- loss:MultipleNegativesRankingLoss
|
| 10 |
widget:
|
| 11 |
- source_sentence: |-
|
|
@@ -801,6 +800,31 @@ base_model:
|
|
| 801 |
This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model trained on the [code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) dataset with
|
| 802 |
[<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with in-batch negatives. Model can be used for code retrieval and reranking.
|
| 803 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 804 |
## Model Details
|
| 805 |
|
| 806 |
### Model Description
|
|
@@ -808,16 +832,7 @@ This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/Moder
|
|
| 808 |
- **Maximum Sequence Length:** 8192 tokens
|
| 809 |
- **Output Dimensionality:** 768
|
| 810 |
- **Similarity Function:** Cosine Similarity
|
| 811 |
-
|
| 812 |
-
|
| 813 |
-
### Full Model Architecture
|
| 814 |
-
|
| 815 |
-
```
|
| 816 |
-
SentenceTransformer(
|
| 817 |
-
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
|
| 818 |
-
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
|
| 819 |
-
)
|
| 820 |
-
```
|
| 821 |
|
| 822 |
## Usage
|
| 823 |
|
|
|
|
| 5 |
- feature-extraction
|
| 6 |
- dense
|
| 7 |
- generated_from_trainer
|
|
|
|
| 8 |
- loss:MultipleNegativesRankingLoss
|
| 9 |
widget:
|
| 10 |
- source_sentence: |-
|
|
|
|
| 800 |
This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model trained on the [code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) dataset with
|
| 801 |
[<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with in-batch negatives. Model can be used for code retrieval and reranking.
|
| 802 |
|
| 803 |
+
## Perfomance on code retrieval benchmarks
|
| 804 |
+
|
| 805 |
+
**RTEB**
|
| 806 |
+
|
| 807 |
+
On 14.10.2025 the model is **6th** on RTEB leaderbord among models with <500M parameters:
|
| 808 |
+
<details>
|
| 809 |
+
<summary>Click</summary>
|
| 810 |
+
<figure>
|
| 811 |
+
<img src="Rteb_top.jpg">
|
| 812 |
+
</figure>
|
| 813 |
+
</details>
|
| 814 |
+
|
| 815 |
+
Perfomance per task:
|
| 816 |
+
| Model | AppsRetrieval | Code1Retrieval (Private) | DS1000Retrieval | FreshStackRetrieval | HumanEvalRetrieval | JapaneseCode1Retrieval (Private)| MBPPRetrieval | WikiSQLRetrieval |
|
| 817 |
+
|-------|---------------|----------------|-----------------|---------------------|--------------------|------------------------|---------------|------------------|
|
| 818 |
+
| english_code_retriever | 8.04 | 75.36 | 32.42 | 18.30 | 71.82 | 46.59 | 72.06 | 87.92 |
|
| 819 |
+
|
| 820 |
+
**COIR**:
|
| 821 |
+
| Model | AppsRetrieval | COIRCodeSearchNetRetrieval | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCCRetrieval | CodeTransOceanContest | CodeTransOceanDL | CosQA | StackOverflowQA | SyntheticText2SQL |
|
| 822 |
+
|-------|---------------|----------------------------|----------------|----------------|--------------------------|------------------------|------------------|-------|------------------|-------------------|
|
| 823 |
+
| english_code_retriever | 8.04 | 74.23 | 44.01 | 57.79 | 42.71 | 60.68 | 35.16 | 25.56 | 56.53 | 42.79 |
|
| 824 |
+
|
| 825 |
+
more information you cand find [here](https://huggingface.co/spaces/mteb/leaderboard)
|
| 826 |
+
|
| 827 |
+
|
| 828 |
## Model Details
|
| 829 |
|
| 830 |
### Model Description
|
|
|
|
| 832 |
- **Maximum Sequence Length:** 8192 tokens
|
| 833 |
- **Output Dimensionality:** 768
|
| 834 |
- **Similarity Function:** Cosine Similarity
|
| 835 |
+
- **Poling** Mean pooling
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 836 |
|
| 837 |
## Usage
|
| 838 |
|