fyaronskiy commited on
Commit
c1af54a
·
verified ·
1 Parent(s): 5a82deb

add RTEB, CoIR results

Browse files
Files changed (1) hide show
  1. README.md +26 -11
README.md CHANGED
@@ -5,7 +5,6 @@ tags:
5
  - feature-extraction
6
  - dense
7
  - generated_from_trainer
8
- - dataset_size:1880853
9
  - loss:MultipleNegativesRankingLoss
10
  widget:
11
  - source_sentence: |-
@@ -801,6 +800,31 @@ base_model:
801
  This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model trained on the [code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) dataset with
802
  [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with in-batch negatives. Model can be used for code retrieval and reranking.
803
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
804
  ## Model Details
805
 
806
  ### Model Description
@@ -808,16 +832,7 @@ This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/Moder
808
  - **Maximum Sequence Length:** 8192 tokens
809
  - **Output Dimensionality:** 768
810
  - **Similarity Function:** Cosine Similarity
811
-
812
-
813
- ### Full Model Architecture
814
-
815
- ```
816
- SentenceTransformer(
817
- (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
818
- (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
819
- )
820
- ```
821
 
822
  ## Usage
823
 
 
5
  - feature-extraction
6
  - dense
7
  - generated_from_trainer
 
8
  - loss:MultipleNegativesRankingLoss
9
  widget:
10
  - source_sentence: |-
 
800
  This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model trained on the [code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) dataset with
801
  [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with in-batch negatives. Model can be used for code retrieval and reranking.
802
 
803
+ ## Perfomance on code retrieval benchmarks
804
+
805
+ **RTEB**
806
+
807
+ On 14.10.2025 the model is **6th** on RTEB leaderbord among models with <500M parameters:
808
+ <details>
809
+ <summary>Click</summary>
810
+ <figure>
811
+ <img src="Rteb_top.jpg">
812
+ </figure>
813
+ </details>
814
+
815
+ Perfomance per task:
816
+ | Model | AppsRetrieval | Code1Retrieval (Private) | DS1000Retrieval | FreshStackRetrieval | HumanEvalRetrieval | JapaneseCode1Retrieval (Private)| MBPPRetrieval | WikiSQLRetrieval |
817
+ |-------|---------------|----------------|-----------------|---------------------|--------------------|------------------------|---------------|------------------|
818
+ | english_code_retriever | 8.04 | 75.36 | 32.42 | 18.30 | 71.82 | 46.59 | 72.06 | 87.92 |
819
+
820
+ **COIR**:
821
+ | Model | AppsRetrieval | COIRCodeSearchNetRetrieval | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCCRetrieval | CodeTransOceanContest | CodeTransOceanDL | CosQA | StackOverflowQA | SyntheticText2SQL |
822
+ |-------|---------------|----------------------------|----------------|----------------|--------------------------|------------------------|------------------|-------|------------------|-------------------|
823
+ | english_code_retriever | 8.04 | 74.23 | 44.01 | 57.79 | 42.71 | 60.68 | 35.16 | 25.56 | 56.53 | 42.79 |
824
+
825
+ more information you cand find [here](https://huggingface.co/spaces/mteb/leaderboard)
826
+
827
+
828
  ## Model Details
829
 
830
  ### Model Description
 
832
  - **Maximum Sequence Length:** 8192 tokens
833
  - **Output Dimensionality:** 768
834
  - **Similarity Function:** Cosine Similarity
835
+ - **Poling** Mean pooling
 
 
 
 
 
 
 
 
 
836
 
837
  ## Usage
838