measurement.json

README.md

@@ -0,0 +1,222 @@
+---
+license: other
+license_name: llama-3
+license_link: https://llama.meta.com/llama3/license/
+tags:
+- text-generation-inference
+- transformers
+- unsloth
+- llama
+datasets:
+- Replete-AI/code_bagel_hermes-2.5
+- Replete-AI/code_bagel
+- Replete-AI/OpenHermes-2.5-Uncensored
+- teknium/OpenHermes-2.5
+- layoric/tiny-codes-alpaca
+- glaiveai/glaive-code-assistant-v3
+- ajibawa-2023/Code-290k-ShareGPT
+- TIGER-Lab/MathInstruct
+- chargoddard/commitpack-ft-instruct-rated
+- iamturun/code_instructions_120k_alpaca
+- ise-uiuc/Magicoder-Evol-Instruct-110K
+- cognitivecomputations/dolphin-coder
+- nickrosh/Evol-Instruct-Code-80k-v1
+- coseal/CodeUltraFeedback_binarized
+- glaiveai/glaive-function-calling-v2
+- CyberNative/Code_Vulnerability_Security_DPO
+- jondurbin/airoboros-2.2
+- camel-ai
+- lmsys/lmsys-chat-1m
+- CollectiveCognition/chats-data-2023-09-22
+- CoT-Alpaca-GPT4
+- WizardLM/WizardLM_evol_instruct_70k
+- WizardLM/WizardLM_evol_instruct_V2_196k
+- teknium/GPT4-LLM-Cleaned
+- GPTeacher
+- OpenGPT
+- meta-math/MetaMathQA
+- Open-Orca/SlimOrca
+- garage-bAInd/Open-Platypus
+- anon8231489123/ShareGPT_Vicuna_unfiltered
+- Unnatural-Instructions-GPT4
+model-index:
+- name: Replete-Coder-llama3-8b
+  results:
+  - task:
+      name: HumanEval
+      type: text-generation
+    dataset:
+      type: openai_humaneval
+      name: HumanEval
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 
+      verified: false
+  - task:
+      name: AI2 Reasoning Challenge
+      type: text-generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: accuracy
+      value: 
+      name: normalized accuracy
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+  - task:
+      name: Text Generation
+      type: text-generation
+    dataset:
+      name: HellaSwag (10-Shot)
+      type: hellaswag
+      split: validation
+      args:
+        num_few_shot: 10
+    metrics:
+    - type: accuracy
+      value: 
+      name: normalized accuracy
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+  - task:
+      name: Text Generation
+      type: text-generation
+    dataset:
+      name: MMLU (5-Shot)
+      type: cais/mmlu
+      config: all
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: accuracy
+      value: 
+      name: accuracy
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+  - task:
+      name: Text Generation
+      type: text-generation
+    dataset:
+      name: TruthfulQA (0-shot)
+      type: truthful_qa
+      config: multiple_choice
+      split: validation
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: multiple_choice_accuracy
+      value: 
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+  - task:
+      name: Text Generation
+      type: text-generation
+    dataset:
+      name: Winogrande (5-shot)
+      type: winogrande
+      config: winogrande_xl
+      split: validation
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: accuracy
+      value: 
+      name: accuracy
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+  - task:
+      name: Text Generation
+      type: text-generation
+    dataset:
+      name: GSM8k (5-shot)
+      type: gsm8k
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: accuracy
+      value: 
+      name: accuracy
+    source:
+      url: https://www.placeholderurl.com
+      name: Open LLM Leaderboard
+quantized_by: bartowski
+pipeline_tag: text-generation
+---
+
+## Exllama v2 Quantizations of Replete-Coder-Llama3-8B
+
+Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.1.6">turboderp's ExLlamaV2 v0.1.6</a> for quantization.
+
+<b>The "main" branch only contains the measurement.json, download one of the other branches for the model (see below)</b>
+
+Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.
+
+Original model: https://huggingface.co/Replete-AI/Replete-Coder-Llama3-8B
+
+## Prompt format
+
+No chat template specified so default is used. This may be incorrect, check original model card for details.
+
+```
+<|im_start|>system
+{system_prompt}<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant
+
+```
+
+## Available sizes
+
+
+| Branch | Bits | lm_head bits | VRAM (4k) | VRAM (8K) | VRAM (16k) | VRAM (32k) | Description |
+| ----- | ---- | ------- | ------ | ------ | ------ | ------ | ------------ |
+| [8_0](https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2/tree/8_0) | 8.0 | 8.0 | 10.1 GB | 10.5 GB | 11.5 GB | 13.6 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
+| [6_5](https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2/tree/6_5) | 6.5 | 8.0 | 8.9 GB | 9.3 GB | 10.3 GB | 12.4 GB | Very similar to 8.0, good tradeoff of size vs performance, **recommended**. |
+| [5_0](https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2/tree/5_0) | 5.0 | 6.0 | 7.7 GB | 8.1 GB | 9.1 GB | 11.2 GB | Slightly lower quality vs 6.5, but usable on 8GB cards. |
+| [4_25](https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2/tree/4_25) | 4.25 | 6.0 | 7.0 GB | 7.4 GB | 8.4 GB | 10.5 GB | GPTQ equivalent bits per weight, slightly higher quality. |
+| [3_5](https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2/tree/3_5) | 3.5 | 6.0 | 6.4 GB | 6.8 GB | 7.8 GB | 9.9 GB | Lower quality, only use if you have to. |
+
+## Download instructions
+
+With git:
+
+```shell
+git clone --single-branch --branch 6_5 https://huggingface.co/bartowski/Replete-Coder-Llama3-8B-exl2 Replete-Coder-Llama3-8B-exl2-6_5
+```
+
+With huggingface hub (credit to TheBloke for instructions):
+
+```shell
+pip3 install huggingface-hub
+```
+
+To download a specific branch, use the `--revision` parameter. For example, to download the 6.5 bpw branch:
+
+Linux:
+
+```shell
+huggingface-cli download bartowski/Replete-Coder-Llama3-8B-exl2 --revision 6_5 --local-dir Replete-Coder-Llama3-8B-exl2-6_5
+```
+
+Windows (which apparently doesn't like _ in folders sometimes?):
+
+```shell
+huggingface-cli download bartowski/Replete-Coder-Llama3-8B-exl2 --revision 6_5 --local-dir Replete-Coder-Llama3-8B-exl2-6.5
+```
+
+Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski