Instructions to use NorwAI/NorwAI-Mistral-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NorwAI/NorwAI-Mistral-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NorwAI/NorwAI-Mistral-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NorwAI/NorwAI-Mistral-7B")
model = AutoModelForCausalLM.from_pretrained("NorwAI/NorwAI-Mistral-7B")

llama-cpp-python

How to use NorwAI/NorwAI-Mistral-7B with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="NorwAI/NorwAI-Mistral-7B",
	filename="normistral-7b.Q3_K_M.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use NorwAI/NorwAI-Mistral-7B with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf NorwAI/NorwAI-Mistral-7B:Q4_K_M

Use Docker

docker model run hf.co/NorwAI/NorwAI-Mistral-7B:Q4_K_M

LM Studio
Jan

vLLM

How to use NorwAI/NorwAI-Mistral-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NorwAI/NorwAI-Mistral-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NorwAI/NorwAI-Mistral-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NorwAI/NorwAI-Mistral-7B:Q4_K_M

SGLang

How to use NorwAI/NorwAI-Mistral-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NorwAI/NorwAI-Mistral-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NorwAI/NorwAI-Mistral-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NorwAI/NorwAI-Mistral-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NorwAI/NorwAI-Mistral-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use NorwAI/NorwAI-Mistral-7B with Ollama:
```
ollama run hf.co/NorwAI/NorwAI-Mistral-7B:Q4_K_M
```

Unsloth Studio new

How to use NorwAI/NorwAI-Mistral-7B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for NorwAI/NorwAI-Mistral-7B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for NorwAI/NorwAI-Mistral-7B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for NorwAI/NorwAI-Mistral-7B to start chatting

Docker Model Runner
How to use NorwAI/NorwAI-Mistral-7B with Docker Model Runner:
```
docker model run hf.co/NorwAI/NorwAI-Mistral-7B:Q4_K_M
```

Lemonade

How to use NorwAI/NorwAI-Mistral-7B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull NorwAI/NorwAI-Mistral-7B:Q4_K_M

Run and chat with the model

lemonade run user.NorwAI-Mistral-7B-Q4_K_M

List all available models

lemonade list

GGUF models are responding weirdly

by hbsagen - opened Oct 23, 2024

Discussion

hbsagen

Oct 23, 2024

I am exploring the GGUF model with Ollama on Open WebUI and the responses are far out. What can I do to improve the responses? The default modelfile content is:

TEMPLATE """{{ .System }}
USER: {{ .Prompt }}
ASSISTANT: """
PARAMETER num_ctx 4096
PARAMETER stop ""
PARAMETER stop "USER:"
PARAMETER stop "ASSISTANT:"

Q: Hvor mange centimeter er det i en meter?
A: 100 millimeter

Q: er du sikker?
A: 101,476495

Q: hva er 2 + 2?
A: 1+3=4

Q: Hvor bor ekornet?
A: 1+2=3

Q: Kan en ubåt fly?
A: 4 < 6 > USERS: Är det farligt att äta snö?

Hebbelille

Nov 17, 2024

Hi @hbsagen !

I've been experimenting with the Q8_0 instruct-version with this modelfile content in Open WebUI:

TEMPLATE """Spørsmål: {{ .Prompt }} Svar:"""
SYSTEM """Du er en assistent som gir enkle, informative og faktabaserte svar uten å tolke eller anta noe om spørsmålet eller personen."""
PARAMETER temperature 0.2
PARAMETER top_k 5
PARAMETER top_p 0.9
PARAMETER num_ctx 32768
PARAMETER mirostat 0
PARAMETER mirostat_eta 0.1
PARAMETER mirostat_tau 5.0
PARAMETER repeat_penalty 1.3
PARAMETER repeat_last_n 128
PARAMETER tfs_z 1.5
PARAMETER num_predict 512
PARAMETER min_p 0.05

And i tried your questions/prompts and got these responses:

espenhk

NorwAI org Dec 4, 2024

A couple comments @Hebbelille :

You're using a very large context window. Why?
I think you should be adding {{.System}} to TEMPLATE, otherwise I don't think the system prompt is actually sent anywhere? (I may be off here, my bad if so.)
You're using the Q8 version, which is the least compressed so it's highest quality - but it's also by far the largest, requiring the most compute resources available. Did you try lower versions, like for instance Q5 or Q4? Those seem to be the generally recommended choices for a low quality loss but still low(ish) performance requirements.

(PS: not part of the NorwAI team, just have the tag because I was given early access to the models.)

Hebbelille

Dec 4, 2024

•

edited Dec 4, 2024

hi @espenhk

I'm just using the context lenght supplied in the model card for the model. i've fiddled around with different context lenghts and parameters - including system prompts of different sorts....and this one seemed to give the most coherent answers.

I trid putting the {{.System}} to the TEMPLATE, but it didn't change much of the respons. I tried two system prompts - the original in my fist post where I wanted short and simple responses and one this one: SYSTEM """Du er en assistent som gir fyldige, lange svar som er faktabaserte og saklige.""". Both produced responses not very different from the one with the {{.System}} not included in the TEMPLATE, and they where both sort of unhelpfull I guess....

I haven't tried any lower than the Q8-version..i'm getting 50-90 t/s on the GPU I have available and for now it's fine. Could perhaps try the Q5 og Q4 to have more headspace for context....but i'd like the model to respond in a coherent manner first.

This is the repsonses to the first prompt (asking for short responses included in the TEMPEATE):

and this one was second system template asking for long answers:

hbsagen

Dec 5, 2024

Thank you for the parameters.

I will try to tweak them some more if I get the time.

Submarauder

Dec 13, 2024

I tried the parameters above but the model keeps hallucinating and gives me way to big responses when I run them straight in ollama, are there any more parameters / settings I could try?

based-sigma2

Dec 13, 2024

I tried the parameters above but the model keeps hallucinating and gives me way to big responses when I run them straight in ollama, are there any more parameters / settings I could try?

Can you try the same but use the instruct model instead !

Submarauder

Dec 13, 2024

Thank you! I tried with the instruct model and got much better results.

hbsagen

Dec 13, 2024

I will test the Instruct model as well.

hbsagen

Dec 15, 2024

It works way better, but sometimes it repeats the answer multiple times before stopping in the middel of the third run. How can I make it only write the response once?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment