Instructions to use rAIfle/Interim-MN-fp16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rAIfle/Interim-MN-fp16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rAIfle/Interim-MN-fp16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rAIfle/Interim-MN-fp16")
model = AutoModelForCausalLM.from_pretrained("rAIfle/Interim-MN-fp16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use rAIfle/Interim-MN-fp16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rAIfle/Interim-MN-fp16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rAIfle/Interim-MN-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rAIfle/Interim-MN-fp16

SGLang

How to use rAIfle/Interim-MN-fp16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rAIfle/Interim-MN-fp16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rAIfle/Interim-MN-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rAIfle/Interim-MN-fp16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rAIfle/Interim-MN-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use rAIfle/Interim-MN-fp16 with Docker Model Runner:
```
docker model run hf.co/rAIfle/Interim-MN-fp16
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Interim-MN-fp16

This one was a doozy. Three restarted runs of which one failed at the last possible moment... Anyway, the result is decent enough and passes my own Nala-testing so it's not too bad. Not gonna benchmark this myself but if anyone feels the need to then I'd be happy to include numbers here.

Probably best if using normal Nemo settings, no weird stuff. (Or just use the recommended ones in the Recommended Settings repo)

iMat GGUFs courtesy of your local Cartel frens

Downloads last month: 11

Safetensors

Model size

12B params

Tensor type

BF16

Model tree for rAIfle/Interim-MN-fp16

Quantizations

2 models

Collection including rAIfle/Interim-MN-fp16

Home baked

Collection

Home-baked merges and tunes. • 12 items • Updated Mar 2