Instructions to use OVHaiLLM/Olala-v1.0-mini-7A1B-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use OVHaiLLM/Olala-v1.0-mini-7A1B-MLX with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("OVHaiLLM/Olala-v1.0-mini-7A1B-MLX") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use OVHaiLLM/Olala-v1.0-mini-7A1B-MLX with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "OVHaiLLM/Olala-v1.0-mini-7A1B-MLX"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "OVHaiLLM/Olala-v1.0-mini-7A1B-MLX" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OVHaiLLM/Olala-v1.0-mini-7A1B-MLX", "messages": [ {"role": "user", "content": "Hello"} ] }' - Atomic Chat
Olala-v1.0-mini-7A1B-MLX
These weights need our mlx-engine fork. Upstream mlx-lm and LM Studio do not support the
olalaarchitecture yet and cannot load them. Install the fork as shown in Run on a Mac.
MLX versions of Olala-v1.0-mini-7A1B for local inference on Apple Silicon Macs.
- Runs on a 16 GB MacBook Air M5: the mixed 5/6-bit folder uses 5.4 GB and generates about 86 tokens/s.
- Faster than llama.cpp on a Mac: custom Metal kernels for the Mamba-3, attention and mixture-of-experts layers.
- Same model: reasoning with four effort levels and tool calling, through an OpenAI-compatible server.
Quantized versions
Each folder is a complete MLX model (weights, config, tokenizer, chat template). Download only the one you need.
| Folder | Quantization | Bits / weight | Size |
|---|---|---|---|
bf16/ |
None (BF16, as in the source checkpoint) | 16 | 13.6 GB |
q8/ |
8-bit, group 64 | 8.56 | 7.2 GB |
q5e6/ |
Routed experts 5-bit, everything else 6-bit, group 64 | 5.76 | 4.9 GB |
q4e8/ |
Routed experts 4-bit (group 32), everything else 8-bit (group 64) | 5.75 | 4.9 GB |
Speed
Run on a Mac
Upstream mlx-lm, mlx-vlm and LM Studio do not support the olala architecture yet. Use the
mlx-engine fork (a fork of LM Studio's
mlx-engine), with Python 3.11 on macOS 14 or later. The Mamba
state is kept in float32 automatically; there is no flag to set.
Sampling parameters: temperature=0.8, top_p=0.9, top_k=40.
Reasoning effort: disabled, low, medium (default) or high. It is a chat-template argument
(reasoning_effort=... in apply_chat_template()). In the server, pass it inside chat_template_kwargs.
Apple M1: run
export OLALA_MOE_V3=0before Python or the server. Without it, M1 Macs crash on the first generated token ("Thread group size โฆ > 384"). M2 and later need nothing.
1. Install the fork and download a folder
git clone https://github.com/OVHai-LLM/mlx-engine.git
cd mlx-engine
python3.11 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/hf download OVHaiLLM/Olala-v1.0-mini-7A1B-MLX --include "q5e6/*" --local-dir ./olala-mlx
2. Chat from Python (mlx-lm)
Run from the mlx-engine checkout:
from mlx_engine.models.olala import register
register() # registers the olala architecture with mlx-lm
from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("./olala-mlx/q5e6")
prompt = tok.apply_chat_template(
[{"role": "user", "content": "Explain how a heat pump works."}],
add_generation_prompt=True, reasoning_effort="medium",
)
for r in stream_generate(model, tok, prompt, max_tokens=4096,
sampler=make_sampler(temp=0.8, top_p=0.9, top_k=40)):
print(r.text, end="", flush=True)
This prints the raw output with its channels (reasoning in analysis, the answer in final, tool calls in
tools). For a single question from the terminal: .venv/bin/python demo.py --model ./olala-mlx/q5e6 --no-system --prompt "..." (--no-system because the chat template already has its own system prompt).
3. Serve an OpenAI-compatible API
export MLX_ENGINE_API_KEY=choose-a-key # required; clients send it as a Bearer token
.venv/bin/python -m mlx_engine.server --model ./olala-mlx/q5e6 \
--host 127.0.0.1 --port 8000 --context-length 8192 --parallel-sessions 1
All these options are required. --context-length goes up to 32,768 and is shared by the
--parallel-sessions; raise them if you have the memory. Use --host 0.0.0.0 to reach the server from other
machines. The server holds one model.
curl -sN http://127.0.0.1:8000/v1/chat/completions \
-H "Authorization: Bearer $MLX_ENGINE_API_KEY" -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "Explain how a heat pump works."}],
"stream": true, "temperature": 0.8, "top_p": 0.9, "top_k": 40, "max_tokens": 4096,
"chat_template_kwargs": {"reasoning_effort": "medium"}
}'
With the OpenAI Python client, top_k and chat_template_kwargs go through extra_body:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="choose-a-key")
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get the current weather for a city.",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
stream = client.chat.completions.create(
model="olala", # ignored: the server holds one model
messages=[{"role": "user", "content": "What's the weather in Lyon?"}],
tools=tools, stream=True, temperature=0.8, top_p=0.9, max_tokens=4096,
extra_body={"top_k": 40, "chat_template_kwargs": {"reasoning_effort": "low"}},
)
for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if getattr(delta, "reasoning_content", None):
print(delta.reasoning_content, end="", flush=True)
if delta.content:
print(delta.content, end="", flush=True)
for call in delta.tool_calls or []:
print("\ntool call:", call.id, call.function.name, call.function.arguments)
# tool call: call_0 get_weather {"city": "Lyon"}
To continue after a tool call, send the conversation back with the assistant message (and its tool_calls) and
a {"role": "tool", "tool_call_id": ..., "name": ..., "content": ...} message.
Differences from the OpenAI API
- Responses are streamed only:
"stream": true,temperatureandtop_kare required. - The reasoning effort goes in
chat_template_kwargs. - Deltas carry
reasoning_content(the reasoning), thencontent(the answer). Tool calls arrive as onetool_callsdelta at the end of the turn, withfinish_reason: "tool_calls". tool_choiceaccepts"auto"and"none"only.seed,logprobs,logit_biasand the presence and frequency penalties are rejected with a 400 error.- When a response reaches
max_tokens, the stream ends with an error event (MLX generation ended without a stop condition.) instead offinish_reason: "length". The text sent before it is valid. - Send a system message only to add instructions: the chat template already contains the model's system prompt.
- Health check:
GET /healthwith the same Bearer token.
How the versions were made
With the fork, from the BF16 safetensors of Olala-v1.0-mini-7A1B:
python -m mlx_engine.models.olala.convert --hf-path Olala-v1.0-mini-7A1B --mlx-path q5e6 --recipe q5e6
Recipes: bf16, q8, q6, q5e6, q4e8, q4. Plain 4-bit (q4) loses too much quality and is not published. bf16/ holds the source weights unchanged; projections that read the same input are concatenated at
conversion, so the folders do not need the Hugging Face remote code.
Uses and limitations
The uses and limitations of Olala-v1.0-mini-7A1B apply to these files. In addition:
- They need our mlx-engine fork. Upstream mlx-lm and LM Studio cannot load them yet.
- Apple Silicon only.
- Apple M1: set
OLALA_MOE_V3=0before running Python or the server. On an M1 Pro with 16 GB and q5e6, the card's examples then run at about 62 tokens/s. - Quantize with our recipe only. Generic recipes produce NaN on realistic inputs.
Before you deploy: read the public summary of training content and our copyright policy.
License
| Licence | Apache License, Version 2.0 (SPDX: Apache-2.0) |
| Licensor | OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France |
| Copyright | Copyright 2026 OVH LLM SAS |
| Release date | 19 November 2026 |
| Distribution | Open weights |
| Notices | NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0) |
| Training content (EU AI Act) | Public summary of training content, on the European Commission template (Art. 53(1)(d)) |
Citation
@techreport{ovhaillm2026olala,
title = {Olala v1.0 mini Technical Report},
author = {{OVHai LLM Research Team}},
institution = {OVHcloud},
year = {2026},
url = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}
Quantized
Model tree for OVHaiLLM/Olala-v1.0-mini-7A1B-MLX
Base model
OVHaiLLM/Olala-v1.0-mini-7A1B-base