Olala-v1.0-mini-7A1B-MLX

These weights need our mlx-engine fork. Upstream mlx-lm and LM Studio do not support the olala architecture yet and cannot load them. Install the fork as shown in Run on a Mac.

MLX versions of Olala-v1.0-mini-7A1B for local inference on Apple Silicon Macs.

  • Runs on a 16 GB MacBook Air M5: the mixed 5/6-bit folder uses 5.4 GB and generates about 86 tokens/s.
  • Faster than llama.cpp on a Mac: custom Metal kernels for the Mamba-3, attention and mixture-of-experts layers.
  • Same model: reasoning with four effort levels and tool calling, through an OpenAI-compatible server.

Quantized versions


Each folder is a complete MLX model (weights, config, tokenizer, chat template). Download only the one you need.

Folder Quantization Bits / weight Size
bf16/ None (BF16, as in the source checkpoint) 16 13.6 GB
q8/ 8-bit, group 64 8.56 7.2 GB
q5e6/ Routed experts 5-bit, everything else 6-bit, group 64 5.76 4.9 GB
q4e8/ Routed experts 4-bit (group 32), everything else 8-bit (group 64) 5.75 4.9 GB

Speed


Left: generation speed on a MacBook Air M5 with 16 GB (q8 69, q5e6 86, q4e8 64 to 68 tokens per second). Right: on a Mac Studio M3 Ultra, MLX against llama.cpp (BF16: 191 against 149 tokens per second; 8-bit: 266 against 178).

Run on a Mac


Upstream mlx-lm, mlx-vlm and LM Studio do not support the olala architecture yet. Use the mlx-engine fork (a fork of LM Studio's mlx-engine), with Python 3.11 on macOS 14 or later. The Mamba state is kept in float32 automatically; there is no flag to set.

Sampling parameters: temperature=0.8, top_p=0.9, top_k=40.

Reasoning effort: disabled, low, medium (default) or high. It is a chat-template argument (reasoning_effort=... in apply_chat_template()). In the server, pass it inside chat_template_kwargs.

Apple M1: run export OLALA_MOE_V3=0 before Python or the server. Without it, M1 Macs crash on the first generated token ("Thread group size โ€ฆ > 384"). M2 and later need nothing.

1. Install the fork and download a folder

git clone https://github.com/OVHai-LLM/mlx-engine.git
cd mlx-engine
python3.11 -m venv .venv && .venv/bin/pip install -r requirements.txt

.venv/bin/hf download OVHaiLLM/Olala-v1.0-mini-7A1B-MLX --include "q5e6/*" --local-dir ./olala-mlx

2. Chat from Python (mlx-lm)

Run from the mlx-engine checkout:

from mlx_engine.models.olala import register
register()  # registers the olala architecture with mlx-lm

from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("./olala-mlx/q5e6")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain how a heat pump works."}],
    add_generation_prompt=True, reasoning_effort="medium",
)
for r in stream_generate(model, tok, prompt, max_tokens=4096,
                         sampler=make_sampler(temp=0.8, top_p=0.9, top_k=40)):
    print(r.text, end="", flush=True)

This prints the raw output with its channels (reasoning in analysis, the answer in final, tool calls in tools). For a single question from the terminal: .venv/bin/python demo.py --model ./olala-mlx/q5e6 --no-system --prompt "..." (--no-system because the chat template already has its own system prompt).

3. Serve an OpenAI-compatible API

export MLX_ENGINE_API_KEY=choose-a-key   # required; clients send it as a Bearer token
.venv/bin/python -m mlx_engine.server --model ./olala-mlx/q5e6 \
    --host 127.0.0.1 --port 8000 --context-length 8192 --parallel-sessions 1

All these options are required. --context-length goes up to 32,768 and is shared by the --parallel-sessions; raise them if you have the memory. Use --host 0.0.0.0 to reach the server from other machines. The server holds one model.

curl -sN http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $MLX_ENGINE_API_KEY" -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Explain how a heat pump works."}],
  "stream": true, "temperature": 0.8, "top_p": 0.9, "top_k": 40, "max_tokens": 4096,
  "chat_template_kwargs": {"reasoning_effort": "medium"}
}'

With the OpenAI Python client, top_k and chat_template_kwargs go through extra_body:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="choose-a-key")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Get the current weather for a city.",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]

stream = client.chat.completions.create(
    model="olala",  # ignored: the server holds one model
    messages=[{"role": "user", "content": "What's the weather in Lyon?"}],
    tools=tools, stream=True, temperature=0.8, top_p=0.9, max_tokens=4096,
    extra_body={"top_k": 40, "chat_template_kwargs": {"reasoning_effort": "low"}},
)
for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    if getattr(delta, "reasoning_content", None):
        print(delta.reasoning_content, end="", flush=True)
    if delta.content:
        print(delta.content, end="", flush=True)
    for call in delta.tool_calls or []:
        print("\ntool call:", call.id, call.function.name, call.function.arguments)
# tool call: call_0 get_weather {"city": "Lyon"}

To continue after a tool call, send the conversation back with the assistant message (and its tool_calls) and a {"role": "tool", "tool_call_id": ..., "name": ..., "content": ...} message.

Differences from the OpenAI API
  • Responses are streamed only: "stream": true, temperature and top_k are required.
  • The reasoning effort goes in chat_template_kwargs.
  • Deltas carry reasoning_content (the reasoning), then content (the answer). Tool calls arrive as one tool_calls delta at the end of the turn, with finish_reason: "tool_calls".
  • tool_choice accepts "auto" and "none" only. seed, logprobs, logit_bias and the presence and frequency penalties are rejected with a 400 error.
  • When a response reaches max_tokens, the stream ends with an error event (MLX generation ended without a stop condition.) instead of finish_reason: "length". The text sent before it is valid.
  • Send a system message only to add instructions: the chat template already contains the model's system prompt.
  • Health check: GET /health with the same Bearer token.
How the versions were made

With the fork, from the BF16 safetensors of Olala-v1.0-mini-7A1B:

python -m mlx_engine.models.olala.convert --hf-path Olala-v1.0-mini-7A1B --mlx-path q5e6 --recipe q5e6

Recipes: bf16, q8, q6, q5e6, q4e8, q4. Plain 4-bit (q4) loses too much quality and is not published. bf16/ holds the source weights unchanged; projections that read the same input are concatenated at conversion, so the folders do not need the Hugging Face remote code.

Uses and limitations


The uses and limitations of Olala-v1.0-mini-7A1B apply to these files. In addition:

  • They need our mlx-engine fork. Upstream mlx-lm and LM Studio cannot load them yet.
  • Apple Silicon only.
  • Apple M1: set OLALA_MOE_V3=0 before running Python or the server. On an M1 Pro with 16 GB and q5e6, the card's examples then run at about 62 tokens/s.
  • Quantize with our recipe only. Generic recipes produce NaN on realistic inputs.

Before you deploy: read the public summary of training content and our copyright policy.

License


Licence Apache License, Version 2.0 (SPDX: Apache-2.0)
Licensor OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France
Copyright Copyright 2026 OVH LLM SAS
Release date 19 November 2026
Distribution Open weights
Notices NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0)
Training content (EU AI Act) Public summary of training content, on the European Commission template (Art. 53(1)(d))

Citation


@techreport{ovhaillm2026olala,
  title       = {Olala v1.0 mini Technical Report},
  author      = {{OVHai LLM Research Team}},
  institution = {OVHcloud},
  year        = {2026},
  url         = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}
Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OVHaiLLM/Olala-v1.0-mini-7A1B-MLX

Collection including OVHaiLLM/Olala-v1.0-mini-7A1B-MLX