Swift-1.5-Ornith-9B — W4A16 Quantized Collection

This repository is part of a unified W4A16 (4-bit weight, 16-bit activation) quantized release of ukisai/Swift-1.5-Ornith-9B, tuned using Intel AutoRound (SignRound) on an NVIDIA H200 GPU with a custom 512-sample, 4096-token reasoning, coding, and agentic tool-use calibration mix.

ukisai/Swift-1.5-Ornith-9B is UkisAI's reasoning-efficient derivative of ornith-ai/Ornith-1.5-9B, reducing thinking token usage by ~33.1% while improving GPQA-Diamond accuracy (+1.52 pp over base).


Available Quantized Variants

All three repositories share the exact same SignRound-optimized weights (W4A16, group_size=32, symmetric) exported simultaneously into three runtime serialization schemas:

Repository Serialization Format Target Runtimes & Kernels
Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound auto_round (auto-round:auto-gptq) vLLM (GPTQ/Marlin), Hugging Face transformers (auto-round backend)
Vishva007/Swift-1.5-Ornith-9B-W4A16-GPTQ auto_gptq vLLM (GPTQ/Marlin), TGI, AutoGPTQ-compatible engines
Vishva007/Swift-1.5-Ornith-9B-W4A16-LLM-Compressor llm_compressor (compressed-tensors) vLLM (compressed-tensors W4A16 Marlin kernels)

Quantization Methodology & Precision Architecture

Optimization Configuration

  • Algorithm: Intel AutoRound (SignRoundConfig)
  • Precision & Granularity: W4A16, group_size=32, sym=True
  • Tuning Hyperparameters: 1200 iterations per block, batch_size=2, gradient_accumulate_steps=4 (effective global batch size = 8), seqlen=4096, nsamples=512

Mixed-Precision Layer Preservation (Hybrid Qwen3.5 Architecture)

Swift-1.5-Ornith-9B uses a hybrid Qwen3_5ForConditionalGeneration architecture combining 24 GatedDeltaNet linear-attention blocks with 8 full-attention blocks, a vision encoder, and a 1-layer Multi-Token Prediction (MTP) head. To prevent accuracy degradation in recurrent state updates, vision embeddings, and speculative drafting, 128/359 linear layers are quantized to INT4 while 231 sensitive modules remain in BF16:

Component Precision Details
Full-Attention & MLP Projections INT4 (W4A16, g=32) Quantized across language model blocks (q/k/v/o_proj, gate/up/down_proj)
GatedDeltaNet (linear_attn) BF16 in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, out_proj across all 24 linear-attention layers
Vision Encoder (visual & merger) BF16 All 27 vision transformer blocks and patch merger preserved (quant_nontext_module=False)
MTP Speculative Head (mtp) BF16 All 15 MTP tensors preserved in model_extra_tensors.safetensors for native speculative decoding
Language Modeling Head (lm_head) BF16 Kept unquantized for logit precision

Calibration Dataset Mix (512 Samples @ 4096 Tokens)

All 512 calibration sequences were verified to exceed 4,096 tokens (min=4,103, max=51,063) with chat templates and <think> traces intact:

Dataset Samples Share Domain & Formatting
open-thoughts/OpenThoughts3-1.2M 180 35.16% STEM, math, and code reasoning traces (<think>)
nvidia/OpenCodeReasoning (split_0) 110 21.48% Competitive coding with chain-of-thought reasoning
openbmb/UltraData-SFT-Agent-2609 102 19.92% Multi-turn agentic trajectories & XML tool definitions
nvidia/OpenCodeInstruct 70 13.67% Direct code generation and execution instruction pairs
HuggingFaceH4/ultrachat_200k 50 9.77% General multi-turn conversational alignment

Serving with vLLM

Requires vLLM ≥ 0.19.1 and Transformers ≥ 5.8.1.

1. Standard High-Throughput Serving (200K Context + FP8 KV Cache)

Note: Replace Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound below with -AutoRound or -LLM-Compressor depending on your preferred checkpoint format.

vllm serve Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --max-model-len 200k \
  --gpu-memory-utilization 0.80 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --prefix-caching-hash-algo sha256 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8k \
  --max-num-scheduled-tokens 8k \
  --max-num-seqs 16 \
  --max-num-active-seqs 8 \
  --long-prefill-token-threshold 8384 \
  --scheduler-reserve-full-isl \
  --linear-backend auto \
  --enable-flashinfer-autotune \
  --async-scheduling \
  --max-cudagraph-capture-size 256 \
  --kv-offloading-size 8 \
  --kv-offloading-backend native \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":25769803776,"lazy_offload":false}}'
  • Thinking Mode Toggle: The command above sets '{"enable_thinking": true}' by default. Remove --default-chat-template-kwargs (or pass {"enable_thinking": false} per request via extra_body={"chat_template_kwargs": {"enable_thinking": false}}) to disable <think>...</think> reasoning traces routed to reasoning_content.

2. Enabling Speculative Decoding (MTP)

Because all 15 BF16 MTP head tensors are preserved in the checkpoint, you can append the MTP speculative decoding flag to any of the variants:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Recommended Sampling Parameters

  • General & Reasoning Tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
  • Precise Coding & Agentic Tool Use: temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

Python Client Example (OpenAI-Compatible API)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound",
    messages=[
        {"role": "user", "content": "Write a Python function to verify if a directed graph is a DAG."}
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)

msg = response.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.15 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.15-runpod 0rsw686010 Deploy to RunPod
PyTorch 2.15 (CUDA 13.4) 13.4 vishva123/cuda-13.4-pytorch-2.15-runpod 0s0rhdfaqy Deploy to RunPod

PyTorch 2.14

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.14 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.14-runpod d7lxsa4w9m Deploy to RunPod
PyTorch 2.14 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.14-runpod yk0y6j6rpg Deploy to RunPod
PyTorch 2.14 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.14-runpod gsp4gwx0nw Deploy to RunPod

PyTorch 2.13

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.13 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.13-runpod gmlupxnxfk Deploy to RunPod
PyTorch 2.13 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.13-runpod y3j8xvk4f4 Deploy to RunPod
PyTorch 2.13 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.13-runpod vigpissn5w Deploy to RunPod

PyTorch 2.12

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.12 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.12-runpod ctmz86zmf0 Deploy to RunPod
PyTorch 2.12 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.12-runpod qjko5yiwzi Deploy to RunPod
PyTorch 2.12 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.12-runpod ifg6xmye0f Deploy to RunPod

License & Acknowledgements

@misc{swift-1.5-ornith-9b,
  title  = {Swift 1.5 Ornith 9B},
  author = {UkisAI},
  year   = {2026},
  url    = {[https://huggingface.co/ukisai/Swift-1.5-Ornith-9B](https://huggingface.co/ukisai/Swift-1.5-Ornith-9B)}
}

@misc{ornith_1_5,
  title  = {{Ornith-1.5}: From Self-Scaffolding to Self-Improvement},
  url    = {[https://ornith.ai/ornith_1_5.html](https://ornith.ai/ornith_1_5.html)},
  author = {{Ornith Team}},
  year   = {2026}
}
Downloads last month
63
Safetensors
Model size
4B params
Tensor type
I32
·
BF16
·
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound

Quantized
(6)
this model

Collection including Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound