Instructions to use Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound") model = AutoModelForMultimodalLM.from_pretrained("Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound
- SGLang
How to use Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound with Docker Model Runner:
docker model run hf.co/Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound
Swift-1.5-Ornith-9B — W4A16 Quantized Collection
This repository is part of a unified W4A16 (4-bit weight, 16-bit activation) quantized release of ukisai/Swift-1.5-Ornith-9B, tuned using Intel AutoRound (SignRound) on an NVIDIA H200 GPU with a custom 512-sample, 4096-token reasoning, coding, and agentic tool-use calibration mix.
ukisai/Swift-1.5-Ornith-9B is UkisAI's reasoning-efficient derivative of ornith-ai/Ornith-1.5-9B, reducing thinking token usage by ~33.1% while improving GPQA-Diamond accuracy (+1.52 pp over base).
Available Quantized Variants
All three repositories share the exact same SignRound-optimized weights (W4A16, group_size=32, symmetric) exported simultaneously into three runtime serialization schemas:
| Repository | Serialization Format | Target Runtimes & Kernels |
|---|---|---|
Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound |
auto_round (auto-round:auto-gptq) |
vLLM (GPTQ/Marlin), Hugging Face transformers (auto-round backend) |
Vishva007/Swift-1.5-Ornith-9B-W4A16-GPTQ |
auto_gptq |
vLLM (GPTQ/Marlin), TGI, AutoGPTQ-compatible engines |
Vishva007/Swift-1.5-Ornith-9B-W4A16-LLM-Compressor |
llm_compressor (compressed-tensors) |
vLLM (compressed-tensors W4A16 Marlin kernels) |
Quantization Methodology & Precision Architecture
Optimization Configuration
- Algorithm: Intel AutoRound (
SignRoundConfig) - Precision & Granularity:
W4A16,group_size=32,sym=True - Tuning Hyperparameters:
1200iterations per block,batch_size=2,gradient_accumulate_steps=4(effective global batch size =8),seqlen=4096,nsamples=512
Mixed-Precision Layer Preservation (Hybrid Qwen3.5 Architecture)
Swift-1.5-Ornith-9B uses a hybrid Qwen3_5ForConditionalGeneration architecture combining 24 GatedDeltaNet linear-attention blocks with 8 full-attention blocks, a vision encoder, and a 1-layer Multi-Token Prediction (MTP) head. To prevent accuracy degradation in recurrent state updates, vision embeddings, and speculative drafting, 128/359 linear layers are quantized to INT4 while 231 sensitive modules remain in BF16:
| Component | Precision | Details |
|---|---|---|
| Full-Attention & MLP Projections | INT4 (W4A16, g=32) |
Quantized across language model blocks (q/k/v/o_proj, gate/up/down_proj) |
GatedDeltaNet (linear_attn) |
BF16 | in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, out_proj across all 24 linear-attention layers |
Vision Encoder (visual & merger) |
BF16 | All 27 vision transformer blocks and patch merger preserved (quant_nontext_module=False) |
MTP Speculative Head (mtp) |
BF16 | All 15 MTP tensors preserved in model_extra_tensors.safetensors for native speculative decoding |
Language Modeling Head (lm_head) |
BF16 | Kept unquantized for logit precision |
Calibration Dataset Mix (512 Samples @ 4096 Tokens)
All 512 calibration sequences were verified to exceed 4,096 tokens (min=4,103, max=51,063) with chat templates and <think> traces intact:
| Dataset | Samples | Share | Domain & Formatting |
|---|---|---|---|
open-thoughts/OpenThoughts3-1.2M |
180 | 35.16% | STEM, math, and code reasoning traces (<think>) |
nvidia/OpenCodeReasoning (split_0) |
110 | 21.48% | Competitive coding with chain-of-thought reasoning |
openbmb/UltraData-SFT-Agent-2609 |
102 | 19.92% | Multi-turn agentic trajectories & XML tool definitions |
nvidia/OpenCodeInstruct |
70 | 13.67% | Direct code generation and execution instruction pairs |
HuggingFaceH4/ultrachat_200k |
50 | 9.77% | General multi-turn conversational alignment |
Serving with vLLM
Requires vLLM ≥ 0.19.1 and Transformers ≥ 5.8.1.
1. Standard High-Throughput Serving (200K Context + FP8 KV Cache)
Note: Replace
Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRoundbelow with-AutoRoundor-LLM-Compressordepending on your preferred checkpoint format.
vllm serve Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--max-model-len 200k \
--gpu-memory-utilization 0.80 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--prefix-caching-hash-algo sha256 \
--enable-chunked-prefill \
--max-num-batched-tokens 8k \
--max-num-scheduled-tokens 8k \
--max-num-seqs 16 \
--max-num-active-seqs 8 \
--long-prefill-token-threshold 8384 \
--scheduler-reserve-full-isl \
--linear-backend auto \
--enable-flashinfer-autotune \
--async-scheduling \
--max-cudagraph-capture-size 256 \
--kv-offloading-size 8 \
--kv-offloading-backend native \
--reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": true}' \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":25769803776,"lazy_offload":false}}'
- Thinking Mode Toggle: The command above sets
'{"enable_thinking": true}'by default. Remove--default-chat-template-kwargs(or pass{"enable_thinking": false}per request viaextra_body={"chat_template_kwargs": {"enable_thinking": false}}) to disable<think>...</think>reasoning traces routed toreasoning_content.
2. Enabling Speculative Decoding (MTP)
Because all 15 BF16 MTP head tensors are preserved in the checkpoint, you can append the MTP speculative decoding flag to any of the variants:
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Recommended Sampling Parameters
- General & Reasoning Tasks:
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0 - Precise Coding & Agentic Tool Use:
temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0
Python Client Example (OpenAI-Compatible API)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="Vishva007/Swift-1.5-Ornith-9B-W4A16-AutoRound",
messages=[
{"role": "user", "content": "Write a Python function to verify if a directed graph is a DAG."}
],
temperature=0.6,
top_p=0.95,
max_tokens=2048,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
msg = response.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
PyTorch 2.15
PyTorch 2.14
PyTorch 2.13
PyTorch 2.12
License & Acknowledgements
- Base Model:
ornith-ai/Ornith-1.5-9B(MIT License) - Fine-Tuned Model:
ukisai/Swift-1.5-Ornith-9B(Apache License 2.0) - Quantization: Quantized and packaged by Vishva007 using Intel AutoRound under Apache License 2.0.
@misc{swift-1.5-ornith-9b,
title = {Swift 1.5 Ornith 9B},
author = {UkisAI},
year = {2026},
url = {[https://huggingface.co/ukisai/Swift-1.5-Ornith-9B](https://huggingface.co/ukisai/Swift-1.5-Ornith-9B)}
}
@misc{ornith_1_5,
title = {{Ornith-1.5}: From Self-Scaffolding to Self-Improvement},
url = {[https://ornith.ai/ornith_1_5.html](https://ornith.ai/ornith_1_5.html)},
author = {{Ornith Team}},
year = {2026}
}
- Downloads last month
- 63