cloverx-id/LuminaV-Optimizer-Paper

LuminaV Optimizer

We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them

Official Upstream & Standalone Codebase | Current Version: v1.2.1 | Check `Files and Versions`

Official Research Paper

LuminaV Optimizer Theory & Mechanics
Read LuminaV.pdf (Local Mirror) | Primary Paper Archive

Click the preview above to read or download the official paper PDF.

Notice: Official Upstream Repository

This repository (cloverx-id/LuminaV-Optimizer-Paper) is the official standalone and living development repository for the LuminaV optimizer family.

While LuminaV was originally conceived and validated as the core engine for the XoneLM-1.0 language model series, all subsequent optimizer upgrades, low-precision Triton kernels, PyTorch standards compliance, and bug fixes are actively maintained and released directly in this repository.

What's New in v1.2.1 (Latest Release)

The v1.2.1 release delivers critical GPU pipeline acceleration, eliminates host-device blocking synchronization, and ensures 100% compliance with modern PyTorch compilation and strict determinism:

Zero-Sync In-Kernel Parameter Norm Reduction: Fused the parameter Euclidean norm squared ∑(p_i²) calculation directly into Pass 1 Triton reduction kernels (_lumina_v2_pass1_kernel and _lumina_v1_pass1_kernel). Pass 2 and Pass 3 load and resolve norm_p entirely within GPU SRAM/registers. Completely eradicated p_contig.float().norm().item(), saving 200–400 synchronous PCIe roundtrips and pipeline stalls per training step on deep Transformer models.
CUDA Graphs & torch.compile Compatibility: Eradicated all host-blocking .item() calls, resolving fatal TorchDynamo graph breaks and making the optimizer step fully captureable via CUDA Graphs (torch.accelerator.Graph).
Stateless CPU Golden-Ratio PRNG Hashing: Replaced dynamic GPU RNG tensor allocation with a pure CPU integer multiplicative hash using Knuth's 32-bit golden ratio constant: (step * 0x9E3779B9 + i * 10007) & 0x7FFFFFFF. Generates uncorrelated 31-bit integer seeds per layer in nanoseconds without launching GPU kernels.
Strict Deterministic Compliance (torch.use_deterministic_algorithms): Injected dedicated per-device generator caches (self._generators) into fallback stochastic rounding (_sr_update). Prevents stochastic rounding from mutating PyTorch's default CUDA RNG state, ensuring bit-for-bit reproducibility for RLHF, Dropout, and DataLoader shuffling.
Consolidated Batch Scratch Buffer Reset: Replaced hundreds of per-slice memset micro-calls with a single consolidated scratch.zero_() reset per step, removing up to 400–800 driver launch overheads per iteration.

For the full version history and detailed patch notes, see CHANGELOG.md.)

Overview

LuminaV is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (FP16 / BF16) without maintaining redundant 4-byte FP32 master weights.

By combining Centered Innovation Variance, Hyperbolic Tangent (tanh) Coordinate Bounding, a Directional Traffic-Cop Mask, and On-Chip Bitwise Stochastic Rounding, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while avoiding weight freezing and gradient shocks.

Key Features

Zero Master-Weight Copies: Directly mutates parameter weights in native FP16 or BF16, eliminating the 4-byte FP32 master weight allocation.
On-Chip Bitwise Stochastic Rounding (SR): Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
Hyperbolic tanh Bounding Envelope: Maps normalized momentum through a (-1.0, 1.0) transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
The Traffic-Cop Directional Gate: Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (u_t · g_t ≤ 0).
Centered Innovation Variance: Tracks centered innovation dispersion (g_t - m_t)² rather than uncentered raw second moments, suppressing variance inflation during confident descent.
Automatic FP16 Cliff Governor: Built-in asymptotic boundary governor that dampens steps near the IEEE-754 FP16 overflow limit (> 65,504), enabling stable pure FP16 training without external schedulers or clipping.
Dual Execution Engine: Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ torch._foreach multi-tensor fallbacks.

Installation

From PyPI (Recommended)

pip install luminav

For GPU acceleration via OpenAI Triton:

pip install luminav[triton]

From Source (Editable Mode)

git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper
pip install -e .

Direct File Drop-in

Alternatively, you can copy luminav.py directly into your working project directory without packaging overhead:

wget https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/raw/main/luminav.py

Quickstart

Standard Instantiation

import torch
from luminav import LuminaV

# Instantiate your model in native low precision (e.g. BF16 or FP16)
model = YourModel().to(device="cuda", dtype=torch.float16)

# Initialize LuminaV v1.2.1
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,                    # or 8e-5 / 8e-6 for fine-tuning
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    alpha_ss=0.5,
    cautious=True,
    cautious_clamp_min=0.5,     # Exact power-of-two ceiling (2.00x)
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Extreme Low VRAM)
    stochastic_rounding=True,
    bound=True,                 # Smooth asymptotic step bounding
    bound_type="radial",        # "radial" (preserves 100% angular direction) or "coordinate"
    bound_ratio=0.03,
    execution="auto"
)

# Standard training step
optimizer.zero_grad(set_to_none=True)
loss = model(inputs, targets)
loss.backward()
optimizer.step()

Loading from `config.json`

import json
import torch
from luminav import LuminaV

with open("config.json", "r") as f:
    config = json.load(f)

# Initialize with verified default configuration
optimizer = LuminaV(model.parameters(), **config["default_params"])

Parameter Reference

Parameter	Type	Default	Description
`params`	`iterable`	Required	Iterable of parameters to optimize or dicts defining parameter groups.
`lr`	`float`	`8e-4`	Learning rate (η).
`betas`	`Tuple[float, float]`	`(0.9, 0.999)`	Coefficients (β₁, β₂) for running momentum and centered innovation variance.
`eps`	`float`	`1e-8`	Numerical stability term (ε). Automatically floored to `1e-4` in FP16 to prevent subnormal underflow.
`weight_decay`	`float`	`8e-2`	Decoupled weight decay coefficient (λ).
`tau`	`float`	`0.8`	Analytical bias correction temperature parameter (τ).
`alpha_ss`	`float`	`0.5`	Softsign dampening factor (`α_ss`) used in single-buffer mode (`buffer=1`).
`cautious`	`bool`	`True`	If `True`, enables Traffic-Cop directional verification masking.
`cautious_clamp_min`	`float`	`0.5`	Safety floor density clamp (`γ_min`) enforcing a power-of-two maximum energy scaling ceiling (2.00x, 2¹) and preventing division by zero.
`buffer`	`int`	`2`	Buffer mode: `2` (Dual-buffer tracking `m_t` and `v_t`) or `1` (Single-buffer scalar RMS tracking).
`stochastic_rounding`	`bool`	`True`	Enables bitwise stochastic rounding on native FP16/BF16 weights.
`bound`	`bool`	`True`	If `True`, enables smooth asymptotic parameter bounding to prevent divergence in deep networks.
`bound_type`	`str`	`"radial"`	Asymptotic bounding formulation: `"radial"` (direction-preserving squashing using `tanh(r)/r`) or `"coordinate"` (elementwise squashing).
`bound_ratio`	`float`	`0.03`	Maximum allowed step displacement ratio relative to parameter norm or magnitude (`R = bound_ratio * ‖p‖`).
`execution`	`str`	`"auto"`	Execution engine: `"auto"`, `"triton"`, `"foreach"`, or `"single"`.

Operational Modes

LuminaV-2 (Dual-Buffer Default: `buffer=2`)

Maintains first moment m_t and centered innovation variance v_t:

$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$

$v_t = \beta_2 v_{t-1} + (1 - \beta_2)(g_t - m_t)^2$

Updates are bounded through the hyperbolic tangent envelope:

$u_t = \tanh\left(\frac{\tilde{m}_t}{\sigma_t}\right)$

LuminaV-1 (Single-Buffer Extreme-Poverty Mode: `buffer=1`)

Collapses variance tracking into a scalar Root-Mean-Square (RMS) across the entire tensor, saving 50% optimizer state memory by maintaining only a single state buffer (m_t):

$\text{RMS}(\tilde{m}_t) = \sqrt{\frac{1}{N} \sum_{i=1}^N \tilde{m}_{t,i}^2 + \epsilon}$

$u_t = \tanh\left(\frac{z}{1 + \alpha_{ss}|z|}\right), \quad z = \frac{\tilde{m}_t}{\tau \cdot \text{RMS}(\tilde{m}_t) + \epsilon(1 - \beta_1^t)\tau}$

Contributors & Acknowledgements

LuminaV is developed and maintained by Silver Moon (@cloverxion) and the Lumina Moon community contributors.

For the complete list of individuals who have contributed code, mathematical analyses, and experimental validation, please refer to CONTRIBUTORS.md.

Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion) and Lumina Moon Contributors}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face / PyPI},
  version      = {1.2.1},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}

License

Apache License 2.0. See LICENSE for full terms.

Downloads last month: 394

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support