Hugging Face's logo

cloverx-id
/
LuminaV-Optimizer

PyTorch
Rust
English
optimizer
custom-optimizer
luminamoon
luminav
clm
td-clm
lso-eval
simd
avx512
vulkan
triton
memory-efficient
stochastic-rounding
low-vram
bf16
fp16
cuda
rocm
xpu
llm
transformer
deep-learning
research
Model card Files
xet
 Community

LuminaV Optimizer v1.6.0 (luminamoon v1.0.9)

Low-Precision BF16/FP16 Deep Learning Optimizer with Dual-Buffer Convex Lookahead Momentum (CLM), Single-Buffer Triangle-Defect Convex Lookahead Momentum (TD-CLM), Rational Padé [13/6] Bounding, and Stateless 32-Bit MurmurHash3 Stochastic Rounding

Official Codebase: github.com/cloverx-id/luminamoon | Docs Portal: cloverx-id.github.io/luminamoon

PyPI luminamoon Documentation Portal Changelog LSO-Eval v1.1.0 Config JSON License

PROJECT LOOMY

PyPI version CI Status LSO-Eval v1.1.0 Documentation Portal Python Versions License

Loomy (luminamoon) is the reference implementation and central repository of PROJECT LOOMY, providing low-precision numerical optimization, dataset text preprocessing and data processing utilities, and an evaluation suite developed by Silver Moon (@cloverxion) and Lumina Moon (@cloverx-id) contributors.

Built with a modular Rust and Python architecture under PROJECT LOOMY, luminamoon provides components for CPU/GPU parameter updates, text dataset preprocessing, and numerical verification:

LSO-Eval v1.1.0 Evaluation & Verification Notebook: Open in Google Colab — Empirical Stage II–IV benchmarks executed on LuminaV v1.5.3 under native torch.bfloat16 (centered_defect=False).

Projects & Components

Project / Component Path Status Description Documentation
LuminaV Optimizer optimizer/ Active (Spec v1.6.0) LuminaV v1.6.0 optimizer implementation with CPU SIMD (AVX-512/AVX2), Vulkan UMA, Triton AOT/JIT kernels, Triangle Family (buffer="T1", "T2"), optional contiguous bucket fusion, and Triton JIT pre-warmup. Optimizer Guide
LuminaMoon Data Engine data/ Active (v1.0.9) Modular text preprocessing library: AVX-512/AVX2/NEON SIMD text normalization, dual-layout StringTensor (CPU StringView & GPU contiguous buffers), Shannon entropy filtering, MinHash/XXH3-64 deduplication, PII masking, and 128B Ping-Pong checkpoint recovery. Data Engine Guide
LSO-Eval Suite eval/ Active (v1.1.0) Algorithm-agnostic optimizer evaluation & verification suite (LSO-Eval v1.1.0): Z3 SMT solver checks, 2D trajectory benchmarks, numerical parity tests, and Two-Class 4B LLM & Vision evaluation. LSO-Eval Guide
Changelogs changelogs/ Active Version histories separating Software Package releases (v1.0.0–v1.0.9) and LuminaV Algorithm Specifications (v1.0.0–v1.6.0). Changelogs
Documentation docs/ Active Documentation portal, developer guides (docs/optimize/, docs/dataloader/), and dependency version audits. Docs Portal
Python Package python/luminamoon/ Active (v1.0.9) Unified free-threaded (abi3t) Python namespace exposing LuminaV Optimizer, DataStreamLoader, and PyTorch/NumPy/HuggingFace integration. API Reference

Quickstart

Installation

Install the compiled package wheel from PyPI:

pip install luminamoon

Or build from source using Maturin and the Rust 2024 toolchain:

git clone https://github.com/cloverx-id/luminamoon.git
cd luminamoon
pip install maturin
maturin develop --release

Basic Usage (LuminaV Optimizer) 

import numpy as np
from luminamoon import LuminaV

# Initialize parameters
weights = np.random.randn(64, 10).astype(np.float32)
biases = np.zeros(10, dtype=np.float32)

# Instantiate LuminaV optimizer (uses default hyperparameters)
optimizer = LuminaV([weights, biases], lr=8e-4, weight_decay=0.08, buffer=2)

# Execute optimization step with explicit gradients
grad_w = np.random.randn(64, 10).astype(np.float32)
grad_b = np.random.randn(10, dtype=np.float32)
optimizer.step(grads=[grad_w, grad_b])

For PyTorch deep learning workflows, see the PyTorch Integration Guide.

LuminaV Hyperparameter Quick Reference

All default values below are exposed in IDE autocompletion and LuminaV.__init__ docstrings:

Parameter Type Default Description
lr float 8e-4 (0.0008) Learning rate (η).
betas Tuple[float, float] (0.9, 0.999) Exponential moving average factors (β₁, β₂) for first moment and second moment / Triangle Defect.
eps float 1e-8 Numerical stability constant (ε) in the adaptive denominator.
weight_decay float 0.08 (8e-2) Decoupled weight decay coefficient.
tau float 0.8 Soft-projection temperature threshold (τ) for denominator damping.
lam float 0.3 Defect weight (λ) for the Triangle Defect in 1-Buffer TD-CLM (buffer=1) and Triangle Family (buffer="T1", "T2").
cautious bool True Enables sign-aligned cautious update masking and lagged EMA scaling.
cautious_clamp_min float 0.5 Lower bound in (0.0, 1.0] for cautious alignment scaling.
buffer int | str 2 State mode: 2 ("dual", "clm") for 2-Buffer CLM, 1 ("single", "td-clm") for 1-Buffer TD-CLM, "T2" for 2-Buffer Triangle Family (tanh-free), or "T1" for 1-Buffer Triangle Family (tanh-free).
stochastic_rounding bool True Applies stateless 32-bit MurmurHash3 stochastic rounding when writing to bfloat16 / float16 weights.
bound bool True Enables adaptive update bounding.
bound_type str | int "coordinate" Clipping mode: "coordinate" (element-wise clip) or "radial" (global norm ratio).
bound_ratio float 0.05 Maximum update magnitude relative to parameter scale.
master_weights str | bool "none" Mixed-precision master weight mode for bfloat16/float16: "none" (False), "semi", or "full" (True).
execution str | int "auto" Hardware backend: "auto", "triton", "vulkan", "simd", or "scalar".
fused_single_pass bool True Fuses parameter update and scalar reduction statistics into a single kernel pass.
fuse_tensors bool False Coalesces parameters into contiguous 1D memory buffers to reduce kernel launch latency on fragmented architectures (e.g. Vision models / ViT). Not recommended for large LLMs or distributed FSDP training.
bucket_cap_mb float 500.0 Maximum memory capacity per contiguous bucket chunk in megabytes when fuse_tensors=True. Partitions large models into bounded chunks. Emits a UserWarning if explicitly specified when fuse_tensors=False.
prewarm_jit bool True Pre-compiles Triton GPU kernels on 64-element dummy buffers (lr=0.0) during initialization to avoid first-step JIT compilation delay.
ema_decay str | float "auto" EMA decay for lagged cautious/radial scalars ("auto" resolves to 0.80 for buffer=2/"T2" and 0.95 for buffer=1/"T1").
fsdp_sync bool False Synchronizes global reduction scratch buffers across torch.distributed FSDP shards.
seed int 1337 Deterministic PRNG seed for stateless stochastic rounding.
version Optional[str] "1.6.0" Selects the active LuminaV mathematical specification version.

Software vs. Algorithm Versioning (v1.0.9 / Spec v1.6.0)

luminamoon maintains two separate version tracks (documented in changelogs/):

All version resolution operates offline using local compatibility profiles:

import luminamoon
from luminamoon import LuminaV
import luminamoon.data as lmd

# 1. Inspect Software Package Version vs. Algorithm Specification Version
print(luminamoon.__version__)    # "1.0.9" (Software distribution)
print(lmd.get_version())         # "1.0.9" (Data Engine module)
print(LuminaV.ALGORITHM_SPEC)    # "1.6.0" (Mathematical specification)
print(LuminaV.getVersion())      # "1.6.0"

# 2. Select a specific algorithm version per-instance or globally
opt = LuminaV([weights, biases], lr=8e-4, version="1.5.5")
opt.setVersion("1.5.5")          # Instance-level version selection
LuminaV.setVersion("1.5.5")      # Class-level default

# 3. Default specification (Note: Specifications prior to v1.5.0 are deprecated)
opt_default = LuminaV([weights, biases], version="1.6.0")

Basic Usage (LuminaMoon Data Engine: Fused Text Preprocessing Pipeline) 

from luminamoon.data import FusedPipeline, StringTensor, create_resumable_collate, ResumablePipeline

# 1. Fused single-pass text preprocessing in Rust
pipeline = FusedPipeline(
    trim_whitespace=True,
    normalize_newlines=True,  # AVX-512 / AVX2 / NEON SIMD
    min_chars=10,
    min_entropy=2.0,
)
string_tensor = pipeline(["  sample 1\r\n", "sample 2 with proper prose text"])

# 2. Export to Apache Arrow, Polars, and PyTorch buffers
polars_series = string_tensor.to_polars()
arrow_array = string_tensor.to_arrow()
offsets_tensor, payload_tensor = string_tensor.to_torch_gpu(device="cuda")

# 3. Stream dataset with background prefetching & 128B Ping-Pong checkpointing
pipe = ResumablePipeline(
    source=raw_data,
    fused_pipeline=pipeline,
    batch_size=64,
    prefetch_batches=4,
    checkpoint_path="./cache/checkpoint.meta",
)
for batch in pipe:
    # Training step...
    pipe.commit()  # Saves Dual-Slot A/B checkpoint and deduplication WAL

For detailed architecture, Shannon entropy filtering, MinHash deduplication, and PyTorch collate_fn adapters, see the Data Engine Guide.

Verification & CI/CD

All pull requests and releases are tested across Linux, Windows, and macOS:

# Run static analysis across the Rust workspace
cargo clippy --workspace --all-targets -- -D warnings

# Run Rust workspace unit and parity tests (Optimizer + Data Engine)
cargo test --workspace

# Run Python static type check
npx pyright python/luminamoon

# Run standalone NumPy, PyTorch, and Data Engine test suites
python optimizer/tests/test_standalone_numpy.py
python optimizer/tests/test_end_to_end_training.py
python -B tests/test_data_modular.py
python -B tests/test_gpu_vs_cpu_strings.py

# Run LSO-Eval verification suite
python -m eval

Developer Resources

For contributors working on PROJECT LOOMY (luminamoon):

Authors & Maintenance

PROJECT LOOMY (luminamoon) is developed and maintained by:

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

LuminaV Optimizer v1.6.0 (luminamoon v1.0.9)

LuminaV is a numerical optimization library implementing the LuminaV algorithm for low-precision training (BF16/FP16) without requiring FP32 master weights. Using bitwise stochastic rounding, it supports hardware execution across CPU SIMD (AVX-512, AVX2), Vulkan compute shaders, and Triton GPU kernels, with buffer interoperability for NumPy and PyTorch.

Features

  • Contiguous Bucket Fusion & JIT Pre-Warmup: Optionally coalesces parameter groups into 1D contiguous allocations (fuse_tensors=True) partitioned into bounded chunks (bucket_cap_mb=500.0) to reduce kernel launch overhead on models with many tensors, and pre-warms Triton JIT kernels during initialization (prewarm_jit=True).
  • Dual-Buffer CLM, Single-Buffer TD-CLM & Triangle Family (T-Family, T1 / T2): Implements Convex Lookahead Momentum (CLM) (buffer=2), Triangle-Defect Convex Lookahead Momentum (TD-CLM) (buffer=1), and the tanh-free Triangle Family (buffer="T2" 2-Buffer Triangle Family Mode and buffer="T1" 1-Buffer Triangle Family Mode with Triangle Defect and rational update bounds). Single-buffer modes (1 and "T1") reduce optimizer state memory by 50% compared to two-buffer modes (2 and "T2").
  • Optional Framework Dependencies: The core package has dependencies = [] with a compiled binary under 1 MB. PyTorch, NumPy, and Triton are optional extras.
  • Hardware Acceleration:
    • x86_64 CPU: AVX-512 and AVX2 SIMD kernels parallelized across cores via Rayon.
    • Vulkan 1.3/1.4: Unified Memory Architecture (UMA) SPIR-V compute shaders for integrated and discrete GPUs.
    • Triton AOT / JIT: Compiled kernels for NVIDIA CUDA (SM75–SM120) and AMD ROCm (RDNA3/CDNA).
    • Scalar Core: Portable Rust fallback for other architectures (ARM/aarch64).
  • Framework Integration:
    • NumPy 2.x: Direct buffer access via the Python buffer protocol and __array_interface__.
    • PyTorch: Standard torch.optim.Optimizer interface with FSDP collective synchronization and optional FP16/BF16 master weights.
  • Runtime Compatibility: Built with CPython Stable ABI (abi3 and abi3t), supporting Python 3.10 through 3.15+ including free-threaded execution (PEP 703).

Architecture & Dispatch

The runtime dispatcher selects the execution backend based on tensor device, memory size, and available hardware:

Workspace Crate Structure 

optimizer/
├── Cargo.toml
├── src/
│   ├── lib.rs              # Rust crate entry point
│   ├── optimizer.rs        # Core PyO3 optimizer implementation
│   └── python_ffi/         # Python-Rust FFI bridge (mod.rs, config_parser.rs, extract.rs, worker.rs)
├── crates/
│   ├── luminav-core/       # Mathematical formulas, PRNG, and state structs
│   ├── luminav-simd/       # AVX-512 and AVX2 explicit SIMD intrinsics
│   ├── luminav-vulkan/     # Vulkan 1.3/1.4 UMA compute pipeline (Ash)
│   ├── luminav-triton-aot/ # Triton AOT runner for CUDA and ROCm HIP
│   └── luminav-dispatch/   # Hardware capability detection and routing
├── tests/                  # Standalone NumPy and PyTorch verification tests
└── tools/                  # AOT compilers for SPIR-V and Triton

Installation & Build

From Wheel (PyPI) 

pip install luminamoon

From Source (Maturin) 

git clone https://github.com/cloverx-id/luminamoon.git
cd luminamoon
pip install maturin
maturin develop --release

Optional Dependencies 

# PyTorch integration
pip install "luminamoon[torch]"

# Triton GPU acceleration (CUDA / ROCm)
pip install "luminamoon[gpu]"

# All optional dependencies
pip install "luminamoon[all]"

Quickstart

Standalone NumPy (No PyTorch Required) 

import numpy as np
from luminamoon import LuminaV

w = np.random.randn(64, 10).astype(np.float32)
b = np.zeros(10, dtype=np.float32)

optimizer = LuminaV(
    [w, b],
    lr=1e-2,
    betas=(0.9, 0.999),
    weight_decay=1e-2,
    buffer=2,  # 2 = Dual-Buffer CLM, 1 = Single-Buffer TD-CLM
)

grad_w = np.random.randn(64, 10).astype(np.float32)
grad_b = np.random.randn(10).astype(np.float32)

optimizer.step(grads=[grad_w, grad_b])

PyTorch Integration 

import torch
import torch.nn as nn
from luminamoon import LuminaV

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.Sequential(
    nn.Linear(128, 256),
    nn.ReLU(),
    nn.Linear(256, 10),
).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,
    betas=(0.9, 0.999),
    weight_decay=8e-2,
    buffer=2,
    bound=True,
    bound_type="coordinate",
    cautious=True,
    fuse_tensors=False,    # Coalesce parameter groups into contiguous buckets
    bucket_cap_mb=500.0,   # Maximum chunk allocation size in MB
    prewarm_jit=True,      # Pre-compile Triton JIT kernels at initialization
)

inputs = torch.randn(64, 128, device=device)
targets = torch.randint(0, 10, (64,), device=device)

optimizer.zero_grad()
loss = criterion(model(inputs), targets)
loss.backward()
optimizer.step()

Hardware Verification

To inspect detected hardware backends and active dispatch targets:

from luminamoon import LuminaV

print(LuminaV.detect_hardware())

Documentation & Library Audits

Running Tests 

# Lint workspace
cargo clippy --workspace --all-targets -- -D warnings

# Rust workspace unit and parity tests
cargo test --workspace

# Standalone NumPy test
python optimizer/tests/test_standalone_numpy.py

# PyTorch integration test
python optimizer/tests/test_end_to_end_training.py

References

  • Triangle Inequality Defect & Lookahead: L. Maligranda, "Some remarks on the triangle inequality for norms," Banach J. Math. Anal., vol. 2, no. 2, pp. 31–41, 2008; M. R. Zhang et al., "Lookahead optimizer: k steps forward, 1 step back," NeurIPS, 2019.
  • Belief Variance & Decoupled Weight Decay: J. Zhuang et al., "AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients," NeurIPS, 2020; S. Taniguchi et al., "ADOPT: Modified Adam can converge with any β₂ with the optimal rate," NeurIPS, 2024; I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," ICLR, 2019.
  • Cautious Alignment & Trust-Region Clipping: K. Liang et al., "Cautious optimizers: Improving training with one line of code," ICLR, 2026; A. Brock et al., "High-performance large-scale image recognition without normalization," ICML, 2021.
  • Padé Rational Approximants & Stochastic Rounding: A. Molina et al., "Padé activation units: End-to-end learning of flexible activation functions in deep networks," ICLR, 2020; S. Gupta et al., "Deep learning with limited numerical precision," ICML, 2015; P. Zamirai et al., "Revisiting BFloat16 training," arXiv:2010.06192, 2021.

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

LSO-Eval v1.1.0: Formal Verification & 4B LLM Qualification (Qwen3.5-4B-Base)

LuminaV (v1.5.3) is formally verified and empirically qualified under the LSO-Eval (v1.1.0) standard across Stage I (12/12 Z3 SMT solver UNSAT safety proofs & 9/9 SymPy/TOST precision checks), Stage II (B1–B5 non-convex surfaces, λ* = 0.3008 equilibrium, and 1-Pass EMA parity), Stage III (MProxyLLM F₁ ∧ F₂ ∧ F₃ screening), and Stage IV (Qwen/Qwen3.5-4B-Base 100% from-scratch pretraining in native torch.bfloat16 with cryptographic XXH3-64 zero-leakage validation).

Empirical Version Attribution (LuminaV v1.5.3): The Stage II–IV empirical qualification tables below were evaluated on LuminaV v1.5.3 under native torch.bfloat16 (dtype_mode=1, centered_defect=False). In the v1.5.3 evaluation data loader script, parallel dataset records were sampled by filtering uniqueness based on joint (source, domain) pairs rather than allocating unique records evenly per domain up to each domain's total target allocation; because all candidate and baseline optimizers shared this identical deterministic stream schedule, relative cross-optimizer comparisons in the v1.5.3 tables remain strictly controlled.

Stage IV-B: 2048 Context Full Schedule (20,000 Steps, Grad Accum 1x, 450,560 Held-Out Tokens, SHA256=108e517e1c181cb6)

LSO-Eval Stage IV-B Qwen3.5-4B Qualification (ctx=2048)
Optimizer Configuration Class Final EMA Val PPL ΔPPL VRAMpeak VRAMopt Max Rt,l Step / Opt Time LSO-Eval Status
LuminaV_Buffer2_Dual (8e-4) Class I (Dual) 4.5549 89.14 -43.43 43.89 GB 15.86 GB 0.0189 297.1 / 48.31 ms PASSED (Class I)
LuminaV_Buffer1_TDCLM_3e4 (3e-4) Class II (Single) 4.9173 165.13 +32.56 35.97 GB 7.94 GB (-50.0%) 0.0087 283.5 / 36.99 ms PASSED (Class II)
AdamW (Native BF16, 3e-4) Baseline 4.7621 132.57 +0.00 43.89 GB 15.86 GB 0.0037 288.0 / 41.48 ms PASSED
AdamW_FP32_Master (3e-4) Master Ref 4.8116 124.43 -8.14 80.11 GB 52.08 GB 0.0045 370.1 / 123.33 ms PASSED

Stage IV-A: 512 Context Qualification (4,800 Steps, Grad Accum 8x, 450,560 Held-Out Tokens, SHA256=4d278cb87e4abbe1)

LSO-Eval Stage IV-A Qwen3.5-4B Qualification (ctx=512)
Optimizer Configuration Class Final EMA Val PPL ΔPPL VRAMpeak VRAMopt Max Rt,l Step / Opt Time LSO-Eval Status
LuminaV_Buffer2_Dual (8e-4) Class I (Dual) 4.7621 141.96 -39.21 36.75 GB 15.88 GB 0.0047 841.6 / 48.37 ms PASSED (Class I)
LuminaV_Buffer1_TDCLM_3e4 (3e-4) Class II (Single) 5.2569 195.58 +14.41 28.81 GB 7.94 GB (-50.0%) 0.0120 829.4 / 36.77 ms PASSED (Class II)
LuminaV_Buffer1_TDCLM (8e-4) Class II (Single) 5.2953 221.07 +39.90 28.81 GB 7.94 GB (-50.0%) 0.0191 830.7 / 36.46 ms PASSED (Class II)
AdamW (Native BF16, 3e-4) Baseline 5.4497 181.17 +0.00 36.75 GB 15.88 GB 0.0034 835.4 / 41.58 ms PASSED
AdamW_FP32_Master (3e-4) Master Ref 5.0234 153.78 -27.39 75.36 GB 54.49 GB 0.0040 919.5 / 123.35 ms PASSED

VRAM Decomposition: VRAM_peak = VRAM_base + VRAM_opt, where the fixed non-optimizer baseline (VRAM_base = W_BF16 + G_BF16 + A_ctx) is 20.87 GB at ctx=512 and 28.03 GB at ctx=2048.

Citation 

@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion) and Lumina Moon Contributors}},
  organization = {Lumina Moon (cloverx-id)},
  title        = {{LuminaMoon & LuminaV Optimizer: Native Rust SIMD, Vulkan, and Triton Deep Learning Suite}},
  year         = {2026},
  publisher    = {GitHub / Hugging Face / PyPI},
  version      = {1.0.9},
  url          = {https://github.com/cloverx-id/luminamoon}
}
Downloads last month
499
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for cloverx-id/LuminaV-Optimizer