docs/en/01_getting_started.md

Getting Started

Setup guide, system requirements, source compilation, hardware verification, and quickstart examples for the LuminaV Optimizer v1.6.0 and LuminaMoon Data Engine v1.1.0.

System Requirements

• Python: Version 3.10 or later (compatible with Python 3.10 through 3.15+, including free-threaded runtime abi3t / PEP 703).
• Operating Systems: Linux (x86_64, aarch64), Windows (x86_64), or macOS (Apple Silicon & Intel).
• Source Compilation (Optional):
  • Rust Toolchain: rustc and cargo version 1.85+ (Edition 2024).
  • Python Build Tool: maturin >= 1.15, < 2.0.
• Hardware Accelerators (Optional):
  • CPU SIMD: x86_64 processor supporting AVX2 or AVX-512 (F, DQ, BW, VL) for both Optimizer and Data Engine, or ARM64 NEON for the Data Engine byte-scanning pipeline (with scalar fallback for the Optimizer on ARM64).
  • Vulkan Compute: Vulkan 1.3 / 1.4 capable driver with compute shader support (SPIR-V).
  • Triton JIT: NVIDIA CUDA (SM75–SM120) or AMD ROCm (RDNA3/CDNA) with triton >= 3.0.0 (or triton-windows).

Installation

1. Standard Installation

The core luminamoon package has no mandatory Python dependencies (dependencies = []), providing both LuminaV and the luminamoon.data processing pipeline:

pip install luminamoon

2. Installation with Optional Extras

If you plan to use LuminaMoon with specific ML frameworks or GPU backends, install with extras:

# PyTorch integration
pip install "luminamoon[torch]"

# Triton JIT GPU acceleration (CUDA / ROCm)
pip install "luminamoon[gpu]"

# NumPy 2.x standard
pip install "luminamoon[numpy]"

# Install all optional dependencies
pip install "luminamoon[all]"

3. Compiling from Source

To compile the native Rust workspace (optimizer + data) locally:

git clone https://github.com/cloverx-id/luminamoon.git
cd luminamoon

# Create and activate virtual environment (optional)
python -m venv venv
# Windows:
.\venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate

# Install maturin build tool
pip install "maturin>=1.15,<2.0"

# Compile native extension in release mode
maturin develop --release

The compiled unified native library _luminamoon_optimizer_rs (exposing both Optimizer and Data Engine symbols) is placed into the luminamoon/optimizer/ subpackage.

Hardware Capability Detection

LuminaMoon provides static runtime probes for both the Optimizer and Data Engine subsystems:

import luminamoon as lm
from luminamoon import LuminaV

# 1. Inspect Optimizer SIMD, Vulkan UMA, and Triton AOT capabilities
hw = LuminaV.detect_hardware()
print("Optimizer Hardware:", hw)

# 2. Inspect Data Engine CPU SIMD byte-scanning instruction set ('AVX-512', 'AVX2', 'NEON', or 'SCALAR')
print("Data Engine CPU ISA:", lm.data_cpu_isa())

Quickstart: Standalone NumPy (Without PyTorch)

LuminaV can be used directly for numerical computing or custom neural networks using NumPy without requiring PyTorch:

import numpy as np
from luminamoon import LuminaV

# 1. Initialize parameters as contiguous FP32 NumPy arrays
rng = np.random.default_rng(42)
weights = rng.standard_normal((64, 10), dtype=np.float32)
bias = np.zeros((10,), dtype=np.float32)

# 2. Instantiate LuminaV optimizer
optimizer = LuminaV(
    [weights, bias],
    lr=0.01,
    betas=(0.9, 0.999),
    weight_decay=0.01,
    buffer=2,  # 2: Dual-Buffer CLM, 1: Single-Buffer TD-CLM
    bound=True,
    cautious=True,
)

# 3. Training loop with explicit gradients
for step in range(1, 11):
    grad_w = rng.standard_normal((64, 10), dtype=np.float32) * 0.1
    grad_b = rng.standard_normal((10,), dtype=np.float32) * 0.1

    optimizer.step(grads=[grad_w, grad_b])
    print(f"Step {step:02d} executed via Rust SIMD / Vulkan UMA.")

Quickstart: PyTorch Integration

Works directly with PyTorch nn.Module using the standard torch.optim.Optimizer interface:

import torch
import torch.nn as nn
from luminamoon import LuminaV

class Regressor(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.net(x)

model = Regressor()

optimizer = LuminaV(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    weight_decay=1e-2,
    buffer=2,             # 2 = Dual-Buffer CLM
    master_weights="full", # Maintain FP32 master weights and states (alias: "fp32", or "semi" for 16-bit states)
    stochastic_rounding=True,
    cautious=True,
)

criterion = nn.MSELoss()
x_batch = torch.randn(32, 64)
y_target = torch.randn(32, 10)

for step in range(1, 11):
    optimizer.zero_grad(set_to_none=True)
    out = model(x_batch)
    loss = criterion(out, y_target)
    loss.backward()
    optimizer.step()

    print(f"Step {step:02d} | Loss: {loss.item():.6f}")

Quickstart: LuminaMoon Data Engine (Fused SIMD & Zero-Copy Pipeline)

Preprocess and stream datasets with FusedPipeline, dual-layout StringTensor buffers, 128-byte Ping-Pong checkpoint recovery (StateTracker), and two-stage xxh3_64 deduplication:

from luminamoon.data import FusedPipeline, StringTensor, create_resumable_collate, ResumablePipeline

# 1. Fused single-pass preprocessing in Rust SIMD
pipeline = FusedPipeline(
    trim_whitespace=True,     # AVX-512 / AVX2 / NEON SIMD whitespace trimming
    normalize_newlines=True,  # CRLF to LF newline normalization
    min_chars=10,
    min_entropy=2.0,          # Shannon entropy quality filter
)

# 2. Process batch into dual-layout StringTensor (CPU StringView & GPU Contiguous Buffers)
raw_texts = ["  Sample training text\r\n", "Another valid high-entropy training document line."]
tensor = pipeline(raw_texts)

# 3. Zero-copy export to Apache Arrow IPC, Polars Series, or PyTorch GPU tensors
polars_series = tensor.to_polars()
arrow_array = tensor.to_arrow()
# offsets_tensor, payload_tensor = tensor.to_torch_gpu(device="cuda")

# 4. Resumable batch processing with 128B Ping-Pong (Slot A/B) checkpoint recovery
# pipe = ResumablePipeline(source=raw_texts, fused_pipeline=pipeline, batch_size=32, checkpoint_path="./cache/train.meta")
# for batch in pipe:
#     # Training step...
#     pipe.commit()  # Atomically persists Ping-Pong Slot A/B and .dedup WAL