True2456/Gemma-4-12B-Agentic-LoRA

Gemma 4 12B — Agentic Specialist LoRA (Expert 1)

Specialist LoRA adapter fine-tuned on top of mlx-community/gemma-4-12b-it-bf16 for multi-step autonomous tool calling, structured JSON execution, and multi-turn software engineering workflows.

Designed as Expert 1 within multi-specialist MoE fusion architectures or for standalone high-precision agent execution on Apple Silicon via Apple MLX (mlx_lm).

---

Model Specifications

Parameter	Specification
Base Model	mlx-community/gemma-4-12b-it-bf16
Adapter Architecture	LoRA (Low-Rank Adaptation)
Target Layers	48 Transformer Layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
LoRA Rank (r)	16
LoRA Alpha (α)	32
LoRA Scale	10.0
Dropout	0.05
Max Sequence Length	8192 tokens
Training Framework	mlx-lm on Apple Silicon Metal

---

Training Dataset Composition

Trained on the curated gemma12b_agentic_specialist pack containing 107,761 training steps organized into three quality tiers:

1. Tier S (Personal Gold Trajectories — 40,080 steps):
   • Curated multi-turn reasoning and tool-orchestration trajectories.
   • Oversampled 40x (2,240 base samples / 56 long-context trajectories) to enforce precise tool call formatting and execution discipline.
2. Tier A (Flawless Native Pack — 7,782 steps):
   • Verified native tool calling traces with strict JSON schema adherence and error recovery patterns.
3. Tier B (Open SWE Trajectories — 59,899 steps):
   • Real-world software engineering execution traces covering file inspection, patch generation, test execution, and debugging loops.

---

Usage with Apple MLX (mlx-lm)

Installation

pip install mlx mlx-lm

Python Inference

from mlx_lm import load, generate

model_path = "mlx-community/gemma-4-12b-it-bf16"
adapter_path = "True2456/Gemma-4-12B-Agentic-LoRA"

model, tokenizer = load(
    model_path,
    adapter_path=adapter_path
)

prompt = tokenizer.apply_chat_template([
    {"role": "system", "content": "You are an autonomous engineering agent with tool access."},
    {"role": "user", "content": "Inspect the repository structure and list files in src/."}
], tokenize=False, add_generation_prompt=True)

output = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=512,
    verbose=True
)
print(output)

---

Training Configuration & Hardware Notes

• Hardware: Apple Silicon M5 Max / M4 Max (128 GB Unified Memory).
• Optimization: Trained with MLX gradient checkpointing enabled (grad_checkpoint: true) at --max-seq-length 8192 to prevent activation spilling and memory swap.
• Intended Use: Autonomous coding workflows, structured API tool invocation, and multi-turn agent execution loops.