Llama-3.2-1B-4B-Quad-MoE

This is a Mixture of Experts (MoE) model based on unsloth/Llama-3.2-1B-Instruct. It merges a base model, an instruct model, a coding specialist, and a math specialist into a single sparse architecture.

Model Details

• Total Parameters: 3.65B
• Active Parameters per Token: 1.24B
• Base Model: unsloth/Llama-3.2-1B-Instruct
• Experts:
  1. unsloth/Llama-3.2-1B-Instruct (General Purpose)
  2. unsloth/Llama-3.2-1B (Creative/Prose)
  3. cutelemonlili/Llama3.2-1B-Instruct_Lean_Code (Coding Specialist)
  4. prithivMLmods/Llama-Express.1-Math (Logic/Mathematics)

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Fu01978/Llama-3.2-1B-4B-Quad-MoE")
tokenizer = AutoTokenizer.from_pretrained("Fu01978/Llama-3.2-1B-4B-Quad-MoE")