Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Paper • 2606.20945 • Published • 80
Abhay kumar
akanyaani
AI & ML interests
LLMs, GenAI, Transformers
Recent Activity
liked a model 5 days ago
FrontiersMind/Lumma-0.6B-Tool liked a model 18 days ago
FrontiersMind/Lumma-0.6B-Base updated a model 18 days ago
FrontiersMind/Lumma-0.6B-BaseOrganizations
authored a
paper 28 days ago
authored 4
papers over 1 year ago
ZClip: Adaptive Spike Mitigation for LLM Pre-Training
Paper • 2504.02507 • Published • 90
A Refined Analysis of Massive Activations in LLMs
Paper • 2503.22329 • Published • 14
Komodo: A Linguistic Expedition into Indonesia's Regional Languages
Paper • 2403.09362 • Published • 11
Variance Control via Weight Rescaling in LLM Pre-training
Paper • 2503.17500 • Published • 5