Haiderkhan64/H64LM

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

语言：Jupyter Notebook · GitHub 星标：19 · Fork：5

话题：deep-learning · from-scratch · gqa · language-model · llm · mixture-of-experts · moe · nlp

在 GitHub 上查看原仓库