ongunm/llama-moe-cache

Expert cache + predictive prefetch for MoE inference in llama.cpp. A 12GB GPU can run a 120GB model at native speed. At 3% sparsity, a single workstation could theoretically run a 1T parameter model. 500 LoC, zero training, 99.5% cache hit rate.

语言：C++ · GitHub 星标：9 · Fork：5

在 GitHub 上查看原仓库