kekzl/imp

Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.

语言：C++ · GitHub 星标：124 · Fork：5

话题：ai-agents · blackwell · cuda · inference-engine · llm-inference · local-llm · mixture-of-experts · nvfp4

在 GitHub 上查看原仓库