intentee/paddler

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

语言：Rust · GitHub 星标：1,685 · Fork：97

话题：ai · llamacpp · llm · llmops · load-balancer

在 GitHub 上查看原仓库