ferrumox/fox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

语言：Rust · GitHub 星标：190 · Fork：28

话题：cuda · gguf · inference-engine · llama-cpp · llm · local-llm · mlx · ollama

在 GitHub 上查看原仓库