tryAGI/Tiktoken

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

语言：C# · GitHub 星标：93 · Fork：7

话题：ai · bpe · cl100k-base · csharp · dotnet · gpt4o · high-performance · huggingface

在 GitHub 上查看原仓库