guillaume-be/rust-tokenizers

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

语言：Rust · GitHub 星标：345 · Fork：33

话题：deep-learning · rust-lang · tokenizer · transformer

在 GitHub 上查看原仓库