OpenNMT/Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support

语言：C++ · GitHub 星标：343 · Fork：83

话题：bpe · cpp · icu · machine-translation · natural-language-processing · python · sentencepiece · tokenization

在 GitHub 上查看原仓库