Every millisecond matters. We’re open sourcing the tokenizer we built and deployed on production; that’s far efficient than huggingface and sentencepiece. https://x.com/perplexity_ai/status/2059664738087469511

関連プロジェクト

HuggingfaceThe Tokenizer