mlc-ai/tokenizers-cpp

The tokenizers-cpp project provides a cross-platform C++ tokenizer binding library that wraps HuggingFace tokenizers and sentencepiece, offering a unified interface for language model applications. It's designed for developers building native applications on platforms like iOS, Android, Windows, Linux, and web browsers who need efficient, dependency-light tokenizer deployment. The library is particularly useful for LLM chat applications and other NLP tasks requiring tokenization across multiple platforms.