sctg-development/sentencepiece-js

SentencePiece-js is a JavaScript library that ports the SentencePiece tokenization library to WebAssembly, enabling text tokenization directly in the browser or Node.js without server dependencies. It's designed for developers building applications like AI chatbots, language models, or text processors who need efficient, client-side tokenization, such as counting tokens for models like Meta Llama 3.1. The library supports loading pre-trained models and provides functions for encoding and decoding text tokens.