📐 Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
TypeScript
137
6
MIT License
Tokenx is a lightweight JavaScript library that provides fast token count estimation for large language models, achieving 95-98% accuracy while maintaining a minimal 2kB bundle size. It's designed for developers who need quick token approximations for text processing, chunking, or validation without the overhead of full tokenizers, supporting multiple languages including CJK and accented characters.
Total donated
Undistributed
Share with your subscribers:
Recipients
How the donated funds are distributed
Support the dependencies
Support the repos that depend on this repository
Top contributors
Recent events
Kivach works on the Obyte network, and therefore you can track all donations.
No events yet
