schoolhouse-world/gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (o1, o3, o4, gpt-4o, gpt-4, etc.). Port of OpenAI's tiktoken with additional features.

MIT License

gpt-tokenizer is a high-performance JavaScript library for tokenizing text using OpenAI's BPE (Byte Pair Encoding) algorithm. It provides fast encoding/decoding for GPT models like GPT-4o, GPT-3.5, and GPT-4, with features including cost estimation, token limit checking, and support for chat messages. Designed for developers building applications that interact with OpenAI's APIs, it offers the fastest and smallest footprint tokenizer available for JavaScript environments.

Total donated

Undistributed

Share with your subscribers: