microsoft/tokenizer

This project provides TypeScript and .NET implementations of the Byte Pair Encoding (BPE) tokenizer used by OpenAI's large language models, enabling developers to tokenize prompts before sending them to LLMs. It's designed for developers working with OpenAI's APIs in Node.js or .NET environments who need to accurately calculate token usage or optimize prompt costs. The C# implementation has been superseded by `Microsoft.ML.Tokenizers`, which offers improved performance and is the recommended library for .NET developers.