OpenAI Tokenizer
Updated Free mock REST & WebSocket APIs
Type or paste text to see exactly how OpenAI models split it into tokens. Each token is highlighted, and you can switch to the numeric token IDs the model actually receives — with the same o200k_base and cl100k_base encodings OpenAI uses.
🔒 Tokenized in your browser with tiktoken's official BPE ranks — nothing is sent to OpenAI.
How to use the OpenAI tokenizer
- Paste your text. Tokens are color-coded as you type; hover one to see its ID.
- Choose o200k_base for GPT-4o, GPT-4.1, o-series, GPT-5.x and GPT-6, or cl100k_base for GPT-4 and GPT-3.5 Turbo.
- Tick “Show token IDs” to see the integers, and copy them as a JSON array for debugging logit_bias or caching.
Which encoding does each model use?
| Encoding | Vocabulary | Models |
|---|---|---|
o200k_base | ≈200,000 tokens | GPT-4o, GPT-4o mini, GPT-4.1, o1, o3, o4-mini, GPT-5 family, gpt-oss |
cl100k_base | ≈100,000 tokens | GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, text-embedding-3-small/large, text-embedding-ada-002 |
p50k_base / r50k_base | ≈50,000 tokens | Legacy GPT-3 (davinci, codex) |
OpenAI hasn't published a new encoding for GPT-5.x and GPT-6; tiktoken maps the GPT-5 family to o200k_base, so it's the right choice for current models. The usage field in each API response is always authoritative.
Things the visualizer makes obvious
- Leading spaces belong to the next word.
" hello"and"hello"are different tokens — that's why the dot (·) appears at the start of most pieces. - Numbers split in groups of up to three digits, so long IDs and prices cost more than they look.
- Emoji and many CJK characters take several tokens, because the tokenizer works on UTF-8 bytes. 👋🏽 is multiple tokens.
- o200k is cheaper for non-English text — its larger vocabulary covers more languages in fewer tokens than cl100k.
Count tokens in code with tiktoken
# Python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
len(enc.encode("hello world")) # 2 → [24912, 2375]
// JavaScript (js-tiktoken)
import { getEncoding } from "js-tiktoken";
getEncoding("o200k_base").encode("hello world").length; // 2Related tools
Frequently asked questions
Is my text sent to OpenAI, Anthropic or anyone else?
No. Everything runs in your browser: the tokenizer files are downloaded once and your text never leaves your device. Nothing is logged or stored.
Is this the same tokenizer OpenAI uses?
Yes. It runs js-tiktoken with OpenAI's published BPE rank files, so token IDs match tiktoken in Python and the token counts the API bills for plain text.
Why does my API usage show more tokens than this?
Chat requests add framing tokens around each message (about 4 per message plus 3 for the reply), and tool definitions, images, files and reasoning tokens are counted on top of your raw text.
What tokenizer does GPT-5 or GPT-6 use?
OpenAI's tiktoken library maps the GPT-5 family to o200k_base and no newer encoding has been published, so o200k_base is the best available count for GPT-5.x and GPT-6 text.
Your business here? Sponsor this tool from $16/week. See placements