Token Counter: Why JSON and Code Cost More

Short answer: a tokenizer keeps long pieces for text it met often in training and cuts everything else into short pieces. Common English words are the cheap case. Digits, symbols and non-English scripts break into more pieces, and you pay per piece.

A scope note first. The sources behind this page print token splits for words, arithmetic and Japanese. They print none for a JSON document or a source file. So for JSON and code this page offers a mechanism and a script, not a measured count.

Splits printed in the Cookbook

The OpenAI Cookbook page on tiktoken encodes tiktoken is great! into six tokens, ids 83, 8251, 2488, 382, 2212, 0, and prints the pieces: t, ikt, oken, is, great, !. It also counts the same string with o200k_base and gets 6. Two details. A leading space belongs to the token, so is and great are one piece each. And the invented word costs three pieces, t, ikt and oken, while the common word great costs one.

Arithmetic is closer to what JSON values look like. The same page splits 2 + 2 = 4, nine characters:

Encoding	Tokens	Pieces
p50k_base	5	2, +, 2, =, 4
cl100k_base	7	2, +, , 2, =, , 4
o200k_base	7	2, +, , 2, =, , 4

Source: OpenAI Cookbook, tiktoken page.

In the two newer encodings, the space before the second digit and before the last one is cut off as a piece of its own. Nine characters, seven tokens. A newer encoding is not automatically a cheaper one for every string.

The Japanese greeting お誕生日おめでとう, also nine characters, comes out at 14 tokens in p50k_base, 9 in cl100k_base and 8 in o200k_base, per the same page. The oldest of the three spent more than one token per character, and the newer ones cut that a lot. The encoding matters as much as the text.

What follows for JSON and code

Take the mechanism seriously, but call it a prediction. In {"id": 42} the brace, the quotes, the colon and the digits are all candidates for separate pieces, and indentation is whitespace, which is exactly what the arithmetic example lost tokens on. Plain prose has fewer of these boundaries per character. That is why a payload tends to cost more than a sentence of the same length, and why you should measure instead of trusting this paragraph:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
samples = {
    "sentence": "The order for customer 42 shipped on Monday.",
    "json": '{"order": {"customer": 42, "status": "shipped", "day": "Monday"}}',
}
for name, text in samples.items():
    ids = enc.encode(text)
    print(name, len(text), "chars", len(ids), "tokens")
    print([enc.decode_single_token_bytes(i) for i in ids])

The second print line shows the pieces, so you can see where the extra tokens come from. The package is the tiktoken project, MIT licensed.

From tokens to dollars

Assume, and this is an assumption, that your JSON payload runs 40% heavier in tokens than the same facts as a sentence. A pipeline sending 50 million sentence-sized tokens a month would send 70 million. At $2 per million input tokens, the GPT-6 Sol input rate on OpenAI's pricing page as of October 22, 2026, that is $100 against $140 a month.

Before you trust any count, find out which encoding your model uses: the Cookbook's model list stops at the GPT-4o family. A count from tiktoken also says nothing about Claude, which has its own tokenizer.

When the structure is yours to choose, send the fields the model needs rather than the whole record. Short keys are cheaper than long ones. The site's calculator and docs page work out a cost for a pasted prompt, which is enough for a before-and-after on your own text.