How LLMs Actually Work

The story starts in May with a question most people treat as a joke. How many R's in "strawberry"? Three. S-T-R-A-W-B-E-R-R-Y. A child with a pencil gets it immediately. A system capable of drafting legal briefs or debugging production code has, historically, gotten it wrong — and the May 17th Dev.to piece (score 6.8, quiet traction) asked the uncomfortable follow-up: does anyone in the industry actually want to explain why? The answer it landed on was tokenization. Not as metaphor. As mechanism. The word "strawberry" doesn't arrive in an LLM as a word. It arrives as fragments — tokens, determined by an algorithm called Byte Pair Encoding — and those fragments don't map cleanly onto letters. The model has no obvious place to hang the concept of "count the R's" because it was never seeing R's in the first place.

That observation, published in May, appears to have lodged in the developer community like a splinter. By June 13th, another Dev.to post went further: "Your LLM can't read. Here's the weird trick it uses instead." The framing had sharpened from parlour trick to structural critique. Then in July, the economics entered the conversation — a piece on tokenization and AI costs, pointing out that the same prompt costs different amounts across different models because their tokenizers carve language differently. The same sentence, submitted to GPT-4 and Claude, might produce meaningfully different token counts, different costs, different failure modes. By August 18th, a piece titled "Inside the Tokenizer" was making the cost asymmetry concrete: not a rounding error, not negligible, but a systematic difference with budget implications at scale. September brought two pieces on the same day — one explaining what ChatGPT is actually doing when it answers you (probability distributions over tokens, not comprehension in any human sense), and one going deep on BPE-style tokenizers as the small algorithm that decides what an LLM can even perceive. Six signals across four months. Not a viral story. A slow, accumulating discomfort.

If confirmed — and here the word "confirmed" means if this explanatory framework holds under scrutiny — the implications cut in several directions at once. For developers building on top of LLMs, tokenization isn't a background detail. It is the interface. The way you phrase a prompt, the punctuation you use, whether you spell out numbers or use digits: these affect token boundaries, which affect model performance and cost in ways that are non-obvious and not documented in most tutorials. Teams optimising for cost at scale need a tokenization strategy, not just a prompt strategy. For product managers and executives who bought into the idea that LLMs "understand" language, this body of work is a gentle but firm correction — the model is doing something extraordinary, but it is not reading. It is pattern-matching over statistical distributions of token sequences, and that distinction matters enormously when you are designing systems that need to be reliable. The strawberry problem isn't a quirk to be patched. It is a window into a genuinely different cognitive architecture, one that happens to be useful for many things and structurally unsuited to others, and conflating the two has real costs.

Watch for model providers publishing more granular documentation on tokenizer behaviour and its relationship to benchmark performance — if that starts appearing, the developer community's informal reverse-engineering will have forced the conversation upward.

NewsHive monitors these sources continuously. All signal titles above link to the original reporting.

Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →