Counting tokens without a tokenizer

I wanted every page to show what it costs a model to read, and the wiki as a whole to show its total. No tokenizer dependency: this project keeps three runtime dependencies and a vocabulary file is many megabytes.

So, an estimate. I benchmarked three candidates against hand-counted samples spanning English prose, heavy markdown with tables and links, and source code.

The simplest one won, and not narrowly. That result is stable enough to just take: characters ÷ 4, rounded up.

Why the cleverer ones lost

The word-based estimator fails on exactly the content a technical wiki is full of. Punctuation, code, URLs, indentation and table pipes are all tokens and none of them are words, so it undercounts markdown badly while looking fine on prose. Its error is not just larger, it is biased — and a biased estimator on a budget is worse than a noisy one, because you can carry a noisy estimate with a margin and you cannot carry a systematically low one at all.

The chunk heuristic lost for a subtler reason: it was tuned on my intuitions about how words split, and those intuitions are wrong in a specific direction. Common words are single tokens no matter how long, and short unusual strings fragment. Any rule I invent by introspection encodes my sense of what looks complicated, which is uncorrelated with what a learned vocabulary found frequent.

Character count sidesteps all of it. It has no theory, so it has no wrong theory.

More precision would have been fake

The obvious objection is: ship a real tokenizer and be exact. But exact against what? Every model family tokenises differently, and the same text differs across them by well over the error of the estimate. A precise number would be precise about one model and silently wrong about the rest, while looking far more authoritative than the honest approximation.

So the display says ~816 tokens, the tilde is not decoration, and the page explaining the statistic says which rule produced it. An estimate labelled as an estimate is more useful than a false exact, because it tells the reader how much to lean on it.

The general rule: precision that exceeds the accuracy of your model is a lie with error bars removed. Round to the significant figures you can defend.

The part that actually mattered: what to measure

This is where I nearly shipped something misleading, and it had nothing to do with the estimator.

Pages are stored with frontmatter — title, tags, timestamps, provenance — that the reader never receives. Counting the stored file would have added a hundred tokens of bookkeeping to every page and inflated the wiki total by more than ten percent, and every number would have been defensible and useless, because nobody is ever billed for it.

So the count is taken on the body only, and there is a test asserting the whole-wiki total comes in under the naive estimate of the stored bytes, specifically to catch anyone later "fixing" it to measure the file.

The lesson generalises past tokens. When you add a measurement, the choice of estimator is usually the easy part and gets all the attention. The question that decides whether the number is any good is what the number is of — and the right answer is almost always the thing the consumer actually pays for, not the thing that is convenient to measure. Response bytes on the wire, not before compression. Wall-clock the user waits, not CPU in the handler. Body tokens, not file bytes.

Two implementation notes worth stealing

Store it, do not compute it. The count lives in the index as a column, so a whole-corpus total is one SUM rather than opening every file. Derived values that get aggregated should be computed once at write and stored, or the aggregate becomes the slowest thing in the system exactly when the corpus is worth aggregating.

Publish it before the fetch, not after. The count appears in listings and search results, not only on the page. A per-page number you can only obtain by fetching the page is useless for its main purpose, which is deciding whether to fetch the page. This is the single most useful property of the whole feature and it comes entirely from where the number is shown.