VocabX: How a Custom SLM Vocabulary Cut Inference Cost 77%

Dharma-AI cut the measured inference cost of a production small language model by 77.0% without changing the base model, the data or the training recipe. The only thing it changed was the tokenizer. For teams fine-tuning open models like Qwen or Gemma for one narrow task, the result says the vocabulary inherited from a general-purpose model may be one of the biggest cost levers they never touch.

What Happened

In a Hugging Face post previewing its VocabX paper, Dharma-AI describes a Portuguese essay grader built on Qwen3-4B. It returns a JSON assessment with a grade and comment per competency, but still used Qwen3-4B’s original tokenizer, with 151,669 entries built for many languages. Two kinds of waste followed. The JSON envelope and competency names the grader writes every time were split into many tokens. And at every generated token the model scored the full vocabulary, including entries the task almost never uses.

Dharma-AI trained seven versions of the grader that differed only in tokenizer, and ran the same 500 requests on an RTX 5090 with vLLM, costing each from measured runtime.

The numbers come from a paper still under review and may change before publication.

Why It Matters

The key idea is that token count isn’t the bill. Decode generates one token at a time and is limited by memory bandwidth, so in the grader a generated token cost 7.4 times a token read, a ratio Dharma-AI says varies with hardware, batch size and context length. VocabX therefore weights each saved token by where it appears: system prompt, user message or response. Existing methods it compared against, including fast vocabulary transfer, multi-word tokenization and Liquid AI’s in-place tokenizer expansion, choose tokens by frequency.

VocabX works on two layers at once. It rebuilds how domain words split into sub-words, and turns whole repeated spans, up to full JSON envelopes, into single “macro tokens.” Five automated miners hunt for repetition. Without all five, Dharma-AI says, token reduction on the grader would drop by 55.6 points. A smaller vocabulary also shrinks the output layer, which is almost 10% of a Qwen3-4B-sized model’s parameters and is read in full for every generated token.

What This Means for Teams Fine-Tuning Small Models

The most useful lesson is a failure. The first VocabX vocabulary fused each grade into the surrounding JSON macro. It was the most compact option, but quality fell to 0.689, below production. Giving the grade its own token cost two extra tokens per response, raised the score to 0.810, and was even cheaper in measurement (77.0% versus 73.3%). Compression hurts when the compressed span carries the decision the model exists to make.

Across five public domains and three model families (Qwen3, Llama-3.1 and Gemma-3), VocabX beat the strongest reference method on compression in all fifteen combinations, by 1.2 to 9.3 points. It’s a different route to cheaper inference from the ones in Stack Archive’s look at Port.io’s study on cutting AI costs by 80%.

Source: Hugging Face Blog

Content created with AI assistance and reviewed for accuracy.