Dharma-AI says changing the tokenizer cut its essay grader's inference cost 77%
The VocabX preview targets repeated output and oversized vocabularies in specialized small models, reporting an RTX 5090 result ahead of the full paper.
Dharma-AI says it cut a specialized essay grader’s measured inference cost by 77% by rebuilding the vocabulary around what the model spends money writing. Its October 21 VocabX preview reports the result for a Portuguese-essay grading system based on Qwen3-4B, ahead of a full paper the team says is still under review.
The comparison retains the same base model, training data, recipe and seed while training versions with different tokenizers. It is not a claim that an unchanged deployed checkpoint becomes cheaper by swapping a tokenizer file. The authors tested seven grader versions on the same 500 requests using an RTX 5090 and vLLM, calculating cost from measured runtime. Their reported average response shrank from 350 tokens to 101, while a task-specific bench score rose from 0.703 to 0.810.
The target is expensive output, not just fewer tokens
A general tokenizer carries many tokens a narrow application rarely uses and can fragment repeated JSON structures or grading phrases into multiple pieces. VocabX combines domain-specific subwords with macro tokens representing repeated spans. It weights a token’s value by the field where it appears, rather than counting all occurrences equally.
In this grader’s measured setup, the authors say generating a token cost 7.4 times as much as reading one. That ratio depends on hardware, batching, context and workload. The method therefore emphasizes repeated output while reducing the vocabulary scored at each generation step, from roughly 152,000 entries to 65,000 in the grader.
The team also reports a useful failed approach: fusing the grade itself into surrounding JSON compressed more aggressively but lowered the bench score to 0.689. Giving the grade its own token added two tokens per response and improved the score. A vocabulary can compress away a decision the application actually needs to express.
A research preview with a specific measurement boundary
The 77% figure is the authors’ runtime-derived result for this grader, not a guaranteed reduction in a hosted API bill or every small-model deployment. Broader comparisons across five domains and three model families report token compression and estimated inference-cost advantages; those are distinct from the grader’s measured runtime result.
The English article says the full method, experiment tables and discarded interventions will appear in the forthcoming paper, and its current numbers may change. AITrending read the preview and inspected its results chart but did not reproduce the training or benchmark. The practical idea worth testing is whether a narrow workload’s repeated output and vocabulary overhead offer savings before moving to a different model.