Token cost and why it is getting the heat
Money Management
Your CFO/leadership just sent a one-liner: “Our token bill is growing faster than our MAUs.” If that stings, you’re not alone. As AI moves from pilots to production, token costs become the line item everyone starts watching. The good news: you don’t need the highest-performing model for every request. The better news: smart routing can cut costs dramatically without hurting outcomes.
The core idea
Not all tasks need frontier intelligence. Many are routine, short-context, or formulaic.
Token cost scales with model choice, prompt length, and response length. Control those, and you control spend.
Route by use case and confidence, not by hype.
A practical routing playbook
Map tasks by difficulty and risk
Low complexity, low risk: FAQs, simple transformations, keyword extraction, templates.
Medium complexity or domain: summarization with structure, basic QA over known docs, entity extraction.
High complexity or risk: legal/medical reasoning, multi-step planning, code generation, open-ended analysis.
2. Define quality bars and failure modes
What does “good” look like? Precision/recall, factuality, tone, latency.
What happens if the model is wrong? Harmless annoyance or regulatory incident?
3. Pick a default small model, add controlled escalation
Start with a lighter/cheaper model for most requests.
Escalate to a stronger model only when:
Input is ambiguous or long.
Confidence is low (self-rating or rule-based).
The task is on a high-risk list.
Cap escalations per session or per user to protect budgets.
4. Use a policy you can test
Build a labeled evaluation set per use case.
Shadow-route a slice of traffic to compare small vs. large model outputs offline.
Tune thresholds until you hit target quality at acceptable cost.
5. Monitor unit economics
Track cost per successful outcome, not just tokens.
Watch escalation rate, average tokens per call, and cache hit rate.
Set alerts for runaway prompts and outlier sessions.
An example routing map
FAQ and template replies: small model + semantic cache; no escalation.
Structured summarization of known docs: medium model with strict instructions; escalate only if source count > N or confidence < threshold.
RAG question answering: small/medium model + retrieval; escalate when retrieval yields weak matches.
PII redaction and policy filters: specialized or fine-tuned smaller model; never escalate to a general model if data sensitivity is high.
Contract or compliance analysis: start medium, escalate to top-tier for edge clauses or conflicting signals.
Code generation or complex reasoning: go straight to high-performing models; consider tool use and tests to bound retries.
Cost levers beyond routing
Shorten prompts and responses:
Move policy and style into a compact system prompt.
Replace long context with retrieval snippets; deduplicate overlaps.
Ask for structured JSON to reduce wordy prose.
Cache aggressively:
Exact-match cache for prompts.
Semantic cache for near-duplicates.
Precompute common summaries and embeddings.
Fine-tune or distill:
A small fine-tuned model often outperforms a general mid-tier model on your niche tasks at a fraction of the cost.
Constrain work with tools:
Call databases, search, or calculators instead of “thinking” in tokens.
Stream and early-stop once a confidence or completeness signal is met.
Govern generation:
Set max tokens, conservative temperatures, and retry limits per use case.
Strip boilerplate from every turn.
What “good” looks like in production
60–90% of traffic handled by small/medium models with high cache hit rates.
Automatic, explainable escalation for the remainder.
Quality measured with a living eval set; thresholds adjusted monthly.
Clear budgets and guardrails by product surface and user tier.
Bottom line: Organizations rarely need frontier models for everything. They need the right model for the right job, with a routing layer that’s measurable, reversible, and cost-aware. Treat token spend like any other cloud resource: tier it, cache it, cap it, and continuously optimize it.