How we reduced AI infrastructure costs by 60%
Calculating cost per successful task (not just token cost) and practical caching and routing strategies.
Cost Per Successful Task vs Cost Per Million Tokens
Most engineering teams obsess over vendor token price charts. In practice, token cost is almost never the primary budget killer. The real cost multiplier is retry loops caused by low-quality model completions.
A $0.50/M model that requires 3 retries and 4 fallback corrections costs significantly more than a $2.50/M model that gets it right on the first turn with half the context overhead.
We shifted our FinOps metrics from 'Cost per 1M tokens' to 'Cost Per Resolved Customer Outcome'. This immediately highlighted where ultra-compact models were leaking money through runaway tool retries and where high-reasoning models paid for themselves within a single turn.
Never benchmark models on raw input token pricing alone. Benchmark on end-to-end cost to complete the task with zero human intervention.
Tiered Model Routing and Semantic Vector Caching
We cut 60% of our production inference spend by implementing a two-tier model router paired with a Redis vector cache:
1. Fast Classifier Tier: Compact models (e.g., Gemini 2.5 Flash / Claude Haiku) determine intent, validate required slot parameters, and perform quick evaluations in under 200ms. 2. Deep Reasoning Tier: Frontier models (e.g., Gemini 2.5 Pro / Claude Sonnet) are invoked exclusively when multi-step planning or ambiguous contract negotiation requires it.
Combined with Redis-backed semantic vector caching for high-frequency queries, 35% of all incoming requests bypass LLM inference entirely.
// Multi-tier model router with semantic cache
interface RouteDecision {
modelTier: "FAST_CLASSIFIER" | "DEEP_REASONER";
cacheKey?: string;
}
export async function routeInferenceRequest(
prompt: string,
complexityScore: number,
cacheClient: SemanticCache
): Promise<RouteDecision> {
// 1. Check semantic vector cache (cosine threshold >= 0.94)
const cachedHit = await cacheClient.findSimilar(prompt, 0.94);
if (cachedHit) {
return { modelTier: "FAST_CLASSIFIER", cacheKey: cachedHit.id };
}
// 2. High-complexity planning requires frontier reasoning
if (complexityScore > 0.65 || prompt.length > 1200) {
return { modelTier: "DEEP_REASONER" };
}
return { modelTier: "FAST_CLASSIFIER" };
}Dynamic Tool Schema Injection & Context Pruning
When examining production traces, we discovered that 70% of tokens in our requests came from the static system prompt and massive JSON schema definitions for tools the model never needed in that specific state.
By dynamically injecting tool schemas only when entering states where those tools are valid, we slashed prompt token payloads by 65% across every turn.
Ephemeral Scratchpad Stripping
During multi-turn execution, agents generate intermediate reasoning chains and raw JSON tool payloads. Leaving these in conversation history balloons token usage turn-over-turn. We strip intermediate tool dumps down to structured 2-line state diffs before calling subsequent turns, keeping conversational context lean.
Measured Impact Across 100,000+ Production Sessions
Over a 90-day evaluation window across 100,000 live conversations, these four optimizations combined to deliver:
• 62.4% reduction in monthly inference expenditure • 38% decrease in P95 user-facing latency • 99.4% task resolution parity with unconstrained frontier routing
Cutting AI costs does not require sacrificing output quality; it requires disciplined routing and context hygiene.
Rohit Anand
Engineering Leader & Builder building enterprise platforms, Voice AI, and Agentic systems. Dedicated to evolutionary architecture, team judgment, and delivery excellence.