How we built a production-ready multi-agent system at scale
Architecture, trade-offs, state machines, and real-world lessons from building multi-agent systems that operate reliably beyond the demo.
The Demo Trap: Why Agent Prototypes Fail in Production
Building an AI agent demo that wows in a 3-minute video takes an afternoon. Building one that operates reliably for enterprise customers across tens of thousands of conversations is an entirely different engineering discipline.
In early prototypes, models appear omniscient because inputs are clean, context is narrow, and failures are dismissed as quirks. In production, customers ask ambiguous questions, external APIs experience partial outages, and conversational drift causes agents to loop or hallucinate invalid tool parameters.
When autonomous agents are given unrestricted control over conversational loops, subtle model non-determinism compounds geometrically. If each step in an 8-step workflow has a 96% success rate, the cumulative probability of completing the chain without error is only 72%. In enterprise operations, a 28% failure rate is an immediate Sev-1 outage.
Never let an agent operate with unbounded execution loops. Every step must have explicit budget caps on latency, token count, retry attempts, and state transition boundaries.
Finite State Machines Over Unconstrained Loops
Our first major architectural breakthrough was constraining our multi-agent framework into deterministic Finite State Machines (FSM). While the LLM is brilliant at semantic reasoning within an individual state, allowing the LLM to control global system transitions results in non-deterministic deadlocks.
By defining strict state transitions in code and treating the LLM as a state evaluator, we reduced catastrophic loop errors to zero while retaining complete flexibility within each conversational task.
Each state defines exact pre-conditions, admissible tool schemas, allowable exit routes, and deterministic fallback targets. If the model fails to satisfy the criteria for advancing to the next state after two attempts, execution shifts automatically to a recovery state rather than spiraling into a retry loop.
// Deterministic state gate pattern
interface AgentTransition {
from: AgentState;
to: AgentState;
condition: (context: ConversationContext) => boolean;
maxRetries: number;
}
const transitions: Record<AgentState, AgentTransition[]> = {
[AgentState.INTENT_DETECTION]: [
{ from: AgentState.INTENT_DETECTION, to: AgentState.KNOWLEDGE_RETRIEVAL, condition: (ctx) => ctx.intent === "FAQ", maxRetries: 2 },
{ from: AgentState.INTENT_DETECTION, to: AgentState.MUTATION_PLANNING, condition: (ctx) => ctx.intent === "TRANSACTION", maxRetries: 2 },
{ from: AgentState.INTENT_DETECTION, to: AgentState.HUMAN_ESCALATION, condition: (ctx) => ctx.uncertaintyScore > 0.45, maxRetries: 1 },
],
[AgentState.MUTATION_PLANNING]: [
{ from: AgentState.MUTATION_PLANNING, to: AgentState.USER_CONFIRMATION, condition: (ctx) => ctx.plannedAction.requiresApproval, maxRetries: 2 },
],
};
const dispatchNextState = (current: AgentState, ctx: ConversationContext): AgentState => {
const allowed = transitions[current] || [];
const next = allowed.find(t => t.condition(ctx));
if (!next) throw new StateTransitionError("Unrecoverable agent drift — escalating to human fallback");
return next.to;
};Tool Execution: Sandboxing, Idempotency, and Partial Failures
The most dangerous part of any autonomous agent is tool invocation. When an LLM generates a function call, it is producing raw text formatted as JSON. Passing that payload directly into backend databases or third-party webhooks is an architectural catastrophe waiting to happen.
We enforce three invariant layers around every tool call:
1. Runtime Schema Coercion & Semantic Refinement: Every tool output is piped through strict Zod schemas with custom .refine() checks. If an agent attempts to book an appointment with a negative duration or a date in 1970, the schema rejects it immediately with a structured semantic error message passed back to the model.
2. Mandatory Idempotency Keys: Every state-mutating tool (e.g., booking, cancellation, payment capture) generates a deterministic idempotency key derived from the conversation ID, user ID, and intent timestamp. If an agent retries an API call during a transient timeout, the backend deduplicates the operation without double-charging or double-booking.
3. Read/Write Privilege Separation: Specialized specialist agents only receive tools appropriate to their responsibility scope. The conversational FAQ agent does not have access to billing write endpoints; the mutation agent only runs in states requiring confirmed transactional change.
Treat LLM tool outputs like untrusted user input from an external public form. Never execute state mutations without runtime schema coercion and idempotency guards.
Context Window Hygiene and Semantic Compaction
As multi-agent conversations extend across 15 or 20 turns, context windows become cluttered with massive tool output payloads, discarded scratchpads, and repetitive assistant pleasantries. This leads to Context Window Degradation: the model forgets instructions placed in the early system prompt, attends to irrelevant past outputs, and hallucinates.
To solve this, we implemented Ephemeral Tool Output Stripping and Semantic Compaction.
Once an agent finishes a tool execution and synthesizes the user-facing response, the full raw JSON payload (which may be 3,000 tokens of raw database records) is removed from the active context buffer and replaced with a compact 40-token semantic summary. The original payload is persisted to an external Postgres trace table for debugging, but never kept in prompt memory.
Observability, Telemetry, and Synthetic Regression CI/CD
If you cannot inspect every hop of a multi-agent decision chain, you cannot debug failures in production. We implemented distributed trace spans capturing the exact prompt template, tool schema, raw model response, latency, and token cost for every invocation.
Combined with automated synthetic regression suites run on every pull request, we catch 90% of prompt regressions before a single real user encounters them.
Whenever a production conversation results in user correction or negative sentiment, the session is tagged, scrubbed of PII, and automatically added to our synthetic evaluation benchmark. Before any engineer merges a prompt modification or model version upgrade, the system tests the proposed changes against 500+ verified edge cases.
Rohit Anand
Engineering Leader & Builder building enterprise platforms, Voice AI, and Agentic systems. Dedicated to evolutionary architecture, team judgment, and delivery excellence.