Model Routing Is Simple. Until It Isn’t.

Integrating a router into an agentic workflow is often pitched as a "quick win" for developers. The logic seems bulletproof: direct straightforward queries to cost-effective, smaller models, reserve the heavy hitters for complex reasoning, or leverage specific model strengths—such as Claude for coding or Gemini for multimodal tasks. By employing a simple heuristic or classifier, you theoretically slash costs while maintaining performance.

However, in practice, this approach frequently collapses. While many developers view model selection as a straightforward classification challenge, the reality of building production-grade agentic systems reveals that routing is, in fact, a complex systems optimization problem.

The Three Pillars of Routing Complexity

Our experience has shown that routing is far more nuanced than simply matching a prompt to a model. Three specific dimensions consistently complicate the process, turning what should be a simple decision into a multi-variable engineering hurdle.

1. Cost Is More Than Model Pricing

We once operated under the assumption that GPT-4.1 would be more economical than Claude Sonnet 4.6. The data proved otherwise. During a rigorous test across 417 tasks using the AppWorld Test Challenge and a CodeAct agent, Sonnet incurred a total cost of $79 ($0.19 per task), whereas GPT-4.1 cost $155 ($0.37 per task)—nearly double the expense.

On the surface, this defies logic. GPT-4.1 boasts lower token pricing for both input and output, and Sonnet requires roughly three times the reasoning steps to complete identical tasks. The missing variable? Caching.

Agentic workloads frequently reuse massive segments of context across multiple steps. When cache hit rates are high, the effective cost of inputs plummets. Sonnet’s superior cache-read pricing allowed it to outperform the "cheaper" model by a wide margin, effectively neutralizing the cost of its longer trajectories.

"A router that only looks at pricing sheets is optimizing against the wrong numbers. Actual cost depends on the interaction between the model, the workload, and the serving infrastructure."

2. Complexity Is More Than Task Difficulty

A common routing strategy involves estimating task difficulty and assigning the "hard" ones to the most capable models. This intuitive approach fails for two primary reasons:

• Difficulty is often invisible at the start: A seemingly simple prompt like "summarize this contract" may trigger a cascade of retrieval, compliance verification, tool execution, and iterative refinement. Conversely, highly technical code might be handled with ease by a specialized, smaller model. You rarely know the true "difficulty" of a task until the execution is already in motion.
• Competing Constraints: Even with perfect difficulty estimation, it remains just one signal among many. In production, routers must juggle cost, latency, model specialization, and reliability simultaneously. Enterprise environments add further layers: data residency, privacy mandates, and governance-approved model lists. A router must navigate these constraints gracefully, often forcing a task to a model that isn't the "ideal" choice for performance, but is the only choice for compliance.

3. Latency Is More Than Model Speed

It is a common misconception that latency is purely a function of model size. While larger models are generally slower, the user experience is dictated by the entire stack.

Routing itself introduces overhead. Infrastructure variables—such as hardware allocation, cache warmth, and endpoint congestion—often dictate end-to-end response times more than the model’s inherent speed. Furthermore, the granularity of your routing matters. While routing once per task adds negligible latency, routing at every individual step provides greater flexibility but introduces a cumulative tax on performance. A router that ignores the state of the serving system is effectively optimizing against a fantasy.

Shifting to Optimization

These challenges forced us to rethink our architecture. We stopped treating routing as a classification problem and pivoted toward a systems optimization model. Instead of asking, "Which model is best for this task?", our algorithm now balances cost, quality, and latency in real-time, all while remaining lightweight enough to avoid becoming a system bottleneck.

By mapping out a cost-accuracy frontier, we found that our router provides a spectrum of operating points. For instance:

• Latency-Optimized Configuration: Achieved 84% accuracy at $93 and 83 seconds—a 21% reduction in cost and a 9% reduction in latency compared to running Opus alone, with only a marginal 4% dip in accuracy.
• Cost-Optimized Configuration: Pushed expenses even lower while maintaining a competitive accuracy profile.

Crucially, standard difficulty-based routers often fail to explore this full tradeoff space. Because our optimization logic is incredibly lean—requiring only 6 ms and 2 kB of memory per task—it provides the benefits of sophisticated routing without the performance penalty.

The Bigger Picture

The ultimate takeaway from our work is that routing is not about selecting the "best" model; it is about optimizing the entire system. Models are merely one variable in a complex equation that includes caching behavior, infrastructure state, compliance requirements, and workload patterns.

When routing is successful, it is because the system has identified the optimal operating point for the entire environment. While this is a significantly more difficult challenge than simple classification, it is the only one worth solving for those building the next generation of agentic systems.

We look forward to sharing more technical details on our approach in the near future. In the meantime, we are eager to hear from the community: what tradeoffs are you encountering as you scale your own agentic architectures?

*

Acknowledgement: This work was shaped by extensive discussions with colleagues whose insights and feedback were instrumental in refining our approach.