Adaptive LLM routing for effective agentic workflows

My MEng at Stellenbosch was on adaptive LLM routing, completed as a case study with Spatialedge. This post is a short summary of what the work set out to do and what came out of it.

The problem

Deploying large language models in an enterprise setting means balancing three things that pull against each other. High-capability proprietary models reason well, but they are billed per token and run slowly, and an agent that reasons over several steps multiplies both. Smaller open-source models that can be self-hosted reduce cost and keep data in-house, but they may not be capable enough for the harder queries. No single model sits at a good point on all three axes, so there is a case for a system that selects an appropriate model per query rather than committing to one.

The target task was natural language to SQL. The agent translates a natural-language question into a SQL query, executes it, and explains the result. It is a useful setting to study routing in because query difficulty varies considerably, from a simple aggregation to a multi-table join with nested conditions.

The framework

I built a routing and orchestration framework around a ReAct-style NL2SQL agent, implemented with LangChain and LangGraph. It maintains a curated repository of thirteen models from Google Vertex AI, drawn from several providers and spanning a range of capability tiers, from small open models to large frontier ones.

The framework implements three routing strategies, each based on a different selection principle.

The Matrix Router trains a shallow neural network on pairwise preference data, projecting query and model embeddings into a shared space where their compatibility can be scored. The query is routed to the highest-scoring model. The loss is weighted by inverse model frequency to counter class imbalance in the training data.

The RAG Router performs no training. Each training query is embedded and stored with its winning model as metadata. For a new query, a cosine similarity search retrieves the nearest stored query, and the model attached to it is selected. Its assumption is that semantically similar queries benefit from similar model capabilities.

The Supervisor Agent Router delegates the decision to a supervisory LLM, which reasons over structured capability descriptions for each candidate and nominates a model. If that model fails, the failure is fed back and the supervisor selects an alternative.

Evaluation

Evaluation used the Defog SQL-Eval benchmark: eleven databases, split roughly seventy to thirty into a training set of 193 questions and a hold-out set of 78 questions on schemas unseen during training. Every model and every router ran the same questions in the same agent configuration, with each run traced in MLflow.

Answers were scored from one to five by an LLM judge on SQL accuracy, answer correctness and answer faithfulness. The judge was deliberately chosen from a provider not represented among the routing candidates, which mitigates the risk of it favouring outputs from its own family. Total cost and execution time were recorded alongside the quality scores.

Results

On the hold-out set the RAG Router ran at $0.31, against $17.40 for the most expensive model in the set, a factor of fifty-six. It also scored higher than that model on SQL accuracy, 4.27 against 3.92, and on faithfulness, 4.68 against 2.72, and finished roughly three times faster. It sits on the Pareto front for cost against combined quality, and it distributed work across ten of the thirteen models, reaching for the most expensive one on only about five per cent of queries.

The Matrix Router reached 0.78 validation accuracy and then failed to generalise. On the hold-out set it selected a single high-capability model for all 78 queries. With 193 training examples and thirteen candidates to separate, and a training distribution already skewed towards that model, it learned the label frequency rather than the underlying relationship between a query and its best model. The inverse-frequency weighting was not sufficient to prevent this.

The Supervisor Agent Router achieved the highest answer correctness of the three routers, but the additional LLM call before every agent run raised its cost by roughly an order of magnitude and its latency by a factor of seven. Passing the worker's output back through the supervisor also reduced its faithfulness score. Repeating the experiment with a different supervisor model produced a noticeably different routing distribution, which indicates how much the outcome depends on that one choice.

What I took from it

None of the routers consistently outperformed the best standalone models on every metric. One mid-priced model scored higher on answer correctness at a lower cost than the RAG Router, and on the metric that matters most it was the better choice. Much of the routing literature reports cleaner advantages than this, but that work typically routes between two models, one strong and one weak. Separating thirteen heterogeneous models from a small training set is a harder problem, and the data was not sufficient for the trained approaches.

The conclusion I drew is that routing does not buy you a better model. It produces a cost, latency and quality profile that no single model on the list offers on its own, which is what an enterprise deployment usually needs when it has a fixed budget, a latency target, and no model that lands where it needs to. The framework is available on GitHub.