Adaptive LLM routing for effective agentic workflows

Thirteen models answered the same 78 questions. The cheapest run cost one cent. The dearest cost $17.40. My master’s was about whether you can pick between them per question, and what that buys you.

The problem

Deploying language models in an enterprise means trading three things off against each other. The large proprietary models reason well, but they are billed per token and they are slow, and an agent that reasons over several steps multiplies both. Smaller open models can be self-hosted, which cuts cost and keeps data in-house, but they may not cope with the harder questions. No single model sits well on all three axes, so there is a case for choosing a model per query rather than committing to one.

The task was natural language to SQL. An agent turns a question into a SQL query, runs it, and explains the result. It is a good place to study routing because difficulty varies a lot, from a single GROUP BY to a multi-table join with a window function inside a CTE.

The setup

I built a routing and orchestration framework around a ReAct-style NL2SQL agent in LangChain and LangGraph, with thirteen models from Google’s Vertex AI Model Garden: Gemini, Claude, Llama and Mistral, from Flash-Lite up to Claude Opus 4.

The questions came from the Defog SQL-Eval benchmark: eleven databases, split by database rather than by question, so the 78 hold-out questions (atis, broker and yelp) came from schemas the routers had never seen. The other 193 were for training. Every answer was scored from one to five by an LLM judge on SQL accuracy, answer correctness and faithfulness, with cost and time recorded alongside.

Training labels came from an arena. Every model ran every training question, and a weighted decision matrix picked the winners: answer correctness 0.4, cost 0.2, and 0.1 each for time, SQL accuracy, faithfulness and a general Arena-Hard score. Anything scoring one out of five on correctness was disqualified outright.

Three ways to pick a model

The Matrix Router trains a shallow network on those preference labels, projecting each query and each model into a shared space and sending the query to the best-scoring model.

The RAG Router trains nothing. Every training query is embedded and stored with its winner. A new query retrieves its nearest stored neighbour and goes to whichever model won that one. The bet is that similar questions want similar models.

The Supervisor Agent hands the decision to an LLM, which reads a short description of every candidate and nominates one, and tries another if the first fails.

Results

56×

cheaper than Claude Opus 4 ($0.31 against $17.40)

3.3×

faster per run than Opus (9.0s against 30.0s)

10

of the 13 models were used by the RAG Router

4.68

faithfulness, the best of anything tested

The RAG Router came out with the best balance of the three. Plotting cost against a combined quality score (the mean of the three judged metrics), four things sit on the Pareto front, where nothing is both cheaper and better: two Gemini Flash models at the bottom, gemini-2.5-pro at the top, and the RAG Router between them.

fig. 1 · Cost against quality, 78 hold-out questionscombined score: mean of SQL, correctness, faithfulness

• Router
• Single model
• Pareto front

Combined score (1 to 5)

2.02.53.03.54.04.5$0.01$0.10$1$10gemini-2.5-proclaude-opus-4RAG Routergemini-2.0-flash-001Matrix RouterSupervisor Agentgemini-2.0-flash-lite-001

Total cost, log scale

Against Claude Opus 4, the most expensive model in the set, the RAG Router was better on SQL accuracy (4.27 against 3.92) and on faithfulness (4.68 against 2.72), ran in 9 seconds against 30, and cost $0.31 against $17.40 for the whole hold-out set. Its faithfulness was the highest of anything tested, router or model.

It was not better on everything. Opus scored higher on answer correctness, 4.15 against 3.94, and so did gemini-2.5-pro, at 4.17, for $0.55. That is 24 cents more across 78 questions. If correctness is the only thing you are paying for, just pick gemini-2.5-pro.

Router or model	Cost	Time	SQL	Correct	Faithful	Combined
gemini-2.5-proon the front	$0.55	20.1s	4.43	4.17	4.60	4.40
RAG Routeron the front	$0.31	9.0s	4.27	3.94	4.68	4.30
gemini-2.0-flash-001on the front	$0.04	5.7s	4.50	3.92	4.13	4.18
Matrix Router	$0.52	20.3s	4.42	3.82	4.26	4.17
claude-sonnet-4	$3.36	25.3s	4.00	4.29	3.46	3.92
claude-3.7-sonnet	$3.37	20.2s	3.94	4.36	2.82	3.71
claude-opus-4	$17.40	29.9s	3.92	4.15	2.72	3.60
gemini-2.5-flash	$0.12	5.7s	3.90	3.48	3.13	3.50
claude-3.5-haiku	$0.76	19.8s	3.89	3.54	2.96	3.46
mistral-large	$0.81	11.9s	3.91	3.44	2.91	3.42
gemini-2.0-flash	$0.05	5.2s	3.63	3.50	2.94	3.36
mistral-small	$0.58	22.7s	3.85	3.32	2.76	3.31
gemini-2.0-flash-lite-001on the front	$0.01	2.6s	2.48	2.44	4.44	3.12
Supervisor Agent	$3.01	61.1s	2.83	4.16	1.97	2.99
gemini-2.0-flash-lite	$0.02	3.1s	2.62	2.35	2.18	2.38
llama-4-maverick	$0.21	18.2s	2.51	1.39	2.66	2.19

Who did the work

The more interesting result is where the router sent things. gemini-2.5-pro won 42.5% of the training queries, more than any other model by a distance, yet the RAG Router sent it only four of the 78 hold-out questions. It spread the work across ten of the thirteen models, mostly cheap ones, and never once reached for Opus.

fig. 2 · Who did the workshare of queries, by model

• Won in training (193 queries)
• Sent there by the RAG Router (78 queries)

• gemini-2.0-flash24.9%26.9%
• gemini-2.5-flash6.7%17.9%
• mistral-large0.5%16.7%
• gemini-2.0-flash-lite4.1%12.8%
• llama-4-maverick6.7%9.0%
• gemini-2.5-pro42.5%5.1%
• mistral-small7.8%3.8%
• claude-3.7-sonnet2.1%2.6%
• claude-sonnet-41.6%2.6%
• claude-3.5-haiku2.1%2.6%

That is the retrieval doing its job. The router does not learn the overall winner; it looks at what won on the nearest similar question. On these three schemas the nearest neighbours were mostly won by the Flash models, so that is where the work went, and the quality held.

The two that did not work

The Matrix Router looked fine on paper: 0.78 validation accuracy and an F1 of 0.76. The training loss told the real story. It barely moved, from 0.6931 to 0.6919 over fifteen epochs, which is a model that has learned almost nothing. On the hold-out set it sent all 78 questions to gemini-2.5-pro. With 193 examples, thirteen candidates and a label distribution already skewed towards one model, it learned the label frequency rather than anything about the questions, and inverse-frequency weighting did not stop it.

The Supervisor Agent had the best answer correctness of the three routers, 4.16, close to gemini-2.5-pro itself. It paid for it. The extra LLM call before every run made it nearly ten times the cost of the RAG Router and almost seven times slower, at 61 seconds a question. Passing the worker’s answer back up through the supervisor cost faithfulness too: its 1.97 and its SQL accuracy of 2.83 were the lowest of anything tested. Swapping the supervisor from gemini-2.5-pro to GPT-5 changed its picks noticeably. The share sent to gemini-2.5-pro dropped from 48.7% to 30.1%, and gemini-2.5-flash became the favourite at 37%. The router is only as neutral as the model doing the routing.

What I took from it

None of the routers beat the best single models on every metric. Most of the routing literature reports cleaner wins than that, but it usually routes between two models, one strong and one weak. Telling thirteen heterogeneous models apart from 193 examples is a harder problem, and it was too little data for the trained approach.

What routing does buy is a profile no single model on the list offers. The RAG Router sat on the Pareto front with the best faithfulness in the set, at a fifty-sixth of the price of the most expensive model and a third of its time. That is what an enterprise deployment usually needs: a fixed budget, a latency target, and no single model that lands exactly where it has to.

It is also why I think about this as an industrial engineering problem as much as a machine learning one. The useful finding was not a router; it was measuring all thirteen models on the same line first, which is what made the trade-off visible at all. I wrote more about that in The factory moved into the inbox.