AI Gateway

The field converged in Q2. Every delivery pattern now routes through a gateway, the open-source gateways reached parity with the commercial ones on provider coverage, and our routing experiment found that static task-class rules capture nearly all the cost saving that learned routers promise. What clients actually buy the gateway for is cost attribution per use case, not routing. The remaining work is operational: adapter drift on streaming and tool-call formats recurs every provider release, and that is a tooling problem, not a research one. We expect to dissolve this field into practice by Q4.

Telco and banking clients ask for one view of spend across two or three model gardens before they ask anything about models. Without a gateway they control, a client cannot route to an open-weight or on-prem model when sovereignty or cost demands it, and cannot answer 'what does use case X cost' at all. A hyperscaler gateway answers that question for the hyperscaler's garden only. The choice is made once per client and is expensive to reverse.

One gateway the client controls — routing, observability and cost attribution over Bedrock, Vertex, Azure, direct APIs and on-prem — is table stakes; standardising on a vendor's gateway trades a small build for a large lock-in and loses the on-prem and open-weight legs. That is the lab's one-line position on it, which is not the same as an explanation.

The shape the field is converging on, from the most authoritative source in it: Learned routers underperform static task-class rules outside their training distribution.signal

This is the section a page most needs a person for, and the one composition is worst at. Nobody has written the plain-language version — what the idea is, in words that assume nothing — and it is the first thing a reader who has never met the term needs.

The lab opened this field on 2025-10-06 and it reached mainstream awareness on 2025-06-10. The gap between those two dates is the lead time the lab is measured on.

What shipped: Hyperscaler gateway adds 'cross-provider routing' — to models inside its own garden only (Cloud provider changelog, 2026-04-15). Tooling arriving is what moves a field from argument to something a team could try.signal

What changes, concretely: Static task-class routing across Bedrock, Vertex and a direct API cut cost 34% at equal eval score on four of six task classes (x-gateway-routing). A learned router added 2 points on top and lost 9 outside its training distribution.signal

Measured rather than argued: Validated. Routed configuration cut blended cost per task by 34% with no family dropping more than 1 point of pass rate. The vendor router comparison failed our extraction eval on 38% of calls. Published as r-gateway-default; g-single-gateway-vendor superseded.experiment

There is already a shipped default — gateway.routing.default = task-class-static · gateway.tags = use-case,cost-centre · skill: cavendish/gateway-routing — so a team adopting this is changing a setting rather than starting a project.

The binding constraint is tooling: it works and is affordable, and nobody can operate it yet. Everything upstream of that is solved and everything downstream of it is waiting.

Workforce readiness is medium: Delivery teams can stand it up; adapter drift still lands on the lab. Agent-estimated. A recommendation needing skills the firm does not hold is an aspiration rather than an action, and it routes to the enablement agenda instead of the delivery one.

Something here has already been killed: One commercial gateway over every model garden, on the routing experiment showed the value sits in the routing policy and the cost ledger, both of which we could not get out of the vendor gateway, so the recommendation became a gateway we control.graveyard

The argued case against it is the red team's, further down this page, and it is deliberately one-sided — this section is what stands in the way mechanically, not what somebody thinks of it.