An enterprise that builds with LLMs eventually has to decide whether all model traffic goes through one gateway or each platform runs its own, and I think the argument tends to stall because it is held at the wrong layer. The more useful question is how many data planes sit under one control plane, where the control plane is where the rules live (policy, identity, the approved model catalog, budgets and the usage ledger) and a data plane is the component each request actually passes through on its way to a model. Several control planes give several answers to the same audit question, so in practice the decision is about data planes.
There is a second distinction to make first, because "AI gateway" now names two different components that carry different traffic.
Model gateways and agent gateways
A model gateway sits between callers and LLMs, and an agent gateway sits between agents and the tools, APIs, data and other agents they call. They do different jobs and usually live in different places:
| Model gateway | Agent gateway | |
|---|---|---|
| Traffic | Prompts and completions to LLMs | Tool calls (MCP, APIs) and agent-to-agent (A2A) calls |
| Where it lives | Usually one, enterprise-wide, next to the default models | One per cloud or runtime, next to the agents and tools |
| Checks every call | Who is calling? Still approved? Allowed this model? Within budget? | Who is calling? Still approved? Allowed this tool and this action? |
| Its own heavy lifting | Safety screening of prompts and responses, token metering, quotas, routing and fallback across models, caching | Destination allow-lists, credential brokering (the agent never sees the tool's secret), action authority (allow, deny or ask a human) |
| Fails badly when | It is bypassed with direct provider keys | It is missing and agents call tools with shared keys |
An agent gateway is not an orchestrator, because planning, choosing tools, looping and memory belong to the agent runtime or framework, and the gateway is the turnstile on the way out rather than the part deciding where to go. For example, when an agent decides to call a ticketing API, the decision was made in the runtime, and the gateway only checks that this agent may take that action, brokers the credential and logs the call. Calling the agent gateway "orchestration" invites scope creep and vendor confusion.
Both gateways read the same registry, where the agent registry is the HR file for an agent (owner, risk class, approved models and tools, status) and the identity provider issues the badge. Both are control plane, and both gateways check them on every call from cache rather than a live lookup (signing keys cached, registry status cached briefly, suspensions pushed), so suspending an agent once blocks it at both.
One agent task crosses both gateways in turn, model call -> tool call -> tool result -> next model call. First, the model call goes through the model gateway. Second, the tool call the model asks for goes through the agent gateway. Third, the tool's result is fed into the next model call, which passes through the model gateway again, and that is where a poisoned tool result gets screened.
Most of what follows concerns the model gateway, and the last section covers how agent gateways affect the choice.
Three operating models
The three operating models differ in two numbers, namely how many control planes you run and how many data planes.
- Centralized: one control plane and one data plane, so every model call, from every cloud, laptop and data center, crosses one enterprise gateway.
- Fully federated: a gateway per platform, each with its own rules, which means many control planes and many data planes. It is fast to start, and it leaves several partial audit trails that never reconcile.
- Hub-and-spoke (split plane): one control plane, with the central gateway as the default data plane and "resident" gateways on other platforms allowed to carry traffic only while they meet the same standard.
In the diagram, dashed lines are policy, catalog and budgets pushed down from the control plane, and solid lines are the requests themselves. The word "federated" is used both ways, sometimes for hub-and-spoke and sometimes for fully decentralized, so it is worth asking which is meant before agreeing to it.
Choosing between them
I would default to central and make every exception earn its way out. Centralizing fits when any of the following holds:
- Audit and risk ask "which AI, used by whom, on what data, under which controls?" and need one answer rather than several partial ones.
- You need per-user and per-app attribution, quotas and budgets before anything reaches production.
- You want provider keys out of application code, one egress point, and model swaps as a configuration change.
- One team has to run it all, since every extra gateway is another runtime to patch, monitor and staff.
A spoke is justified when one of these holds instead:
- A second cloud carries real volume and you have measured that the extra hop hurts.
- Platform-native agent features need the model next to the data, for example grounding, memory and context caching.
- The work is latency-critical or bulk, for example real-time voice, batch inference and embeddings.
- Custom or self-hosted models stay where they run, while still being registered, metered and shared through the central gateway.
Fully federated is rarely chosen deliberately and is more often the state an enterprise is in before it decides, and I would accept it only for sandboxes on synthetic or public data.
A routing rule turns this into something a team can apply without a meeting. The questions are asked in order and the first "yes" decides:
- Is it SaaS with embedded AI, such as a copilot inside a vendor app? Then it is not a gateway decision, because the traffic cannot be routed, so it is governed through third-party risk.
- Is it a sandbox on synthetic or public data? Then it runs as an approved sandbox with minimum controls, and it must meet the standard to go live.
- Is it enterprise-level traffic? Then it goes through the central gateway, and any one of these qualifies:
- it crosses a cloud, network or organizational boundary, or calls a public provider API
- it is used across business units or draws on shared model capacity, for example developer coding tools
- it is customer-facing, highly autonomous, or touches confidential data
- its spend or volume is above a set threshold
- Does its local gateway meet the standard? Then it may stay resident.
- If none of the above applies, it goes through the central gateway.
Latency is worth measuring before it is used as an argument for a spoke. In my experience safety screening adds more delay than the network hop, since a cross-cloud hop costs tens of milliseconds against LLM calls of hundreds of milliseconds to seconds, so time to first token and gateway overhead per hop settle it faster than a slide does.
The standard a resident gateway has to meet
A resident gateway earns traffic by meeting a written standard rather than by asking for an exception, and this is the checklist I would hold a spoke to:
| Control | What the spoke must prove |
|---|---|
| Safety policy | The same screening floor as central, covering streamed responses and multi-turn context; fails closed |
| Identity | A workload identity per app or agent, no static keys; the end user's identity carried through |
| Attribution | An app or agent ID on every call |
| Telemetry | The common log schema, shipped to the same SIEM |
| Tokenomics | A daily usage feed into the one ledger; budget approved before production |
| Catalog | Approved models and versions only |
| Network | Private access, no internet egress, provider credentials in the enterprise vault |
| Registry | Registered, with an owner and a risk tier |
| Effectiveness | Periodic red-team evidence that the controls actually work |
If a spoke stops conforming, its traffic moves to the central gateway within an agreed window, and without that clause drifting from the standard has no consequence.
The same standard is also how an enterprise answers frameworks like the NIST AI RMF, which ask for evidence rather than products, for example an inventory of AI use, production monitoring and the ability to switch a model off. A gateway is the cheapest place to produce that evidence, and the standard makes every gateway produce it the same way.
Each operating model also carries costs of its own, along with the usual mitigation:
| Model | Cost | Mitigation |
|---|---|---|
| Central | Every prompt and response in one log makes a high-value target | Restricted index, field-level redaction, access owned by security |
| Central | A single point of failure, plus an onboarding queue | Regional failover, self-service onboarding, published SLAs |
| Resident | Drift, as templates, logs and quotas slowly diverge from central | The standard, re-tested on a schedule |
| Resident | Partial audit trails, duplicated effort, spend outside the central ledger | Common logs and the daily ledger feed as conditions of entry |
| Fully federated | Every resident cost, with no standard to pull it back | Move to central or hub-and-spoke and write the standard |
Agent gateways and the hybrid end state
Two consequences follow once both kinds of gateway are in the picture.
First, centralized is hub-and-spoke with zero spokes. If you write the standard and the routing rule and no local gateway meets the standard yet, everything routes central, so model traffic is centralized for now with a path open for a spoke that proves itself. The architecture is the same either way and only the number of spokes changes, so one design is enough and the evidence decides how many spokes it has.
Second, agent gateways are federated by nature, because tools live next to the agents that call them, in every cloud and data center. So even a fully centralized model gateway sits inside a hub-and-spoke system once agent gateways are counted, and the target state I would argue for is models central and tools local, with one registry and one identity provider governing both.
That splits the work so that the central model gateway owns model traffic, safety screening and one alerting view, while the local agent gateways own registry checks, tool enforcement and credentials. Three conditions keep the split from leaking:
- The central gateway checks the registry too, because if only the agent gateways do, a suspended agent is blocked from its tools but can still call models.
- The agent gateways feed the same alerting view in the same log format, or the single pane of glass shows half the picture and tool-call blocks and credential misuse surface somewhere else.
- Data leaving to a tool is checked locally. Prompts, responses and poisoned tool results are screened centrally on the next model call, but sensitive data going out in an API call never passes the central gateway, so that check belongs at the agent gateway.
When to revisit the decision
The model-gateway decision is worth revisiting only on evidence, for example:
- Measurements show the extra hop materially harms a major traffic class, or a second cloud becomes the main home for LLM workloads.
- The central gateway cannot deliver multi-provider routing, caching and token accounting on the timeline the platforms need.
- Security selects a primary inline screening engine that integrates materially worse with your gateway than with an alternative.
The order of work that follows from all this is to settle the vocabulary first (one control plane, a model gateway for model traffic, agent gateways for tools), then write the standard and the routing rule, after which central versus hub-and-spoke is decided by measurements of specific traffic classes rather than by which team owns the gateway.
0 Comments
Leave a Comment