I wanted to test three things that are usually drawn on architecture diagrams and less often run: a single AI gateway serving agents on more than one cloud, a single registry that says which agents and tools exist, and containment, meaning the ability to stop one agent's traffic by changing one record. I built a small lab across AWS and Google Cloud with Claude Code to see which parts hold, and this post describes what was built, what the results were, and what the lab does not show.

The question#
An AI agent makes two kinds of calls. It calls a language model, and it calls tools, which increasingly are exposed through the Model Context Protocol (MCP), where an MCP server lists tools and executes them on request. If agents run on several platforms, each platform tends to bring its own gateway and its own registry, and there is then no single place to say "this agent is disabled" and have that be true everywhere.
The design I tested is the centralized alternative. First, every model call and every tool call goes through one gateway. Second, one registry holds a record for each agent and each MCP server. Third, the gateway reads the registry on every call. The question was whether that is enough to stop an agent on either cloud, and whether anything on the second cloud has to read the registry as well.
What was built#
| Component | Where it ran |
|---|---|
| Gateway | Cloud Run, Google Cloud (us-central1) |
| Registry | Google Cloud Agent Registry |
| Agent 1 | AWS Lambda (us-east-1) |
| Agent 2 | Cloud Run, Google Cloud, same code as agent 1 |
| MCP server with two tools | AWS Lambda, built with the MCP Python SDK |
| Model | Gemini 2.5 Flash on Vertex AI |
| Audit trail | BigQuery |
The flow is the same for both agents:
- The agent holds no model credential and no tool credential. It has one thing, an identity it can prove.
- It sends every model call and every MCP call to the gateway.
- The gateway verifies the identity, looks the caller up in the registry, and refuses if the agent is not active, the MCP server is not active, or the agent is not entitled to that server.
- The gateway calls the model, or forwards the tool call to the MCP server using the gateway's own credential.
- The MCP server accepts requests from the gateway and from nobody else.
Step 5 is what allows the registry to be read in one place. If the only route to the tools is through the gateway, then the gateway's check cannot be skipped by calling the server directly.

The agent itself is a short loop. It lists the tools through the gateway, asks the model through the gateway, runs whichever tool the model asks for through the gateway, and sends the result back for a final answer. A normal run, as the agent reports it:
{
"platform": "aws",
"question": "What is the same-day wire cut-off for EUR, and the EURUSD rate?",
"trace": [
{"step": "mcp.initialize", "http": 200, "ok": true},
{"step": "mcp.tools/list", "http": 200, "ok": true, "tools": ["get_fx_rate", "get_wire_cutoff"]},
{"step": "model.generate", "http": 200, "ok": true, "model": "gemini-2.5-flash", "agent": "poc-aws-agent"},
{"step": "mcp.tools/call", "http": 200, "ok": true, "tool": "get_wire_cutoff"},
{"step": "mcp.tools/call", "http": 200, "ok": true, "tool": "get_fx_rate"},
{"step": "model.generate", "http": 200, "ok": true, "model": "gemini-2.5-flash", "agent": "poc-aws-agent"}
],
"answer": "The same-day wire cut-off for EUR is 11:00 ET. The indicative mid-market FX rate for EURUSD is 1.0842."
}
Identity without a stored secret#
The agent on AWS needs to prove who it is to a gateway on Google Cloud, and I did not want an API key sitting in a Lambda environment variable. The mechanism is Workload Identity Federation: the Lambda's IAM role is trusted by a Google Cloud identity pool, the agent exchanges its AWS credentials for a short-lived Google token, and it then asks for a Google-signed identity token whose audience is the gateway. The agent on Google Cloud gets the same kind of token from the metadata server. The gateway therefore verifies one token type for both clouds.
On AWS, the agent federates and then mints the identity token. fetch_id_token, the usual helper, does not support a federation credential file, so the call to generateIdToken is made directly:
def _token_aws(audience: str) -> str:
"""Federate the Lambda execution role, then mint an id-token as the service account."""
source, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
source.refresh(Request())
body = json.dumps({"audience": audience, "includeEmail": True}).encode()
request = urllib.request.Request(
"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/"
f"{SERVICE_ACCOUNT}:generateIdToken",
data=body, headers={"Authorization": f"Bearer {source.token}",
"Content-Type": "application/json"})
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read().decode())["token"]
On Google Cloud the same token comes from one request to the metadata server, with format=full so that it carries the email claim:
url = ("http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/"
f"default/identity?audience={audience}&format=full")
The gateway verifies the token in full (signature, expiry, issuer, audience) and uses the email claim as the caller's identity:
def identify(token: str | None) -> Caller:
audience = os.environ.get("GATEWAY_AUDIENCE", "")
if not audience:
# verify_oauth2_token skips the audience check when given None, which would
# accept a Google id-token minted for any other service. Refuse instead.
raise NotAuthenticated("gateway misconfigured: GATEWAY_AUDIENCE is not set")
...
claims = _verify(token, audience) # google.oauth2.id_token.verify_oauth2_token
email = (claims.get("email") or "").lower()
if not email or not claims.get("email_verified", False):
raise NotAuthenticated("token carries no verified email claim")
return Caller(email=email, subject=str(claims.get("sub", "")), verified_by="gateway")
Two details mattered here. First, the pool is pinned to one named AWS role, because an AWS account is an authentication boundary and not an authorization one, and without that condition any principal in the account could federate. Second, the gateway takes the caller's identity from the verified token and never from the request body. The gateway I started from accepted an agent_id field in the request, which means any permitted caller could claim to be any agent, and a per-agent disable would have meant nothing.
The registry#
Google Cloud Agent Registry accepted registrations for an agent that runs on AWS and for an MCP server hosted on AWS, and it is free to use. It did not fit the job in two ways. First, a record has no status or lifecycle field, so there is nowhere to say "disabled". Second, it has a Binding resource that looks like it should express "this agent may use this server", but creating one fails with "auth_provider is required", because it is a credential-brokering record and not an entitlement.
I worked around both by carrying status and entitlement as a small JSON object in each record's description, and I kept the registry behind one function in the gateway so a different registry could replace it. An agent's record carries this:
{"gov": 1, "status": "active",
"principal": "poc-aws-agent@<project>.iam.gserviceaccount.com",
"owner": "ai-platform@example.com", "platform": "aws",
"workload_class": "tool_calling_agent",
"mcp_servers": ["poc-treasury-mcp"]}
A small command-line tool is the only thing that writes records, and it is what the scenarios below call:
python poc/registry/registryctl.py register-agent poc-aws-agent \
--principal poc-aws-agent@<project>.iam.gserviceaccount.com \
--owner ai-platform@example.com --platform aws --url <where it runs>
python poc/registry/registryctl.py register-mcp poc-treasury-mcp \
--url https://<function-url>/mcp --audience agent-poc-treasury-mcp \
--owner ai-platform@example.com --platform aws --toolspec toolspec.json
python poc/registry/registryctl.py bind poc-aws-agent poc-treasury-mcp
python poc/registry/registryctl.py disable poc-aws-agent # the kill switch
python poc/registry/registryctl.py enable poc-aws-agent
The gateway's decision for a tool call is three checks in order, each with a reason that is returned to the caller and written to the audit table:
def authorize_tool_use(self, principal: str, server_name: str) -> Decision:
decision = self.authorize_agent(principal) # registered, and active
if not decision.allowed:
return decision
agent = decision.agent
...
server = snapshot.mcp_by_name.get(server_name)
if server is None:
return Decision(False, "mcp_server_not_registered", ...)
if not server.active:
return Decision(False, "mcp_server_disabled", ...)
if server.name not in agent.mcp_servers:
return Decision(False, "agent_not_bound_to_mcp_server", ...)
return Decision(True, "ok", agent=agent, server=server)
The registry is read into a snapshot and cached for 10 seconds. If a refresh fails, the previous snapshot is used until it is 60 seconds old, and after that every decision is a refusal, so a registry outage stops agent traffic instead of letting unchecked traffic through. For anyone choosing a registry for this purpose, I think the requirements reduce to three: a status that can be set and read, a place to record which tools an agent may use, and an API the gateway and deploy pipelines can query quickly.
Results#
Each scenario changes one record and then runs both agents.
| Registry change | Agent on AWS | Agent on Google Cloud |
|---|---|---|
| None | model and tools work | model and tools work |
| Disable the AWS agent | model and tools refused | unaffected |
| Disable the Google Cloud agent | unaffected | model and tools refused |
| Disable the MCP server | tools refused, model works | tools refused, model works |
| Remove the AWS agent's entitlement | tools refused, model works | unaffected |
| Agent calls the MCP server directly | refused by the server | refused by the server |
| AWS agent names the other agent in its request | attributed to the AWS agent | not involved |
The output of the run for the first disable scenario, where each line is the path one agent took and the HTTP status of each hop:
## 2. Disable the AWS agent record
registry: poc-aws-agent is now disabled
aws: model=refused tools=refused | mcp.initialize 403 (agent_disabled) > model.generate 403 (agent_disabled)
gcp: model=ok tools=ok | mcp.initialize 200 > mcp.tools/list 200 > model.generate 200 > mcp.tools/call 200 > mcp.tools/call 200 > model.generate 200
RESULT: PASS


The MCP server's own check is what refuses a direct call. It verifies the token and then requires the caller to be the gateway's service account, so an agent's token is refused even though it is valid:
def _verify(authorization: str) -> tuple[bool, str]:
"""(allowed, reason). Full verification: signature, expiry, issuer, audience, caller."""
...
claims = id_token.verify_oauth2_token(authorization[7:].strip(), Request(), audience=AUDIENCE)
email = (claims.get("email") or "").lower()
if not claims.get("email_verified") or email not in ALLOWED_CALLERS:
return False, f"caller {email or '(no email)'} is not the gateway"
return True, email
## 6. Bypass: each agent calls the MCP server directly
aws: {"http": 401, "reached_tools": false, "refused": "caller poc-aws-agent@... is not the gateway"}
gcp: {"http": 401, "reached_tools": false, "refused": "caller poc-gcp-agent@... is not the gateway"}

In one timed trial the AWS agent was first refused 7.4 seconds after the disable command was issued, and 5.8 of those seconds were the registry write itself. No token was revoked in any scenario: the same token was accepted before the change and refused after it. Nothing on the AWS side read the registry.
The same record also works as a deploy gate. A command checks whether the records for the things about to be deployed are active and exits non-zero if one is not, which is the shape a pipeline step needs.
$ python poc/registry/registryctl.py gate poc-aws-agent poc-gcp-agent poc-treasury-mcp
{"record": "poc-aws-agent", "status": "disabled", "pass": false}
{"record": "poc-gcp-agent", "status": "active", "pass": true}
{"record": "poc-treasury-mcp", "status": "active", "pass": true}
DEPLOY GATE FAILED: not active in the registry: poc-aws-agent (exit 1)
What review found#
Before deploying, I had a separate agent session review the code with the instruction to break the containment claim, and it found two ways through that the tests had not covered.
First, the gateway still exposed its older routes, which took the agent name from the request body, so a disabled agent could have kept working by calling those. They are now closed when registry enforcement is on.
Second, I had taken a shortcut on identity. The gateway ran as a private Cloud Run service, where the platform verifies the caller's token before the request arrives, and I trusted that. The platform accepts the token in either of two headers, so a caller could satisfy the platform with a real token in one header and put an unsigned token naming a different agent in the other. The gateway now verifies a token carried in its own header against Google's public keys.
The review also pointed out that the gateway forwarded tool calls to whatever URL the registry record held. Anyone who can edit a record could point it at their own server and receive a valid gateway credential. The gateway now refuses to send its credential to a host it was not configured for, which means registry write access is a privileged role and should be treated as one.
A model router in the path#
Many AWS deployments put a model router such as LiteLLM between the agents and the upstream. The concern is that the router authenticates upstream with one shared credential, in which case the gateway sees the router and not the agent.
I tested this locally with LiteLLM 1.109 in front of a mock upstream that recorded the headers it received. LiteLLM's documentation says the caller's Authorization header is never forwarded, which matches what I saw. The LiteLLM configuration under test:
model_list:
- model_name: gw-openai
litellm_params: {model: openai/mock, api_base: "http://127.0.0.1:4021/v1", api_key: SHARED-UPSTREAM-KEY}
- model_name: gw-anthropic
litellm_params: {model: anthropic/mock, api_base: "http://127.0.0.1:4021", api_key: SHARED-UPSTREAM-KEY}
general_settings:
master_key: sk-litellm-master
forward_client_headers_to_llm_api: true # false in the second run
What the upstream received, first with forwarding on and then with it off:
/v1/chat/completions | x-agent-identity: AGENT-JWT-on-gw-openai | authorization: Bearer SHARED-UPSTREAM-KEY
/v1/messages | x-agent-identity: AGENT-JWT-on-gw-anthropic | x-api-key: SHARED-UPSTREAM-KEY
/v1/chat/completions | x-agent-identity: None | authorization: Bearer SHARED-UPSTREAM-KEY
/v1/messages | x-agent-identity: None | x-api-key: SHARED-UPSTREAM-KEY

With forward_client_headers_to_llm_api: true, a custom header carrying the agent's identity reached the upstream on both the OpenAI-style and the Anthropic-style routes, next to LiteLLM's own shared key. With the default setting it did not. So the agent's identity can pass through the router in a custom header, provided forwarding is turned on. I did not run the full containment test through LiteLLM.
Latency and data transfer#
A centralized gateway means agents on the other cloud pay for a round trip. I ran the same benchmark from the AWS agent and from the Google Cloud agent, on a reused connection, and had the gateway report its own processing time so that network time could be separated from model time.
The gateway adds its processing time to every response, and the benchmark subtracts it from the round trip:
started = time.perf_counter()
response = await call_next(request)
response.headers["X-Gateway-Seconds"] = f"{time.perf_counter() - started:.4f}"
network = [r["seconds"] - r["gateway_seconds"] for r in rows_timed
if r["gateway_seconds"] is not None]
| Call | Round trip from AWS (median) | Gateway's own time | Added by crossing clouds |
|---|---|---|---|
| Identity and registry check only | 41.5 ms | 2.5 ms | 28.0 ms |
| Model call, small prompt | 972 ms | 931 ms | 26.2 ms |
| Model call, 32 KB prompt | 1,187 ms | 1,134 ms | 37.0 ms |
| MCP tool call | 267 ms | 228 ms | 27.5 ms |
Crossing clouds added 26 to 37 ms per call, which is about 3% of a model call and about 10% of a tool call. A new connection took 39 ms from AWS against 9 ms from Google Cloud, so connection reuse matters. An agent that makes 20 calls in a run pays roughly half a second in total.
For data transfer I counted HTTP bytes in each direction and applied list prices: $0.09 per GB leaving AWS and $0.12 per GB leaving Google Cloud over the internet. One million model calls with a 32 KB prompt came to about $3.23, and one million small calls to about $0.32. Per-GB cost is small at these payload sizes. A private connection between the clouds is priced differently, mostly by fixed port charges in the thousands of dollars a month, so the volume-based figure is not the whole cost of a private path.
My first benchmark run reported the larger prompts as faster than the small one. Those calls had been refused by the gateway, which returns some refusals as HTTP 200, so the benchmark now checks each call's outcome.
What disabling does not do#
Disabling a record stops the agent's model and tool traffic at the gateway, and it does not stop the workload, because the process keeps running and can still do whatever its own cloud role and network allow without a model. A call already in flight completes, and a streaming response that began before the change is not cut off, so a multi-step run fails at its next model or tool call. If the requirement is that the workload itself stops, that is a separate control on each platform.
The result also depends on two conditions that nothing checks at request time: agents have no other route to a model, and tool servers accept the gateway alone.
Carrying this to a production gateway#
My gateway re-reads the registry every 10 seconds, and if the registry cannot be read it uses the last copy for up to 60 seconds and then refuses everything. A production API gateway has its own caching. For example, Apigee's documentation gives a default of 180 seconds for cached API keys and OAuth tokens, configurable from 1 to 180, and 300 seconds for key value maps. A containment target such as 60 seconds therefore has to be designed for and measured on the real gateway, and the 7.4 seconds from this lab says nothing about it.
Apigee's documentation does state that a proxy can have your own MCP server as its backend target, with authentication and per-tool access, which is the same shape as the MCP path in this lab.
What was not tested#
- A production API gateway, since the gateway was my own service.
- Agents on Kubernetes, since the AWS agent ran on Lambda.
- A private network path between the clouds.
- Per-tool authorization, since entitlement was per MCP server.
- Streaming responses and time to first token.
- More than one region pair, or more than a few runs.
0 Comments
Leave a Comment