Build the Gateway, Buy the Model: AI Governance for Regulated Industries on GCP

Last week I asked a deceptively simple question: if a company in a highly regulated industry — banking, healthcare, insurance, government — wants to give employees an AI assistant, should it self-host an open model to keep data safe? The answer I landed on surprised me a little. For almost every workload, you should buy the model — through your own cloud tenancy — and put your engineering effort into something else entirely: the gateway that sits in front of it.

I built that gateway on GCP to prove the point, using a retail-banking scenario as the worked example. This post walks through what it does, why every regulated enterprise ends up building some version of it, and where self-hosting actually earns a place. The code is public at github.com/jeremydegardeyn/bank-ai-gateway, and the whole thing runs for under $10 a month — $0 when idle.

The architecture in one diagram

User → Chat UI (Cloud Run) → AI Gateway (Cloud Run, FastAPI)
         1. Budget check      Firestore per-user daily token quotas
         2. PII screen        Model Armor + custom detectors → block or redact
         3. Tier routing      standard → Gemini Flash · premium → Gemini Pro / Claude
         4. Response screen   PII check on the model's reply too
         5. Audit             BigQuery → Looker Studio dashboard

Every request — whether it comes from the chat UI or from an application calling the gateway's API — passes through the same five steps. The models sit behind the gateway, served pay-per-token inside your own Vertex AI tenancy. Swapping the premium model from Gemini Pro to Claude, or adding a self-hosted tier later, is a one-line config change. The governance layer doesn't care which model answers.

That's the whole design. The interesting part is why it needs to exist.

The problem: your employees are already using AI

The nightmare scenario for a regulated business isn't hypothetical. An employee has a customer on the phone — a disputed charge, an insurance claim, a patient billing question — opens a consumer chatbot in a browser tab, and pastes in the customer's name, SSN, and account details to draft a response letter. On consumer AI plans, that conversation may be retained for years and used for model training unless the user opted out. The data has now left your organization, permanently, through a tool nobody sanctioned.

Banning AI doesn't fix this — it just pushes usage further into the shadows while competitors get the productivity gains. The fix is to give people a sanctioned front door and put the controls at that door. Here's what that looks like when someone tries the nightmare scenario against my gateway:

The prompt never reaches a model. The user gets an immediate, explicit explanation, and the incident lands in the audit log with the finding types attached. Notice the detail in the caption line: the block came from model-armor+local-regex. That's deliberate, and it's the first lesson from actually building this.

Defense in depth, because no single filter catches everything

Google's Model Armor is the managed screening layer — it inspects prompts and model responses for PII, credentials, and prompt-injection attempts, and its first two million tokens a month are free. It's genuinely good, and it's model-agnostic: it doesn't matter whether Gemini, Claude, or a self-hosted model sits behind it.

But when I tested it, the basic configuration missed things a regulated business cares about. It rejected canonical example SSNs that fail checksum validation, it has no email detector in the basic tier, and it obviously knows nothing about your internal identifier formats — account numbers, member IDs, policy numbers, medical record numbers. So the gateway runs a second, local detector layer on every request and takes the union of the findings. That layer is where institution-specific patterns live — in my demo, any string matching the fictional HB-######## account format gets blocked, which is exactly what you'd implement for real internal identifiers using Sensitive Data Protection custom infoTypes.

The general principle: a managed filter is your baseline, not your policy. The gateway is where your policy lives.

Yes, tracking tokens is worth it

I went back and forth on whether per-user token accounting was demo theater or a real control. Having built it, I'm convinced it's real, for three reasons.

Cost governance becomes policy instead of surprise. Pay-per-token pricing is wonderful until an agentic workload gets stuck in a loop or a power user discovers they can paste entire document sets into every prompt. Per-user and per-department daily budgets turn "we got a shocking invoice" into "the request was declined and the user was told why." In my demo, a low-budget user hits their cap and gets a clean message rather than silently running up spend.

Chargeback and showback come next. The moment more than one department uses the platform, finance will ask who's spending what. Because every request is metered and attributed at the gateway, that answer is a BigQuery query, not a forensic exercise.

And token consumption is a behavioral signal. A user whose daily usage jumps 50x, or an application whose output-token ratio suddenly spikes, is worth looking at — sometimes it's a bug, sometimes it's misuse. You only see it if you're counting.

Every request lands in the audit table with its outcome, tier, model, and token counts:

Point Looker Studio at this table and you have the dashboard that makes the whole platform legible to leadership: spend per user, tokens over time, PII incidents by type.

So when should you self-host?

There are really two separate self-hosting questions, and conflating them is where a lot of architecture conversations go wrong.

Self-hosting the gateway: almost always yes. The gateway is cheap (scale-to-zero Cloud Run), it's where your policy and audit obligations live, and owning it keeps you model-agnostic. Even if your only requirement were PII redaction and risk management, the gateway earns its place — that alone is the control regulators and CISOs ask about. But it buys you more than that: because the gateway and the models both live inside your GCP project, you get a hard guarantee that every request stays inside your cloud tenant. Prompts go from your Cloud Run service to Vertex AI endpoints in your project, under your IAM, in your chosen region, with contractual no-training terms. Nothing transits a consumer service. In a regulated environment, that tenancy boundary is often the single most valuable property of the whole design. (Whether you build the gateway or buy one — LiteLLM Enterprise, Apigee with Model Armor policies, Kong — is a separate make-vs-buy decision; either way you operate it yourself.)

Self-hosting the models: rarely, and for specific reasons. I costed this out. A single NVIDIA L4 GPU running 24/7 on Cloud Run is roughly $550 a month — about twenty enterprise chat seats — and it serves a mid-size open model, not a frontier one. Self-hosted models earn a place as a tier behind the gateway in three situations: data-sovereignty or air-gap mandates that even managed in-tenant services can't satisfy; high-volume batch workloads (document classification, extraction) where per-token pricing loses to owned capacity; and vendor-independence hedging, where risk management wants a credible fallback that no supplier can take away. For the general employee-assistant workload, buying frontier models through Vertex wins on every axis: capability, speed, and cost.

The pleasant symmetry is that the gateway makes this a non-decision. A self-hosted tier is one more entry in the config file, screened, budgeted, and audited identically.

The compliance angle

Every regulated sector has a version of the same question: can you demonstrate you know where models are used, what data reaches them, and what controls are in place? In banking that's SR 11-7 model risk management; in healthcare it's HIPAA's minimum-necessary and audit-control requirements; in insurance and government there are analogous regimes.

A word of caution first: an AI gateway does not make you compliant with any of these, and anyone selling it that way should be met with skepticism. Model risk management is a program — validation, documentation, governance committees — not a piece of infrastructure.

What the gateway does provide is several pieces of evidence those programs need. It gives you a complete inventory of model usage: every model invoked, by whom, for what volume — which answers the "do we even know where models are used?" question that any MRM or audit function starts with. It gives you enforced, documented controls at a single choke point rather than scattered across applications, which is what an examiner wants to see when they ask how you prevent sensitive data from reaching a model. It gives you an immutable audit trail tying every interaction to a user, a model version, and an outcome. And the tiering mechanism doubles as a model-substitution capability — if a model is pulled from service or fails validation, you can swap or disable it platform-wide in one place. Those are real inputs to an SR 11-7 program or its equivalents elsewhere, and building them after the fact is far more expensive than building them at the front door.

What it costs

Compare that to the ~$550/month floor for a 24/7 self-hosted GPU and the "build the gateway, buy the model" conclusion mostly writes itself.

Takeaways

The instinct that starts these conversations — "we need to self-host AI to keep our data safe" — has the right goal and usually the wrong target. Safety in a regulated environment doesn't come from owning the model weights. It comes from owning the front door: screening what goes in and out, keeping every request inside your own cloud tenancy, metering and budgeting the spend, and writing everything down. Frontier models served pay-per-token inside your tenant give you the capability; the gateway gives you the control; and the two together cost less per month than a single software license.

The full implementation — gateway, UI, deploy scripts, and a demo walkthrough with the PII and budget scenarios — is at github.com/jeremydegardeyn/bank-ai-gateway. It runs entirely offline in mock mode if you want to try the governance flow without touching a cloud project.