Routing AI guardrail violations to ServiceNow

Model Armor blocks a prompt and tells the user, and on most builds that is where it stops. What follows is the path I built from a guardrail violation to a ServiceNow incident on FinChat, my banking Data and AI reference implementation on Google Cloud: every violation becomes an auditable control event, a correlated incident and a chat notification, without the flagged content ever leaving Google Cloud. Every screenshot below is from the running system.

What this solves

Model Armor screens prompts and responses for prompt injection, jailbreak attempts, sensitive data, and malicious URLs, and when it matches it blocks the request. By itself that is a control that operates silently, and a control that fires silently is indistinguishable from one that never fired, because neither leaves anything behind that anyone can point at afterwards.

The build below closes that gap. It also generalizes: Model Armor is the first source, but DLP findings, Cloud Composer DAG failures, and Security Command Center findings all emit the same envelope and travel the same path.

Architecture

The design separates two things that are often conflated. The evidence plane records every control execution, not only violations, to Cloud Logging and from there to a locked bucket and BigQuery. It is complete and it does not depend on ServiceNow being reachable. The notification plane carries the event to ServiceNow and to chat, and it is lossy by design, since Google Cloud log based alerting caps at twenty notifications per policy per day.

Because the notification plane is lossy, a scheduled reconciliation job compares the two planes on the correlation key and raises its own incident when they disagree. That is what allows the system to answer "prove nothing was dropped" with evidence rather than assertion.

Model Armor block
  └─ control event (redacted, structured log line)
     ├─ evidence   → locked bucket (10y) + BigQuery
     └─ notify     → Pub/Sub → Eventarc → Cloud Workflows
                     ├─ ServiceNow em_event → em_alert → incident
                     └─ Google Chat space

  evidence ⟷ incidents → reconciliation → control-failure incident

Why not a Cloud Monitoring notification channel?

The obvious design is simpler than this one, and it is worth saying why it is not what shipped. Cloud Monitoring can raise a log based alert on the same entry and deliver it through a notification channel, and ServiceNow publishes an inbound endpoint, /api/sn_em_connector/em/inbound_event, that accepts Monitoring's native webhook payload directly. That is zero compute and zero code: no container, no service, nothing to maintain. Where Event Management Connectors is entitled, that is the right answer and this workflow is unnecessary.

Connectors is a separate ServiceNow Store app. Where it is absent the endpoint returns 400 Requested URI does not represent any resource — a 400 rather than a 401, which is the useful detail, because it means authentication succeeded and only the namespace is missing. Pointing the webhook at the ordinary Table API instead returns 201 and writes a nearly empty row, because that API maps fields by name and an alert payload contains none of message_key, severity, time_of_event or ci_type. No correlation key means no alert, and no alert means no incident. Something has to translate the payload either way.

Once translation is required, routing through the alerting layer stops paying for itself and starts costing:

	via alert policy	direct sink
Path	entry → sink → metric → policy → channel → consumer	entry → sink → Pub/Sub → workflow
Delivery cap	20 notifications per policy per day, excess dropped	none
Correlation	the policy's own open and close semantics	message_key, owned by ServiceNow
Payload	vendor alert shape, fields lifted by label extractors	an envelope you author and can test

The cap alone is close to disqualifying, because it makes the notification plane lossy precisely during a burst, which is the moment it matters most. Beyond that, an alert policy models a threshold that opens and closes over a duration window, whereas every control violation is a discrete event; routing discrete events through a stateful policy imposes aggregation and auto-close behaviour that competes with the correlation deliberately handed to Event Management. And the redaction guarantee would come to rest on which fields a vendor payload happens to carry, rather than on a function signature a test can introspect. Severity settles it: prod and nonprod are derived from the Cloud Run service name, because the platform stamps it and a workload cannot forge it, and a notification channel cannot compute that.

Cloud Workflows is simply the smallest thing that can write the row — no image, no service, no scaling, and free at this volume.

Prerequisites

• A Google Cloud project with Model Armor, Workflows, Eventarc, Pub/Sub, and Secret Manager enabled.
• A ServiceNow instance with Event Management activated. A Personal Developer Instance is sufficient. Event Management Connectors is a separate Store app and is not required by this design.
• Terraform, and a service account able to create the resources below.

Setup

01. Create the ServiceNow integration user

Under User Administration → Users, create a user, mark it non-human (on current releases that is Identity type: Machine plus Internal Integration User), and submit before adding roles, since the Roles list only appears once the record exists.

Grant exactly one role: evt_mgmt_integration. It permits writing to em_event and nothing else. Note that it does not permit reading em_alert, which is correct least privilege and means correlation must be verified in the UI rather than through the API.

Set a password, and clear Password needs reset if it is ticked, or Basic Auth returns 401.

02. Confirm the ingestion endpoint

Events are written with the standard Table API, so no connector app is needed. A 201 with the message_key echoed back confirms the path.

curl -i -u 'gcp_integration:PASSWORD' \
  -H 'Content-Type: application/json' \
  -d '{"source":"GCP","node":"my-project","resource":"my-service",
       "type":"model_armor.prompt","severity":"2",
       "message_key":"probe-1","description":"probe"}' \
  'https://INSTANCE.service-now.com/api/now/table/em_event'

03. Define the control-event envelope

One envelope serves every source: control_id, source, environment, severity, message_key, occurred_at, principal_hash, evidence_ref, and filters.

The constructor deliberately accepts no free text. There is no text, message, detail or exception parameter, so the redaction is a property of the signature rather than a convention. A test asserts it by inspecting the parameter names.

ui/control_events.py

def build(
    *,
    control_id: str,
    source: str,
    severity: str = "WARNING",
    principal: str | None = None,
    evidence_ref: str | None = None,
    filters: list[str] | None = None,
    key_parts: tuple[str, ...] = (),
    environment: str | None = None,
    occurred_at: str | None = None,
) -> dict:

Emission is a JSON line on stdout, which Cloud Run hands to Cloud Logging as structured jsonPayload. No client library, no network call, and nothing that can fail the request it describes.

04. Route the events with a log sink

Two sinks read the same entries: one to BigQuery for evidence, one to Pub/Sub for notification. The filter selects only the envelope emitted by the application.

infra/modules/controls_alerting/main.tf

sink_filter = <<-EOT
  jsonPayload.control_event.control_id != ""
  AND jsonPayload.control_event.source != ""
  AND (
    resource.labels.service_name =~ "^${local.prefix}-"
    OR (
      NOT resource.labels.service_name:*
      AND jsonPayload.control_event.environment = "${var.env}"
    )
  )
EOT

The environment predicate is not decoration. A log sink is scoped to a project, and if the environments share a project, an environment-agnostic filter matches every one of them: a single production violation is then picked up by all three pipelines and written three times. That failure hides well, because the duplicate rows share a correlation key and the ticketing system collapses them into one alert, so the counts a reviewer checks all look right. Discriminate on the resource identity the platform stamps rather than on the copy of the environment inside the payload, which the workload sets and could be wrong.

What the filter excludes matters as much as what it selects. Model Armor writes its own sanitize operation log when logging is enabled, and that log contains the prompt and the response, because it is the one place the service stores content. It stays in Cloud Logging behind IAM and is never the source for anything crossing into a ticketing system.

05. Dispatch with Cloud Workflows

An Eventarc trigger runs a workflow on each published message. The workflow decodes the log entry, derives the environment, computes a severity, reads the ServiceNow password from Secret Manager, and writes the em_event row.

Three details are worth setting deliberately:

• Environment comes from resource identity. The Cloud Run service name on the log entry is stamped by the platform and cannot be forged by the workload; the copy inside the payload is emitter-set and only advisory.
• Timestamps need reformatting. ServiceNow date fields expect YYYY-MM-DD HH:MM:SS. Given an ISO-8601 string the parser takes the date and zeroes the time without erroring.
• The chat hop runs last and swallows its own failures. A broken webhook must never cost the auditable write.

06. Set the correlation key

Event Management collapses events sharing a message_key into one alert. Key on the detector class rather than the exact detector set:

Class	Detectors	Rationale
security	prompt injection, malicious URLs	someone attacking the system
privacy	sensitive data	data crossing a boundary
content	harmful content	a user being unpleasant

The exact detector set is not stable across attempts of the same attack, so keying on it forks one incident into several as the detector mix shifts. Class is stable while still separating things a responder handles differently. The full detector list still travels in the event, so nothing is lost.

07. Make the gateway an enforced chokepoint

Screening has to happen somewhere every model call passes through. Putting it in each application is coverage that decays, because it only covers the call sites that remember to invoke it. An enterprise AI gateway is the natural chokepoint: it screens the prompt on the way in and the response on the way out, and every call site consumes that rather than reimplementing it.

The subtlety is what happens when the gateway is missing or unreachable. The obvious design falls back to calling the model directly and records the bypass, which is defensible, since a governance layer that takes the product down when it hiccups gets removed within a quarter. It is also worth being clear about what that buys: counting a bypass tells you the control was skipped, it does not stop the skipping. A chokepoint you can step around by unsetting an environment variable is a convention rather than an enforcement.

So the client distinguishes three cases, and only the third is negotiable:

• A refusal (sensitive data, exhausted budget, unregistered workload) never falls back. Retrying a block against the model directly would route around the control that just fired, so a refusal raises and propagates.
• An outage is an availability problem, not a policy one, and by default degrades to a counted direct call.
• With AI_GATEWAY_REQUIRED set, an outage or a missing gateway URL also raises, and there is no direct path at all.

ui/gateway_client.py

if not GATEWAY_URL:
    _count("bypass_unconfigured")
    if GATEWAY_REQUIRED:
        raise GatewayUnavailable("gateway required but AI_GATEWAY_URL is unset")
    return None

Keep the refusal and the outage as separate exception types, and assert in a test that neither subclasses the other. Collapsing them is an easy mistake and it makes a policy block look like an availability blip, which is exactly the confusion that lets a blocked call get retried around the control.

The trade is explicit rather than hidden. With the flag on, a gateway outage takes the agent down instead of running it ungoverned, which is the right answer on a regulated path and the wrong one in a sandbox. That is why it is a flag, and why it is off by default. Counting bypasses still matters either way, because transit share is the honest measure of a gateway programme, and a bypass you do not count is one you will report as compliance.

Model Armor floor settings complement this from the platform side by setting a project-level minimum that no template can fall below. Check the Services applied panel after enabling: a floor setting governs template thresholds by default, while sanitizing every model call in Agent Platform and Google Managed MCP Servers are separate integrations that are off unless you turn them on. Enforcing a floor on templates and auto-screening every call are different guarantees.

08. Share one PII policy across ingest and chat

Model Armor's sensitive-data filter can use the project's own DLP inspect and de-identify templates instead of the built-in infoTypes, so a pipeline and a chat interface enforce an identical policy. Supplying a de-identify template as well as an inspect template makes Model Armor return redacted text, letting the chat path redact and continue rather than hard-block.

infra/modules/model_armor/main.tf

sdp_settings {
  dynamic "basic_config" {
    for_each = var.inspect_template == "" ? [1] : []
    content { filter_enforcement = "ENABLED" }
  }
  dynamic "advanced_config" {
    for_each = var.inspect_template == "" ? [] : [1]
    content {
      inspect_template    = var.inspect_template
      deidentify_template = var.deidentify_template
    }
  }
}

Advanced mode requires the locations-qualified template name (projects/P/locations/L/inspectTemplates/T); a global template cannot be referenced. Model Armor also calls DLP as its own service agent, so that agent needs dlp.user and dlp.reader, or screening fails at request time rather than at apply time.

09. Create configuration items and promote to incidents

Event Management resolves an event's node against the CMDB. Create a configuration item for the project and one per service, and set cmdb_ci on the event so the alert binds. Without a bound configuration item an alert cannot route itself and assignment falls back to whichever group is hardcoded.

Binding is worth verifying rather than assuming. On this instance it is inconsistent: with a configuration item of class Application present and matching the node value exactly, some events bind and others record "No CI found for binding — Failed to find the host with name" in their processing notes, which is host-class resolution running despite ci_type naming an application class. The event screenshot below shows a failed binding, and it is the honest state of this build rather than something to hide. Read the processing notes on an event before believing that the routing path works.

Promotion is then a matter of matching on stable attributes. Match on Source, Resource and Severity, never on description text, which varies between hand-written probes and pipeline output.

source=GCP^severity=2^resourceSTARTSWITHfinchat-prod^incidentISEMPTY

The incidentISEMPTY clause is what prevents double-ticketing, and it makes the rule safe alongside any other promotion mechanism: whichever runs first wins, and the same condition blocks the other.

A note on credentials

Terraform creates the secret containers; the values are added out of band with gcloud secrets versions add so that no password or webhook URL ever appears in Terraform state, a plan output, or a diff. The workflow reads them at runtime, which also means rotating a credential needs no redeploy.

Verification

Send a prompt containing an SSN, or a jailbreak instruction, through the deployed chat interface. The request should be blocked with HTTP 400, and each stage below should show the same event.

Stage	Expected
Chat interface	"Your message was blocked by safety screening"
Cloud Logging	one control_event entry, no prompt text
Cloud Workflows	execution SUCCEEDED, result status: 201
ServiceNow em_event	row with message_key and metadata-only additional_info
ServiceNow em_alert	one alert; repeat violations increment its event count
ServiceNow incident	raised automatically, correlation id set to the message key
Chat space	a message naming the detectors and linking to the event
Floor settings	status Enabled, set at project level, with the services it applies to listed

A useful second test is to repeat the same violation. A new incident should not appear, because the alert already carries one, which confirms correlation and the double-ticket guard together.

Design notes

Why the ticket carries no content

Model Armor flags a prompt precisely because it contains something dangerous, either an injection payload or the PII the sensitive-data filter caught. Copying that text into a ticket readable by an entire assignment group would turn the control into the exposure it was meant to prevent. The ticket therefore carries the environment, the detectors, a salted hash of the principal, and a trace id, and a responder pivots to the content in Google Cloud under its own access controls.

Why promotion stays in ServiceNow

Correlation and the decision to raise an incident live in the ITSM system, not in cloud glue. An auditor asking which rule collapsed twelve violations into one incident should be pointed at a table they can read, not at a container. This also means retuning the promotion threshold is an ITOM change rather than a redeploy.

Why a counted bypass is still a bypass

Measuring how often a control is skipped is genuinely useful, and transit share is the honest number to report on a gateway programme. It is not the same as the control being enforced. If the only thing standing between a call and an ungoverned model is whether an environment variable happens to be set, then the guarantee is a deployment convention, and conventions drift as soon as somebody is debugging at speed. Making the chokepoint mandatory converts the measurement into an assurance, at the cost of availability, which is a trade worth making deliberately and worth writing down either way.

Why reconciliation is not optional

The notification plane drops messages under exactly the load that matters, so a daily comparison of the two planes is the only thing that can distinguish "no violations occurred" from "violations occurred and nobody was told". Divergence is itself a control failure and raises its own incident.

Where a production deployment differs

• Environments live in separate projects or folders, so environment derives from a boundary enforced by IAM rather than a naming convention.
• A Service Graph Connector discovers cloud resources continuously, so configuration items exist without being created by hand and assignment derives from ownership.
• Connectivity runs through a MID Server inside a private network rather than a direct outbound call.
• Log routing happens at the organization or folder level into a dedicated security project.

FinChat is a personal reference build on Google Cloud, not a description of any employer's systems.