I have been working through how to stack governance layers on a data and AI platform and tie them together, and I found it genuinely daunting. Not because any single layer is hard, but because you end up with seven or eight of them, added at different times for different reasons, all answering some version of "is this allowed". Keeping them straight in your head is the actual work.
So here is the simplest breakdown I have arrived at, in case it helps anyone else in the same position.
What is actually in the way
| Layer | Governs | Binds at | Owned by |
|---|---|---|---|
| IAM and roles | which identity may do what | every API call | platform engineering |
| Column tags and masking | which columns return values, blanks, or nothing | every query | steward classifies, product owner grants |
| Curated views | what analytics can see at all | the view definition | the data model owner |
| Catalog and access requests | discovery, quality, how access is asked for | when data is found | data steward |
| Agent instructions | what an assistant may say or attempt | inference | AI governance |
| AI gateway | model choice, spend limits, content screening | every model call | platform engineering |
| CI policy checks | ownership, tool scope, infrastructure posture | merge | platform engineering, reviewed by risk |
Most write-ups stop at the first two columns. The last two are the ones that made this manageable for me.
Two questions sort everything
Where does it bind? A rule in CI only applies to changes that go through CI. A rule in IAM applies to everyone, including someone clicking around in a console who never opened a pull request. A rule written into an assistant's instructions applies to nobody: it asks.
That gives you an ordering. Put a rule as close to the data as it will go, and treat anything further upstream as an early warning rather than a control. It also gives you a reporting discipline. A green check from a CI layer is not evidence that a control is operating, and calling it one ends the conversation about the gap instead of closing it.
Who owns it? Not documentation, a design constraint. Classification and granting are usually different people: a steward decides a column is sensitive, a product owner decides who reads it unmasked. That separation is correct, but if you have not written it down, you get access requests sitting with someone who has no button to press.
A layer with no owner drifts, because nobody reviews it. A layer with two owners diverges, and the version that matters is whichever one is enforced rather than whichever one is documented.
Four things I would apply
Prefer structure to enforcement. The cheapest layer above is curated views, because it is not a check. If a column is not in the view, there is no rule to evaluate and nothing to misconfigure. Masking costs more than leaving something out; an access rule costs more than masking. Reserve enforcement for what genuinely has to stay reachable.
Treat assistant instructions as advice. Telling a model not to do something does improve its answers and is worth maintaining properly. It will still do the thing sometimes, especially a managed model you do not run. Anywhere the wrong answer is expensive, pair the instruction with a plain check on what actually came back. This is the one place a requirement genuinely belongs in two layers at once.
Keep entitlements in code. Anything granted by hand is invisible to any rule you write about your pipeline, and your repository quietly stops describing reality. Every conclusion you draw from reading it after that is unsound.
Say which layer refused, and who can change it. This is the one I underrated. With seven layers, any failure has seven plausible causes, and people will guess until something tells them. An error that states a cause it did not actually observe is worse than one admitting uncertainty, because it sends someone confidently in the wrong direction. If it points them at an owner who cannot grant what is missing, that is an ownership problem showing up as a message problem.
Is it worth it
Three different answers, and the split is the useful part.
Layers the platform enforces are worth it without qualification. IAM, column security and simply leaving data out of a view hold no matter what calls them, cost nothing to keep once set, and fail safely.
Layers enforced in your pipeline are worth it with the caveat said out loud. They catch real problems early and cheaply, and they only bind people using the pipeline. Treat them as a fast first opinion and migrate the rules that deserve it down into platform settings over time.
The instruction layer is worth having and is not a control. It improves the typical answer and cannot be trusted for the expensive one.
The cost nobody puts in the diagram is attribution. Every layer you add is another place a failure can appear to have come from, so unless you spend part of each step on saying which layer refused and who owns it, defense in depth turns into diagnosis in depth. You pay that either up front when designing your errors, or later in evenings.
I would still build all of it. I would build it in a different order, starting with knowing which layer is talking, rather than adding that once the layers made it necessary.
0 Comments
Leave a Comment