Most of what I work on right now is new. Agent gateways, MCP servers, model risk controls, tokenized data for training, none of it has a ten-year-old playbook, and yet the expectation is that we build it at enterprise scale, secure it, and stand behind it in front of risk partners and examiners on the first pass. At the same time there is an enormous amount of AI-generated material about all of these topics, and I have noticed that most of us, myself included, have gotten very good at producing collections of it. A deep research report, a slide deck, a wiki page, a fifty-item recommendation list, each one takes about ten minutes now.
What I think is quietly getting worse is the ability to articulate any of it. I can generate a target-state architecture in an afternoon, but if someone in a review points at one box on the slide and asks why it is there, what it talks to, and what control it satisfies, the honest answer is often that I read it once and it seemed reasonable. That gap between what I can produce and what I can explain is the thing I have been trying to correct, and the most effective tool I have found is embarrassingly simple: after the AI does the research, I have it turn around and interview me.
I use this in three ways.
1. Sequencing and maturity, checked against what actually exists
The first is for large, multi-wave programs where the plan is going to be wrong at the far end no matter how carefully it is written. I start by having the AI do the deep research and build the sequencing: what posture we need at each wave, which tools land when, what the compliance story is at each step, and what "mature" would look like in a year or two. The important part is that I lean the research and the recommendations against the real material, meaning our actual repos, design docs, control definitions, runbooks, and wikis, rather than letting it reason from industry averages.
Then, instead of reading the output, I have it walk me through the sequence one step at a time and interview me about each one. For example, the plan says wave two introduces centralized policy enforcement at the gateway, so it asks me what is enforcing policy today, where that is documented, and whether the document matches the code. What I found doing this is that there is nearly always a three-way gap: what the data collection actually shows, what the documentation claims, and what the target state assumes. The interview is what surfaces it, because I have to answer out loud with a specific finding instead of nodding at a paragraph.
I also re-run the interview at the start of each maturity wave. The early waves are usually well defined and the later ones are fuzzy and a bit too ambitious, and re-interviewing with the findings from the wave just completed is how the later waves get dialed in over time rather than staying aspirational.
2. Closing a real knowledge gap so you can defend the box on the slide
The second use is for the topics where I have a genuine gap, not a vague one. There, I have the AI research the space, make recommendations grounded in industry practice, and then align those recommendations to our stack specifically, because a generic best practice that does not account for the identity provider, the data platform, or the change process we actually have is not a recommendation I can act on.
Then I have it coach and quiz me on the target state until I can explain it without the notes. For me this has been the most valuable of the three, because a solid architectural recommendation is not the same thing as a good idea mentioned in a meeting, a deck, or an article I forwarded. It has to be something I can hold up under questioning. If a partner points at the box labeled "policy decision point" and asks what happens when it is unavailable, I should be able to answer that, and the quizzing is what gets me there. It also tends to expose the places where I was repeating a phrase I did not really understand, which is a useful and humbling thing to learn in private.
3. Interview and meeting prep
The third is the most literal one. Before an interview or a high-stakes meeting, I give the AI the job description or the agenda, lean it against my actual qualifications and experience, and have it interview me in a style that is much more scrutinous than any real panel would be. It pushes on every claim, asks for the metric, asks what I personally did versus what the team did, and does not let a vague answer pass.
It is not a free-flowing conversation, and it does not replace one. What it does is drill the key points until they are compact and consistent, and, more usefully, it forces a thesis to emerge. After a couple of rounds I usually have a north star, a one-sentence version of what I think and why, that I did not have when I started, and the real conversation tends to feel easier by comparison.
The common thread
All three come down to the same move. The AI is good at producing material and I am not going to out-produce it, so I stopped trying. What I can do is make sure that whatever it produced, I can explain, defend, and correct against what actually exists. Having it interview me is how I check that, and it has done more for the quality of my recommendations than any amount of additional reading.
0 Comments
Leave a Comment