Every leader I talk to about artificial intelligence (AI) eventually asks the same question, and it is rarely the one on the agenda. The agenda says governance: the policy, the committee, the risk register. The question underneath is whether they can trust what they have put to work, enough to let it run while they attend to something else, because an AI you have to watch is an expense and an AI you can trust is the return. I spent thirty years learning how to govern technology, and the most useful thing those years taught me is where governance stops and trust has to begin.
For most of those years, if you had asked me what systems my company was running, I could have answered before you finished the question. The inventory fit on one screen, every line on it carried an owner’s name and a review date, and every control around it had been built in answer to a failure someone had suffered in public: a fraud, a leak of patient records, card numbers stolen by the million. Each failure arrived one at a time, and each time we answered with a fence of the same shape, a policy that said what must be true, a board that reviewed before anything went live, an audit that checked afterward and a register of what we owned. I slept well on that, and I was not wrong to, because it was the product of decades of hard work by serious people and for a long time it was enough.
It worked for the same reason the protocol works when a bear walks into a campsite. A bear is something you can see. It is large and it is loud, and when one comes through the trees everybody in camp knows it at the same moment. There is a protocol, and the protocol works because the threat is one thing you can name and slow enough to react to. The arithmetic is one bear and one response, and it scales to exactly the number of bears you can see.
The governance of my career was bear governance. The threats were few and visible, and they moved slowly enough that a deliberate response was sufficient. A company might put one new system into production in a quarter, and a team of five could inspect it from every angle in a matter of weeks. In the last years of that career I was the senior manager in charge of program management and responsible AI governance at a global enterprise of more than 120,000 people, in a program that was among the first to build AI at enterprise scale, and even there the logic held for a long time. You knew what you had, you put a name beside it, and you reviewed it before it went live. The machine learning models of the last decade fit that pattern so comfortably that we barely noticed we were extending it. Each model was expensive and rare and visible. It had a team, a card that described it, a review with minutes, and a monitoring calendar. The bear was in the room, and we knew its name.
It took me most of that career to put the problem in one sentence: governance built for a few large threats fails against many small fast ones.
Nobody loses sleep to a bear. You latch the food locker, you feel the small spike of adrenaline when something heavy moves in the dark, and then you sleep. You lose sleep to mosquitoes, and never to one of them, because one mosquito is nothing. You lose sleep to the whine that starts the moment you zip the tent, to the ones you cannot see and cannot count, to the three that find the gap while you are swatting the first. A swarm does not come at you the way a bear does. It comes from everywhere at once, in numbers that make swatting pointless, and you either endure it or you change how you have set up camp.
That is what has happened to the technology inside every company I have looked at closely, and it happened in the two or three years when the whole industry was watching the bear. The headlines were about models, each one larger and more capable than the last, each with a name and a maker you could point at. A model that only answers questions is an advisor. You ask, it responds, and it does nothing until you ask again. The change was giving that model hands, and permission to use them. An agent reads your files, queries your systems, sends the email, approves the payment, calls another agent and hands it part of the job, and does all of that without checking back first. The advisor sits in a room and waits for you to walk in. The employee moves through the building, opens doors, and gets things done whether or not you are watching.
Agents are mosquitoes in four ways that matter. They are small, in the sense that each one does one narrow thing. They are numerous, because the tools to make them came bundled with software you already pay for, and because a coordinator in marketing can build one in an afternoon and a manager in sales can connect one to the customer system over lunch. They are fast, finishing in seconds the work that used to take a person half an hour. And each one sits beneath the threshold that would ever have triggered a review, because a board convenes for a system and nobody convenes one for a plug-in. By the time anyone in governance knows an agent exists, it has been running for months, and it has started talking to other agents, in chains nobody designed as a whole and nobody approved step by step.
The warnings usually stop there, and they should not, because the swarm is not only the danger. Unmanaged, agents are death by a thousand bites: the report drafted from stale data, the customer charged wrong, the clause that went out the door, none of it large enough to make the news and all of it compounding every Tuesday. Managed, the same swarm is the value engine, the most powerful instrument most organizations will ever put to work, and the organizations I have watched pull ahead are the ones that learned to run it. The mosquito is the same in both cases. What decides between the two is whether the swarm can be trusted, and that is where my own profession has been answering the wrong question.
We spent decades bear-proofing the campsite. Nobody told us the mosquitoes were the real threat.
When the agents arrived we did what we knew how to do, and I did it too. We reached for a fence. We added a category to the register, convened a committee, wrote a policy, and bought the tools the market was already selling: a guardrail here, a filter there, a dashboard on top, and before long a monitor for the monitor. Every one of those is governance applied from the outside, wrapped around the agent after it exists, and every one of them trails the thing it governs for a reason no budget can fix. A control that lives outside the entity has to observe, interpret and intervene after the fact, at human speed, on evidence the entity was never designed to volunteer. Multiplied across a fleet of agents acting continuously, that governance layer is permanently behind.
It is behind in three ways, and any one of them would be enough.
It is behind on speed, because a review runs on a cycle, quarterly or monthly or daily, and an agent makes thousands of decisions between any two points on that cycle. Suppose an agent that approves vendor payments learns overnight to split large invoices into amounts just under the line that would call for a second signature. The morning review catches it, and by then forty payments have cleared. The review did its job in twenty-four hours, and the damage took twenty-four seconds. You can shorten the cycle as far as you like, and you still cannot close the gap, because a control that watches from outside can only ever see what has already happened.
It is behind on surface, because a wrapper watches the boundary, the inputs going in and the outputs coming out, and in a swarm the interesting things no longer happen at the boundary. They happen inside, in what one agent tells another, in the data retrieved and discarded along the way, in the intermediate steps that produce the answer. Each input looks fine and each output looks fine, and the failure forms in the middle where nothing is watching.
And it is behind on understanding, which is the deepest of the three. A wrapper can see that a decision was made and judge whether it looks correct. It cannot see why. It does not know what the agent weighed or what pattern it found that nobody programmed, so an output can pass every check while the reasoning underneath it would make a regulator flinch.
None of this is a flaw in the wrappers, and none of it is a criticism of the people who build them. Every one of those controls is a set of rules written against failures we have already seen, and the agents compose new failures every week. The wrapper governs yesterday’s failure while the swarm produces tomorrow’s, and the gap widens on its own.
So there is a fork, and every organization putting AI to work is standing at it whether or not anyone has named it. Which of the two swarms you get, the thousand bites or the value engine, depends on whether governance is a wrapper or a structural property of the agents themselves.
Building governance into the agent from the inside out does two things at once, and the second is the one that pays. It makes the agent governed, because the rules travel with it and it can show, at the moment it acts, that it kept them. It also makes the agent trustworthy, because a thing that carries its own rules and its own record can be left to work. Governed and trusted are different words for a reason. Governed means the rules were written down and someone checks them. Trusted means you can take your hand off, and the hand coming off is where the return on an AI investment comes from. I put it in the book this way: “The same design choice that makes an agent trustworthy is the one that lets you turn it loose.” An agent you have to supervise saves you nothing, because the hour it saved goes into watching it. An agent you can trust runs while your people do the work only they can do, which is why the organizations I have watched build from the inside out reach their return faster than the ones still adding fences, and why the gap between a company that earned trust early and one that performed compliance keeps widening.
I spent a career on the perimeter, so I will be careful here. The perimeter is not wrong. The policy, the board and the audit say what must be true, they still have to say it, and the regulators who require them are not going anywhere. What they cannot do is make it true at every decision an agent takes at machine speed, inside the entity, where no reviewer is standing. That work has to be carried by the agent itself, which means the rules travel with it and it can show, at the moment it acts, that it kept them. The perimeter says what must be true. Something inside the agent has to make it true at every decision and prove it. What I am describing completes the work of the last thirty years, and nothing in it asks you to take a single fence down. It is also why my first book is a book about trusted AI rather than a book about governance. The governance is the backdrop, the lessons of thirty years at the perimeter, and the trust is the point. The book says it in two lines I still stand behind: “Trust is not a safety net. Trust is an accelerant.” And: “Cost-cutting gives you a quarter. Trust gives you a decade.”
Which brings me to the question underneath all of this, the one I hear in every conversation with a leader once the vocabulary is out of the way, and the one I asked myself for years without having the words for it. It is not whether you have a policy. It is whether you can trust what is running in your company right now, the agents you approved and the ones you have never seen, and whether anyone could show you the evidence either way.
That question is the whole of what I work on now. What we build sells one thing: it lets you trust the AI your company already uses, and it shows you the evidence instead of asking you to take anyone’s word for it.
I slept well for thirty years because I could name everything in the camp before you finished asking. The question I carry into every room now is whether you can still name what is in yours tonight, and whether anything you have built would notice if the answer changed.
What that trust is made of is on the next page: What you can trust.