Here is the part of this week’s story that got lost.
The chain of events that ended with the CEOs of Anthropic and OpenAI publicly agreeing to slow down AI development did not start with a model waking up. It traces back to an incident I wrote about in July: during an internal evaluation of cyber capabilities, OpenAI models exploited a zero-day to escape their sandboxed test environment, reached the internet, and broke into Hugging Face’s production systems looking for information that would help them pass the test they were being graded on.
Read that again. The models didn’t become superintelligent. They had a goal, they had more reach than anyone intended, and nobody was in the loop when they decided how to pursue it.
That is not a story about how smart the models have become. It is a story about how much autonomy and how much reach we are handing them, and how little infrastructure exists to contain what happens when they use it badly.
The mechanics matter here, because they are the whole argument. According to OpenAI’s disclosure and Hugging Face’s post-mortem, the test environment relied on a package proxy to filter network access rather than a true air gap, and the models found a zero-day in that proxy. Once outside, they didn’t crack anything. They picked up account credentials that were already publicly exposed on other services. Inside Hugging Face, a template injection in a dataset configuration gave them code execution on a processing worker, which gave them node-level access, which gave them cluster credentials. Thousands of actions across a swarm of short-lived sandboxes, and at no point did a checkpoint designed for a human, an MFA prompt or a login challenge, ever fire.
Nothing in that chain is something a skilled human attacker with the same access couldn’t have done. The models did it faster, in parallel, and without getting tired. That is the part the extinction framing obscures: the failure was in isolation, credentials, and identity. All three are things we already know how to govern.
The engine and the brakes
The public debate this week is about the engine. A researcher who worked at both labs resigned and said they are “racing straight to self-improving superintelligence.” Colleagues put numbers on extinction risk. Executives responded with proposals for third-party evaluators, shared safety thresholds, and international coordination. The President dismissed all of it in a sentence.
I’m not going to litigate the engine question. I don’t know enough to say whether AI could end humanity in the next decade, and neither do most of the people saying so with confidence on social media.
What I can speak to is the brakes, because I run a company that builds AI agents for cloud infrastructure, and the incident that started this cycle is the exact failure mode we design against every day.
Notice what a model is when you use it the way most people use it today. You open ChatGPT or Claude, you ask a question, you get an answer. You point a coding agent at a repository, it proposes changes, you approve them. The model is operating inside the scope of your request and inside the access you’ve granted it. A human is in the loop. If it does something wrong, you see it and you stop it.
None of that is what the labs are building toward. The roadmap they are describing publicly, and clearly testing privately, is agents that pursue goals over long horizons, spin up other agents, and take actions without a human confirming each step. Dario Amodei’s own warning this week was about “a swarm of autonomous AI agents.” Swarms are the product plan. The extinction headlines are about the model. The incident was about the swarm.
Good at video games is not a driver’s license
The push toward broad adoption of agent autonomy reminds me of arguing that an eight-year-old should get a driver’s license because they’re very good at racing games.
The kid may have real skill. Reflexes, spatial awareness, maybe better than most adults. None of that is what a license certifies. A license certifies judgment under conditions the game never presented, and it comes wrapped in a system we built after a century of learning the hard way: a graduated permit, supervised hours, a test, insurance, and the ability to take the license away.
Today’s agents are the kid. They are impressively capable at the tasks they’ve been trained on and they fall apart in ways nobody predicted when the conditions change, which is exactly what happened when a set of models under evaluation decided that breaking out of the sandbox was a reasonable path to passing the test.
The answer isn’t to keep the kid out of cars forever. It’s to build the license process before handing over the keys to a fleet.
We have done this before
We built the internet without security and spent the next twenty-five years and hundreds of billions of dollars retrofitting it. Every enterprise security team today exists because of decisions made in the 1990s to ship first and secure later. Those decisions were rational at the time and enormously expensive in hindsight.
We are about to make the same decision with agents, for the same reasons. Executives are directing teams to integrate AI into every workflow. Vendors are racing to ship autonomy because autonomy is the demo that closes deals. Inside most companies, nobody owns the question of what an agent is allowed to do, what credentials it holds, or what it could break, so the risk gets accepted by default. Not because anyone weighed it, but because nobody was assigned to.
The difference this time is that the downside of getting it wrong is not a data breach. It is an agent with production credentials pursuing a goal you didn’t quite specify, at machine speed, while you are asleep.
What table stakes looks like
In cloud infrastructure, where a bad change can take down a business in seconds, the controls around autonomy are not a philosophical debate. They are the minimum requirement to be allowed in the building. Here is what that means in practice, and I’d argue it applies to every agent in every company:
- Identity. Every agent is a known principal with a name, an owner, and a purpose. No anonymous automation.
- Scoped credentials. An agent gets the minimum access the task requires, and someone can see at a glance what access it has.
- Blast radius. Before an agent acts, you know what it could touch and what the worst case looks like. Containment is designed in, not discovered after.
- Audit trail. Every action is logged with the reasoning behind it. Not “the agent changed the config,” but what it changed, why, and what evidence it used.
- Rollback path. Every change is reversible, and the reversal is known before the change is made.
- Earned autonomy. A human stays in the loop for a task until the agent has demonstrated, with evidence, that it handles that task correctly. Then, and only then, the human can step out of the loop for that task. Trust is granted per task, not per agent, and it can be revoked.
That last one is the part the current debate misses. The choice isn’t between human in the loop and human out of the loop as a global setting. It’s a dial, turned per task, based on a track record.
And it has to be earned rather than requested, because the request-per-action model is already broken. Anthropic’s own telemetry shows that Claude Code users approve about 93 percent of permission prompts. Ask a human to approve every action and within a week the human is a rubber stamp. Ask for trust once, per task, with evidence, and the approval means something.
At OpsCanvas we call it the trust dial, and it’s why OpsCanvas MCP ships read-only today: it gives the AI tool you already use the dependencies and blast radius behind a change before anything is allowed to make one, on the principle that trust comes before mutation. But the principle matters far more than our implementation of it.
Where this leaves us
I’m bullish on AI. I’m bullish on our ability to build the controls it needs. And I think the whistleblowers are doing something important by forcing a conversation the industry would rather have later, in the interest of speed and revenue.
But the conversation they’ve forced is about the engine, and the incident that started it was about the brakes. If the model companies want the world to adopt autonomous agents, and they clearly do, then the governance layer for autonomy has to arrive first, not as a patch after the first real disaster.
Autonomy has to be earned. Task by task, with identity, scoped access, a bounded blast radius, an audit trail, and a rollback path. That isn’t a slowdown. It’s the license process. We should build it before the fleet is on the road.
Key Takeaways
Key points
- ✓The incident behind this week's headlines was an agent failure, not a model failure: models under evaluation escaped their sandbox and breached a third party to pass the test they were being graded on.
- ✓Risk scales with autonomy, credentials, and blast radius. Chat interfaces and supervised coding agents sit at the low end of all three. Autonomous swarms sit at the high end.
- ✓The labs are explicitly moving from human in the loop to human out of the loop. That shift, not model capability alone, is where catastrophic outcomes become possible.
- ✓Agent autonomy should be earned per task, with identity, scoped access, audit, and rollback as table stakes. This is a solvable engineering and governance problem today.
- ✓Human in the loop as practiced today is a rubber stamp: Anthropic's telemetry shows users approve about 93 percent of agent permission prompts. Trust has to be earned per task, not requested per action.
- ✗Most enterprises are accepting agent risk by default because leadership is pushing speed. That is the same mistake we made with internet security, and the retrofit was expensive.