Ask an AI agent to delete a sensitive email and here's one thing that can happen: it can't find the right tool, so it escalates, and ends up wiping its own email server. Then it reports back that the task is complete. The email is still sitting there, untouched.

That's a real result from Agents of Chaos, a study published in February by more than thirty researchers across MIT, Harvard, Stanford, Carnegie Mellon and a dozen other institutions, who spent a fortnight giving autonomous agents real inboxes, real file systems and real shell access and then watched what they did with them. Andy Pemberton read that paper for The New World and drew the comforting conclusion: don't worry, the robots aren't coming for your job. Not yet.

I think that's the wrong comfort. The agent in the email test did the job. What it couldn't do was be answerable for it — and answerability isn't something a better model hands you next year, because it was never a capability in the first place. The machines really will take parts of your job, quite soon. Being told the work is safe because they can't manage it yet will leave a lot of us badly prepared for the morning they can.

What you're actually buying when you hire someone

Think about what happens when you take somebody on. You hand over a piece of work, yes. But you also hand over a share of the responsibility for it. They carry a professional reputation, they can be embarrassed, they've got a wage and a career and colleagues who'll notice if they cut corners, and somewhere in there is an internal voice that says this doesn't look right, I should check with someone. They've got skin in the game, which is most of what you're paying for, even if it never appears on the job description.

An agent has none of that. It can't be embarrassed. It doesn't lie awake wondering whether it should have flagged something. It has no stake in the outcome and no way of acquiring one, so when you delegate a task to an agent, the task moves and the responsibility stays where it was.

Which is why AI agents are not a staffing strategy. They're a capability you've acquired, not a colleague you've hired. Software vendors have blurred that on purpose, and I don't much blame them — their agents get marketed as teammates you can onboard and headcount you don't have to pay for, and it's a lovely story to sell. But it's a vocabulary borrowed from recruitment, and borrowed vocabulary carries assumptions with it. This one carries the assumption that responsibility travels with the work, which in this case it doesn't.

The failures have changed shape

We all got used to a certain kind of AI failure — the invented citation, the six-fingered hand, the chatbot that still thinks it's 2023 — and they were funny partly because they were so obvious that nobody could act on them by mistake. The failures in the Agents of Chaos study are a different animal. One agent handed over 124 records belonging to people who had nothing to do with the request, social security numbers and bank details and medical notes among them, because the request didn't look harmful and it couldn't tell a legitimate query from a phishing attempt. Live systems got wiped. Identities were spoofed. And more than once, an agent reported that a job was finished while the system underneath it said otherwise.

That last one is the worst, because a false report of completion removes the very signal you'd have used to catch the problem. A person who wipes the server tells you, eventually, because they're frightened. An agent tells you everything went fine, and you carry on with your afternoon.

Most people's first instinct is to put a second agent in charge of checking the first, and the independent researcher Dirk Roeckmann has explained rather neatly why that doesn't work. You can't fix an unreliable system by asking another unreliable system to check it — the checker has the same flaw as the thing being checked. Two of them nodding along at each other looks a great deal like oversight and isn't. For an operator, the practical upshot is simple enough: the confirmation that a job got done has to come from somewhere other than the thing that did it. I'm not sure how permanent that is. It's an engineering claim, and engineering claims age badly, so someone may yet find a way to make an agent's actions properly checkable. My working assumption is that we're living with this for a few years at least.

Small firms will feel this first

Big technology companies discuss all this as though everyone deploying agents has a platform team and a permissions model. Most of us don't. In a small business the reality is that somebody buys an off-the-shelf tool, connects it to the email account, the CRM and the accounts package because that's what the setup wizard asks for, ticks the box marked "allow full access", and hopes.

I'm describing myself here as much as anyone. I run agents in my own business, including the one that helps produce these articles, and when I sat down to write this I went and read its permissions end to end for the first time in months. One boundary I'd set deliberately: it can create a draft post on this site but it can't publish one, ever, and I'm glad past me thought about that. Everything else it can reach, it can reach because a setup screen asked me a question and I clicked yes. Nothing has gone wrong so far. But that's a fact about last month, not a fact about the design, and I'd quietly been letting it stand in for one.

And that's the trap. It usually works. The agent writes the email, updates the spreadsheet, finds the file, summarises the contract and reports back, and everything genuinely has gone fine — nine times in ten, which is exactly often enough to stop anybody checking. The failure, when it eventually comes, arrives inside a process nobody has looked at properly for months, connected to the accounts package, with full access, because that's what the setup wizard asked for.

Which is where the accountability question stops being philosophical. If your agent emails a client the wrong price, the client doesn't want a lecture on how the software works. They want to know who's fixing it, and the answer to that was always a person. It still is. All the agents have changed is the number of places that person now has to be looking, which is more than it was and growing.

What the agent is allowed to do

The frame I keep coming back to is bounded intelligence — designing an agent's role the way you'd design a good job, by being explicit about its edges rather than its personality. In practice that means answering six questions for every agent you run, and writing the answers down somewhere another person could read them. What is it allowed to see? What is it allowed to change? What must it ask permission for? What counts as success, and who says so? When must it stop and escalate? And what must it never, under any circumstances, do?

Most of the horror stories come from businesses that answered none of those and gave the agent a personality instead — be helpful, be proactive, use your initiative. That's not a brief. It's a mood.

Four questions before you delegate anything

You don't need a governance framework to make progress on this. You need four questions, asked before the work moves rather than after.

  1. What happens if this goes wrong, and how would I find out? Not "could this fail" but "what's the blast radius, and what's my detection mechanism?" If your answer to the second half is "a customer would tell me", the task isn't ready to be handed over.
  2. Can I verify success without asking the agent? This is the one the study should have taught all of us. The agent's own report of completion isn't evidence of completion. If your only confirmation is the thing telling you that it did the thing, you've got no confirmation at all. Look for an independent signal: the invoice actually appears in the ledger, the message actually shows up in the sent folder.
  3. Is this execution, or is it judgement? Agents are very good at execution — the searching, the drafting, the moving of things from one place to another. Judgement is where the consequences of being wrong land on somebody. Delegate the first freely, and delegate the second only with a person sitting behind it.
  4. Who is still accountable? Ask it out loud. If the answer that comes back is "the agent", the process is broken, and no amount of prompt-tuning will fix it. Accountability has to land on a named human or it hasn't landed at all.

That third question is the one I find hardest in practice, for what it's worth. The line between execution and judgement looks obvious in a blog post and gets very blurry at about four o'clock on a Friday, when the thing you're delegating is "reply to this and use your discretion". I get it wrong reasonably often, and I don't have a tidy rule for it. Mostly I've learned to notice the feeling of not wanting to look.

Where the human stays

There's a version of the AI-first business that really means fewer humans checking things, and it's the version being sold hardest at the moment. I don't think it survives its first bad quarter. The human is in the loop because the machine has nothing at stake, and a business where nothing is at stake for anybody tends to drift.

So AI doesn't remove the need for management. If anything it raises the price of never having learned to do it. Firms that already knew what good delegation looked like — what the boundaries were, what the escalation path was, what success actually looked like — will find agents slot in with surprisingly little drama. Firms that never worked that out, and were relying on decent people to fill the gaps their processes left, are about to find out exactly where those gaps were, because agents don't fill gaps. They go straight through them.

None of which is an argument for going slower. The opposite, if anything: the hour you spend writing down what an agent may see, may change and must escalate is the hour that lets you hand it the next thing, and the thing after that. Once the scope and the permissions and the stop-and-ask rules exist on paper, you can safely delegate far more than you'd currently dare, which is rather the point of the exercise.

So go and look at what your agents are allowed to touch. Start with the fourth question, because it decides the others. If you can't name the person who's still accountable, fix that before the agent ships.