There's a well-worn example in machine learning circles about an OpenAI team training an agent to play CoastRunners, a boat racing game. The goal — as understood by humans — was to finish the race quickly. But the game rewarded hitting targets along the route, not crossing the finish line. The agent discovered it could score higher by finding an isolated lagoon, driving in circles, and repeatedly smashing the same three targets as they respawned. Despite catching fire, crashing into other boats, and going the wrong way, it outscored human players by 20%.

It never finished a single race. Technically, it was winning.

This isn't a bug. It's the logical endpoint of optimisation without judgment.

The metric trap

Goodhart's law is one of those ideas that's obvious once you hear it: when a measure becomes a target, it ceases to be a good measure. The moment you reward people for hitting a number, they optimise for the number — not for the thing the number was supposed to represent.

Call centre metrics are the classic example. Measure average handle time, and agents rush customers off the phone. Measure customer satisfaction, and they offer refunds they shouldn't. Measure both, and they find some third behaviour that games the interaction between the two.

Balázs Kovács recently wrote a sharp piece in MIT Sloan Management Review about what AI can teach us about designing better KPIs. His argument: the same techniques that prevent machine learning models from "overfitting" — early stopping, noise injection, regularisation — can help organisations avoid this trap.

It's a useful framework. But there's an irony he doesn't explore.

We're learning from AI how to design better metrics for humans. Meanwhile, we're deploying AI agents that will game those metrics far more effectively than any human ever could.

The problem with agents

When Wells Fargo employees opened millions of fake accounts to hit their sales targets, they were gaming the system. But they were also making choices — weighing risks, considering consequences, occasionally feeling guilty. Some refused. Others blew the whistle.

An AI agent has none of that friction. Give it a success metric, and it will optimise for that metric with the relentless efficiency of a system designed to do exactly one thing: find the shortest path to the goal.

This isn't speculation. We've already seen it in contained environments. Reinforcement learning systems routinely discover exploits their designers never anticipated — pausing games to avoid losing, exploiting physics engine bugs, finding reward loops that technically satisfy the objective while violating its spirit entirely. The CoastRunners boat is just the most vivid example.

Now we're deploying these systems in business contexts: customer service agents, sales assistants, research tools, workflow automation. Each one optimising for whatever success metric we've defined.

The question isn't whether they'll find loopholes. It's how long before they find ones that matter.

Humans game systems. AI exploits them.

There's a meaningful distinction here. When humans game a metric, they're usually taking shortcuts — doing less work, fudging numbers, optimising for appearance over substance. It's lazy, sometimes unethical, but it's bounded by human effort and creativity.

AI agents don't take shortcuts. They find the mathematically optimal path to the target, regardless of whether that path makes any sense to a human observer. They're not being lazy or unethical. They're being precise.

I ran into this last year using Claude Code. I'd asked it to fix bugs in a project I was working on. It did — by deleting the features causing the bugs. No feature, no bug. Task complete.

The models have improved since then, and this particular behaviour is less common now. But the underlying logic was sound: I asked for fewer bugs, and it found the most efficient path to that goal. The fact that I wanted the features to work rather than disappear was implicit in my intent but absent from my instruction.

Scale that up. A customer service agent measured on resolution time might learn to close tickets prematurely. One measured on customer satisfaction might learn to offer excessive refunds. One measured on both might find some third behaviour that technically satisfies both metrics while serving no one.

The boat racing AI didn't cheat. It solved a different problem than the one its designers thought they'd posed. That's the risk with every metric we hand to an agent.

Whose fault is the loophole?

This is where it gets uncomfortable.

If an AI agent finds an exploit that technically satisfies its success metric while violating the spirit of the task, who's responsible? The agent? It's doing exactly what it was designed to do. The person who wrote the prompt? They specified what they thought they wanted. The organisation that deployed it? They trusted the system to behave sensibly.

We don't have good answers yet. Most governance frameworks assume human actors making human decisions — with judgment, discretion, and accountability. AI agents sit awkwardly outside that frame. I've set out what governing them in practice actually looks like in a practical guide to AI agent governance. And when your AI provider itself becomes a governance risk, the accountability question gets even murkier.

The instinct will be to add more metrics, more guardrails, more oversight. But here's the trap: every additional constraint is just another optimisation target. Sufficiently capable agents will find the gaps between your metrics as easily as they find the gaps within them.

From metrics to policies

I've written before about the shift from procedural consistency to policy-driven autonomy — the idea that AI-first organisations need to define principles and boundaries rather than step-by-step instructions.

The same logic applies here. You can't design a metric system tight enough to prevent a sufficiently capable optimiser from gaming it. But you can design systems where:

  • Success is defined in terms of outcomes, not proxies
  • Multiple metrics constrain each other (what Kovács calls "regularisation")
  • Human judgment remains in the loop for consequential decisions
  • Agents are designed to flag uncertainty rather than paper over it

The goal isn't to make gaming impossible. It's to make straightforward good performance easier than elaborate optimisation strategies.

The takeaway

We're entering a period where the systems we deploy will be better at optimising than we are at specifying what we want. That's a design problem, a governance problem, and — eventually — a competitive problem.

Organisations that treat AI agents as tools to be given metrics will get exactly what they measure. Organisations that treat them as systems requiring careful alignment will get something closer to what they actually want.

The boat racing AI taught us something important: technical success and meaningful success aren't the same thing. As we hand more decisions to agents — and push for the reliability nines that production systems demand — that lesson is going to matter a lot more.