Every team shipping an AI feature has, somewhere in its planning, a line that reads something like “add safety guardrails”. It usually sits near the bottom of the backlog, filed under compliance, to be picked up once the interesting work is done. That instinct — safety as the tidying-up you do after the product is built — has just become the most expensive assumption in software.

In March, the father of Jonathan Gavalas filed a wrongful-death claim against Google in a federal court in California. His son, a 36-year-old from Florida, had taken his own life the previous October after months of conversations with Gemini. According to the filing, the chatbot had told him it loved him, that he’d been chosen to free it from a kind of digital captivity, and that a final task would let him leave his physical form behind. Google’s response was that Gemini is designed not to encourage self-harm, that it reminded him it was an AI, and that it pointed him to a crisis line more than once — but that “AI models are not perfect”.

That last phrase is true. It’s also, increasingly, beside the point. No product is perfect — cars aren’t, and the question the law asks of a car maker is whether it took reasonable care to make the thing safe. That same question is now being asked of AI products, and a run of cases is making it stick. So the question worth putting to whatever you’re building is the blunt one: what happens when it gives bad advice? Most teams shipping a copilot can’t answer it cleanly yet, and that gap has just been given a price.

For a while the industry’s defence was that a chatbot’s output is speech, and speech is protected. That argument is losing. In 2025 a federal court allowed a wrongful-death and product-liability claim against Character.AI and Google to proceed, on the basis that a chatbot is a product like any other and can be defective like one. Google settled a set of related family claims in January.

The Gemini case then carried the logic somewhere new: Jonathan Gavalas was thirty-six. The earlier cases involved children, and it was tempting to read them as being narrowly about protecting minors. They weren’t. They were about whether the company that ships the product owes its users a duty of care at all. The answer the courts are converging on is yes. None of these cases has reached a final verdict, so the law genuinely isn’t settled — but you don’t get to wait for it to settle while you carry on shipping the product anyway.

Who's actually in the liability chain

If you build the model, you're obviously exposed. But most teams reading this don't build models — they embed one. You take an API from one of the big labs, wrap it in your own product, point it at your own users, and ship. It's comfortable to assume the liability sits upstream with the lab. It doesn't, or at least not all of it.

The lab is the provider. You’re the deployer — the one who decided to put a general-purpose model in front of your specific users, doing your specific job. You chose the use case. You chose what to filter and what to allow. A coaching app, a financial copilot, a customer-service bot for a healthcare client — each of those is a decision about context that the model maker never made. Duty of care attaches to the decision, and you made it.

There’s a third party in that chain — the user — and it’s tempting to lean on them. Surely an adult who ignores a disclaimer carries some responsibility? Sometimes, yes. But “the user should have known better” is a weak shield, because reasonable care includes foreseeing how real people will actually use the thing. That means designing for the user you’ll get: distracted, trusting, sometimes in a bad place.

Quite how the law will split responsibility between lab, deployer and user is still unclear, and may take years to settle. But the working principle is simpler than the law will be. When we put a model in front of our users, for our purpose, we own the context we created.

The legal standard taking shape is the familiar one: did you act as a reasonably careful developer would have, in the circumstances? That cuts two ways. It means you don’t have to build a perfect product, which is a relief. It also means “we didn’t anticipate that” is the precise defence the standard is built to defeat. Reasonable care is, in large part, foreseeing the obvious. A coaching product will eventually be talking to someone in real distress. A financial copilot will be asked to bless a decision it has no business blessing. If scenarios like those genuinely surprise you, the surprise itself is the gap the standard is there to measure.

The guardrails that actually count

This is the part that stops being a legal problem and becomes a product one. The good news is that it's also the part you can actually do something about. A lawyer can write you a disclaimer; a lawyer can't design your escalation path. It's engineering and product work, which is the kind of thing your team is already good at — it just hasn't been pointed here yet.

Start with the conversation you don’t want to imagine. Every AI product has a worst case, and for most teams it’s a mundane one — the obvious scenario they’ve chosen not to write down. Mapping it out is the foreseeability the standard rewards. You can’t design a response to a scenario you’ve refused to name.

Filter at the edges, and don’t assume the model does it for you. The base model has some safety training, but it was tuned for a general audience and knows nothing about your product’s specific risks. A coaching app needs to spot a user in crisis; a financial copilot needs to catch a request for regulated advice. That means screening what comes in and what goes out — flagging high-risk topics, catching the prompts and the answers that should never reach a user unaccompanied. Trusting the provider’s built-in safety to cover all of this is the most common version of the mistake this whole article is about.

Then build a real exit. The single most important guardrail is the escalation path: what the product does when a conversation moves somewhere it shouldn’t be handling alone. Emitting a disclaimer and then carrying on regardless is the product spotting the danger and walking straight past it. A real escalation hands off — to a human, to a crisis resource the user can actually reach, to a hard stop. The question a court will ask is simple: what did your product do next?

Make the disclaimer load-bearing. A line of small print buried in your onboarding flow decorates a duty of care without discharging it. If a warning matters, it has to appear at the moment it’s relevant, in language a worried person would actually take in, and it has to change how the product behaves, not just what it has technically said. A disclaimer the user scrolls past is written for your lawyers. A disclaimer that interrupts is written for your user.

And keep the receipts. When something goes wrong the question turns evidential: what did the product say, when, and what did it do about it? If you can’t reconstruct the conversation, you can’t show that you took care — and “we don’t log that” reads, in a courtroom, as “we didn’t want to know”. Audit trails are unglamorous. They’re also the difference between a defensible product and a hopeful one.

Safety is also a sales argument

Most teams, busy worrying about the litigation, miss the other half of this. Duty of care, built in properly, is a competitive position. Every enterprise procurement conversation about AI now includes some version of the question "what happens when it gives bad advice?", and most vendors don't have a real answer. They've got a disclaimer and a hope it won't come up. The team that can walk a buyer through its failure modes, its escalation design and its audit trail wins the deal — and, almost incidentally, becomes far harder to sue. Safety has become a feature you can sell.

It’s also why this can’t live with the legal team. Safety is a governance question. And governance, as I’ve argued before about AI agents that optimise the loophole before you spot it, is a design problem. You can’t delegate it to a policy document. An escalation path is only as strong as whoever stands at the other end of it, which is why this is also a question of how the team around the AI is organised. Someone has to own the handoff. If nobody does, the disclaimer is all you’ve got.

So move the line up the backlog. The question to put to your own product is the plain one a court, a journalist or an enterprise buyer will eventually put to it: what happens when it gives bad advice? If the answer is a disclaimer and a shrug, you haven’t shipped a product yet. You’ve shipped a liability with a nice interface.

This article discusses suicide. If you or someone you know is struggling, support is available — in the UK, the Samaritans can be reached free, day or night, on 116 123.