In March, Microsoft put a second AI model to work checking the first one’s homework inside Copilot, and its own testing said the research came out better for it. On 19 October it told administrators that the feature was being retired, with the switch-off due to finish by the middle of this month. Microsoft hasn’t said why. My reading is that the checking cost more to run than a flat-priced product could comfortably carry — and if that’s right, it tells us something about every AI product we use.

The accuracy any AI vendor gives us by default is the accuracy that suits its margin, and if we want more than that, the cheapest route is a second opinion we arrange ourselves.

What the second model did

Microsoft called the two features Critique and Council, and it offered them inside Researcher, the deep-research agent in Microsoft 365 Copilot, to customers in its Frontier early-access programme. With Critique, OpenAI’s GPT drafted the research report and Anthropic’s Claude reviewed it for accuracy, completeness and the quality of its citations before it was delivered. Council ran the same question through both models side by side and handed over both reports, along with a third model’s summary of where they agreed, where they differed and what each had found that the other hadn’t.

Microsoft’s own testing put Researcher with Critique almost 14% ahead of the next-best system it measured, Perplexity’s deep-research tool, on its internal deep-research benchmark. That’s a vendor marking its own homework, so I’d hold the exact figure loosely, but the direction is what you’d expect. A single model will write a wrong answer in the same confident tone as a right one, so you can’t tell from reading it. A model from a different company is unlikely to share all of the first one’s blind spots.

Where the two disagree, the disagreement doesn’t tell you which of them is right, but it does tell you where to look. That makes it a partial answer to the sycophancy tax I described last month, the habit these tools have of telling whoever is asking what they’d like to hear.

Why it went

The retirement notice says only that the preview is concluding and that what Microsoft learned may help bring “similar value” to Copilot in future, without naming anything. Everything after this point is my own reading of the economics.

Running a second top-tier model over every research task roughly doubles the model costs of every answer, and Council added a third model to compare the two. All of it adds to the wait as well. Copilot is sold to businesses at a flat price per seat, which means every extra model call comes out of Microsoft’s margin and never appears on the customer’s bill. For the person reading the report, a better answer at twice the running cost is a good deal. For the company selling the licence, it’s a poor one.

None of this makes Microsoft unusual. Every AI vendor is choosing, product by product, how much checking and how much reasoning to put behind each answer, and each makes that choice against its own cost of serving us. We get the default that works for its business. Nobody shows us the dial.

Your own second opinion

The good news for a firm of five or fifty people is that you don’t need Microsoft for any of it. Most of us already pay for, or could cheaply add, a business account with an assistant from a second company, and the check that Copilot offered is something we can run by hand in a few minutes. What it does need is a rule about which work gets one, because checking everything is how a good habit dies in its second week.

Start with the documents where a wrong number or a wrong fact costs real money: quotes, contracts, proposals, anything going to a customer and anything going to HMRC. When a draft in one of those categories has been written or researched with one assistant, paste it into one made by a different vendor. Include the figures and sources it was built from, and give it the same instruction every time:

Check every number, every statement of fact and every source in this draft against the material provided. List anything that’s wrong, anything you can’t verify from what you’ve been given, and anything you’d push back on.

The fixed wording matters more than it looks, because a free-flowing chat is exactly where the agreeable habit creeps back in. Use business accounts whose data settings you’ve checked, and take out anything a customer wouldn’t want shared before it goes anywhere. Contracts and tax still deserve a qualified human at the end. The second model’s job is to show that person which lines need a closer read.

It’s also worth keeping a few test questions whose answers you’ve already checked, and putting them to both assistants once a month. Models get updated without much notice, and a monthly test is the quickest way to spot when one of them has started doing worse at the work you rely on it for. Put one person’s name against the rule and the test. It’s the same principle as routing most everyday AI work to cheaper models: someone in the business has to own the quality decision, because the vendor’s product team is optimising for a different number.

I don’t know how much of Microsoft’s improvement a small firm would see by doing this by hand, and I suspect it varies a great deal with the kind of work. A modest gain on the documents that carry money is still worth a few minutes each.

Whatever Microsoft does with the idea next, the vendors will go on deciding what accuracy comes in the box, and what leaves our building unchecked is still our decision. So pick the one document type where a wrong number costs you most, and from this week have a second model read every one of them before it goes out.