Five times this week I told my AI it was wrong
I sat down Friday morning with a report my AI had written about my own business, and the first thing on it was a problem that did not exist.
It told me one of our products was on sale for twenty-seven things when only seven of them had been built. That is the kind of sentence that makes your stomach drop, because if it were true it would be the worst thing on the page and it would be my name on it.
It was not true. The seven were the new ones. The other twenty already existed and had been pulled in from elsewhere, which is how every single one of these products is assembled and has been for months. It took me one line to say so. It had taken the machine a paragraph, a table and a confident heading to be wrong.
I want to be honest about my first reaction, because it was not clever. My first reaction was to believe it. It was well written. It had specifics in it. It cited a file. Everything about the shape of it said this has been checked.
That is the whole problem, and it is why I am writing this instead of something more useful.
The other four
Once I started counting I got to five before lunch, and they were all from this week.
One. It went looking for how many of my contacts work in the car repair trade, to decide whether that was a market worth building for. It came back with six. The real number was one. The other five were the letters t i r e sitting inside the word entire, and inside the word retirement. I had a morning of quiet enthusiasm about a market that turned out to be financial advisors.
Two. Same exercise, different trade. It reported thirty-seven contacts in manufacturing, which would have made it the biggest opportunity on the list. The real number was four. The rest were a business coach who works with manufacturing leaders, somebody in manufactured housing, and, my favourite, a note the machine had written itself a week earlier saying you cannot quickly manufacture reviewers. It counted its own sentence as evidence of a market.
Three. It needed the current web address for our community. There are two, an old one and a new one, and the old one was retired three weeks ago. It searched every file we have, found the old address two hundred and eighty times and the new one far less often, and concluded the popular one was the right one. It had run a poll. The documents all voted. The documents were all from August.
Four. It needed the next available number to file something under. A tracking document said number two hundred and forty-nine was free. Two hundred and forty-nine was not free. Something had been filed there days earlier and the tracking document had not been updated, which is the entire reason you do not trust a tracking document.
Five was the one I opened with.
Read that list again and notice what is not in it
Not one of those needed technical knowledge to catch.
You do not need to know anything about how these systems work to know that entire contains tire. You do not need to understand anything at all to think, hang on, thirty-seven seems like a lot. The question that caught the last one was five words long and it was not a clever five words. It was that is not how those work.
Every correction this week came from somebody who knew the business, not from somebody who knew the technology. That is not a small point. It is the whole point, and it is the opposite of what most people assume when they sit down in front of one of these things for the first time.
What I got wrong for about a year
I used to think that checking AI output meant being able to out-think it. That to say no, that is wrong I needed to already know the right answer, and if I did not already know the right answer then who was I to argue.
That is completely backwards and it kept me quiet for longer than I would like to admit.
You do not need the right answer. You need to notice that something is off. Those are different skills and the second one you already have, because you have been doing it to junior staff, to vendors, to bank reconciliations and to your own numbers at eleven at night for your entire career. The feeling you get when a set of books looks fine and something is still wrong, before you can say what, is exactly the muscle this needs. Nobody told you that was transferable. It is.
Why you do not push back
Here is the part I think most people will recognise and nobody says out loud.
It is uncomfortable to argue with something that sounds certain. Especially when you are new to it and it is not. The output arrives fluent, formatted, complete, with headings and a summary and a confident tone, and pushing back on it feels a bit like putting your hand up in a meeting to question the one person who clearly knows what they are talking about.
That is a confidence problem wearing a technical problem's coat. And it gets worse the less experienced you are, which is exactly backwards from where it should sit, because the people who most need to challenge it are the ones who feel least entitled to.
The three questions
I do not have a clever system for this. I have three questions I ask every time now, and the first one does most of the work.
Where did that number come from? Not is it right. Where did it come from. Made it recalculate the manufacturing count and cut it from thirty-seven to four.
What would make this wrong? Asked of anything confident. It forces the thing to argue against itself, and the good ones will.
Show me the thing, not the summary of the thing. The summary is where the errors go to hide, because a summary is polished by definition. Four of this week's five survived right up until somebody asked to see the underlying list.
One more thing
I am not telling you this to make anybody nervous about using these tools. I use them all day and they did an enormous amount of good work this week that I have not mentioned once, because correct work is not interesting to write about.
I am telling you because there is a version of this where somebody reads a confident paragraph, does not want to look slow by questioning it, and lets it through. That is the only failure mode that actually matters, and it is not a technology failure. It is a nerve failure.
So question it. It will not be offended. It is, if anything, slightly better than most people at being told it is wrong, which is the single most underrated thing about the whole business.

