I’ve touched on this in passing across a few posts now, but it deserves its own space. Because as useful as AI has been for me, it is not sunshine and rainbows.
AI is a great tool. I genuinely believe that. But I also think the future gets pretty dystopian if people don’t use these tools with their eyes open. And right now, a lot of people aren’t.
One thing I read recently that stuck with me: it can do super advanced calculations but it can’t tell time right. That sounds like a joke but it isn’t. Some of the things I’ve asked it, it will be absolutely insistent it’s correct. You interrogate it because something smells off, and eventually it folds. Oh yeah, you’re right, I was wrong. I’ve had situations where I knew it was wrong, kept pushing, and it took a surprisingly long time before it admitted it.
So what I tell people, my kids, colleagues at work, is this. Use it. But do not trust it. Assume it’s going to lie to you. If you go in with that mindset and you scrutinise the output, it can be really good. But you have to be able to scrutinise it. That’s the part people skip.
That’s also the part that makes it genuinely dangerous in the wrong hands. I can ask it something about cybersecurity and I’ll know pretty quickly if the answer looks right or completely off. But I can’t ask it to do my taxes. I don’t know tax law. So if it tells me I can do something, I have no idea if it’s true. That’s a problem. And it’s why you see things like lawyers submitting court filings with citations that don’t exist because a judge caught them using AI and not checking the output. People just throw stuff in and take whatever comes out.
I ran into this myself about a year or so ago when I was doing some budget planning. Nothing super sensitive, just the savings pots I set aside for predictable expenses throughout the year so I’m not hit with an inconsistent spend later. I do it all in a spreadsheet and it gets involved. I figured let me see if AI can handle it.
It did it, and then it didn’t. The numbers were inconsistent. Flat out wrong in places. I tried for a few months and I just could not rely on it. So I stopped and went back to my spreadsheet.
More recently I tried again with a different model, and I’ll get into that in the next post. But it’s actually working now. Two months in and the output is consistently accurate. I’ve also gotten smarter about how I prompt it, asking it to show its work and export the data in a way I can verify. So it’s a combination of the models improving and me getting better at using them.
But the overall lesson hasn’t changed. The reading of the tea leaves is the hard part. Sometimes the output is exactly what you wanted and better than you could have done yourself. Sometimes it’s close but slightly off in a way that’s easy to miss. And sometimes it’s just wrong and completely confident about it.
The tool is getting better. That’s real. But so is the risk of people treating it like it’s infallible. It isn’t. Not even close.