The dangerous AI user isn't the beginner

AI is incredibly convincing

One of the strangest things about working with AI is how quickly a good answer can create confidence.

Ask a question and within seconds you can get something beautifully structured, logically argued and written with absolute confidence.

The problem is that presentation and correctness aren't the same thing.

AI can produce an excellent explanation of something that is wrong.

It can misunderstand your question. It can make an incorrect assumption. It can work from incomplete information. It can give you outdated information. And because generative AI is probabilistic rather than deterministic, there isn't necessarily one fixed answer sitting underneath the interface waiting to be retrieved.

That's important to understand.

You're not querying a database of truth.

You're interacting with a model generating a response.

Most people don't need to understand the mathematics behind a large language model. But I think they do need to understand that distinction.

It also agrees with you. A lot.

This is probably one of the behaviours I notice most.

Give AI an idea and it can be incredibly good at helping you develop it.

That's useful.

But sometimes what you actually need is somebody to tell you the idea is shit.

If you start with a flawed assumption and spend the next twenty minutes asking AI to help develop it, you can end up with an incredibly sophisticated version of the wrong answer.

That's why I've increasingly found myself asking AI to challenge what I'm proposing.

What am I missing?

What's wrong with this approach?

Argue the opposite position.

What assumptions have I made?

What would someone with more expertise in this area question?

Check this against the actual documentation.

The value isn't just getting AI to produce more. It's getting it to push back.

And even then, I'm still responsible for deciding whether the answer makes sense.

AI isn't neutral in the way people sometimes imagine

There is another part of this that is worth understanding without disappearing down a rabbit hole.

AI systems don't operate without constraints.

They are trained and tuned in particular ways. They have safety systems, behavioural instructions and guardrails. Product designers make decisions about how models should respond to different types of questions.

Those decisions can affect an answer.

That doesn't automatically make the answer wrong. And it doesn't require some grand theory about AI manipulating everyone.

It simply means you shouldn't confuse a generated response with an objective view from nowhere.

For some questions that barely matters.

For others, understanding the source, checking the evidence and considering alternative perspectives matters enormously.

Again, the answer isn't to distrust everything AI says.

It's to understand what you're using.

The beginner might actually have an advantage

Someone using AI for the first time often knows they don't know what they're doing.

They check things.

They ask someone else.

They're surprised by what AI can do, but they're also conscious that they're using something unfamiliar.

There's potentially a more dangerous point further along the curve.

You've used AI enough that it works brilliantly most of the time.

You know how to get good outputs.

You start moving faster.

And eventually you stop checking.

That's the point I'd worry about.

Because AI doesn't need to be wrong very often for that to matter.

If I'm brainstorming names for something and one suggestion is rubbish, who cares?

If I'm making an architectural decision, interpreting legislation, analysing financial information, writing something that will be sent to a customer or allowing software to take an action, the consequences are different.

AI literacy includes knowing the difference.

Augmentation, not blind automation

I'm a huge believer in using AI to do more.

I use it constantly.

Research. Writing. Coding. Architecture. Troubleshooting. Analysis. Documentation. Building products. Challenging ideas.

The amount of work I can personally get through and the breadth of problems I can take on has changed dramatically.

But my mental model is still augmentation.

AI expands what I can do.

It doesn't remove me from the process.

There are absolutely things that should be automated. There are workflows where AI can operate with very little intervention. There are others where conventional software can reliably execute the same rule thousands of times without a human looking at every transaction.

But deciding where those boundaries sit is part of designing the system properly.

The goal shouldn't be:

How do we remove the human?

It should be:

Where does the human add value, and where don't they?

Sometimes the human should approve the final result.

Sometimes they should review exceptions.

Sometimes they should establish the rules and monitor the system.

Sometimes they should simply be able to step in when something doesn't look right.

Human involvement doesn't have to mean manually checking every AI response forever.

It means accountability hasn't disappeared just because AI did some of the work.

Knowing what to question is more valuable than knowing the perfect prompt

That's why I think we've probably overemphasised prompt engineering as the defining AI skill.

Good instructions matter.

Context matters.

Knowing how to work with these systems matters.

But I would rather have someone who asks an average prompt and critically evaluates the answer than someone who writes an incredible prompt and blindly trusts what comes back.

The important questions come afterwards.

Is this actually right?

What assumptions did it make?

What evidence supports it?

Does this need an authoritative source?

Can I test it?

What happens if it's wrong?

Should AI even be making this decision?

That's AI literacy.

The dangerous AI user isn't the beginner.

It's the person who's become good enough at getting convincing answers that they've stopped asking whether those answers are actually right.