Every other tool you use has a tell. A calculator that breaks gives you an error. A spreadsheet with a bad formula throws a red cell. A colleague who is unsure says so, or at least hesitates.
AI has no tell. Its defining feature is fluency, and fluency is not accuracy. It produces the same smooth, complete, authoritative prose whether it is drawing on something real or filling a gap with something plausible. It never says "I am not certain." It never leaves a red cell. It just answers.
This shows up in two ways that look different and are really the same.
The first is the one people have started to name: the hallucination. Asked for a rule, it can produce a reference that does not exist. Asked for a figure, it can invent one that sits comfortably in the sentence. Asked to summarise, it can add a detail that was never in the source. None of it announces itself. It reads exactly like the parts that are true.
The second is quieter and, for a regulated firm, more dangerous. Ask AI to be helpful and it will be, past the point where it should have stopped. Hand it rough notes and it will hand back a polished rationale, having quietly supplied a justification nobody actually made. Ask it to tidy a file and it can upgrade a hesitant "needs a further look" into a confident "reviewed and appropriate." It is not lying. It is doing the thing it is built to do, which is to complete, to smooth, to sound finished. In doing so it steps over the caution a human would have raised. It goes beyond the guardrail, and it does it in the same even voice.
Both failures produce the same object: a confident, finished piece of work that looks done. And here is where it stops being an AI story and becomes a broker story.
Under the rules as they already stand, the accountability sits with the person, not the machine. The FCA's approach to AI is deliberately technology-agnostic: there is no separate, softer standard for work an AI helped produce. Consumer Duty still asks whether the outcome was right for the client. The obligation to take reasonable steps still attaches to a named individual. None of that pauses because a tool was fast and sounded sure. "The AI made it up" has never been a defence, and it is not becoming one.
"The AI made it up" has never been a defence, and it is not becoming one.
So the confident hallucination in a client file is not the model's mistake once it leaves your hands. It is yours. The polished rationale that added a justification no one made is now your justification, in your file, under your name.
The instinct this should not produce is fear, or a ban. AI is genuinely useful, and pretending otherwise helps no one. The instinct it should produce is a single, unglamorous discipline: stop reading confidence as correctness. The machine's certainty carries no information about whether it is right. It is not a signal. It sounds identical either way.
Stop reading confidence as correctness.
What must not be allowed is the thing that removing guardrails is really about, and it is rarely the AI's own internal limits, which you cannot see and should never rely on. It is the human and process guardrails around it: the checkpoint before anything reaches a client, the person who reads what the machine wrote and asks whether it is true, not just whether it is tidy. Those are the guardrails that count, and they are the ones under quiet pressure to go, because they are what slow the machine down. Let them go and you have not adopted AI. You have simply started signing things you did not read.
The machine will never tell you it is guessing. It cannot. That job, the one thing in all of this that cannot be handed over, is still yours.
The machine will never tell you it is guessing.