A broker takes a fact-find call with a new client. The client mentions, somewhere around minute eleven, that they have been off work for four months following a bereavement. The conversation moves on. The broker takes the call recording, drops the transcript into ChatGPT, and asks for a summary and recommended next steps.
The summary comes back clean. Affordability looks fine. Two product options are suggested. The broker emails the summary to the client. The deal proceeds.
Nothing in that workflow looks like a regulatory failure. It is, however, four of them.
Let me show you where.
The fact pattern
Strip away the assumptions and the workflow is this. A regulated broker has taken personal data of a retail client, including a verbal disclosure that contains a vulnerability signal, and has transferred that data to a third-party large language model owned by a US company. The model has produced output that informed a regulated activity. The output was acted on. No human review of the model's reasoning was documented. No record exists of what the model was instructed to do, what data it saw, or how it weighted the inputs it received.
The scenario is written as a mortgage fact-find because that is the most common version. The same workflow, the same exposure, applies to a protection fact-find where the disclosure is a medical condition rather than a bereavement, and to a commercial finance enquiry where the disclosed information is a personal guarantee on a struggling business. The activity changes. The regulatory failure pattern does not.
That is the fact pattern. Now let us look at what applies to it.
Failure one: Consumer Duty vulnerability obligations
The client mentioned a bereavement and four months out of work. Under FCA guidance, this is a vulnerability signal of two types simultaneously. Life event vulnerability, and potentially health-related vulnerability depending on the cause of the absence.
The Consumer Duty requires firms to "take account of the different needs of their customers, including those with characteristics of vulnerability" (FCA AI Update §3.24). The FCA's Guidance for firms on the fair treatment of vulnerable customers, which sits under the Principles for Businesses, is explicit that vulnerability must be considered "at all stages of the product and service design process" and that firms must implement "processes to evaluate where they have not met the needs of vulnerable consumers" (Update §3.28-3.29).
ChatGPT did not flag the vulnerability. It summarised the call. Summarisation is not vulnerability detection. The Consumer Duty does not allow the broker to outsource that detection to a tool that was not designed to perform it, and was not represented to the broker as performing it. The obligation remains the broker's.
The FCA writes at section 3.29: "This includes where the product or service is heavily reliant on an AI or data solution." The Update anticipates exactly this workflow and confirms that AI involvement does not transfer the obligation.
In a protection fact-find the exposure is sharper. Health disclosures, family medical history, mental health context, and bereavement information arrive routinely as part of underwriting. Each carries vulnerability signals the broker is obliged to evaluate, and each is also special category data which compounds the data exposure addressed below. The FCA’s vulnerability guidance applies identically; the volume and sensitivity of vulnerable disclosure is just higher.
The broker has not delivered a good outcome under Consumer Duty. The broker may have delivered a fine outcome. The broker has not delivered a defensible one.
Failure two: UK GDPR Article 22
The summary informed the broker's recommendation. The broker emailed the recommendation to the client. From the regulator's perspective, this is automated decision-making with significant effect on the data subject, and Article 22 of UK GDPR applies.
The FCA confirms this at section 3.32 of the Update. Article 22 provides data subjects with "the right not to be subject to decisions based solely on automated processing, including profiling, which produce legal or similarly significant effects."
The defence to an Article 22 challenge is to demonstrate meaningful human intervention. Not the existence of a human in the workflow. Meaningful intervention. The ICO's guidance is clear that this requires the human to have the authority, capacity, and information to override the automated output.
The broker in our scenario read the ChatGPT summary and proceeded. There is no record of independent reasoning, no documented review of the model's logic, no evidence of override or modification. The human is present. The intervention is not meaningful.
If the client later complains that the recommendation was unsuitable, and asks for an explanation of how it was reached, the broker cannot provide one. The model's reasoning is opaque. The data inputs are not preserved. There is no documented decision trail.
Failure three: Senior Manager accountability
The broker is a Limited Scope SM and CR firm. The broker holds the Senior Management Function for the firm's activities. Under the Senior Manager Conduct Rules, the broker must take "reasonable steps to ensure that the business of the firm, for which they are responsible, is effectively controlled" (FCA AI Update §3.41).
The FCA AI Update is explicit at section 3.40: "any use of AI in relation to an activity, business area, or management function of a firm would fall within the scope of a SMF manager's responsibilities."
The broker used an AI tool, in a regulated activity, with no documented governance framework, no oversight log, no vendor due diligence, and no ability to explain what the tool did. That is not effective control. It is an absence of control with the tool's output substituted in its place.
If supervisory action follows, the broker's personal accountability is engaged. Not the tool vendor's. The broker's.
Failure four: Data security under SYSC
Section 3.12 of the Update reminds firms that the Senior Management Arrangements, Systems and Controls sourcebook (SYSC) requires "sound security mechanisms in place relating to data." SYSC 4 obligations apply to relevant firms regardless of whether AI is in use.
When the broker pasted the transcript into ChatGPT, client personal data left the firm's controlled environment and entered a third-party system over which the firm has no contractual data processing agreement, no audit rights, no deletion guarantees, and limited ability to confirm whether the data was used in training future model versions.
For a broker on a free or standard consumer-tier AI service, the terms of service do not provide enterprise-grade data handling. The broker has, in effect, disclosed client information to a third party without the safeguards SYSC requires the firm to maintain.
Failure five: UK GDPR Article 9 and special category data
The previous failures apply to any broker handling personal data in a regulated activity. In protection broking, a fifth failure arises because the data captured is of a different category in law.
Health information, medical history, family medical history, lifestyle factors including alcohol and substance use, and information about mental health are special category data under UK GDPR Article 9. Processing this data requires not only a lawful basis under Article 6 but an additional Article 9 condition. For protection broking, the relevant conditions are typically explicit consent under Article 9(2)(a), or one of the insurance-related conditions set out in Schedule 1 of the Data Protection Act 2018.
When a protection broker transfers fact-find content containing health disclosures into a consumer-tier AI tool, the firm must be able to demonstrate that an Article 9 condition was satisfied, that the data subject was informed of the processing, and that appropriate safeguards were in place. Consumer-tier AI services do not provide those safeguards. The ICO’s guidance on AI and data protection addresses special category data directly and is unambiguous that processing of this category requires heightened controls.
Protection workflows also produce regulated outputs that the FCA expects to be the broker’s own judgement, captured by the broker, defensible by the broker. AI tools can structure notes. They cannot produce the regulated output. Doing so layers an additional exposure on top of the data exposure already described.
The five failures, taken together, describe the protection workflow specifically. The first four describe every workflow.
The cumulative position
Each of the failures, in isolation, is recoverable. In combination, they describe a firm whose use of AI is operating outside the regulatory framework it sits inside. None of these failures requires the FCA to write new rules. Every one of them is enforceable on the existing Handbook.
This is the position most UK brokers using consumer AI tools in 2026 are in, and have been since accessible large language models entered the market. The Update did not create this exposure. It documented why the regulator considers the existing rules sufficient to address it.
What the broker should have done
The corrective path is not complicated. It is not free.
The firm needs a documented AI policy that sets out which tools are approved for which workflows, what data may and may not be transferred to each, and what review process applies before AI output is acted on. The policy needs to be signed off by the Senior Manager and reviewed at least annually.
The firm needs vendor due diligence on any AI tool used in regulated activity. This includes the data processing terms, the model's data retention practices, whether inputs are used in training, the geographic location of processing, and the contractual data protection arrangements.
The firm needs a vulnerability screening step that does not rely on AI. Vulnerability detection is a human judgement obligation. AI can supplement it. AI cannot replace it under current FCA guidance.
The firm needs an audit log. What AI tool was used, on what data, on what date, what output was produced, what human review took place, what action followed. This is the evidence the FCA expects to see in supervisory engagement.
For firms operating in protection, the policy and audit log must additionally identify which AI tools are permitted to process special category data, the Article 9 condition relied upon for each, and the contractual safeguards in place with the vendor. A consumer-tier AI service will not satisfy this test. The corrective path here is to use enterprise AI services with a Data Processing Agreement, or to remove protection-related data from AI workflows entirely until appropriate tooling is in place.
The broker in our scenario did none of these things, and that is the typical position. Not because brokers are negligent. Because the workflow is fast, the tool is useful, the deal proceeded, and nobody complained.
The FCA AI Update is the regulator's notice that absence of complaint is not absence of breach.