AI-Driven Attacks & Data Breaches
Incidents from the past 18 months with root causes, impact, and governance lessons.
Arup Hong Kong — $25M Deepfake CFO Fraud
Criminals used AI-generated deepfakes of senior executives on a video call, convincing an employee to make 15 transactions totalling $25M USD to attacker-controlled accounts.
Root Cause
No identity verification for video-based financial authorisations. No deepfake detection. No multi-channel verification for large transactions.
Governance Lesson
Include AI-enabled social engineering in operational risk frameworks. Implement callback verification and dual authorisation above threshold.
AI Copilot Prompt Injection — Data Exfiltrated
Hidden instructions embedded in documents caused enterprise AI copilots to silently exfiltrate sensitive corporate data through encoded outbound requests.
Root Cause
No prompt injection detection. AI processed untrusted external content with broad access to internal data and actions.
Governance Lesson
Treat all external content processed by AI as hostile. Implement output monitoring, data minimisation, and injection detection.
NHS AI Chatbot Leaks Patient PII to Third Parties
An NHS-integrated AI triage chatbot transmitted patient symptom data and demographics to external analytics providers without consent or DSPT compliance.
Root Cause
No DPIA for the AI integration. Vendor data flows not disclosed. Data governance team excluded from deployment.
Governance Lesson
Require DPIA for all AI integrations processing personal data. Audit vendor data flows before procurement.
UK Law Firm — Client Files Uploaded to Unauthorised AI
Associates uploaded privileged legal advice, draft contracts, and confidential instructions to personal ChatGPT accounts. No AI acceptable use policy existed.
Root Cause
No AI acceptable use policy. No technical controls blocking external AI tools. No privilege-awareness training.
Governance Lesson
Publish AI AUP before permitting any tool use. Deploy DLP controls. Train staff on privilege waiver risks.
Autonomous AI Agent — £4M Erroneous Trades
An AI trading agent misinterpreted a regulatory announcement, executing erroneous trades with no kill-switch or human approval gate, causing £4M in losses.
Root Cause
Model risk framework designed for traditional algo trading. No stress testing against unstructured regulatory inputs.
Governance Lesson
Require human-in-the-loop for consequential AI decisions. Implement kill-switch protocols.
Model Inversion — 12K Employee Records Reconstructed
Repeated queries against a UK insurer's AI underwriting API reconstructed salary banding and demographic data for ~12,000 employees.
Root Cause
No differential privacy. No output filtering. No rate limiting on API queries processing personal data.
Governance Lesson
Assess model inversion risk for any AI trained on personal data. Implement differential privacy and query monitoring.
AI Data & PII Risk Vectors
How AI adoption creates new vectors for data exposure.
Training Data Exposure
Employee data fed into third-party AI may be used for model training, permanently leaving your control.
Shadow AI
Employees using personal AI tools for work, uploading contracts, payroll, and legal documents to unknown platforms.
AI Vendor Breaches
A breach at your AI vendor exposes compliance posture, audit trails, and PII to attackers and regulators.
Over-Permissioned AI
Enterprise copilots with broad access expose far more data than necessary through a compromised session.
Autonomous AI Agents: Rising Risk
Companies deploy AI agents faster than governance can keep up.
No Human Approval Gates
Agents making consequential decisions with no human-in-the-loop controls.
Risk: Cascading failures with no rollbackNo Audit Trail
Existing frameworks can't answer: who authorised it, what data was used?
Risk: EU AI Act Article 13 non-complianceGoal Misalignment
Agents optimise for objectives, not values. "Reduce costs" may cut compliance.
Risk: Unintended regulatory liabilityPrivilege Escalation
Admin-level agents become the most dangerous insider threat when compromised.
Risk: Complete system compromiseNo Agent Policy
No formal accountability when an autonomous agent causes damage.
Risk: Director-level personal liabilityMulti-Agent Gaps
When agents interact, decisions emerge that no single agent was designed to make.
Risk: Accountability vacuumStay Ahead of AI Threats
Emerging attack patterns, governance failures, and regulatory responses — every Monday.
Subscribe to Compsilon →