Why a security team should care at all
Regulation (EU) 2024/1689, the AI Act, is built like product-safety law: it regulates AI systems according to what they are used for and assigns duties to whoever places them on the market (providers) and whoever uses them professionally (deployers). It is not a cybersecurity framework, and most of its text is about fundamental rights, transparency and conformity.
Security teams still end up owning a slice of it, for three reasons:
- Article 15 makes cybersecurity an explicit legal requirement for high-risk AI systems, and names AI-specific attacks.
- Your organization is almost certainly a deployer of general-purpose models (chat assistants, copilots, LLM features inside your SIEM or ticketing tools), and someone has to know what those systems do with data.
- The SOC is increasingly a heavy AI user itself: alert summarization, phishing triage, query generation, enrichment agents. When legal asks "where do we use AI and what is the risk", the SOC is part of the answer.
If you are already working through NIS2, DORA and GDPR, the AI Act sits next to them rather than replacing anything. Our NIS2, DORA and GDPR overview covers how those three overlap; the AI Act adds a layer on top that is scoped by use case, not by sector.
The risk tiers, briefly
The Act sorts AI into buckets. The bucket determines the obligations.
| Tier | Examples | What it means |
|---|---|---|
| Prohibited (Article 5) | Social scoring, manipulative techniques exploiting vulnerabilities, untargeted scraping of facial images, emotion recognition at work or school (with narrow exceptions) | Banned outright |
| High-risk (Article 6, Annexes I and III) | Safety components of regulated products; Annex III uses such as biometrics, critical infrastructure safety components, employment and worker management, access to essential services, law enforcement | Full set of requirements: risk management, data governance, logging, human oversight, accuracy, robustness, cybersecurity, conformity assessment |
| Transparency (Article 50) | Chatbots interacting with people, generated or manipulated content (deepfakes) | Disclose that it is AI, mark synthetic content |
| Minimal | Spam filters, most internal tooling | No specific obligations beyond general ones such as AI literacy |
General-purpose AI (GPAI) models, the foundation models behind most LLM products, have their own separate chapter, covered below.
A SOC triage assistant is usually not high-risk. Summarizing alerts or drafting detection queries does not appear in Annex III. Be more careful with anything that scores or monitors employees: Annex III covers AI used to monitor and evaluate the performance and behaviour of workers, and some insider-threat or UEBA use cases come close to that line. That is a question for your legal team, not for a blog post.
Article 15: accuracy, robustness and cybersecurity
For high-risk systems, Article 15 requires an appropriate level of accuracy, robustness and cybersecurity throughout the lifecycle, with performance metrics declared in the instructions for use. Three parts matter to security people.
Robustness. Systems should be resilient to errors, faults and inconsistencies, including those arising from interaction with people or other systems. Technical redundancy and fail-safe plans are named as possible measures. For systems that keep learning after deployment, the Act specifically asks providers to address feedback loops, where biased outputs become future inputs.
Cybersecurity. The system must resist attempts by unauthorized third parties to alter its use, outputs or performance by exploiting vulnerabilities. The text explicitly lists the AI-specific threats to address where appropriate:
- data poisoning (manipulating the training set)
- model poisoning (manipulating pre-trained components)
- adversarial examples or model evasion (inputs crafted to cause mistakes)
- confidentiality attacks
- model flaws
Prompt injection is not named as such, but it fits squarely within "inputs designed to cause the model to make a mistake" and attempts to alter use or outputs. Any assessor will expect you to have thought about it.
Proportionality. The measures must be appropriate to the circumstances and the risks. The Act does not prescribe a control catalogue; harmonised standards (being developed by CEN-CENELEC JTC 21) are meant to give a presumption of conformity once published.
Article 15 does not live alone. Article 12 requires high-risk systems to support automatic logging of events, Article 14 requires effective human oversight, and Article 9 requires a documented risk management process. The new part is that the threat model must include the model and its data, not just the API in front of it.
GPAI obligations: mostly your vendor's problem
Providers of general-purpose AI models carry their own obligations (Article 53): technical documentation, information for downstream providers who integrate the model, a policy to comply with EU copyright law, and a public summary of training content. Models classified as having systemic risk (presumed above a training compute threshold of 10^25 FLOPs, or designated by the Commission) add model evaluation including adversarial testing, systemic risk assessment and mitigation, serious incident reporting, and an adequate level of cybersecurity protection for the model and its infrastructure (Article 55). A voluntary GPAI Code of Practice was published in 2025 as a way to demonstrate compliance.
If you only call a model through an API, you are not the GPAI provider. You become relevant in two cases: you fine-tune or substantially modify a model and redistribute it, or you integrate a model into a system that is itself high-risk. In both cases, the documentation your model vendor must provide is your input, so ask for it.
The timeline (get this right in your slides)
The Act entered into force on 1 August 2024 and applies in phases.
| Date | What starts applying |
|---|---|
| 2 February 2025 | Prohibited practices (Article 5) and AI literacy (Article 4) |
| 2 August 2025 | GPAI model obligations, governance structure, penalties regime |
| 2 August 2026 | Most remaining provisions, including Annex III high-risk systems and Article 50 transparency |
| 2 August 2027 | High-risk systems that are safety components of products under Annex I legislation; GPAI models placed on the market before August 2025 must be compliant |
Check the current status before you plan around August 2026. In late 2025 the Commission proposed, as part of its digital omnibus package, to adjust the application of the high-risk rules and tie them to the availability of standards and support tools. Whether and how that changes the dates depends on the legislative outcome. Verify against the Official Journal, not a vendor webinar.
Penalties scale with the violation: the highest tier (prohibited practices) reaches up to EUR 35 million or 7% of worldwide annual turnover, whichever is higher, with lower ceilings for other breaches.
What a SOC using LLMs should document
Even when nothing you run is high-risk, deployer duties and plain good governance push in the same direction. AI literacy under Article 4 already applies: staff operating AI systems need sufficient understanding of them. Here is the minimum record we would want for every LLM use in a SOC.
1. An inventory entry per use case
use_case: phishing-triage-summary
owner: soc-lead
model: vendor-hosted LLM (name, version, region)
role: deployer
inputs: reported email headers and body, URL and hash enrichment results
outputs: draft verdict and summary for L1 analyst
decision_authority: analyst decides; model never closes tickets
data_classes: personal data (sender, recipient), possibly sensitive content
retention: prompts and outputs kept with the case record
ai_act_tier: minimal (not Annex III); reviewed by legal on 2026-09-01
The tier field is the point: write down why you think it is not high-risk, who decided, and when. If the use case drifts (the triage bot starts auto-blocking senders, or starts scoring employees), the assessment has to be redone.
2. A threat model that includes the model
Treat every LLM input as attacker-controlled when it can contain external content, which in a SOC is almost always. A phishing email is literally written by the adversary. Document:
- Prompt injection paths: email bodies, web pages fetched by an agent, file contents, log lines. An attacker can plant instructions in any of them.
- Tool permissions: what the model can do. Read-only enrichment is low risk; an agent that can close tickets, block IPs or send mail needs a human approval step.
- Data exposure: what leaves your perimeter in prompts, where the provider processes it, and whether it is used for training.
- Output trust: model-generated IOCs, queries and verdicts are suggestions until validated. A hallucinated hash or a confidently wrong regex is a real failure mode. Test generated patterns on /tool/regex before they go into a rule.
3. Human oversight that is real
"A human reviews the output" only counts if the human can realistically disagree. Record the cases where analysts overrode the model. If the override rate is zero across thousands of cases, either the model is perfect or nobody is reading. Both deserve a look.
4. Logs
Keep prompts, model version, retrieved context, outputs and the final human decision with the case. For high-risk deployers, Article 26 requires keeping automatically generated logs for an appropriate period of at least six months; for everyone else it is simply what you will need to explain a bad call after the fact.
5. An incident path
Add "AI misbehaviour" to your incident taxonomy: injected instructions acted upon, sensitive data in a prompt that should not have left, a model update that silently changed verdict quality. Providers of high-risk systems must report serious incidents (Article 73), and deployers must inform the provider when they identify one, so you want this flowing through the same process as any other incident.
How this fits with what you already do
Most of what Article 15 asks for maps to controls you already run. The table below is the translation we use.
| AI Act expectation | Existing security practice to extend |
|---|---|
| Resilience against manipulation of inputs | Input validation, content isolation, red teaming with injection payloads |
| Protection against data and model poisoning | Supply-chain controls on datasets and model artefacts, hashing and provenance, see SBOM tracking |
| Confidentiality attacks | DLP on prompts, access control on retrieval sources, tenant isolation |
| Logging (Article 12) | SIEM ingestion of model calls and tool actions |
| Human oversight (Article 14) | Approval gates in SOAR playbooks |
| Lifecycle accuracy | Regression test sets for verdict quality after each model update |
The genuinely new work is the regression test set. Classic software does not change behaviour when the vendor updates a dependency you never see; hosted models do. Keep a fixed set of a few dozen labelled cases (benign, malicious, tricky) and rerun it whenever the model or the prompt changes.
What to do Monday
- List every place AI touches the SOC, including features hidden inside existing tools. Assign an owner and a tier with a written rationale.
- Flag anything that scores or monitors employees for legal review.
- Strip action permissions from any agent that processes attacker-written content without a human approval step.
- Start logging prompts, model versions and human decisions alongside case records.
- Build a small labelled regression set and run it on every model or prompt change.
- Brief the team: Article 4 AI literacy is already in force, and a one-hour session on prompt injection and hallucinated IOCs covers most of what a SOC needs.
- Put the August 2026 and August 2027 dates in the risk register with a note to recheck them against any adopted amendments.
The AI Act does not ask security teams for anything exotic. It asks you to treat the model as part of the attack surface and to be able to prove it.
