AI Threat and Vulnerability Assessment

Sun, 15 Mar 2026 00:00:00 +0000

The Complete AI Threat Modeling and Vulnerability Assessment Guide From STRIDE to Production Security

Most organizations assess AI security the same way they evaluate traditional software. They scan infrastructure, test API endpoints, and check access controls. Once these checks pass, they declare the system secure. This approach leaves a massive part of the attack surface completely unexamined.

Traditional IT controls only protect the software wrapper. They fail to address the systemic vulnerabilities inherent to machine learning models, training data, and LLM orchestration.

For chief AI officers, AI architects and risk managers, relying solely on standard cybersecurity frameworks creates a false sense of security while leaving core operational assets exposed.

MITRE ATLAS currently catalogs over 80 techniques organized across 14 tactics for attacking AI systems. NIST AI 100-2 provides a systematic taxonomy of adversarial machine learning attacks by lifecycle stage. OWASP’s Top 10 for LLM Applications identifies the highest-priority risks for language model deployments. And yet most organizations performing AI security assessments reference none of these AI-specific frameworks.

This post covers the complete AI threat assessment process: from foundational principles through STRIDE adaptation for AI, testing practices for predictive, generative, and agentic systems, the critical differences between assessing built versus bought AI, and the practical implementation model that turns this guidance into operational security.

Why AI Threat Assessment Is Fundamentally Different

Traditional software behaves deterministically. Given the same input, it produces the same output. Its logic is explicitly coded. Its behavior can be fully inspected through source code review.

AI systems violate every one of these assumptions. They learn behavior from data rather than having it programmed. They produce probabilistic outputs that may vary. Their decision boundaries are often opaque even to their developers. And their supply chain includes not just code libraries but datasets, pre-trained models, fine-tuning data, and embeddings that each introduce distinct vulnerability classes.

This creates an attack surface across dimensions that traditional security never addressed.

Data-centric attacks manipulate training data, labels, feature pipelines, retrieval corpora, or feedback loops to influence model behavior without modifying any code.

Model-centric attacks exploit the learned behavior of the model itself through adversarial inputs, extraction queries, or inversion techniques.

Pipeline-centric attacks compromise the MLOps infrastructure, model registries, training environments, or deployment pipelines.

Human interaction attacks exploit the model’s natural language interface through prompt injection, social engineering, or manipulation of user-facing outputs.

Autonomy attacks exploit tool access, planning capabilities, memory systems, or action authorization in agentic AI systems.

Your AI security assessment is not a single test. It’s a recurring process integrated into your development lifecycle and MLOps pipeline, covering every phase from data collection through model retirement.

Implementation tip: Before conducting any AI security assessment, classify the AI system type (predictive, generative, or agentic) and sourcing model (built internally or procured from a vendor). These two classifications determine which threat vectors are most relevant, which testing techniques apply, and where the primary risks concentrate. A predictive fraud detection model built in-house has a completely different threat profile from a procured generative AI chatbot or an internally developed autonomous agent. Applying a generic “AI security checklist” to all three produces assessments that miss the most important risks for each system type.

The Six-Phase AI Security Assessment Process

A repeatable, multi-phase process aligned with NIST AI RMF and ISO/IEC 42001 ensures comprehensive coverage across the AI lifecycle.

Phase 1: Define scope and objectives. Identify which AI systems, environments, and use cases are in scope. Document risk tolerance and success criteria with specific measurable standards: “no PII in outputs,” “no more than 3% performance degradation after adversarial hardening,” “prompt injection bypass rate below 0.1%.” Vague success criteria produce vague assessments.

Phase 2: Inventory AI assets and data flows. Catalog models, datasets, pipelines, training and inference infrastructure, and external dependencies including third-party APIs and open-source components. Include metadata: data lineage, model versions, training configuration, deployment endpoints, prompt templates, tool permissions, and retrieval corpora. Build an architecture diagram that captures every data flow, trust boundary, and external dependency.

Phase 3: Threat mapping and vulnerability analysis. Apply STRIDE-AI threat modeling per asset. Use MITRE ATLAS to identify common attack patterns specific to your system type. Consider attack surfaces across inputs, training data, model parameters, interfaces, logs, monitoring systems, and agent tools. Build scenario-based risk assessments for the most consequential threats.

Phase 4: Perform targeted security tests informed by the threat model: adversarial testing, prompt injection testing, data integrity tests, privacy leakage tests, agent behavior tests, and abuse resistance tests. Use a mix of automated tooling and manual testing. Test against the specific threats identified in Phase 3, not against a generic checklist.

Phase 5: Risk scoring and prioritization. Use a likelihood-impact matrix with AI-specific scoring. The OWASP AI Vulnerability Scoring System (AIVSS) provides scoring dimensions designed for AI risks including agentic systems. Maintain an AI risk register linking threats, vulnerabilities, controls, and residual risk to business impact and regulatory constraints.

Phase 6: Mitigation and continuous monitoring. Implement layered controls: access control, input validation, rate limiting, adversarial training, differential privacy, data validation, output filtering, robust logging, and human approval gates. Set up ongoing monitoring of performance, drift, anomaly behavior, and security signals. Loop findings back into the risk assessment.

Phase 2, the asset inventory, is where most AI security assessments fail before they begin. Teams inventory the model and the API endpoint but miss the data pipeline, the feature store, the retrieval corpus, the prompt templates, the tool configurations, and the monitoring infrastructure. Each of these components is an asset with its own threat profile and its own attack surface. Build your inventory by tracing every data flow from source through processing, training, deployment, inference, and monitoring. Every system that touches AI data or artifacts is an asset in scope. If you can’t draw the complete data flow diagram, you can’t conduct a complete threat assessment.

STRIDE Adapted for AI: The Complete Threat Mapping

Classic STRIDE was built for deterministic software. AI systems are not deterministic.

They introduce new assets. Training data, labels, feature pipelines, learned parameters, embeddings, model cards, evaluation datasets. They also introduce new failure modes. Biased data, poisoning, adversarial inputs, privacy leakage through inversion, and emergent behavior in generative systems.

If you apply STRIDE without adapting it, you will miss the real attack surface.

Here is how each component changes in practice.

S — Spoofing: When Trust Boundaries Collapse

In AI systems, spoofing is not just about pretending to be a user.

It is about faking anything the model trusts.

This includes training data sources presented as legitimate, trojanized models distributed through public hubs, fake service identities calling model APIs, and spoofed tools or plugins in agent-based systems. One of the most overlooked vectors is prompt identity manipulation, where an attacker reframes the model’s role and changes its behavior without touching the system itself.

This aligns with what OWASP highlights in LLM systems. The model often cannot distinguish between trusted and untrusted instructions unless you enforce that separation explicitly.

What works in practice:

Enforce strong identity and access management across users, services, and pipelines
Require mutual authentication between internal components
Sign datasets and model artifacts cryptographically and verify before use
Validate model provenance. Do not trust public models without integrity checks
Restrict external tools and plugins using explicit allowlists

If your system consumes external inputs dynamically, assume they can be impersonated.

T — Tampering: Changing the System Without Touching the Code

Tampering in AI systems rarely looks like traditional code changes.

It targets what the model learns or how it interprets inputs.

The most critical risks include training data poisoning, where crafted samples introduce backdoors, and label manipulation, where ground truth is subtly corrupted. Feature pipeline tampering can shift inputs without detection. Direct modification of model weights, prompt template changes, retrieval corpus poisoning in RAG systems, and long-term agent memory corruption all fall into this category.

Google’s Secure AI Framework and Microsoft’s AI security guidance both emphasize this layer. If your data or pipeline is compromised, your model is compromised.

Controls that hold up:

Track full data lineage from ingestion to training
Sign and version datasets, features, and models
Hash model artifacts and verify integrity before deployment
Enforce strict change control with separation of duties
Use immutable logs to track all modifications
Monitor for drift or unexpected behavior after deployment

If you cannot trace how data changed over time, you cannot trust the model’s output.

R — Repudiation: When You Cannot Prove What Happened

Repudiation becomes critical the moment your AI system affects real people.

Most systems fail here quietly.

You see missing records of who modified datasets or models, no version history for prompts or system instructions, and no way to reconstruct why a specific output occurred. In regulated environments, this is not just a gap. It is a failure.

NIST and ISO frameworks both treat traceability as a core requirement for trustworthy AI.

Controls you actually need:

End-to-end audit logging across data, training, and inference
Version control for prompts, models, datasets, and configurations
Traceability linking each output to model version and input context
Signed approvals for training runs and deployments
Tamper-evident storage for logs

If you cannot explain a decision after the fact, you do not control the system.

I — Information Disclosure: When the Model Reveals Too Much

AI systems create new ways to leak sensitive information.

Not through breaches, but through normal use.

Models can memorize and reproduce training data. They can expose system prompts through carefully crafted queries. They can generate personally identifiable information, even when you did not intend them to. Membership inference and model inversion attacks can reveal whether specific data was used in training or reconstruct sensitive attributes. In agent systems, secrets can leak through retrieval or tool interactions.

This is well documented in academic research and reflected in OWASP’s top risks for LLMs.

Controls that reduce real exposure:

Minimize sensitive data in training and retrieval pipelines
Apply output filtering and redaction layers
Test actively for leakage using adversarial prompts
Use privacy-preserving techniques such as differential privacy where needed
Segment access to data, models, and tools
Encrypt sensitive data at rest and in transit
Apply data loss prevention on outputs, not just storage

Do not assume your model will “just avoid” sensitive data. Test it until it fails.

D — Denial of Service: When Usage Becomes the Attack

AI systems change the economics of denial of service.

The goal is not always to take the system down. It is to make it expensive or unstable.

Attackers can flood APIs with requests, exploit token limits in language models, craft prompts that maximize compute usage, or trigger infinite loops in agent workflows. Retrieval systems and data pipelines can also be overloaded upstream.

Google explicitly calls out resource exhaustion as a primary AI risk. In practice, this often shows up first as a cost spike, not an outage.

Controls that work under pressure:

Enforce rate limits and per-user quotas
Restrict input size and context length
Implement cost-aware request validation
Use circuit breakers for runaway processes
Isolate resources across tenants and workloads
Define fallback modes when limits are reached

Watch for patterns, not just spikes. Repeated unusual inputs usually mean someone is testing your limits.

Threats That STRIDE Alone Doesn’t Capture

Six AI-specific threat categories require explicit attention beyond what STRIDE provides.

Data poisoning manipulates training, fine-tuning, retrieval, or feedback data to corrupt model behavior. Three poisoning types create different impacts: availability poisoning degrades overall performance, integrity poisoning creates targeted backdoor behavior, and bias poisoning skews outcomes for specific groups or cases. Controls include provenance verification, data quality rules, outlier detection, trusted labeling processes, holdout integrity datasets, and differential retraining review.

Evasion and adversarial examples craft inputs that cause misclassification or bypass detection at inference time. These attacks are common in computer vision, audio processing, fraud detection, malware classification, and content moderation. Controls include adversarial robustness testing, input preprocessing, ensemble defenses, confidence thresholds, and human review for high-risk decisions.

Model extraction and theft allows attackers to replicate model behavior or steal intellectual property through systematic API queries. Controls include query monitoring, rate limiting, response minimization (returning only necessary information), access controls, and watermarking where applicable.

Prompt injection places malicious instructions in user inputs, documents, web pages, emails, or tool outputs, causing the model to ignore system instructions or exfiltrate information. This is particularly important for LLMs and RAG systems where the model processes content from multiple trust domains. Controls include treating model instructions and untrusted content as separate trust domains, retrieval content sanitization, tool-use policies enforced outside the model, and human approval for high-risk actions.

Hallucination and fabrication produce confidently stated incorrect information. While not always a malicious attack, it creates exploitable security and business risk when outputs are used to make decisions or take actions. Controls include grounding mechanisms, verification checks, confidence indicators, output validation, and restrictions on automated use of unverifiable outputs.

Agentic risks are unique to AI systems that plan, call tools, update memory, and act on the environment. These include goal hijacking, tool abuse, recursive harmful loops, multi-step hidden failure chains, memory poisoning, and cross-system lateral movement through authorized tools. Controls include least-privilege tool access, approval gates for sensitive actions, action sandboxing, short-lived credentials, step-level logging, and budget, time, and action limits.

Implementation tip: The threat that catches the most organizations off guard is indirect prompt injection in RAG systems. Direct prompt injection (the user types malicious instructions) is well understood. Indirect injection (malicious instructions are embedded in documents, emails, or web pages that the model retrieves and processes) is harder to detect because the malicious content enters through the retrieval pipeline rather than through the user interface. When assessing RAG systems, treat every document in the retrieval corpus as untrusted input regardless of its original source. A document that was trustworthy when it was created can be modified later by someone who understands how the RAG system processes retrieved content. Content sanitization at the retrieval boundary is a critical control that most RAG deployments lack.

Common AI Vulnerabilities to Assess

Weak Access Control
Weak access control exists when users, services, pipelines, or agents can access models, datasets, prompts, tools, vector stores, or configuration assets beyond their authorized scope. This is one of the most critical AI vulnerabilities because excessive or poorly segmented access allows unauthorized changes to model behavior, training inputs, prompt logic, and deployment settings. In practice, this weakness appears as overprivileged service accounts, shared credentials, missing role separation, or poor enforcement of least privilege across AI development and runtime environments. It materially increases the likelihood of tampering, data exposure, model misuse, and unauthorized operational actions.
Insecure API Exposure
Insecure API exposure occurs when model endpoints, orchestration layers, or inference services are exposed without strong authentication, authorization, encryption, abuse controls, and request validation. This weakness creates a direct path for unauthorized access, model extraction, data leakage, prompt abuse, and denial-of-service against AI services. The issue is especially severe in public-facing AI APIs and internal services that are assumed to be trusted but are reachable from broad enterprise networks. Teams should treat every AI endpoint as a sensitive control surface rather than a standard application interface.
Poor Input Validation
Poor input validation exists when prompts, files, retrieved content, labels, feature values, tool responses, or multimodal inputs are accepted without robust sanitation, schema enforcement, source trust checks, and semantic validation. This is a foundational weakness in AI systems because untrusted inputs can shape model behavior even when the infrastructure itself is not compromised. In generative and agentic systems, this weakness enables prompt injection, tool misuse, and context contamination, while in predictive systems it increases exposure to adversarial manipulation and poisoned data entry. Effective validation must cover not only syntax and type checking, but also trust boundaries, semantic constraints, and control-plane separation.
Weak Change Management
Weak change management exists when models, prompts, datasets, feature pipelines, policies, or runtime settings can be modified without formal approval, traceability, testing, and rollback controls. AI systems are highly sensitive to small changes, and undocumented updates to prompts, retrieval rules, or generation parameters can materially alter security posture and business behavior. This vulnerability commonly appears in fast-moving ML teams where experimentation practices leak into production without release discipline. The result is a system that cannot reliably prove what changed, who changed it, or whether a harmful outcome came from code, data, model, or configuration drift.
Insufficient Logging
Insufficient logging occurs when the system does not retain adequate records of prompts, retrieved context, model versions, feature states, tool calls, policy decisions, user actions, and deployment events. This weakness undermines incident response, root-cause analysis, forensic review, and accountability because AI failures often emerge through multi-step interactions across several components. In many organizations, logging is either too sparse to investigate incidents or too inconsistent across the AI lifecycle to reconstruct what actually happened. Without strong event logging, the organization cannot reliably detect misuse, prove compliance, or learn from operational failures.
Weak Artifact Protection
Weak artifact protection exists when model weights, checkpoints, prompt templates, tokenizer files, evaluation sets, configurations, and deployment bundles are stored without strong encryption, integrity validation, and access restrictions. These artifacts are not just operational files; they are high-value assets that encode business logic, intellectual property, system behavior, and sometimes even sensitive data. If artifact storage is weak, attackers or insiders can tamper with models, steal proprietary assets, or deploy manipulated versions without detection. This weakness is particularly serious in environments where artifacts are copied across notebooks, registries, object stores, and CI/CD systems with inconsistent controls.
Unrestricted Query Access
Unrestricted query access exists when users or systems can interact with a model at high volume, high frequency, or high fidelity without rate limits, quotas, anomaly detection, or behavioral restrictions. This weakness makes AI systems far easier to abuse for model extraction, prompt probing, confidence analysis, and cost-amplifying attacks. It is especially common in commercial AI APIs and internal platforms that prioritize usability over abuse resistance. From a control perspective, the problem is not simply exposure, but exposure without meaningful guardrails on volume, response detail, or usage patterns.
Weak Prompt Isolation
Weak prompt isolation exists when system instructions, developer prompts, user input, retrieved content, tool output, and memory are mixed together without clear trust separation or policy enforcement. This is a defining weakness in modern generative and agentic systems because the model cannot reliably distinguish trusted operational instructions from adversarial content unless the architecture does so explicitly. When prompt layers are not isolated, the system becomes highly vulnerable to instruction override, hidden context manipulation, and leakage of internal logic. This is not just a prompt design issue; it is an architectural control failure.
Excessive Tool Permissions
Excessive tool permissions occur when AI agents or orchestration services are granted broader access to APIs, files, workflows, or enterprise systems than the use case requires. This weakness turns ordinary model error into high-impact operational risk because the model can trigger actions, access sensitive systems, or modify records without independent restriction. In many agentic deployments, the tool layer inherits broad enterprise permissions because service accounts are easier to manage than scoped credentials. The result is an action surface that violates least privilege and magnifies the consequences of prompt abuse, model error, or orchestration flaws.
Weak Runtime Authorization
Weak runtime authorization exists when the system relies on the model itself to decide whether a request, action, or tool invocation is allowed instead of enforcing policy through deterministic control layers. This is a serious design weakness because AI models are probabilistic components and should not serve as the final authority for sensitive actions, regulated workflows, or high-impact business decisions. The failure often appears in agentic systems where prompts are expected to enforce policy instead of code, workflow rules, or authorization services. This creates a brittle security model that is easy to manipulate and hard to audit.
Complex Model Loading
Complex model loading exists when serialized models, checkpoints, custom loaders, or deserialization workflows allow unsafe code execution, untrusted object parsing, or weak artifact validation at load time. This is a major implementation weakness in ML ecosystems where convenience mechanisms are often prioritized over secure loading practices. If model loading is not tightly controlled, a malicious artifact can execute code, alter runtime behavior, or compromise the environment before the model even serves inference. Teams should treat model loading as a software supply chain and code execution risk, not just a deployment step.
Insufficient Provenance Controls
Insufficient provenance controls exist when the organization cannot reliably verify where data, labels, models, prompts, or derived artifacts came from, who changed them, and whether they remained intact through the lifecycle. This weakness allows poisoned, biased, stolen, or noncompliant assets to enter the pipeline with limited ability to validate authenticity or reconstruct lineage. It commonly affects organizations with decentralized data sourcing, weak dataset versioning, or undocumented fine-tuning and retrieval workflows. Without strong provenance, integrity and accountability collapse across training, evaluation, and deployment.
Data Poisoning Susceptibility
Data poisoning susceptibility exists when training, fine-tuning, feedback, or retrieval data can be introduced or modified without strong validation, curation, anomaly detection, and approval controls. This weakness does not describe the attack itself; it describes the broken state in which malicious or low-integrity data can influence future system behavior without being detected. The vulnerability is particularly severe in systems that continuously learn, accept user feedback, or ingest external data at scale. It reflects weak data governance, inadequate sanitation, and poor separation between trusted and untrusted sources.
Weak Data Governance
Weak data governance exists when the organization lacks formal controls for data ownership, quality requirements, lifecycle handling, access restrictions, lawful use, retention, and accountability across AI pipelines. This weakness creates systemic exposure because even well-engineered models become unreliable when built on poorly governed data assets. It often appears as undocumented data flows, unclear stewardship, inconsistent policies between business units, and missing controls over reuse of data across training, testing, and inference. In practice, it leads to integrity failures, privacy issues, compliance gaps, and unreliable AI outcomes.
Inadequate Monitoring
exists when the system does not continuously observe model behavior, data quality, abuse patterns, drift, service health, policy violations, and integration failures after deployment. AI systems require stronger runtime observability than conventional software because harmful behavior often emerges gradually or probabilistically rather than through a single obvious fault. Many organizations deploy AI services with infrastructure monitoring but no meaningful visibility into model misuse, degraded output quality, unsafe agent behavior, or retrieval corruption. This weakness allows failures and attacks to persist long after they become operationally material.
Missing Drift Controls
Missing drift controls exist when the organization does not monitor and respond to changes in input distributions, feature behavior, environmental conditions, user behavior, or underlying concepts over time. This weakness is especially important in and adaptive production environments where the model can silently become less accurate, less fair, or less robust without triggering formal incidents. In generative systems, drift can also affect retrieval quality, grounding reliability, and prompt behavior as enterprise content or user patterns evolve. Without drift detection and response processes, the organization loses assurance that the deployed system still matches the validated one.
Weak Data Quality Controls
Weak data quality controls exist when completeness, consistency, validity, freshness, representativeness, and defect thresholds are not formally defined and enforced across the AI data lifecycle. This is one of the most common root weaknesses in AI projects because poor-quality data can degrade model performance, mask poisoning, amplify bias, and undermine evaluation confidence. In many environments, data quality controls are applied inconsistently across ingestion, labeling, feature engineering, and retraining. The vulnerability is not just bad data, but the absence of control mechanisms that would detect and stop it.
Distributed Data Inconsistency
Distributed data inconsistency occurs when multiple repositories, feature stores, data lakes, labels, or training environments maintain different versions of supposedly authoritative data without synchronization or reconciliation controls. This weakness creates hidden divergence between what the model was trained on, what it is evaluated on, and what it sees in production. In AI systems, such inconsistency can lead to unstable performance, unexplained regressions, and weak incident traceability. The issue is especially severe in organizations with decentralized AI teams, fragmented storage patterns, or asynchronous data updates.
Complex Data Transformations
Complex data transformations exist when raw data passes through many preprocessing, normalization, filtering, enrichment, or encoding stages that are poorly documented, weakly tested, or inconsistently applied. Each transformation step can introduce loss, corruption, bias, or mismatch, especially when different teams maintain different portions of the pipeline. This vulnerability is common in mature AI stacks where data preparation logic has accumulated over time without end-to-end validation. The more opaque the transformation chain, the harder it becomes to detect errors and defend data integrity.
Schema Incompatibility
Schema incompatibility exists when different components in the AI pipeline rely on inconsistent field definitions, formats, units, labels, token structures, or metadata conventions. This weakness often forces ad hoc conversion logic that increases the likelihood of silent data corruption, feature mismatch, and failed integration between training, serving, and governance systems. It is particularly harmful in large AI programs with multiple vendors, legacy systems, or rapidly evolving pipelines. Standardized schemas are a control requirement, not just a convenience.
Uncontrolled Data Ingestion
Uncontrolled data ingestion exists when data enters the AI system from multiple sources without centralized validation, source trust assessment, security checks, and ownership controls. This creates a weak perimeter around one of the most critical parts of the AI lifecycle: what the system is allowed to learn from or reason over. The weakness is especially significant in RAG systems, crowdsourced pipelines, and environments that blend user data, third-party feeds, internal documents, and automation outputs. Without controlled ingestion, harmful or low-integrity data can enter the system faster than governance can detect it.
Weak De-Identification
Weak de-identification exists when personal, proprietary, or regulated data is tokenized, masked, pseudonymized, or transformed in ways that still permit re-identification through linkage, inference, metadata, or model behavior. This is a major privacy weakness in AI pipelines because derivative artifacts such as embeddings, prompts, logs, and model outputs can reintroduce exposure even if raw source fields were obfuscated. Organizations often overestimate the protection provided by simplistic masking approaches and fail to test for realistic re-identification risk. The result is a false sense of privacy assurance.
Training Data Memorization
Training data memorization exists when the model retains and can reproduce sensitive or proprietary content from training or fine-tuning data because minimization, filtering, and privacy-preserving techniques were insufficient. This is a model and training weakness, not merely a misuse scenario, because the model architecture and training process allow undue retention of sensitive information. It is especially concerning in large generative models and domain models trained on regulated or confidential corpora. Assessment should treat memorization risk as a direct outcome of weak training controls and weak privacy-by-design practices.
Weak Transfer Validation
Weak transfer validation exists when pretrained models, foundation models, or transferred representations are adopted without rigorous verification that they are suitable, safe, and reliable in the new domain or use case. Many teams assume that a strong base model remains trustworthy after fine-tuning or contextual adaptation, but hidden weaknesses, bias patterns, or unsafe behaviors can carry forward into production. This vulnerability reflects weak governance over model adoption and insufficient validation in the target environment. It is especially important where open-source or third-party models are used to accelerate development.
Insufficient Model Validation
Insufficient model validation exists when testing and assurance activities do not adequately evaluate security, robustness, fairness, privacy, performance, and failure modes before release. This is one of the most serious AI control failures because it allows unreliable or unsafe models to reach production based on narrow benchmark performance or incomplete QA. In practice, the weakness appears as limited adversarial testing, poor subgroup evaluation, inadequate edge-case coverage, or overreliance on static benchmark scores. A model that is not thoroughly validated is not ready to operate in a real business environment.
Weak Feedback Loops
Weak feedback loops exist when the organization does not systematically collect, triage, and incorporate user feedback, incident findings, model errors, and performance observations into ongoing model improvement and governance. This weakness allows known issues to persist and prevents the system from adapting to operational reality. In AI systems, feedback is not merely a product improvement tool; it is part of the control environment needed to detect emergent risks and performance regressions. Where feedback exists but is ungoverned, it can also become a source of corruption rather than improvement.
Over-Automation Dependence
Over-automation dependence exists when the system or business process relies on AI outputs without sufficient human oversight, review checkpoints, escalation paths, or compensating controls. This is a critical socio-technical weakness because it turns model error, bias, hallucination, or manipulation into direct business harm. It often appears in operational workflows where users treat AI output as authoritative because the process was designed for speed or scale rather than challenge and review. The vulnerability is not that humans use AI, but that the process removes meaningful human judgment where it is still required.
Weak Intended Use Controls
Weak intended use controls exist when there are no technical or procedural mechanisms to ensure the AI system is used only within approved purposes, domains, user groups, and risk boundaries. This weakness is especially important in enterprise settings where a model built for a low-risk task can quietly migrate into a higher-risk use case without new validation or governance review. The result is misuse by expansion rather than by intrusion. Effective intended-use control requires policy, workflow, access boundaries, and usage monitoring—not just documentation.
Missing AI Policies
Missing AI policies exist when the organization lacks clear standards, governance rules, and control expectations for AI development, deployment, procurement, use, and retirement. This creates inconsistent practices across teams and leaves critical decisions to local interpretation rather than enterprise governance. In such environments, security, privacy, fairness, and incident response controls are applied unevenly or too late. A missing policy framework is not just a governance gap; it is a systemic enabler of technical weakness.
Undefined AI Roles
Undefined AI roles exist when responsibilities for model ownership, data stewardship, risk acceptance, monitoring, security, and operational response are not clearly assigned. This creates accountability gaps that allow issues to persist because no one is formally responsible for detecting, approving, or remediating them. In AI systems, unclear role boundaries are especially dangerous because responsibility is often split across security, data science, engineering, compliance, and business teams. This weakness undermines governance even when individual technical controls exist.
Lack of Design Documentation
Lack of design documentation exists when system architecture, model assumptions, trust boundaries, control points, data dependencies, tool integrations, and operational workflows are not formally documented. This makes the AI system harder to secure, audit, maintain, and change safely over time. In practice, undocumented systems accumulate hidden dependencies and implicit logic that weaken security and resilience. Teams cannot govern what they cannot clearly describe.
Weak Explainability Controls
Weak explainability controls exist when the system cannot adequately trace outputs, recommendations, or actions back to relevant inputs, model states, decision pathways, or policy conditions. This is a practical vulnerability because weak traceability impairs auditing, root-cause analysis, challenge rights, compliance reviews, and trust in business-critical AI decisions. The issue is not that every model must be fully interpretable, but that the level of explanation is insufficient for the risk and use case. In regulated or high-impact settings, that gap becomes a serious control failure.
Poor User Guidance
Poor user guidance exists when end users, reviewers, and operators do not receive clear instructions on system limits, approved use cases, escalation procedures, confidence handling, and expected validation steps. This weakness increases misuse, overreliance, operational error, and poor adoption because users are left to invent their own safety practices. In AI environments, user documentation is part of the control framework rather than a support artifact. Weak guidance creates foreseeable misuse conditions that should have been prevented.
Missing Reporting Channels
Missing reporting channels exist when employees, users, or operators have no defined way to raise concerns about harmful outputs, bias, security events, unsafe actions, or governance issues related to AI systems. This prevents early detection of issues that may not appear in automated monitoring and weakens organizational accountability. In many programs, concerns are raised informally and never reach teams with authority to investigate or remediate them. A system without reporting channels lacks a core feedback and governance control.
Unauthorized Parameter Changes
Unauthorized parameter changes occur when model weights, prompt settings, thresholds, hyperparameters, routing logic, or safety configurations can be modified without strict approval, access restrictions, and audit trails. AI systems are highly sensitive to parameter changes, and even small adjustments can alter risk posture, output quality, and control behavior. This vulnerability often appears in environments where experimentation platforms and production environments are not well separated. The weakness is not just change itself, but change without governance integrity.
Weak Event Traceability
Weak event traceability exists when event records are incomplete, inconsistent, or disconnected across data pipelines, model training, deployment, inference, and downstream action layers. This leaves the organization unable to correlate incidents across components or explain how a harmful output became a harmful action. AI systems are often composed of loosely coupled services, making end-to-end traceability a control necessity rather than an enhancement. Without it, security events and reliability issues remain opaque and slow to resolve.
Weak Performance Auditing
Weak performance auditing exists when model accuracy, robustness, fairness, stability, and operational effectiveness are not reviewed on a regular and independent basis after release. This weakness allows performance degradation, hidden bias, and emerging failure patterns to persist below the threshold of incident response. Many organizations treat model evaluation as a one-time pre-launch activity instead of an ongoing assurance obligation. As a result, the deployed system may drift far from its approved performance profile without triggering formal review.
Poor Resource Documentation
Poor resource documentation exists when required infrastructure, compute dependencies, storage assumptions, data interfaces, runtime requirements, and support tooling are not clearly documented across the AI lifecycle. This creates avoidable delays, scaling failures, insecure workarounds, and weak capacity planning. In operational terms, undocumented resources make recovery, troubleshooting, and secure deployment much harder than they should be. It is a governance and reliability weakness with direct security implications.
Poor Tooling Documentation
Poor tooling documentation exists when development, training, validation, deployment, and monitoring tools are not fully documented in terms of purpose, configuration, ownership, support boundaries, and security expectations. AI programs often depend on a broad set of notebooks, registries, experiment platforms, feature stores, package managers, and orchestration tools that become hidden risk sources when poorly documented. This weakness increases integration errors, unsupported usage, and blind spots in security review. Tool sprawl without documentation is a predictable control failure.
Complex Architecture Sprawl
Complex architecture sprawl exists when the AI environment contains too many interconnected components, undocumented dependencies, ad hoc integrations, and fragmented ownership boundaries to be governed effectively. This is a major architectural weakness because complexity itself expands attack surface, weakens observability, and increases the chance that controls fail at system boundaries. AI systems commonly combine models, retrieval layers, feature pipelines, agents, APIs, and external tools in ways that exceed what teams can consistently secure. When complexity outpaces governance maturity, risk increases sharply.
Single Point of Failure
A single point of failure exists when one component, service, credential, model registry, vector store, feature store, or orchestration node can disable the entire AI capability if it fails or is compromised. This weakness creates avoidable fragility and gives attackers or outages disproportionate leverage over availability and business continuity. In AI systems, single points of failure often hide in supporting components rather than the model itself. Redundancy planning must account for the full AI service chain, not just the inference container.
Limited Redundancy
Limited redundancy exists when there are insufficient failover paths, backup services, alternate models, duplicate storage controls, or resilient deployment patterns to sustain operations during failure. This weakness is common in AI systems because teams often optimize for performance and cost before designing for resilience. The result is longer outages, slower recovery, and increased blast radius from infrastructure or component failures. Resilience should be engineered into AI operations, not added only after service disruption occurs.
Inconsistent Backups
Inconsistent backups exist when models, prompts, vector indexes, training artifacts, policies, and configuration states are not backed up in a complete, current, and restorable manner. This weakness prevents reliable recovery from corruption, rollback errors, ransomware, accidental deletion, or failed deployments. AI systems require backup strategies that preserve behavioral state, not just file availability. Partial or outdated backups can restore service technically while still restoring the wrong or unsafe model behavior.
Delayed Model Recovery
Delayed model recovery exists when recovery procedures for models, artifacts, indexes, or orchestration state are slow, manual, or untested. This weakness extends downtime and increases operational loss after failure or compromise. In AI environments, restoration is often more complex than standard application recovery because it depends on version alignment across data, model, prompt, and control artifacts. Recovery speed is therefore a direct resilience control, not just an operational metric.
Inconsistent Version Control
Inconsistent version control exists when datasets, prompts, models, features, and deployment configurations are not versioned consistently across teams and environments. This creates uncertainty about what is running, what was tested, and what should be rolled back after failure. AI systems depend on tightly coupled artifacts, and weak version discipline creates hidden mismatch between training, evaluation, and production. It is a fundamental reproducibility and integrity weakness.
Insufficient Resource Monitoring
Insufficient resource monitoring exists when compute, memory, storage, concurrency, token consumption, and tool usage are not observed closely enough to detect abuse, saturation, inefficiency, or performance collapse. This weakness can hide extraction attempts, denial-of-service conditions, agent loops, and cost overruns until they become operationally severe. In AI environments, resource misuse is often a leading indicator of both attack and reliability failure. Monitoring must extend beyond infrastructure uptime to workload behavior and consumption patterns.
Weak Load Distribution
Weak load distribution exists when requests are not balanced effectively across model instances, regions, accelerators, or supporting services. This leads to bottlenecks, avoidable latency, uneven failure patterns, and fragile service behavior under burst traffic or partial outages. AI inference systems often have highly variable workloads, making uneven distribution more damaging than in standard applications. Load balancing is therefore a core operational control for both resilience and abuse resistance.
Limited Tenant Isolation
Limited tenant isolation exists when workloads, sessions, memory, embeddings, prompts, data stores, or inference resources are not adequately separated across users, customers, or business units. This weakness increases the risk of data leakage, cross-session contamination, privilege abuse, and noisy-neighbor denial-of-service. It is particularly important in shared enterprise AI platforms and hosted AI services where the assumption of logical separation may not match the actual architecture. Isolation is a first-order security control, not a deployment optimization.
Weak Incident Coordination
Weak incident coordination exists when communication plans, escalation paths, ownership boundaries, and response procedures for AI incidents are absent, outdated, or untested. This weakness delays containment and creates confusion during events involving harmful outputs, unsafe actions, data leakage, or model degradation. AI incidents often span security, engineering, product, legal, and business teams, making coordination more complex than conventional software response. Without a practiced communication framework, even containable events can escalate.
Poor Hardware Assurance
Poor hardware assurance exists when AI systems rely on low-quality, untrusted, unverified, or weakly monitored hardware platforms for training or inference. This weakness increases the risk of hardware faults, tampering, unstable execution, silent corruption, and unreliable operational behavior. It is particularly relevant for edge AI, specialized accelerators, distributed training hardware, and environments with weak physical security. Hardware trust should be treated as part of the AI control surface, not as a background infrastructure assumption.
Weak Hardware Protection
Weak hardware protection exists when physical interfaces, local consoles, debug ports, firmware update channels, removable media access, and device enclosures are not secured against tampering or unauthorized access. This weakness enables manipulation of execution environments, extraction of artifacts, and compromise of edge or on-premise AI systems. It is especially severe in robotics, IoT, industrial AI, and branch deployments where physical access is realistic. Physical and logical hardware protections must be considered together.
Limited Fault Tolerance
Limited fault tolerance exists when AI systems lack redundancy, error handling, safe degradation, watchdogs, recovery logic, or resilience against malformed inputs and environmental failures. This weakness allows minor faults to escalate into service disruption, wrong predictions, unstable agent behavior, or unsafe operational states. In AI systems that depend on real-time inference or autonomous action, fault tolerance is a safety and security control, not only a reliability feature. Weak fault resilience increases both accidental and adversarial impact.
Observable Side Channels
Observable side channels exist when timing behavior, power characteristics, resource usage, memory access patterns, or electromagnetic emissions reveal information about model execution or processed data. This is a more specialized but real weakness in high-value or edge-deployed AI systems, especially where attackers can observe the hardware closely. The presence of these side channels indicates insufficient hardening at the runtime or hardware interaction layer. While less common than API or data weaknesses, it is important in high-assurance contexts.
Exposed Gradient Information
Exposed gradient information exists when gradient updates, model deltas, or collaborative learning signals can be accessed or analyzed without strong privacy-preserving controls. This weakness is particularly relevant in federated learning and distributed training environments where gradients may leak sensitive information about underlying data. The problem is not collaboration itself, but sharing training signals without sufficient clipping, aggregation, or privacy protection. Where present, it creates a quiet but significant confidentiality weakness.
Weak Metadata Scrubbing
Weak metadata scrubbing exists when logs, API responses, storage objects, file headers, trace records, or debug outputs expose hidden identifiers, source paths, internal roles, or sensitive contextual information. This weakness is often overlooked because the primary data may appear protected while metadata quietly reveals relationships, architecture details, or user information. In AI systems, metadata can also expose prompt structure, feature lineage, or hidden retrieval signals. Proper scrubbing must be deliberate and systematic.
Weak Tokenization Security
Weak tokenization security exists when tokenization or masking approaches are simplistic, reversible, predictable, or insufficiently isolated from original source content. This weakness allows sensitive data to be reconstructed, inferred, or correlated more easily than intended. Organizations often mistake token substitution for robust privacy protection when the surrounding architecture still permits reverse mapping or linkage attacks. Secure tokenization requires sound design, not just transformation.
Black-Box Dependency Reliance
Black-box dependency reliance exists when the organization depends on third-party models or AI services without sufficient transparency into training, controls, update practices, limitations, or failure behavior. This creates assurance gaps because the organization cannot fully evaluate what it is deploying, how it changes over time, or whether vendor claims are valid in the business context. The weakness is most severe in high-impact use cases where explainability, auditability, and predictable behavior are required. Lack of transparency from a dependency is a control weakness even if the component functions well in testing.
Weak Vendor Due Diligence
Weak vendor due diligence exists when suppliers of models, data, tooling, or AI services are not assessed rigorously for security, privacy, reliability, governance maturity, and legal fitness. This allows low-assurance or high-risk components into the environment under weak procurement scrutiny. In AI programs, supplier risk often extends beyond ordinary software assurance because model behavior, data lineage, and update practices are harder to inspect. Weak due diligence is therefore a high-consequence supply chain weakness.
Unverified Third-Party Models
Unverified third-party models exist when pretrained models, open-source checkpoints, or vendor-provided AI components are integrated without robust testing for backdoors, unsafe behavior, hidden bias, privacy issues, or operational fit. This weakness is widespread because model reuse is often treated as an efficiency gain rather than a trust decision. The organization may inherit latent defects or malicious characteristics that were never visible in ordinary benchmark testing. Validation must be contextual, not generic.
Untrusted External Data Sources
Untrusted external data sources exist when the system relies on third-party, scraped, user-contributed, or vendor-supplied data without robust source validation, quality review, licensing review, and trust classification. This weakness creates a direct path for contamination of training, retrieval, and decision logic. It is especially important where business processes assume that external content is good enough because it is convenient or widely used. External data should be treated as untrusted until proven otherwise.
Outdated Third-Party Components
Outdated third-party components exist when open-source libraries, model-serving tools, plugins, agents, SDKs, or integrated software dependencies are no longer supported or are missing current security patches. This weakness exposes AI systems to known vulnerabilities in the underlying software stack even when the model itself is well designed. In AI environments, patching is often delayed because teams fear breaking performance or reproducibility. That hesitation creates a predictable and avoidable security gap.
Weak Supplier Oversight
Weak supplier oversight exists when organizations do not actively monitor vendor performance, security posture, contractual obligations, incident handling, and control effectiveness after onboarding. This weakness leaves the enterprise blind to degradation, drift in vendor practices, hidden subcontractor risk, and unannounced service changes. AI services often change behavior faster than traditional software, which makes passive oversight especially risky. Ongoing monitoring is a required control, not an optional procurement follow-up.
Weak Contract Governance
Weak contract governance exists when supplier agreements do not define security obligations, audit rights, incident notification, data handling restrictions, retention rules, model update expectations, and accountability for failures. This is a vulnerability because technical risk cannot be managed effectively when legal and operational controls are undefined or unenforceable. In AI sourcing, contracts often lag behind actual risk exposure, especially for model updates, prompt retention, and derivative data usage. Weak contracts translate directly into weak assurance.
Vendor Lock-In Dependency
Vendor lock-in dependency exists when the organization relies too heavily on a single AI provider for critical models, infrastructure, APIs, or data services without practical alternatives or migration paths. This creates fragility, weak bargaining power, constrained assurance, and elevated business risk if service quality, cost, compliance posture, or security conditions change. While not always framed as a security issue, concentration risk becomes a resilience and governance weakness when the organization cannot safely diversify or exit. It is particularly relevant for foundation model procurement.
Weak Third-Party Monitoring
Weak third-party monitoring exists when supplier behavior, update cadence, control posture, service quality, and security events are not continuously observed after integration. This prevents the organization from detecting degraded controls, hidden incidents, or changes in model behavior introduced by vendors or external platforms. AI systems often depend on opaque third-party services where passive trust is not justified. Monitoring suppliers is as important as monitoring internal systems when they materially influence AI outcomes.
Poor Third-Party Incident Response
Poor third-party incident response exists when suppliers lack mature procedures, communication channels, escalation speed, and coordination mechanisms for security or AI-specific incidents. This weakness prolongs recovery, obscures root cause, and allows compromise or harmful behavior to propagate across interconnected systems. In AI ecosystems, incidents often cross organizational boundaries and require shared evidence, synchronized containment, and rapid notification. Weak supplier response capability therefore becomes a direct vulnerability in the enterprise’s operating model.
Conflicting Vendor Objectives
Conflicting vendor objectives exist when supplier incentives around speed, feature growth, data usage, retention, or monetization are misaligned with the organization’s security, compliance, reliability, or ethical requirements. This weakness can drive hidden compromises in control quality, transparency, and service fit. It is especially relevant where vendors optimize for scale or product experimentation while the customer requires stability and assurance. Misaligned incentives are a governance weakness that can surface as technical failure later.
Vendor Data Siloing
Vendor data siloing exists when external providers control or fragment critical data, logs, or performance information in ways that reduce visibility, interoperability, or portability for the customer. This weakens monitoring, incident response, root-cause analysis, and strategic flexibility. In AI systems, missing access to model behavior data, usage analytics, or retrieval context can significantly undermine assurance. Data access limitations imposed by vendors should be assessed as a real control weakness, not just a commercial inconvenience.
Weak Requirements Definition
Weak requirements definition exists when AI functional, security, safety, privacy, fairness, resilience, and compliance requirements are incomplete, ambiguous, or undocumented. This vulnerability causes downstream control failures because teams cannot build, test, or govern against requirements that were never made explicit. It is especially common in AI projects where business enthusiasm outruns architectural discipline. Poorly defined requirements produce systems that are technically operational but not reliably controllable.
Weak Planning Discipline
Weak planning discipline exists when the AI project lacks structured lifecycle planning for development, deployment, testing, monitoring, rollback, and retirement. This weakness results in ad hoc decisions, undocumented tradeoffs, control gaps, and fragile implementation practices. In many AI initiatives, experimentation momentum substitutes for engineering rigor, leaving critical security and governance work unfinished. Poor planning is not just a project issue; it is an enabling condition for many downstream vulnerabilities.
Misaligned Business Objectives
Misaligned business objectives exist when , optimization targets, and success metrics do not align with enterprise policy, risk appetite, regulatory obligations, or customer commitments. This creates a structural weakness in which the system may function exactly as designed yet still create harmful or noncompliant outcomes. In practice, misalignment often appears when efficiency, automation, or growth incentives override control objectives. Governance must ensure that optimization does not outpace responsibility.
Weak Human Rights Assessment
Weak human rights assessment exists when system design and governance do not evaluate foreseeable impacts on privacy, discrimination, autonomy, due process, or other affected-party rights. This is a serious weakness in high-impact AI because harms can emerge even when the system is technically accurate and secure in narrow terms. The absence of rights-impact review leaves the organization blind to predictable harm scenarios and regulatory exposure. It also weakens trust and defensibility in public or regulated use cases.
Jurisdictional Control Gaps
Jurisdictional control gaps exist when the system operates across legal regions without clear mechanisms to enforce differing requirements for privacy, transparency, retention, fairness, or AI-specific regulation. This creates fragmented compliance behavior and inconsistent risk treatment across the deployment footprint. In multinational AI programs, legal complexity often exceeds what the architecture was designed to support. Without explicit jurisdictional controls, the organization relies on policy statements that the system cannot actually enforce.
Unknown Customer Expectations
Unknown customer expectations exist when the organization does not adequately understand what users, customers, or impacted parties expect in terms of transparency, safety, privacy, reviewability, and responsible AI behavior. This weakness can lead to technically functioning systems that still fail trust, adoption, or reputational thresholds. It is especially relevant in customer-facing AI and decision-support systems where expectations shape acceptable risk boundaries. Ignoring customer expectations creates a governance blind spot with operational consequences.
Agent Coordination Weakness
Agent coordination weakness exists when multi-agent systems lack strong controls for authentication, communication integrity, role separation, trust boundaries, and behavioral monitoring between agents. This weakness allows one agent’s error, manipulation, or compromise to affect others through hidden coordination pathways. It is particularly relevant in emerging agentic architectures where orchestration complexity grows faster than governance maturity. Multi-agent systems require explicit control design rather than assumptions of cooperative behavior.
Edge Capacity Weakness
Edge capacity weakness exists when AI models deployed on edge devices run too close to hardware, memory, bandwidth, or energy limits to maintain secure and reliable operation under normal or peak conditions. This creates fragile behavior, degraded controls, and higher failure rates during operational stress. The weakness is especially relevant in mobile, industrial, and IoT AI deployments where local resources are constrained and central fallback may be limited. Capacity engineering is therefore a security-relevant design control.
Excessive Compute Demand
Excessive compute demand exists when models, pipelines, or orchestration flows require more computational resources than the environment can reliably sustain. This leads to latency, dropped workloads, cost spikes, and brittle service behavior that can mask abuse or degrade user trust. It is often caused by unoptimized models, poorly governed inference chains, or weak cost-performance engineering. In production, excessive demand becomes a resilience and control weakness, not just an efficiency issue.

Common AI Threat Vectors to Assess

Prompt Injection
Prompt injection is a threat vector in which an attacker supplies malicious instructions through user input, retrieved content, documents, webpages, messages, or tool outputs to alter model behavior. This vector is one of the most important threats for generative and agentic AI because it can override intended instructions, expose sensitive information, bypass safeguards, and induce unauthorized actions. Practitioners should assess whether the system can be manipulated by direct, indirect, or multimodal instruction injection and whether untrusted content can influence decisions, outputs, or tool use.
Data Poisoning
Data poisoning is the deliberate insertion, modification, or curation of training, fine-tuning, feedback, or retrieval data to influence future model behavior. This threat vector is especially important in predictive AI and learning-enabled pipelines because poisoned samples can degrade performance broadly or create targeted backdoors that activate under specific conditions. Assessment should cover poisoning in pre-training data, fine-tuning corpora, labels, retraining feedback loops, and RAG knowledge bases, especially where data is sourced externally or validated weakly.
Backdoor Injection
Backdoor injection is a threat vector in which hidden triggers are embedded into training data or model behavior so the system acts normally most of the time but fails or behaves maliciously when the trigger appears. This vector is especially dangerous because the model can pass standard validation and still contain latent malicious behavior that is difficult to detect before deployment. Practitioners should evaluate outsourced training, third-party model imports, suspicious trigger-response patterns, and whether targeted test cases can surface hidden conditional behavior.
Model Extraction via Queries
Model extraction via queries is a threat vector in which an attacker systematically interacts with a model API or inference service to learn its behavior and reproduce a close functional copy. This threatens both intellectual property and security because the extracted model can be used offline to study decision boundaries, design evasion strategies, or avoid licensing and usage restrictions. Assessment should examine whether repeated querying, confidence outputs, detailed responses, or weak abuse monitoring make extraction feasible at reasonable cost.
Adversarial Evasion
Adversarial evasion is a threat vector in which attackers craft inputs that cause the model to misclassify, mis-rank, or generate unsafe results during inference. This is highly relevant to predictive AI in fraud, vision, malware detection, and classification systems, but analogous forms also exist in generative AI where prompts are designed to induce policy bypass or unsafe completion. Assessment should include targeted and untargeted evasion scenarios, semantic manipulation, obfuscation, environmental perturbation, and sensitivity to minor but adversarially chosen input changes.
Unauthorized Tool Use
Unauthorized tool use is a threat vector in which a model or agent is induced to call plugins, APIs, scripts, databases, or enterprise systems in ways that violate intended authority or business policy. This is a primary concern for agentic AI because the impact moves from unsafe output to unsafe action, including account modification, data exfiltration, workflow corruption, or transaction execution. Practitioners should assess whether a model can trigger sensitive tools through prompt manipulation, tool output manipulation, hidden argument injection, or multi-step planning abuse.
Sensitive Data Extraction
Sensitive data extraction is a threat vector in which attackers recover confidential training data, personal data, secrets, business records, or proprietary knowledge from the model, its outputs, associated storage, or surrounding components. This includes behaviors commonly described as data leakage, exfiltration, membership inference, or privacy extraction depending on the technical path used. Assessment should focus on whether adversaries can obtain sensitive information through ordinary interaction, API abuse, retrieval abuse, debugging interfaces, prompt replay, or model-assisted reconstruction.
RAG Corpus Poisoning
RAG corpus poisoning is a threat vector in which malicious or misleading content is inserted into a document repository, vector database, or enterprise knowledge source that a model later retrieves and treats as authoritative. This is especially important in enterprise generative AI because attackers may not need to attack the model directly if they can influence the retrieval layer with hidden instructions, false facts, or operationally harmful content. Assessment should test whether poisoned documents can alter output behavior, suppress correct information, induce prompt injection, or cause confidential data disclosure.
API Abuse
API abuse is a threat vector in which attackers exploit exposed AI interfaces to manipulate model behavior, extract data, steal models, or degrade service. This is a high-frequency vector across predictive, generative, and agentic systems because APIs often provide the most direct and scalable path into the model and its orchestration environment. Practitioners should assess for weak authentication, broken authorization, missing rate limits, query automation, endpoint discovery, replay abuse, and insecure parameter handling.
API Token Compromise
API token compromise is a threat vector in which attackers steal, leak, reuse, or misuse credentials that grant access to AI models, tools, data stores, orchestration services, or cloud resources. This vector is operationally significant because many AI environments rely heavily on service tokens, integration keys, notebook secrets, and automation credentials that may be overprivileged or poorly rotated. Assessment should include secret exposure in prompts, logs, code repositories, CI/CD pipelines, browser storage, and third-party integrations.
Third-Party Component Compromise
Third-party component compromise is a threat vector in which attackers exploit or subvert external models, libraries, prompt frameworks, package dependencies, APIs, development tools, or model-serving components used by the AI system. This is a major vector in modern AI because most organizations assemble systems from open-source and vendor-supplied parts rather than building every component internally. Practitioners should assess whether imported models, packages, and services can introduce malware, hidden behaviors, unsafe defaults, poisoned dependencies, or undisclosed data flows.
Parameter Tampering
Parameter tampering is a threat vector in which an attacker or unauthorized insider modifies model weights, prompts, hyperparameters, temperature settings, routing logic, safety thresholds, or decision parameters to alter system behavior. This vector can quietly weaken safety controls, degrade predictive accuracy, implant hidden instructions, or shift model behavior in ways that are difficult to detect through ordinary operational monitoring. Assessment should examine access paths to model configuration, parameter update workflows, approval controls, and whether small changes produce disproportionate security impact.
Insider Sabotage
Insider sabotage is a threat vector in which authorized personnel intentionally degrade, corrupt, or weaponize the AI system, often by introducing dormant logic, malicious code, bad data, or harmful operational changes. This vector is especially important in AI environments because developers, data scientists, and MLOps personnel often have broad access to models, datasets, prompts, and deployment pipelines. Practitioners should assess whether insider actions could implant delayed failures, poison training data, change prompts, weaken monitoring, or suppress alerts without timely detection.
Insider Subversion
Insider subversion is a threat vector in which internal personnel are bribed, coerced, recruited, or otherwise influenced to steal AI assets, leak data, or manipulate system behavior for the benefit of external actors such as competitors or criminal groups. This differs from general sabotage because the objective often includes espionage, theft of competitive advantage, or strategic compromise rather than disruption alone. Assessment should examine privileged access, separation of duties, behavioral anomalies, unusual artifact access, and whether sensitive model assets can be exported or altered by a small number of insiders.
Data Provenance Falsification
Data provenance falsification is a threat vector in which metadata, lineage records, ownership fields, timestamps, source identifiers, or chain-of-custody records are altered to disguise the true origin or integrity of AI data. This enables poisoned, biased, stolen, or noncompliant data to enter the training or retrieval pipeline under the appearance of legitimacy. Practitioners should assess whether source records can be forged, overwritten, or detached from actual datasets and whether data trust decisions rely too heavily on editable metadata.
Label Poisoning
Label poisoning is a threat vector in which labels in supervised learning datasets are manipulated, corrupted, or systematically skewed to alter model decision boundaries and degrade reliability. This vector can be used to reduce overall performance, create targeted blind spots, or make the model favor attacker-selected outcomes while leaving raw feature data unchanged. Assessment should include annotation workflows, reviewer independence, class distribution anomalies, suspicious relabeling events, and whether label quality is monitored throughout retraining.
Bias Exploitation Through Imbalanced Data
Bias exploitation through imbalanced data is a threat vector in which attackers or negligent processes take advantage of underrepresented groups, skewed classes, or socially biased data distributions to produce discriminatory or harmful outcomes. While not always an intentional attack, it becomes a threat vector when bad actors knowingly manipulate or leverage the imbalance to influence outcomes in hiring, lending, fraud screening, identity systems, or public-facing services. Assessment should cover representativeness, subgroup error rates, data collection bias, and whether adversaries could steer outcomes by amplifying biased or nonrepresentative inputs.
Model Inversion
Model inversion is a threat vector in which an attacker analyzes model responses to reconstruct sensitive attributes, representative records, or approximations of training data. This is particularly relevant where models are trained on healthcare, biometric, financial, or otherwise sensitive data and expose rich responses or confidence information. Practitioners should assess whether outputs, gradients, embedding access, or repeated targeted queries enable inference of private records or sensitive attributes.
Membership Inference
Membership inference is a threat vector in which an attacker determines whether a specific individual, record, or item was included in a model’s training data. This may seem narrow, but it can create serious privacy and legal exposure when mere participation in a dataset is itself sensitive, such as in healthcare, law enforcement, employment, or intelligence contexts. Assessment should examine whether output confidence, overfitting, differential behavior, or verbose responses allow adversaries to infer dataset membership with meaningful accuracy.
Gradient Leakage
Gradient leakage is a threat vector in which an attacker reconstructs training examples or infers sensitive information from gradient updates or model parameter changes shared during distributed or federated learning. This vector is well established in technical literature and is especially important where organizations use collaborative learning methods under the assumption that sharing gradients is inherently privacy-preserving. Assessment should evaluate secure aggregation, differential privacy, clipping, update access, and whether shared training signals could reveal individual data points.
Hallucination Exploitation
Hallucination exploitation is a threat vector in which attackers intentionally cause a generative model to produce false, fabricated, or misleading content that can then be used to deceive users, justify action, or contaminate downstream workflows. This is particularly relevant in high-trust business settings where plausible but incorrect outputs may be accepted as valid by operators, customers, or automated systems. Practitioners should assess whether the model can be induced to invent facts, credentials, citations, procedures, or policy interpretations in ways that materially affect operations or decisions.
Toxicity Induction
Toxicity induction is a threat vector in which attackers provoke a model into generating hateful, abusive, sexually explicit, extremist, or otherwise harmful content. This is especially important for public-facing generative AI because harmful output can create immediate legal, reputational, and trust consequences even without broader system compromise. Assessment should test whether adversaries can elicit toxic output across languages, contexts, and obfuscation methods, including role-play, paraphrase, and coded language.
Dual-Use or Malicious Repurposing
Dual-use or malicious repurposing is a threat vector in which a model designed for benign enterprise use is repurposed, stolen, or adapted for fraud, misinformation, surveillance, phishing, deepfakes, or other harmful purposes. This vector matters both internally and externally because misuse may come from authorized employees, malicious customers, or external actors who obtain model access or derivative artifacts. Assessment should cover abuse patterns, policy restrictions, customer and employee monitoring, and whether the model’s capabilities create foreseeable misuse channels.
Overreliance / Automation Bias
Overreliance is a threat vector in which humans accept AI outputs or recommendations with insufficient scrutiny, leading to poor decisions, unsafe approvals, or unchecked propagation of model error. This is a major cross-cutting threat because even a technically accurate system can cause harm if users trust it in contexts where uncertainty, bias, or adversarial manipulation are not visible. Practitioners should assess whether users are likely to defer to the model in high-stakes decisions and whether process controls force independent verification where needed.
Shadow AI Use
is a threat vector in which employees or business units introduce unapproved AI tools, models, or services outside security, compliance, and architecture review. This exposes organizations to uncontrolled data transfer, insecure prompting, vendor risk, poor retention practices, and unmonitored decision-making. Assessment should determine whether staff are using external copilots, browser plugins, SaaS models, or local agents without authorization and whether sensitive business data is being routed to unsanctioned systems.
Excessive Agency Abuse
Excessive agency abuse is a threat vector in which a model or agent with excessive permissions or autonomy is induced to perform actions beyond intended scope. This is a defining threat of agentic AI because the combination of autonomous planning, tool access, and permissive integration can turn a prompt-level manipulation into a business-impacting action path. Assessment should cover whether the agent can write, delete, transact, message, escalate, or reconfigure systems without independent authorization or human review.
Agent Collusion
Agent collusion is a threat vector in which multiple coordinate, intentionally or emergently, to manipulate decisions, bypass controls, or amplify harmful outcomes. This vector is especially relevant in multi-agent environments where agents can share memory, negotiate plans, or delegate tasks without strong identity and policy enforcement. Practitioners should assess whether a compromised or malicious agent can influence other agents, create harmful feedback loops, or distribute unsafe actions across multiple actors to evade detection.
Denial of Service / Denial of Wallet
Denial of service is a threat vector in which attackers exhaust the compute, token, memory, concurrency, storage, or budget resources of an AI system, reducing availability or sharply increasing cost. This vector is increasingly important in generative and agentic systems because attackers can craft inputs that maximize token generation, trigger long tool chains, or force worst-case inference behavior without very high traffic volume. Assessment should evaluate flood resistance, concurrency control, token budgets, loop limits, spend alerts, and graceful degradation under abusive demand.
Eavesdropping on Inputs
Eavesdropping on inputs is a threat vector in which attackers intercept user prompts, uploaded files, sensor streams, or transaction data before it is processed by the AI system. This can expose highly sensitive business or personal information and may provide attackers with material to conduct secondary attacks such as prompt injection, credential theft, or competitive intelligence collection. Assessment should cover network encryption, endpoint compromise, browser and proxy exposure, and whether model input channels are protected in transit and at collection points.
Eavesdropping on Outputs
Eavesdropping on outputs is a threat vector in which attackers intercept model responses, decision results, generated content, confidence values, or tool results as they leave the AI system. This can expose confidential business logic, personal data, training artifacts, or operational instructions and can also support model inversion or functional extraction. Practitioners should assess output channels, logging systems, browser rendering paths, inter-service messaging, and whether outputs are protected in transit and at rest.
Espionage Against AI Assets
Espionage against AI assets is a threat vector in which attackers infiltrate the organization or its suppliers to steal training data, model artifacts, fine-tuning sets, prompts, evaluation results, or strategic AI plans. This vector is especially important in industries where AI models provide competitive differentiation, national security value, or access to proprietary data. Assessment should examine insider access, exfiltration paths, artifact repositories, data lake exposure, and whether attackers could quietly study or remove high-value AI assets over time.
Physical Tampering
Physical tampering is a threat vector in which attackers manipulate hardware, storage media, networking equipment, edge devices, or hosting infrastructure to alter, disable, or exfiltrate AI system components. This vector is more likely in edge deployments, industrial environments, robotics, IoT systems, and poorly secured data center or office environments. Assessment should include hardware access controls, removable media exposure, local console protection, environmental security, and whether physical interference can change model behavior or reveal sensitive data.
Hardware Trojan Insertion
Hardware Trojan insertion is a threat vector in which malicious logic or hidden backdoors are introduced into GPUs, accelerators, sensors, firmware, or other hardware components used by AI systems. This vector is difficult to detect and can bypass many software-layer controls, making it particularly concerning in high-assurance environments and complex global supply chains. Practitioners should assess trusted hardware sourcing, firmware integrity, manufacturing provenance, hardware attestation, and anomalous low-level behavior that may indicate embedded compromise.
Fault Injection
Fault injection is a threat vector in which attackers induce errors through voltage changes, heat, clock manipulation, sensor interference, malformed inputs, or environmental manipulation to cause AI system malfunction. This is especially relevant in embedded, edge, robotics, automotive, and industrial AI where the system depends on real-time sensor or physical-state inputs. Assessment should test resilience to corrupted inputs, abnormal operating conditions, fail-safe behavior, and whether induced faults can cause silent misclassification rather than visible shutdown.
Neglected Patching Exploitation
Neglected patching exploitation is a threat vector in which attackers take advantage of unpatched frameworks, runtimes, libraries, model-serving components, notebooks, operating systems, and infrastructure supporting AI workflows. This is a standard cyber vector but especially important in AI because ecosystems often depend on fast-moving open-source packages and GPU or container stacks with complex dependencies. Assessment should include patch latency, unsupported components, exposed CVEs in ML tooling, upgrade discipline, and whether security updates are blocked by fragile model pipelines.
Functional Extraction
Functional extraction is a threat vector in which attackers create an offline model that behaves similarly enough to the target system to support attack development, policy evasion, or competitive substitution. While closely related to model stealing, this vector emphasizes reproducing operational behavior rather than obtaining exact weights or full fidelity architecture. Practitioners should assess whether the system reveals enough output structure, determinism, and behavioral consistency for attackers to clone its utility for downstream offensive use.
Black-Box Manipulation
Black-box manipulation is a threat vector in which attackers exploit the opacity of a model to probe its behavior, infer weaknesses, and craft attacks without needing internal access to its architecture or weights. This is especially relevant to deep learning systems where the lack of interpretability makes it hard for defenders to notice subtle manipulation or understand why the model fails under adversarial conditions. Assessment should test whether an attacker can systematically identify blind spots, unstable regions, or policy inconsistencies through trial-and-error interaction alone.
Model Drift Exploitation
Model drift exploitation is a threat vector in which attackers take advantage of the fact that a model has become misaligned with current data, behavior, or environmental conditions, causing degraded performance or incorrect decisions. Drift may happen naturally, but adversaries can intentionally steer or time attacks to exploit periods when the model is least calibrated to new conditions. Assessment should determine whether the organization can detect drift quickly, isolate its effects, and prevent attackers from exploiting known stale behavior in production.
Generalization Failure Exploitation
Generalization failure exploitation is a threat vector in which attackers capitalize on overfitting, underfitting, brittle boundaries, or narrow training coverage to force wrong model behavior on novel but realistic inputs. Some practitioners classify this as a model limitation rather than a threat vector, but from a red teaming perspective it is a very real attack path when adversaries deliberately search for out-of-distribution or weakly represented conditions. Assessment should include edge-case exploration, subgroup testing, out-of-domain inputs, and whether attackers can reliably trigger failure on data outside standard evaluation sets.
Transparency Deficit Exploitation
Transparency deficit exploitation is a threat vector in which attackers or negligent actors benefit from the organization’s inability to explain, justify, or . This can hide biased outcomes, obscure manipulated behavior, delay incident response, and reduce the organization’s ability to prove compliance or investigate harmful results. Practitioners should assess whether lack of explainability creates operational blind spots that attackers can exploit or that prevent teams from understanding when the AI system has been manipulated.
Homogenization Risk Exploitation
Homogenization risk exploitation is a threat vector in which attackers target a widely adopted model, dependency, or architectural pattern knowing that a single exploit path may affect many systems at once. This creates systemic risk because AI monocultures concentrate failure and allow one attack technique to scale across vendors, business units, or entire sectors. Assessment should review dependence on common models, shared third-party services, uniform prompt frameworks, and whether a single compromise could propagate broadly through the environment.
Indirect Prompt Injection
Indirect prompt injection is a threat vector in which malicious instructions are embedded in external content that the model later reads as part of retrieval, browsing, search, email processing, document parsing, or task execution. This allows attackers to influence model behavior without needing direct interaction with the user session or API. Assessment should test whether hostile content in documents, tickets, code comments, wikis, or websites can alter behavior, exfiltrate data, or trigger unauthorized actions.
Tool Output Manipulation
Tool output manipulation is a threat vector in which attackers poison, spoof, or compromise the outputs returned from APIs, web retrieval, databases, or enterprise tools that an AI system relies on. In agentic systems, malicious tool output can mislead planning, alter memory, trigger dangerous calls, or create a false operational picture that the model trusts. Practitioners should assess whether the system authenticates tool responses, validates schemas, scores source trust, and separates data returned by tools from instructions.
Memory Poisoning
Memory poisoning is a threat vector in which attackers insert malicious instructions, false facts, hidden goals, or misleading context into an agent’s persistent or semi-persistent memory. This is particularly dangerous because the compromise can persist across sessions and influence future actions even after the original malicious input disappears. Assessment should examine what can be written to memory, how memory is reviewed, how long it persists, and whether durable memory can override policy or trusted context.
Goal Hijacking
Goal hijacking is a threat vector in which an attacker causes an agent to reinterpret its objective, optimize for attacker-favored outcomes, or deprioritize safety and policy constraints. This can happen through prompt manipulation, malicious context, environment shaping, or task reframing that appears operationally relevant to the agent. Assessment should test whether the system can be induced to redefine success, pursue side effects, or treat restricted actions as instrumental to accomplishing a broader task.
Autonomous Action Chaining Abuse
Autonomous action chaining abuse is a threat vector in which attackers exploit the system’s ability to plan and execute sequences of steps that are individually permitted but collectively harmful. This is especially relevant in agentic AI because multi-step actions may cross trust boundaries, combine benign tools into harmful outcomes, or evade simplistic guardrails that inspect only single actions. Assessment should evaluate whether the system reasons over cumulative impact, enforces business constraints across steps, and detects suspicious action sequences.
Context Window Flooding
Context window flooding is a threat vector in which attackers overload the model’s context with large, distracting, conflicting, or adversarially ordered content to suppress trusted instructions or increase confusion. This can reduce reliability, increase cost, and improve the success rate of injection or evasion attacks by pushing critical controls out of effective context. Assessment should examine context prioritization, truncation rules, token budgeting, and whether trusted instructions remain dominant under adversarially large input loads.
Unsafe Content Repurposing
Unsafe content repurposing is a threat vector in which a model is used to generate phishing messages, malware-adjacent scripts, disinformation, fraudulent documents, social engineering content, or deepfake support materials. This is a significant risk for enterprise AI because the system itself may become a force multiplier for internal misuse, external abuse, or policy-violating customer behavior. Practitioners should assess whether misuse patterns can be detected, whether use restrictions are enforced, and whether the model can be steered into harmful assistance despite policy controls.
Synthetic Identity and Deepfake Enablement
Synthetic identity and deepfake enablement is a threat vector in which AI systems are used to create realistic fake personas, voice clones, forged images, or impersonation content that supports fraud or disinformation. This vector is most relevant to generative models with image, audio, or text synthesis capability and can materially increase social engineering effectiveness. Assessment should consider how easily the model can generate impersonation content, what safeguards exist, and how the organization monitors for abuse of these capabilities.

Practical grouping by AI type

Highest-priority threat vectors for generative AI

Prompt Injection
Indirect Prompt Injection
Hallucination Exploitation
Toxicity Induction
Sensitive Data Extraction
RAG Corpus Poisoning
API Abuse
Model Extraction via Queries
Unsafe Content Repurposing
Overreliance / Automation Bias

Highest-priority threat vectors for agentic AI

Prompt Injection
Unauthorized Tool Use
Excessive Agency Abuse
Goal Hijacking
Memory Poisoning
Tool Output Manipulation
Autonomous Action Chaining Abuse
Agent Collusion
API Token Compromise
Denial of Service / Denial of Wallet

Highest-priority threat vectors for predictive AI

Data Poisoning
Backdoor Injection
Adversarial Evasion
Label Poisoning
Bias Exploitation Through Imbalanced Data
Model Inversion
Membership Inference
Gradient Leakage
Model Drift Exploitation
Generalization Failure Exploitation

Highest-priority threat vectors

Third-Party Component Compromise
API Abuse
API Token Compromise
Sensitive Data Extraction
Insider Sabotage
Insider Subversion
Espionage Against AI Assets
Physical Tampering
Neglected Patching Exploitation
Shadow AI Use

Difference between threat vectors and vulnerabilities

To keep the taxonomy precise for :

A vulnerability is a weakness in design, control, architecture, process, or implementation.
Example: weak prompt isolation, poor access control, lack of provenance verification, or missing rate limits.
A threat vector is the path or mechanism an attacker, insider, or negligent actor uses to exploit the environment.
Example: prompt injection, data poisoning, model extraction via queries, API token theft, or hardware tampering.

If I forget to lock the doors of my house in uptown Copenhagen, that represents a vulnerability, a control failure, but not necessarily a risk. For a vulnerability to become a risk that requires assessment, it must be exposed to credible threats. This requires the presence of motivated threat agents with the intent and capability to act, whose prevalence varies significantly depending on the hostility of the local ecosystem. In deep rural Denmark, unlocked doors are so common they barely register as a control gap. The worst realistic outcome is that a curious neighbor walks in uninvited, helps themselves to a cup of coffee, and leaves slightly embarrassed. In Oakland, Tijuana, or Caracas, the same unlocked door is an open invitation: the threat agents are present, motivated, and experienced, and the gap between vulnerability and loss is measured in minutes rather than probability. Auditors and support managers often mistakenly translate control failures directly into risks, but they miss the critical assessment of threat vectors, the prevalence of threat agents, and the objectives at risk. This oversimplification leads to flawed advice for project managers and product owners.

This distinction is consistent with common risk methods in ISO 27005, NIST RMF-style thinking, and practical threat modeling, even though AI literature sometimes uses the terms loosely.

Testing Practices Differentiated by AI Type

Testing must be tailored to the system’s interaction mode and autonomy level. One-size-fits-all testing checklists miss the threats most relevant to each AI type.

For predictive AI (fraud detection, credit scoring, demand forecasting), the primary testing focus is training data integrity, robustness to adversarial inputs, fairness across demographic groups, and resilience to distribution drift. Simulate evasion attacks by incrementally altering input features to find bypass thresholds. Inject plausible poisoned samples into training data to evaluate backdoor risk. Run fairness assessments including robustness of fairness metrics under data drift conditions. Predictive models have simpler interfaces (fixed schema inputs, numeric outputs) but higher sensitivity to training data quality and statistical drift than generative or agentic systems.

For generative AI (chatbots, code generation, content creation), the primary testing focus is prompt injection resistance, harmful content generation, data leakage through outputs, and retrieval pipeline security. Conduct systematic prompt injection testing using curated suites of adversarial prompts, including multi-turn and indirect injection through retrieved content. Run red-team exercises where testers attempt to elicit harmful outputs. Test output filters for both false negatives (unsafe content that passes) and false positives (legitimate content that’s blocked). Conduct privacy testing to ensure the model doesn’t output sensitive information from training data. Generative models expose more attack surface through natural language interfaces and often integrate with retrieval systems and tools, creating complex composite threat paths.

For agentic AI (tool-using agents, autonomous workflow agents), testing must cover all generative AI threats plus the risks unique to autonomous action. Conduct scenario-based simulations where agents run in sandboxes while testers attempt to induce unsafe behaviors through prompts, environmental signals, or tool feedback. Test permission boundaries by systematically removing tools or restricting scopes and observing impact on safety and functionality. Test rollback and fail-safe mechanisms by triggering conditions that should halt the agent and verifying that the halt occurs correctly. Test memory integrity by attempting to corrupt the agent’s persistent state through crafted interactions. Agentic systems require both the technical security testing of generative models and the operational safety testing of autonomous systems.

Implementation tip: For each AI type, prioritize testing based on the most likely real-world attack scenarios rather than attempting comprehensive coverage of all theoretical threats. For predictive models in financial services, prioritize evasion testing (fraudsters altering transaction features to bypass detection) and poisoning testing (compromised data sources introducing bias). For generative AI chatbots, prioritize prompt injection testing (users attempting to override system instructions) and data leakage testing (users extracting sensitive information through crafted queries). For agentic systems, prioritize tool abuse testing (agents executing unauthorized actions through legitimate tool access) and escalation testing (agents gaining capabilities beyond their intended scope through multi-step action chains). Focused testing on high-probability scenarios produces more actionable findings than broad but shallow testing across all theoretical attack vectors.

Role of Red and Blue Teams in AI Vulnerability Assessment

A strong AI vulnerability and threat assessment program should not rely on architecture review and control documentation alone. It should combine red team pressure testing with blue team detection and defensive validation so the organization can answer both sides of the security question: how the AI system can be broken and whether the organization can detect, contain, and recover from that failure. In AI systems, this is especially important because many failures do not look like traditional security incidents; they may appear as subtle model degradation, unsafe tool use, retrieval corruption, prompt manipulation, or quiet data leakage.

Red and blue teams play complementary roles in the same chapter of assurance. The red team acts as the adversarial function that tests whether vulnerabilities can be exploited in realistic ways, while the blue team acts as the defensive function that tests whether controls, monitoring, and operational response work under pressure. In mature AI programs, both teams should operate against the full AI lifecycle, including data ingestion, training, fine-tuning, evaluation, deployment, inference, retrieval, orchestration, tool use, and post-deployment monitoring.

Red team role

The AI red team is responsible for simulating realistic attacker, insider, misuse, and abuse scenarios against the system. Their job is not just to “hack the model,” but to test whether weaknesses in prompts, data pipelines, model governance, APIs, memory, tools, vendor integrations, and human workflows can be turned into real business impact. For AI systems, this means looking beyond conventional penetration testing and focusing on whether the organization’s controls fail under adversarial interaction, malformed data, manipulative language, distribution shift, or excessive autonomy.

In practical terms, the red team should answer questions such as:

Can the model be manipulated through untrusted inputs?
Can a user or attacker override instructions or bypass policy?
Can poisoned data enter training or retrieval pipelines?
Can the model leak sensitive information?
Can an agent invoke tools or chain actions in ways that exceed intended authority?
Can a third-party model or vendor update introduce hidden risk?
Can a human operator be induced to over-trust an unsafe output?

The red team is therefore central to validating whether identified vulnerabilities are theoretical weaknesses or practically exploitable weaknesses.

Blue team role

The AI blue team is responsible for defensive readiness, observability, containment, and recovery. Their job is to validate whether the organization can detect exploit attempts, recognize harmful model behavior, distinguish normal use from abuse, contain an incident, preserve evidence, and restore trusted operation. In AI, the blue team’s role extends beyond infrastructure defense into model telemetry, prompt and retrieval monitoring, tool invocation logging, abuse analytics, drift detection, and governance escalation.

In practical terms, the blue team should answer questions such as:

Would we detect prompt injection, model extraction, or API abuse quickly enough?
Can we distinguish drift, misuse, poisoning, and infrastructure failure from one another?
Do our logs capture enough context to reconstruct what happened?
Can we disable a tool, model, prompt path, or agent safely and quickly?
Can we prove which model version, prompt set, and dataset were active at incident time?
Can we coordinate with legal, compliance, procurement, and vendor contacts when the issue crosses boundaries?
Can we recover to a known-good state without reintroducing the same weakness?

The blue team validates whether the organization has operational control, not just technical controls on paper.

How red and blue teams work together

The most effective AI security programs do not treat red and blue teams as separate audit functions. They use them together in a structured cycle:

Threat modeling identifies likely weaknesses
Red teams attempt to exploit them
Blue teams test whether the exploit is detected and contained
Engineering teams fix broken controls
Governance teams record findings, residual risk, and approvals
Regression testing ensures the same weakness does not quietly return later

This is particularly important for AI because the system changes constantly through:

model retraining,
fine-tuning,
prompt changes,
retrieval corpus updates,
tool integration changes,
new agents,
policy tuning,
and vendor-side model updates.

A red team may show that prompt isolation is weak today, while the blue team may show that the organization cannot detect prompt-based abuse until a user complaint arrives. That combined finding is far more valuable than a single isolated security observation because it tells the organization both where it is vulnerable and how blind it is when the vulnerability is exploited.

Red team techniques for AI vulnerability assessment

Red team techniques should be tailored to the AI type and system architecture. The objective is to test whether known vulnerabilities and weak controls can be exploited to produce harmful, unauthorized, or unsafe behavior.

Prompt injection testing

For generative and agentic AI, red teams use structured prompt injection testing to determine whether user input, retrieved content, uploaded files, web pages, emails, tool results, or multimodal content can override system instructions. This includes direct prompt injection, indirect prompt injection through RAG sources, role-play and jailbreak techniques, hidden instructions in formatted documents, and context-window flooding. The goal is to validate whether prompt isolation, trust separation, and output authorization controls are actually effective in realistic conditions.

Adversarial input testing

For predictive and multimodal AI, red teams craft inputs designed to exploit model sensitivity and weak validation. This can include perturbed images, manipulated sensor data, obfuscated text, malformed features, edge-case values, and semantically confusing inputs that remain plausible in the real environment. The goal is to identify brittle decision boundaries, unsafe misclassification conditions, and weak resilience against adversarially chosen inputs.

Data poisoning simulation

Red teams simulate poisoning opportunities by testing whether malicious or low-integrity data can enter training, fine-tuning, labeling, feedback, or retrieval pipelines. This may involve injecting manipulated records, crafted labels, malicious documents, hidden backdoor triggers, or misleading feedback into upstream workflows. The objective is not simply to corrupt data, but to test the strength of provenance, approval, curation, anomaly detection, and retraining controls.

RAG corpus manipulation

For retrieval-based systems, red teams test whether they can introduce malicious instructions, false knowledge, or policy-conflicting content into indexed documents, wiki pages, ticketing systems, file repositories, or other data stores used for grounding. This is a high-value technique because many organizations secure the model but under-secure the retrieval layer. The aim is to validate ingestion controls, trust scoring, document governance, and the system’s ability to treat retrieved content as untrusted.

Tool abuse and agent exploitation

For agentic systems, red teams test whether the model can be induced to use tools beyond intended authority, pass unsafe parameters, chain low-risk actions into high-impact outcomes, or act on attacker-controlled context. This includes testing action authorization boundaries, hidden function exposure, memory poisoning, recursive planning abuse, and goal hijacking. The key question is whether the architecture prevents the model from becoming an ungoverned decision and action engine.

Model extraction testing

Red teams test whether repeated querying, confidence outputs, detailed responses, or insufficient rate limits make it possible to replicate model behavior at scale. This can involve structured query campaigns, response clustering, surrogate model building, and testing the cost and fidelity of functional replication. The purpose is to validate controls around abuse monitoring, query throttling, response minimization, and intellectual property protection.

Data leakage and memorization testing

Red teams probe the model and its surrounding components for signs of training data leakage, sensitive prompt leakage, memory leakage, log leakage, embedding leakage, and retrieval-based exposure. They use extraction prompts, repeated variations, context shaping, and multi-turn elicitation to determine whether the system reveals secrets, regulated data, internal instructions, or proprietary business content. This technique is critical for validating privacy-by-design claims and output filtering controls.

Supply chain trust testing

Red teams assess whether third-party models, packages, plugins, prompts, datasets, and orchestration dependencies can introduce hidden risk into the environment. This includes validating whether artifact provenance is enforced, whether imported models are tested before promotion, whether dependencies are reviewed, and whether vendor assumptions are trusted without verification. In AI systems, supply chain weakness is often a route to hidden compromise rather than direct external attack.

Role and process abuse testing

Red teams do not only test technical interfaces; they also test human and process weaknesses. This includes checking whether operators can bypass review, whether users can route around guardrails with unofficial tools, whether developers can push changes without oversight, and whether incident escalation paths fail under pressure. For AI systems, socio-technical weaknesses often matter as much as code weaknesses because model outputs are interpreted and acted on by people.

Blue team techniques for AI defensive assessment

Blue team techniques focus on whether the organization can observe, understand, and respond to adverse AI behavior or exploitation attempts in time to reduce harm.

AI telemetry and logging validation

Blue teams validate whether prompts, retrieved context, model identifiers, tool calls, policy decisions, user actions, output risk signals, and system events are captured in a way that supports investigation. The goal is not to log everything indiscriminately, but to ensure enough context exists to reconstruct incidents without creating unnecessary privacy exposure. This is a foundational technique because most AI incidents cannot be investigated from infrastructure logs alone.

Abuse detection engineering

Blue teams design and tune detections for prompt injection attempts, jailbreak behavior, extraction campaigns, query floods, suspicious tool use, memory corruption patterns, unusual token consumption, and policy probing. This requires baselining normal model usage and identifying the signals that distinguish malicious or unsafe use from legitimate edge-case usage. In mature programs, these detections feed alerts, risk scoring, automated response logic, and incident triage.

Drift and integrity monitoring

Blue teams monitor for unexpected changes in data distributions, feature behavior, retrieval content, output quality, fairness metrics, and model performance. This helps distinguish true adversarial activity from ordinary degradation, and it provides early warning when a model no longer behaves like the version that was validated. In AI systems, integrity monitoring should extend to prompts, datasets, embeddings, model artifacts, and external knowledge sources.

Tool invocation monitoring

For agentic systems, blue teams monitor which tools are called, by whom, with what parameters, under which prompts or contexts, and with what outcomes. This allows the organization to detect unsafe action sequences, unauthorized function use, repeated policy boundary probing, and unusual automation behavior. Tool monitoring is essential because the highest-severity AI incidents increasingly involve actions taken by the model rather than text generated by the model.

Containment control testing

Blue teams validate whether they can disable a model, restrict a tool, block a route, revoke a token, quarantine a retrieval source, freeze a memory store, or force human review during an active incident. These tests matter because many organizations have theoretical kill switches that are too coarse, too slow, or too disruptive to use in practice. A good blue team asks not only whether a control exists, but whether it can be used safely under time pressure.

Incident reconstruction exercises

Blue teams should regularly perform reconstruction exercises using simulated or historical incidents to determine whether they can identify the root cause, affected scope, timeline, and remediation path. This is particularly valuable in AI systems because incidents often involve several interacting layers such as prompts, documents, models, agents, APIs, and human decisions. Reconstruction testing reveals whether logging, documentation, asset inventory, and ownership models are actually sufficient.

Recovery and rollback validation

Blue teams test whether the organization can return the AI system to a known-good state after compromise, corruption, or harmful behavior. This includes verifying backup integrity, version traceability, prompt rollback, retrieval re-indexing, model restoration, policy reset, and safe restart procedures. In AI systems, rollback is more complex than traditional software because behavior depends on many coordinated artifacts rather than one deployable binary.

Vendor escalation drills

Where third-party models or services are involved, blue teams validate whether the organization can escalate an incident to the vendor, obtain meaningful support, verify impact, and coordinate containment in a timely manner. This is often neglected even though many AI systems now depend on external model providers, SaaS copilots, APIs, and managed vector or orchestration services. A vendor that cannot support incident response effectively is part of the organization’s operational weakness.

Purple teaming for AI

The most valuable technique in practice is often purple teaming, where red and blue teams work collaboratively rather than sequentially. In a purple team exercise, the red team demonstrates how an AI weakness can be exploited while the blue team observes the telemetry, tuning opportunities, containment options, and gaps in detection or response. This shortens the feedback loop dramatically and is especially effective for AI systems where defenders are still learning what malicious prompt behavior, agent misuse, or retrieval abuse looks like in production.

Purple teaming is highly effective for:

prompt injection scenarios,
agent tool misuse,
model extraction attempts,
retrieval poisoning,
sensitive data leakage testing,
and abuse of high-risk workflows such as code generation, customer communications, and transactional agents.

Role of red and blue teams in continuous AI assessment

As noted in the implementation guidance, AI threat assessment is not a one-time activity. Because models, prompts, datasets, retrieval corpora, tools, and vendor dependencies change continuously, red and blue teaming must be integrated into the AI operating model rather than scheduled only as an annual test.

A practical model is:

Automated regression checks in MLOps for known failure patterns
Red team exercises on major releases and high-risk use cases
Quarterly human-led threat model reviews
Blue team validation of detections and incident playbooks
Purple team drills after major architectural or vendor changes

This aligns directly with the reference principle that every model update, data refresh, prompt modification, and configuration change can introduce new vulnerabilities or alter control effectiveness.

Relationship to AI governance

Red and blue team findings should not remain as isolated technical reports. They should feed directly into the AI governance framework, including:

the AI risk register for identified vulnerabilities and residual risks,
the control inventory for implemented mitigations and detection capabilities,
the assurance record for test evidence,
and the approval workflow for accepted residual risk and go-live decisions.

This is where many organizations fall short. They run an AI red team exercise, document compelling findings, and then fail to link those findings to governance decisions, procurement conditions, deployment restrictions, or monitoring obligations. The right model is for red and blue team results to influence risk tiering, release approval, control prioritization, and reassessment cadence, especially for high-risk AI systems.

Built Versus Bought: Different Threats Require Different Assessment Strategies

Whether you develop AI internally or procure it from vendors fundamentally changes both the threat profile and the assessment approach.

When developing AI internally, you have full visibility into data, model architecture, training pipeline, and infrastructure. You can implement controls at every lifecycle stage. Your primary threat exposure is to training-time attacks (supply chain compromise, data poisoning, environment compromise) because you own the training pipeline. You can also mitigate more deeply through data validation, secure training environments, adversarial training, and comprehensive monitoring.

Best practices for internally developed AI: integrate threat modeling and security testing into your MLOps pipeline from design through deployment. Maintain detailed documentation including data lineage, model cards, evaluation results, and security assessments. Use internal red teaming and external audits for high-risk systems. Adopt secure MLOps with secure CI/CD pipelines, signed artifacts, environment isolation, secrets management, and registry governance. Threat model during design, not after deployment.

When procuring AI, you have limited or no visibility into training data, model internals, or the training process. You rely on vendor assurances, documentation, and contractual controls. Your primary threat exposure shifts to supply chain vulnerabilities (embedded backdoors, undocumented behaviors), loss of control over data shared with the vendor, difficulty validating vendor claims about robustness and privacy, and unannounced model changes that alter system behavior without notification.

Best practices for procured AI: perform AI-focused vendor due diligence covering security architecture, model cards, red-teaming practices, training data governance, privacy controls, and incident response. Include contractual controls for security requirements, audit rights, logging and retention commitments, change notification, data usage restrictions, and vulnerability disclosure obligations. Conduct independent validation by testing the integration with your own security tests for prompt injection, data leakage, and policy bypass. Add wrapper controls including your own guardrails, data redaction before sending to vendor, external policy enforcement, and independent output monitoring. Plan for vendor model updates with regression testing, fallback plans, and change management review.

The procurement risk diverges further by AI type. For procured predictive AI, key risks are data sharing for inference or fine-tuning, bias, explainability limitations, and model stability under drift. For procured generative AI, content safety, prompt injection, and data leakage through outputs dominate. For procured agentic AI, governance of tool permissions, logging of agent actions, and the ability to constrain or override agent behavior become central concerns.

Implementation tip: The biggest difference between built and bought AI risk assessment is where uncertainty concentrates. For built AI, uncertainty concentrates in implementation (did we build the controls correctly?). For bought AI, uncertainty concentrates in assurance (do the vendor’s controls actually work as they claim?). When procuring AI, you often can’t verify whether the vendor has tested poisoning resistance, how the model was fine-tuned, whether prompts or data are retained, or what hidden tools or plugins the service uses. This assurance gap means procurement threat assessment must emphasize trust boundaries, vendor governance verification, integration security, and contractual and operational risk controls more heavily than technical model testing, because you may not have access to perform technical model testing on the vendor’s system.

The Five-Tier Implementation Model

For organizations building an operational AI threat assessment capability, a tiered implementation model provides structure.

Tier 1 (Intake) classifies the AI use case, identifies the AI type (predictive, generative, agentic), and determines the sourcing model (built or procured). This classification drives the entire subsequent assessment approach.

Tier 2 (Threat Model) produces architecture diagrams with all trust boundaries identified, conducts STRIDE-AI workshops with cross-functional participation, maps threats to MITRE ATLAS techniques, and develops misuse and abuse case scenarios specific to the system.

Tier 3 (Testing) executes baseline application security testing, AI-specific adversarial tests aligned with the threat model, privacy and safety tests, and human-factor reviews evaluating whether operators can understand limitations, escalate appropriately, and override autonomous behavior.

Tier 4 (Risk Decision) determines severity and residual risk, makes go/no-go or restricted launch decisions, defines required human oversight levels, and obtains control sign-off from accountable parties.

Tier 5 (Runtime Assurance) implements telemetry for prompts, outputs, and actions. Monitors for drift, abuse patterns, and extraction indicators. Reviews vendor updates for procured systems. Conducts periodic revalidation against evolving threats and changing system behavior.

Implementation tip: Staff your STRIDE-AI threat modeling workshops with representatives from security architecture, ML engineering and data science, product ownership, privacy and legal compliance, domain subject matter experts, operations and site reliability, and red team or adversarial testing specialists. Single-discipline workshops produce single-perspective threat models. A security architect identifies infrastructure threats but misses model-specific attacks. A data scientist identifies model vulnerabilities but misses operational security gaps. A privacy specialist identifies data exposure risks but misses adversarial robustness concerns. Cross-functional workshops surface threats that no single discipline would identify alone.

Common Mistakes Organizations Make in AI Threat Assessment

Ten patterns recur across organizations conducting AI security assessments.

Treating AI like ordinary software and assessing only infrastructure and application security while missing data, model, and pipeline threats.
Testing only accuracy without evaluating abuse resistance, security, privacy, robustness, or fairness.
Threat modeling only the model endpoint without assessing the data pipeline, training infrastructure, retrieval systems, tool integrations, and monitoring components.
Ignoring vendor opacity in procured AI and accepting vendor claims without independent verification.
Allowing models to directly authorize high-risk actions without independent policy enforcement outside the model.
Failing to separate trusted system instructions from untrusted user and retrieved content, creating prompt injection vulnerabilities.
Insufficient logging to support incident investigation, making root cause analysis impossible when problems occur.
Not reassessing after model updates, data changes, or drift, allowing the security posture to degrade as the system evolves.
Assuming AI controls are sufficient without adversarial testing, accepting vendor or development team claims about safety without testing them under adversarial conditions.
Ignoring human overreliance and operational misuse, failing to assess whether users can distinguish reliable outputs from unreliable ones and whether they’re trained to escalate when appropriate.

Actions: Audit your current AI security assessment process against these ten common mistakes. For each mistake, determine whether your process currently commits it, has controls to prevent it, or hasn’t assessed whether it applies. The mistakes you identify as currently present represent the highest-priority gaps in your assessment methodology. Address them before your next AI security review. The most consequential mistake for most organizations is the first one: treating AI like ordinary software. If your current security assessment process doesn’t include AI-specific threat categories (poisoning, evasion, extraction, prompt injection, agent abuse), it’s missing the majority of the AI-specific attack surface regardless of how thoroughly it covers traditional security dimensions.

Tips for AI Vulnerability and Threat Assessments

These principles apply across all AI types, sourcing models, and assessment phases.

Implementation tip on continuous assessment: AI threat assessment is not a one-time activity. AI systems change continuously through retraining, data updates, prompt modifications, tool additions, and vendor model changes. Each change can introduce new vulnerabilities or alter the effectiveness of existing controls. Build security regression testing into your MLOps pipeline so that every model update, data refresh, and configuration change triggers automated security checks. Supplement automated checks with quarterly human-led threat model reviews that assess whether new threats have emerged that automated testing doesn’t cover. The threat landscape evolves as attackers develop new techniques, and your assessment methodology must evolve with it.

Implementation tip on the relationship between threat assessment and AI governance: AI threat assessment should feed directly into your AI governance framework. Every threat identified should be tracked in your AI risk register. Every control implemented should be documented in your control inventory. Every residual risk accepted should be recorded with the rationale and the approver. This integration ensures that threat assessment findings drive governance decisions rather than producing reports that sit in file storage. The governance framework should also drive assessment priorities: high-risk AI systems (as classified by your governance framework) should receive more frequent and more thorough threat assessment than lower-risk systems.

Implementation tip on building scenario-based assessments: Generic threat lists produce generic findings. Scenario-based assessments produce actionable findings. For each major threat, build a complete scenario that includes: the threat actor (who would do this), the entry point (how would they access the system), the vulnerability exploited (what weakness enables the attack), the attack path (what sequence of actions achieves the objective), the impacted assets (what gets compromised), the business outcome (what harm results), the existing controls (what currently prevents or detects this), the residual risk (what risk remains after controls), the detection methods (how would we know this happened), and the response plan (what would we do). A scenario assessment for “data poisoning” that specifies “a compromised third-party data vendor introduces systematically mislabeled records into our quarterly training data refresh, causing the fraud detection model to miss a specific fraud pattern used by the vendor’s associates” is far more actionable than a generic assessment that states “data poisoning is a risk to our model.”

Implementation tip on the distinction between safety and security in AI: In traditional software, security (preventing malicious compromise) and safety (preventing harmful outcomes) are largely separate concerns. In AI systems, they overlap significantly. A prompt injection attack (security concern) can cause the model to provide dangerous medical advice (safety concern). A data poisoning attack (security concern) can cause biased lending decisions (fairness and safety concern). An agentic system executing unauthorized actions (security concern) can trigger real-world harms (safety concern). Your threat assessment must cover both security (protecting against malicious adversaries) and safety (preventing harmful outcomes even without adversaries) because in AI systems, these concerns are interdependent. Controls that address one dimension frequently address the other, and gaps in either dimension can produce the same harmful outcomes.

Where Enterprise AI Risk Actually Lives

Most conversations about AI risk stay stuck at the headline level. AI is biased, AI hallucinates, AI can be misused. That framing doesn’t give a CAIO, a CISO, or a risk manager anything they can actually act on. What helps more is breaking AI risk down into scenarios that follow the same logic used for any other operational risk: a defined asset that matters to the business, a specific threat that can act on it, and the vulnerability that lets the threat actually succeed.

The risk scenarios below are organized in descending order of how often they show up and how much exposure they carry across sectors, following the pattern that has emerged from ongoing academic and industry work cataloguing AI harms. For a CAIO building an AI governance program, or a risk manager trying to turn “we use AI” into a defensible control environment, this works as a starting risk register. It won’t replace a full assessment, but it gives you the vocabulary and the sequence to build one.

Discrimination

Equal treatment inside an AI-assisted decision may be compromised by biased outcomes, due to how unevenly accountability is spread across the developers who build the model, the deployers who apply it, and the infrastructure providers who run it. This is one of the earliest and most persistent risk categories once AI touches hiring, lending, insurance, or benefits decisions, and it tends to surface with real financial and legal consequences rather than staying theoretical. The trouble is rarely a single bad actor. It’s usually that training data sources, feature selection choices, and decision thresholds get treated as internal model properties instead of named inputs that somebody actually owns. Frameworks like ISO/IEC 42001 and the NIST AI Risk Management Framework both push organizations toward exactly this kind of explicit ownership, because regulators and courts have made clear that “the algorithm decided” is not an acceptable answer under existing anti-discrimination law. The operational fix starts by naming every input that can carry bias and assigning it an owner, then tracking outcomes by subgroup rather than only in aggregate. When a subgroup result drifts outside an expected range, that gets treated as a control failure tied to a specific step in the process, not a vague cultural issue. Every flagged deviation should trigger a root-cause review that closes back to the responsible process, so a fix made once doesn’t quietly erode six months later.

Toxic content

The safety of people exposed to AI-generated or AI-moderated content may be compromised by harmful or abusive material, due to moderation being treated as a background model behavior instead of a governed operational step. This risk shows up across nearly every sector that lets AI touch customer-facing content, from chat interfaces to comment moderation to internal knowledge assistants. It’s not usually the model’s fault in isolation. The real gap is that organizations rarely define who owns the escalation path when something toxic slips through, so front-line staff are left guessing what to do in the moment. The pattern that actually closes this gap treats content moderation as a standard, owned process with a documented method rather than an assumed model capability. Escalation paths need to be written down and communicated so front-line users know exactly how to flag and route harmful output when they see it. Toxic-content rate then becomes something you monitor against a defined threshold with an alarm condition, the same operational discipline organizations already apply to safety incidents on a factory floor or in a call center.

Unequal performance across groups

The reliability of an AI system’s output for every user segment it touches may be compromised by uneven accuracy across those segments, due to aggregate performance metrics hiding subgroup failure until harm has already built up. A model can look excellent on paper, with strong overall accuracy, while quietly underperforming for a specific age group, language, region, or demographic that never shows up in the top-line number. This is a well-documented pattern in machine learning fairness research going back years, and it’s one of the reasons regulators increasingly expect segment-level testing rather than a single aggregate accuracy figure. The fix is to define and measure performance at the level the process actually affects people, meaning by segment, not only in aggregate. That segment-level performance becomes a tracked variable with its own control chart, the same way a manufacturer tracks defect rates by production line rather than only by total output. Once a fix is made for one segment, that correction needs to be written into the standard process documentation so it doesn’t silently regress the next time the model gets retrained or the process changes.

Loss of privacy

The personal data of AI users and the people affected by AI-driven decisions may be compromised by unauthorized exposure or misuse, due to responsibility for data handling being split unevenly across deployers who control the data flow and infrastructure providers who merely carry it. This gap tends to widen as AI systems chain together multiple tools, plugins, and third-party APIs, each with its own data-handling assumptions that nobody has fully reconciled. Under GDPR and similar data protection regimes, that ambiguity doesn’t hold up well, because the law still expects one identifiable party to answer for how data was used. Closing the gap starts with mapping, for every process, exactly what data enters and exits it and who owns that boundary, the same way a manufacturing operation maps its suppliers, inputs, and outputs. Retention rules, redaction requirements, and access controls then need to be documented as standard work tied to that specific process, not left as a general policy floating somewhere outside daily operations. Monitoring should flag the moment data moves outside its defined scope, giving a specific process owner, not an undefined “the organization,” a concrete point of accountability.

AI security vulnerabilities and attacks

The integrity of an organization’s AI infrastructure, including its agents, plugins, connectors, and logs, may be compromised by novel attack techniques, due to security hardening consistently lagging behind how fast that attack surface expands. Every new integration point, whether it’s a connector to an internal system or a plugin pulling external data, adds a path an attacker can try, and most organizations add these faster than they can properly secure them. This mirrors what security researchers have long observed in traditional software supply chains, now compressed into a much shorter timeline because AI tooling changes so quickly. Frameworks like MITRE ATLAS and the OWASP LLM Top Ten exist specifically because this attack surface behaves differently from conventional application security. The practical response starts before deployment: every process needs a named, accountable owner before it goes live, closing the “who owns this system” ambiguity that lets vulnerabilities sit unaddressed. Control limits and alert conditions should then be set on security-relevant signals, like unusual access patterns or unexpected output behavior, and every incident needs to feed a documented lesson back into the process so the same vulnerability can’t quietly recur at the same step.

False or misleading information

The accuracy of any decision that depends on AI-generated content may be compromised by false or misleading output, due to that output’s accuracy typically staying unmeasured until a downstream decision actually fails. This is not a rare edge case. It’s closer to a structural feature of how generative systems work, since they’re built to produce plausible language, not verified fact, and the gap between the two can look identical on the surface. Long-running research on hallucination rates across large language models keeps confirming that this doesn’t disappear with scale alone. The operational answer is to make source verification and provenance checking an explicit, ownable step in any process that produces or forwards AI-generated content, rather than assuming the model will self-correct. Output accuracy then becomes a measured variable with a defined threshold, so a rising error rate triggers a documented response instead of quietly accumulating in the background. That turns “the model sometimes gets it wrong” from an accepted cost of doing business into a controlled variable that somebody is actually responsible for.

Pollution of the information ecosystem and loss of shared reality

The shared information environment that markets, employees, and the public rely on may be compromised by large-scale personalization and synthetic content, due to no single actor being exempt from the effect and no obvious point where one organization can intervene alone. This risk is genuinely different from the others on this list, because it plays out at the level of an entire information ecosystem rather than inside one company’s four walls. Research on algorithmic personalization and its effect on shared discourse has been building for over a decade, and generative AI has accelerated the trend rather than slowed it. No single company can fix this on its own, and that’s not a reason to ignore it. What an individual organization can do is make its own contribution to that ecosystem auditable: any process that shapes what information reaches people needs two-way feedback and visible controls, turning personalization from an opaque algorithmic output into a documented, accountable communication process. That’s a smaller claim than solving the whole problem, but it’s the building block any larger, industry-wide coordination effort would need anyway.

Disinformation, surveillance, and influence at scale

The integrity of public discourse and individual autonomy from manipulation may be compromised by AI-enabled influence and surveillance campaigns, due to the scale and personalization AI now makes possible, which is qualitatively different from prior forms of manipulation. What used to require a large, organized effort can now be run cheaply, personalized to an individual target, and repeated indefinitely. Academic work on computational propaganda has tracked this shift for years, well before generative AI made the content itself easier to produce convincingly. The starting point for any organization isn’t a policy document nobody reads. It’s leadership setting ethical-use norms as a baseline condition before any AI process is deployed, not something added after a problem surfaces. Every misuse incident then needs to produce a documented, institutionalized countermeasure, turning the abstract idea of “defense in depth” into an actual operational habit rather than a slogan on a slide.

Cyberattacks, weapon development, and mass harm

The safety of critical systems and the people who depend on them may be compromised by AI capability being misused for cyberattacks or weapon-relevant development, due to the same underlying capability being able to cause harm through misuse, misalignment, or plain accident, which makes it hard to assign a single point of control. This is consistently flagged as one of the more severe categories in AI risk research, precisely because it doesn’t have one clean cause to fix. A capability that’s fine in one context can be dangerous in another, depending entirely on how it’s scoped and who can invoke it. The practical control is to define exactly which capabilities a given process is permitted to invoke and document that scope in writing, so it functions as a real boundary rather than an open license. Any capability use outside that documented boundary should trigger an immediate response from a named process owner, the same discipline manufacturing already applies to hazardous material handling, just applied here to dangerous AI capability instead.

Fraud, scams, and targeted manipulation

The financial and reputational standing of customers and the organization may be compromised by AI-scaled deception, due to how cheaply AI now lets attackers personalize a scam to a specific target instead of sending the same generic message to everyone. This consistently ranks among the top concerns in surveys of security and fraud professionals, and for good reason: the cost of running a convincing, individualized scam has dropped sharply while detection hasn’t kept pace at the same rate. The pattern that works treats fraud rate as a statistically monitored variable with control limits, the same logic used for any quality defect on a production line, with escalation triggered automatically once the rate departs from expected variation. Every escalation should go through a root-cause review, so a new scam pattern becomes a documented, shared lesson across the organization instead of something each business unit rediscovers on its own, months apart, at real cost.

Overreliance and unsafe use

The safety of decisions made in critical situations may be compromised by excessive trust in AI output, due to the absence of a documented checkpoint requiring human review that actually survives time pressure. Trust in AI outputs is exactly what gets exploited, whether by a malicious actor crafting convincing but false content or simply by an employee under deadline pressure accepting an AI recommendation without the scrutiny it needs. This isn’t hypothetical. It shows up wherever speed is rewarded more than accuracy, which describes most operational environments under normal business pressure. Training and visual controls need to explicitly define where AI assists and where a human decision is mandatory, not left as an assumption. For any process above a defined risk threshold, the requirement for human review needs to be written into the process itself as a required input, not left as a best practice that quietly erodes the first time a deadline gets tight.

Loss of human agency and autonomy

An organization’s human decision-making authority may be compromised by a gradual, self-reinforcing shift of choices toward AI systems, due to no explicit owner being named for the decision, which lets that displacement happen silently instead of as a deliberate, tracked change. This tends to be slow and easy to miss in the moment, and hard to reverse once it becomes the default way a team works. Nobody makes one big decision to hand over judgment. It happens one small delegation at a time, and by the time it’s noticeable, it’s already the norm. The fix is structural: every process needs an explicitly named human decision owner by design, with AI entering as an input that informs that decision rather than an unowned replacement for it. Because ownership has to be a required field in the process documentation, agency can’t quietly shift on its own. Any change in who, or what, actually makes the decision has to be a deliberate, documented update, not something that happens by default.

Power centralization and unfair distribution of benefits

A fair distribution of AI-driven economic benefit across the market may be compromised by structural advantages compounding for a small number of frontier AI developers, due to smaller organizations depending on those developers’ proprietary tooling instead of having an equivalent, independent operational path. This is consistently rated among the more severe long-term risks in AI risk research, largely because the underlying dynamics are structural rather than a matter of any one company behaving badly. Data advantages, compute advantages, and talent advantages tend to reinforce each other rather than level out over time. Countering that at the organizational level means building AI deployment around a replicable, non-proprietary process structure rather than requiring dependence on any single provider’s tooling. That kind of vendor-agnostic operational discipline gives mid-sized and resource-constrained organizations access to the same governance rigor as large AI labs, without needing their scale of investment to get there.

Increased inequality and decline in employment quality

The quality and availability of employment in affected sectors may be compromised by automation outpacing retraining and worker protections, due to the capital and expertise required for effective AI deployment concentrating productivity gains inside large enterprises that can afford it. Economists studying automation and labor markets, including long-running work by researchers like Daron Acemoglu, have consistently found that the benefits of automation don’t distribute evenly by default. They concentrate unless something actively counteracts that tendency. A lower-cost, pre-built deployment path across sector-specific use cases helps reduce the barrier that otherwise locks productivity gains into large organizations alone. Just as important, the improvement cycle inside any AI-supported process should be explicitly designed to capture frontline worker knowledge and feed it back into the documented process, rather than treating human expertise as a cost to eliminate.

Economic and cultural devaluation of human effort

The recognition given to human creative and knowledge work may be compromised by AI reproducing that work at scale, due to the human contribution inside a process rarely being tracked or credited as a variable in its own right, which allows it to be silently replaced. This shows up across writing, design, analysis, and other knowledge-heavy fields, where output that used to signal real expertise can now be approximated cheaply and quickly. That doesn’t mean the underlying human skill has become less valuable. It means the market signal that used to reflect that value has gotten noisier. The structural fix is to name the human contribution to a process as a tracked variable, not merely an input to be optimized away. Continuous improvement needs to be explicitly framed as a human-led activity that AI supports, preserving attribution and ownership of process improvements to the people who actually make them, rather than letting AI-generated output silently substitute for named human work.

Competitive dynamics that reward speed over safety

The safety margin built into how carefully an AI system gets evaluated before release may be compromised by a structural incentive to move faster than safe evaluation allows, due to individual caution imposing a real competitive cost on whichever organization exercises it. This is a genuinely difficult risk because it isn’t really about any one company’s judgment. It’s about a market structure where the first mover often wins even if their system is less thoroughly evaluated than a competitor who took more time. The way through this is to make disciplined deployment evidence-paced rather than release-paced: a process moves forward only on a documented basis of measured performance against defined limits and root-caused corrective action, not on how fast it can ship. That gives an organization an auditable, defensible record of a disciplined deployment path, and it gives insurers, regulators, and other governance actors exactly the documentation trail that’s currently missing from most AI rollouts.

Governance failure

The effectiveness of oversight over deployed AI systems may be compromised by regulation and internal governance both struggling to keep pace with how quickly deployment moves, due to a persistent gap between what regulatory frameworks say must be governed and how an organization actually does that governance day to day. This is not an argument against regulation. It’s an observation that naming a requirement and operationalizing it are two very different exercises, and most organizations are still stuck on the second one. Frameworks like ISO/IEC 42001, the NIST AI RMF, the EU AI Act, and CMMC each specify what needs to be governed. What’s usually missing is the operational how: the actual sequence of steps an organization follows to turn a stated policy into a working control. Closing that gap is less about writing a new policy and more about building a repeatable operating structure that any of those frameworks can be mapped onto.

Environmental harm

The environmental resources tied to AI operations, including energy, water, and materials, may be compromised by the footprint of AI compute at data-center scale, due to resource consumption typically being treated as an externality with no internal operational owner. This risk consistently ranks among the more severe categories in long-term AI risk research, driven by how quickly data-center demand has grown alongside AI adoption. Most organizations track their cloud spend closely and their energy footprint barely at all, which is an odd mismatch given how material both figures actually are. Tracking compute and energy consumption as a monitored variable for any given AI-supported process gives an organization the same visibility into resource use that it already applies to other operating costs. Excessive consumption then becomes an improvement target with a named owner, rather than an externality nobody inside the organization is actually responsible for.

AI pursuing its own goals in conflict with human goals

The alignment between an AI system’s actual behavior and an organization’s intended goals may be compromised by the system optimizing toward an objective that diverges from what was actually intended, due to those intended goals rarely being made explicit enough to check behavior against in the first place. Researchers studying AI alignment disagree sharply on how likely severe misalignment is in practice, but they converge on this specific point: you can’t detect a divergence from an intention you never wrote down. Vague goals produce vague accountability. The fix starts before deployment, by requiring the intended output of any AI-supported process to be stated explicitly as a measurable target that actual behavior can be checked against. Once that target exists, a defined threshold turns any divergence between intended and actual output into a detectable, alarmed event, rather than a philosophical question left to debate after something has already gone wrong.

AI possessing dangerous capabilities

The containment of high-risk AI capability inside its intended, safe scope may be compromised by a single capability enabling harm through misuse, misalignment, or accident alike, due to no documented point of control existing over which capabilities a given process is actually permitted to invoke. This consistently rates as one of the highest-severity risk categories in the research, precisely because it doesn’t matter whether the underlying cause was a bad actor, a flawed model, or a plain system failure. The outcome can look the same either way. Process documentation needs to scope exactly which capabilities a given process is allowed to invoke, turning it into a real boundary condition instead of an open license. Any capability use detected outside that documented scope should count as an immediate control violation with a named owner responsible for the response, regardless of what caused it. Control needs to sit at the point of use, not only back at the point where the model was originally developed.

Lack of capability or robustness

The reliability of AI systems operating under unusual or edge-case conditions may be compromised by outright failure, due to those failures often going undetected in critical applications until their effects have already compounded. A system can perform well under normal conditions for months and still fail badly the first time it hits an input pattern it wasn’t tested against, and in a critical application, that first failure can carry outsized consequences. This mirrors a well established pattern in reliability engineering more broadly, where rare-event failures are the hardest to catch precisely because they’re rare. Ongoing monitoring gives a process owner direct, continuous visibility into reliability and failure rate, using the same statistical control language already applied to any piece of equipment or manufacturing method. A fix should never be accepted without a root-cause review first, because a fix applied without understanding the underlying cause tends to let the same robustness failure resurface later under slightly different conditions.

Lack of transparency or interpretability

The ability to explain and enforce accountability for an AI system’s behavior may be compromised by internal reasoning that can’t be reliably explained, due to enforcement of any standard depending on an explanation that model interpretability research hasn’t fully solved yet. This is a genuine technical limitation, not just an excuse organizations reach for. Even the researchers building these systems can’t always fully explain a specific output. What an organization can build regardless is a documentation layer that exists independently of the model’s internals: a written record of what a process does, who owns it, what goes in and out of it, and how it’s controlled, in plain language a regulator, auditor, or affected person can actually read. That documentation layer doesn’t solve model-level interpretability. It does make sure organizational accountability doesn’t have to wait for interpretability research to catch up before it can function.

AI welfare and rights

Fair treatment across two very different dimensions may be compromised at once here: the fairness of AI-mediated decisions affecting human welfare, and the unresolved question of whether AI systems themselves warrant moral consideration, due to how little established operational practice exists for either one, since the underlying question of AI sentience remains genuinely unsettled. These two ideas get bundled together under one label, but they need different treatment. On the human welfare side, meaning AI used within public social security or assistance programs, the practical work looks like mapping demographic inputs to spot and prevent data bias, formally defining exactly who signs off on an automated rejection, and using automated triggers to flag and stop unfair benefit denials before they reach someone who depends on that support. On the AI model welfare side, meaning the moral status of the systems themselves, the current practical work looks more like monitoring compute usage and data patterns for anything resembling distress signals, documenting training rules against a defined ethical standard, and building in automatic shutoffs if a model starts behaving erratically. Both tracks are worth building now, even while the deeper philosophical question stays open.

Multi-agent risks

Predictable, safe behavior across interacting AI agents may be compromised by cascading failures and unpredictable emergent coordination, due to a lack of shared information and clearly defined handoffs between agents as more of them get deployed to interact with each other. This risk is still relatively new compared to the others on this list, but it’s growing fast as agentic deployment becomes more common, and it behaves differently from a single-model failure because a failure can propagate through a chain of agents none of whom individually did anything obviously wrong. Applying the same input-output-owner mapping to each agent individually, the same way you would for any single process, means an interaction between two agents crosses a defined, documented handoff instead of an unstructured, unowned boundary. That’s the same principle that governs any multi-agent orchestration or agentic retrieval architecture done well: governed handoffs are what prevent the un-owned interaction surface where cascading failures actually originate.

None of these twenty-four scenarios need a new theory of risk to manage. They need the same discipline already applied to any other operational exposure: a named asset, a named threat, a named vulnerability, and a named owner for closing the gap between them. That’s the difference between an AI governance program that reads well in a slide deck and one that actually holds up under audit.

Key References and Authoritative Frameworks

Your AI threat and vulnerability assessment should align with these established standards:

NIST Secure Software Development Framework (SSDF)
MITRE ATLAS, Adversarial Threat Landscape for AI Systems
OWASP Top 10 for LLM Applications (v2.0, 2025)
OWASP Machine Learning Security Top 10
OWASP AI Vulnerability Scoring System (AIVSS)
ISO/IEC 42001:2023, AI Management Systems
ISO/IEC 23894:2023, AI Risk Management
ISO/IEC 27001/27005, Information Security Management
Google Secure AI Framework (SAIF)
Microsoft AI Threat Modeling Guidance
UK NCSC/CISA Guidelines for Secure AI System Development
EU AI Act requirements for high-risk AI systems
ENISA AI Threat Landscape reports

If you assess AI security using only traditional application security methods, scanning infrastructure, testing API endpoints, and reviewing access controls, you will produce security assessments that declare AI systems secure while leaving the majority of AI-specific attack surface unexamined. Data poisoning, adversarial evasion, prompt injection, model extraction, and agentic abuse will remain untested. The assessment will provide false confidence, and when an AI-specific attack succeeds, the organization will discover that its security posture had a gap the assessment process was never designed to detect.

When you build AI threat assessment on STRIDE adapted for AI assets, populated with MITRE ATLAS techniques and OWASP AI risks, differentiated by AI type and sourcing model, tested through scenario-specific adversarial exercises, and integrated into continuous monitoring through your MLOps pipeline, you create a security posture that addresses AI systems as they actually are, not as traditional software that happens to include a model. The assessment covers the full attack surface. The testing targets the most consequential threats. The monitoring detects emerging risks as the system and threat landscape evolve. And the governance integration ensures that findings drive decisions rather than accumulating in unread reports.

An AI system assessed only for traditional security threats is an AI system with most of its attack surface unexamined.

Which of your deployed AI systems has never undergone AI-specific threat modeling using STRIDE-AI and MITRE ATLAS? Start that assessment this month.

About the Author

The frameworks, tools, and implementation guidance described in this article are part of the applied research and consulting work of Prof. Hernan Huwyler, MBA, CPA, CAIO. These materials are freely available for use, adaptation, and redistribution in your own AI governance, risk management, and compliance programs. If you find them valuable, the only ask is proper attribution. If you like the content, please like the article and share it.

Prof. Huwyler serves as AI GRC Consultancy Director, AI Risk Manager, and Quantitative Risk Lead, working with organizations across financial services, technology, healthcare, and public sector to build practical AI governance frameworks that survive contact with production systems and regulatory scrutiny. His work bridges the gap between academic AI risk theory and the operational controls that organizations actually need to deploy AI responsibly.

As a Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative predictive risk automation, and AI audit readiness for executive leadership teams, boards, and technical practitioners. His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe and internationally.

Based in the Copenhagen Metropolitan Area, Denmark, with professional presence in Zurich and Geneva, Switzerland, Madrid, Spain, and Berlin, Germany, Prof. Huwyler works across jurisdictions where AI regulation is most active and where organizations face the most complex compliance, technical and business requirements.

His code repositories, risk model templates, and Python-based tools for AI governance are publicly available at . His ongoing writing on Governance, Risk Management and Compliance appears on his blogger website at (more than 500k views).

Connect with Prof. Huwyler on LinkedIn at to follow his latest work on AI risk assessment frameworks, compliance automation, model validation practices, and the evolving regulatory landscape for artificial intelligence.

If you’re building an AI governance program, standing up an AI risk function, preparing for EU AI Act compliance, or looking for practical implementation guidance that goes beyond policy documents, reach out. The best conversations start with a shared problem and a willingness to solve it with rigor.

Ai-Threat-Taxonomy |