Cybersecurity |

How ISO 24970 and prEN 18229-1 Turn Post-Deployment Chaos Into Auditable Evidence

Sun, 28 Jun 2026 00:00:00 +0000

When AI Systems Fail, Logs Tell the Story

Your AI system just flagged 300 legitimate transactions as fraud. A biometric authentication tool locked out half your workforce. A content moderation model started removing benign posts at twice the normal rate. In each case, the first question from your board, your regulator, or your customer is the same: what happened?

Without structured logs, you have no answer. Without a logging framework that captures the right events at the right resolution, you cannot reconstruct the failure, validate your risk controls, or prove you met your oversight obligations. This is the operational gap that ISO 24970 and the European pre-draft standard prEN 18229-1 were built to close.

Why AI Logging Is Different From Application Logging

AI systems generate decisions under uncertainty. A traditional application either executes correctly or throws an error. An AI model can produce a technically valid output that is still wrong, biased, unsafe, or out of scope. The system can drift over time as input distributions shift, adversarial patterns emerge, or model retraining introduces new failure modes.

Standard application logs capture exceptions and transactions. AI logs must capture context, decisions, inputs, outputs, model state, human interventions, and the conditions under which the system operated. They must support not only debugging but also compliance, human oversight, risk detection, bias monitoring, and post-market surveillance.

Designing AI logging architectures based solely on deterministic software practices guarantees blind spots. If your infrastructure fails to capture the exact input distribution and model version during an anomalous inference, you cannot reconstruct the failure or quantify the resulting model risk.

The challenge is that you cannot predict in advance which events will matter. A logged input that seems routine today can become the key evidence in a discrimination claim six months from now. A pattern of outlier detections that you ignored can signal the onset of adversarial attack or domain drift. Logging for AI is not just instrumentation. It is a form of institutional memory that lets you reconstruct what the system knew, what it decided, and what humans did or did not do in response.

Relevance is also not static. As the system interacts with users, encounters new data, or gets deployed in new contexts, the events worth logging can change. Some systems can adapt their logging behavior automatically. Others require human reconfiguration. The standards do not mandate one approach, but they do require that you document your triggers, justify your event selection, and ensure that your logs remain usable across the system lifecycle.

The operational payoff is clear. Logs support monitoring, troubleshooting, strategic planning, and continuous improvement. They feed risk management processes, inform retraining decisions, and provide the evidence base for regulatory filings. But the value depends entirely on log quality, governance, access controls, and the organizational capacity to interpret and act on the data. A poorly designed logging system creates compliance theater. A well-designed one turns operational telemetry into decision support and legal protection.

The Regulatory Context: ISO 24970 and prEN 18229-1

ISO 24970 is the international standard for AI system logging. It defines what to log, when to log it, how to structure log entries, and how to manage log storage and access. The standard is technically precise, format-agnostic, and applicable across sectors and jurisdictions.

prEN 18229-1 is the European counterpart, currently in pre-draft status under CEN-CENELEC Joint Technical Committee 21. It embeds logging into a broader trustworthiness framework that also covers transparency and human oversight. The standard is being developed to support compliance with the EU AI Act, particularly the logging obligations in Article 12 for high-risk systems and the enhanced requirements in Article 14 for remote biometric identification.

The two standards overlap heavily on technical content. Both require event-based logging, traceability through timestamps and identifiers, risk-driven event selection, and governance controls on access and retention. Both treat logs as evidence that must survive audits, support post-market monitoring, and enable deployer oversight.

The main difference is scope and regulatory intent. ISO 24970 is a general-purpose technical foundation. prEN 18229-1 wraps that foundation in a compliance layer designed for EU AI Act obligations, including explicit ties to legal requirements for transparency, human oversight, and post-market surveillance. For organizations deploying high-risk AI in Europe, prEN 18229-1 translates ISO 24970 into a regulatory checklist.

Because prEN 18229-1 is still in pre-draft status, the text is subject to change. The current draft is under enquiry within the European standardization process. It references Directive 2024/1689 (the AI Act) and is expected to be cited in the Official Journal of the European Union once finalized. Organizations building logging systems today should track both standards and design for convergence.

The practical approach is to start with ISO 24970 to define your logging architecture, then map those logs to the compliance and oversight requirements in prEN 18229-1. For high-risk systems, that means aligning your event triggers, log content, retention policies, and access controls with the AI Act from the beginning. Retrofitting logging after deployment is expensive and often incomplete.

EU AI Act Requirements for High-Risk Systems

Article 12 of the EU AI Act mandates automatic logging capabilities for all high-risk AI systems. The logs must capture events that indicate emerging risks or significant modifications to the system under Article 79. They must enable post-market monitoring under Article 72. They must support deployer oversight under Article 26.

Article 12: Record-Keeping: High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system. In order to ensure a level of traceability of the functioning of a high-risk AI system that is appropriate to the intended purpose of the system, logging capabilities shall enable the recording of events relevant for: (a) identifying situations that may result in the high-risk AI system presenting a risk within the meaning of Article 79(1) or in a substantial modification; (b) facilitating the post-market monitoring referred to in Article 72; and (c) monitoring the operation of high-risk AI systems referred to in Article 26(5). For high-risk AI systems referred to in point 1 (a), of Annex III, the logging capabilities shall provide, at a minimum: (a) recording of the period of each use of the system (start date and time and end date and time of each use); (b) the reference database against which input data has been checked by the system; (c) the input data for which the search has led to a match;(d) the identification of the natural persons involved in the verification of the results, as referred to in Article 14(5).

For remote biometric identification systems listed in Annex III, the logging requirements are more specific. You must log precise start and end timestamps for each usage session. You must record the reference database used during input validation. You must log input data that triggered search matches. You must identify the individuals responsible for verifying results, as required by Article 14.

These are not optional features. They are legal obligations. Failure to implement automatic logging, retain the required data, or make logs available to competent authorities can trigger enforcement action, including fines up to 3 percent of global annual turnover for severe violations.

The standards give you the technical blueprint to meet these obligations. But compliance also depends on governance. You need documented policies on what to log, how long to retain it, who can access it, and how to respond when logs reveal risks. You need processes to review logs, escalate anomalies, and update the system when logging reveals gaps or failures. And you need technical controls to prevent log tampering, ensure log integrity, and protect log confidentiality.

Framework Feature	ISO/IEC 24970	prEN 18229-1 (Pre-Draft)
Primary Scope	Global technical logging mechanism	European trustworthiness and compliance
Target Application	General AI system architecture	High-risk AI systems (EU AI Act)
Key Directives	Event triggers, data models, traceability	Post-market monitoring, deployer oversight
Operational Focus	Diagnostic telemetry and error handling	Legal accountability and transparency

Core Concepts: Logs, Log Entries, and Logging Components

A log is a structured repository of log entries. Each log entry is a discrete record that captures a specific event, condition, state, input, output, or decision related to the AI system. Logging is the process of generating, capturing, and managing those entries.

A model is a representation of a system, entity, or process, whether physical, mathematical, or logical. In AI, the model is typically the trained artifact that produces predictions or decisions. But the AI system is larger than the model. It includes data pipelines, serving infrastructure, user interfaces, monitoring tools, and external integrations.

An audit is a systematic, independent process for obtaining and evaluating objective evidence to determine whether audit criteria are met. Internal audits are conducted by the organization. External audits are conducted by customers, regulators, or third-party certification bodies.

Auditability is the capability to collect and make available the evidence needed to conduct an audit. For AI systems, that evidence lives in logs. Without logs, you cannot prove what the system did, when it did it, or under what conditions.

An error is a discrepancy between a computed value and the true or specified value. Errors can be caused by component failures or by the activation of latent faults. In AI, errors also include incorrect predictions, misclassifications, or outputs that violate safety or fairness constraints.

Monitoring is the ongoing observation and assessment of system behavior, outputs, and context. Monitoring can be automated or manual. It detects deviations from expected operation, such as failures, malfunctions, cyberattacks, out-of-domain inputs, or abnormal usage.

A logging component is the part of the AI system, or a linked external system, that enables logging. It can consist of multiple subcomponents that generate, format, filter, or forward log entries. The logging component can be implemented in software, hardware, or a hybrid configuration.

A log user is the organization or entity that accesses, reviews, or analyzes logs. Log users include developers, testers, operators, auditors, deployers, and regulators. Each has different access rights and different purposes.

De-identification is the process of removing or altering data so that individuals or entities cannot be identified, directly or indirectly. De-identification is often required to comply with privacy regulations or to share logs with third parties.

A data principal is the entity to which data relates. This includes persons, organizations, devices, or software applications. The term is broader than personally identifiable information principal or data subject.

An organization is a person or group with its own functions, responsibilities, and objectives. This includes companies, government agencies, nonprofits, and partnerships.

An AI user is the organization or entity that uses AI products or services. A stakeholder is anyone who can affect, be affected by, or perceive themselves to be affected by the AI system. An AI developer is the organization involved in development.

Memory capacity is the maximum number of items that can be held in the logging component’s volatile memory, typically measured in bytes. Storage capacity is the maximum number of items that can be held in persistent storage.

A software error is an erroneous result produced by the use of a software product. This includes incorrect outputs, exceptions, or failures to execute.

A controller is an authorized human or external agent that performs control actions on the AI system. A control point is the part of the system interface where control can be applied, such as a function, switch, or signal receiver.

Control engagement is the process where a controller takes over control points. Control disengagement is when a controller releases control points. Control transfer is the handover of control points from one controller to another.

A governance scheme is the set of rules that defines how the system is managed and controlled. This can be a regulation, standard, guideline, convention, or social norm.

An AI provider is the organization that provides products or services using one or more AI systems.

These definitions matter because they set the boundaries of what must be logged, who has access, and what counts as evidence. If your logging system does not distinguish between a software error and a model prediction error, you cannot diagnose failures. If your logs do not capture control transfers, you cannot prove human oversight. If you do not de-identify logs before sharing them, you violate privacy law.

What Goes Into an AI System Log

An AI system log captures information related to operation, behavior, inputs, outputs, or context. The log can contain structured data like JSON objects, semi-structured data like annotated text, or unstructured data like screenshots. Logs can originate from the AI system itself, its internal components, interacting systems, users, or external observers.

Logs can be generated continuously, periodically, or in response to specific conditions. They serve multiple purposes including monitoring, debugging, auditing, compliance, human oversight, iterative improvement, and accountability.

AI system logs can include time-stamped events, which are recorded occurrences linked to a specific moment. Examples include when a model generates a prediction, an error occurs, or a user interaction takes place. They can include status snapshots, which are point-in-time captures of system conditions such as memory usage, model state, or active components.

Logs can include sensor or input data, meaning information received from external sources like camera images, user inputs, location data, or telemetry. They can include outputs such as classifications, recommendations, predictions, or generated content. They can include decisions, which are discrete choices or actions taken by the system, either autonomously or through human-in-the-loop mechanisms.

Logs can include error messages, which are alerts or diagnostic records indicating failures, exceptions, or issues. They can include environmental context such as network status, sensor readings, user load, or surrounding events. They can include annotations, which are supplementary notes or metadata added manually or automatically to describe behavior, flag anomalies, or provide interpretive context.

AI system logs can be stored persistently for long-term retention, inspection, or regulatory compliance. They can be processed in real time to support live monitoring, alerting, or adaptive behavior. They can be managed under data minimization or privacy constraints to avoid collecting unnecessary personal data, ensure user consent, or comply with legal frameworks.

Logs can be machine-readable, formatted for automated processing using standards like JSON or XML. They can be human-interpretable, presented in a way that allows developers, auditors, or analysts to understand the content without complex tooling.

Logging Components and Their Role

A logging component is the functional part of the AI system or an external system that supports the generation, capture, formatting, storage, or management of log data. It can consist of one or more subcomponents responsible for detecting events, recording log entries, applying data policies like filtering or redaction, or ensuring secure and reliable handling.

Logging components can be internal to the AI system, integrated into model-serving infrastructure or runtime environments. They can be external systems or services such as observability platforms, audit modules, or compliance loggers. They can operate independently or in coordination with other system parts. Complexity varies from a simple event logger to a distributed, multi-service logging pipeline.

The logging component does not assume a fixed structure, automation level, or deployment location. It can be implemented in software, hardware, or hybrid configurations. It is designed to meet different operational, analytical, or regulatory objectives.

The logging component and the storage used for logging are not necessarily part of the AI system itself. They can be separate infrastructure managed by third parties, provided that confidentiality, integrity, and availability are maintained according to applicable regulatory requirements.

Logging in Context: Operational and Management Integration

Management of an AI system in operation is naturally integrated with operation itself. Management and operation share the fundamental goal of navigating uncertainty to achieve purposes. This involves capitalizing on opportunities and mitigating risks through planning, monitoring, decision-making, and learning.

Components of the AI system, including monitoring systems, can use logs to better fulfill the intended purpose, including risk mitigation. A log user can collect and analyze possibly de-identified logs from multiple AI systems to create and improve AI systems.

The logging component logs behaviors of the AI system. Monitoring systems can use the logs to help the AI user assess potential benefits and harms of AI system activities. Logs can be used to assess the continuous fulfillment of various requirements such as accuracy, robustness, security, privacy, safety, and data quality. This assessment can inform the selection of actions.

Organizations can collect and analyze logs from similar AI systems or similar components on the market to support the creation, maintenance, and continuous improvement of the data, AI systems, and components they provide.

Structure and Content of Log Entries

AI system log entries are discrete, identifiable units of information within a log. Each entry captures a specific event, condition, state, input, output, decision, or contextual detail related to the functioning or environment of the AI system.

Log entries are typically composed of a combination of metadata such as timestamps, source identifiers, and severity levels, along with content-specific data relevant to the purpose of the log. Protection of confidential information must be taken into account.

Log entries can vary in structure and content depending on the type of information being recorded and the intended use of the log. Some entries are highly structured, such as a JSON object. Some are semi-structured, such as textual annotations. Some are free-form, such as screenshots.

Components of a log entry can include a timestamp, the date and time at which the logged event or condition occurred. They can include a source identifier, a label or address indicating which component, system, user, or external observer generated the entry. They can include an event or message code, a categorization or classification of the type of event such as an error event, inference event, or user override event.

They can include payload or data content, the core data being recorded such as input features, output values, error traces, or contextual metadata. They can include a severity or priority indicator, a label indicating the importance or criticality of the event, useful for filtering or alerting.

Log entries can be generated automatically by system components or instrumentation. They can be manually created by users, operators, or auditors, such as annotations or overrides. They can be derived from external systems such as monitoring tools or interacting AI components.

To be useful for downstream analysis, log entries should be recorded in a way that ensures traceability, interpretability, and data integrity over time.

The Process of AI System Logging

AI system logging is the process of generating, capturing, recording, and managing information related to the operation, behavior, decisions, or context of an AI system, for the purpose of creating one or more logs.

Logging can be performed automatically by system components, manually by users or operators, or through hybrid methods. It can occur during design, testing, deployment, or post-deployment operation.

Logging can involve the collection of data from a variety of sources including system components such as model execution, middleware, or infrastructure. It can involve user interactions such as input submissions or user overrides. It can involve external observers such as monitoring tools or regulatory systems. It can involve interacting AI systems such as decision handoffs or multi-model coordination.

Logging activities can be continuous, such as telemetry data. They can be event-driven, such as error occurrences. They can be scheduled, such as periodic health checks. They can be conditional, such as events triggered by threshold violations or policy rules.

Logging can include one or more sub-processes. Instrumentation is the implementation of tools, code, or mechanisms to monitor, extract, and capture data from software or hardware components during execution. Serialization is converting data structures or objects into a standardized format such as JSON, XML, or binary for logging, storage, or transmission.

Storage and retention is saving logs to appropriate storage systems with defined retention policies. De-identification processes are applied according to applicable legal or ethical standards. Validation and integrity checking ensures that log data is accurate, complete, and has not been tampered with.

Logging should be guided by clearly defined objectives such as performance monitoring, safety validation, auditability, transparency, compliance, user redress, or support for system improvement.

General Requirements for AI System Logging

AI system logging must provide specific capabilities. Logging functions must enable traceability between multiple events and log entries if necessary to manage risk, relevant to the intended purpose, and technically feasible given the inputs and outputs.

The organization must identify the security and privacy requirements for integrity and confidentiality protection. It must protect information taking into account different purposes of logging for different stakeholders. For example, an AI developer who implements, an AI tester who tests the system, and an AI service provider who monitors service during operation all have different purposes.

Security and privacy requirements include those based on applicable privacy and security regulatory requirements.

Logging functions must enable the recording of events relevant for identifying situations that can result in the AI system presenting a risk according to the risk management process. They must facilitate the monitoring of AI systems as a product or service, proportional to their risks, to enable collection, documentation, and analyzing performance data from initial development to the end of the retirement stage.

Technical Documentation for Logging

The technical documentation for the AI system must explain and justify the specific criteria for determining relevant events. It must explain and justify the specific criteria for logging relevant events. It must specify any interaction with human controllers. It must specify any interaction with automated monitoring.

It must recommend a frequency and scope of monitoring for relevant events. It must recommend a frequency and scope of logging relevant events. It must explain and justify the accuracy and precision of timestamps, where used.

It must explain and justify resource constraints such as memory capacity, storage capacity, and processing power. It must explain and justify constraints related to privacy. It must refer to related legal requirements related to data protection, system accountability, traceability, and transparency.

It must include appropriate information security considerations and data retention policies. It must include specification of failure handling, such as AI system reaction in case of log memory overloading. It must include interfaces with other systems. It must contain a specification of used log data structures.

AI System Logging Fields for Governance Professionals

The following table covers event types with five columns: what field to capture, what to actually record and why it matters, the governance purpose it serves, and whether it’s required, recommended, or optional, with a risk level for each.

Legends

Required, must be captured, non-negotiable
Recommended, best practice, capture when feasible
Optional, adds value in specific contexts

Event type	Field to capture	What to record and why it matters	Governance purpose	Obligation and risk
Operational events, triggered by normal AI system activity
Transaction initiation When a request enters the AI system	event_id	A globally unique ID for this specific request. Used to trace a single transaction through all downstream logs and audit trails. Without this, you cannot link what went in to what came out.	Traceability, audit	Required High risk
	system_id	Identifies which AI system, as a governed unit, processed the request. In shared infrastructure where one platform serves multiple products, this field distinguishes which system the risk assessment applies to.e.g. “loan-approval-v2” not just “ml-cluster-3”	Accountability, risk scoping	Required High risk
	model_id	Full model name and version, as granular as the provider makes available. This is critical for post-incident investigation: if a model version introduced a bias or error, you need to know which transactions were affected.e.g. “gpt-4o-2024-08-06” not just “GPT-4”	Incident response, model versioning	Required High risk
	timestamp	Precise date and time the request was received, in a standardised format (ISO 8601). Include timezone explicitly. For systems without a real-time clock, record a relative counter (e.g. cycles since startup) to preserve ordering.e.g. “2025-11-14T09:32:11.482Z”	Sequencing, forensics	Required Medium risk
	input_ref	A pointer to where the full input is stored, not necessarily the input itself. Capture the source identity (which user, API endpoint, sensor, or system sent this). Enables tracing input provenance in multi-source environments.	Traceability, security audit	Recommended Medium risk
	input_payload	The actual content sent to the model, or a reference to retrieve it. Required when understanding the input is necessary to explain the output. For sensitive inputs, store a reference and apply access controls. Do not log raw personal data unnecessarily.	Explainability, redress	Recommended High risk
	metadata	Contextual parameters that shaped how the model processed the request, for example, temperature settings, prompt version, language of input, encoding type. Without these, reproducing or explaining a result is often impossible.	Reproducibility, debugging	Optional Lower risk
Transaction outcome When the AI system returns a result	output_payload	The actual content returned by the model. Essential for auditing whether the AI system produced harmful, biased, or incorrect outputs. Store a reference if the payload is large or sensitive.	Accountability, bias detection	Required High risk
	confidence_level	The model’s confidence or probability score for its output, where available. Helps identify cases where low-confidence outputs led to consequential decisions, a key signal for human review thresholds.	Risk calibration, oversight	Recommended Medium risk
	correlation_id	Links this outcome back to its originating transaction and any intermediate processing steps. Essential in multi-stage systems where a request passes through several components before a response is returned.	End-to-end traceability	Recommended High risk
Transaction feedback When correctness of an output is established	ground_truth	The correct or intended output for a given input, provided after the fact by a human reviewer, test data, or authoritative source. Used to measure model accuracy over time and detect performance degradation. Not applicable to all AI system types.	Model performance, continuous improvement	Recommended Medium risk
	feedback_source	Who or what provided the ground truth, including a named human reviewer, an automated test suite, or a regulatory authority. Allows weighting of feedback by source reliability and supports audit of the feedback process itself.	Accountability, audit quality	Recommended Medium risk
Anormality and security events triggered by automated monitoring
Software error When the system fails to process normally	error_code	A structured code classifying the error type, such as inference failure, timeout, component crash. Allows filtering and trending of error types across large volumes of logs without reading free-text descriptions.	Reliability monitoring, SLA	Required High risk
	error_message	Human-readable description of what failed. Pair with error_code. Include severity level (critical / warning / informational) and the impact: did this affect the user’s outcome? Was a fallback triggered?	Incident response, debugging	Required High risk
	recovery_action	What the system did in response, retried, switched to fallback, notified the user, escalated to a human, or failed silently. Silent failures with no logged recovery action are a major governance gap.	Resilience, human oversight	Recommended High risk
Outlier input detected When an input falls outside expected distribution	outlier_flag	Indicates the input deviated from the statistical profile of the training domain, e.g. a feature value outside established bounds. Log the specific metric or threshold that was breached, not just a boolean flag.	Risk detection, domain monitoring	Required High risk
Adversarial attack When a deliberate attempt to manipulate the model is detected	attack_type	Classify the detected pattern: prompt injection, model inversion attempt, data poisoning signature, unauthorised access pattern. Detection may be triggered by a single input or a pattern across multiple inputs, note which applies.	Security, incident response	Required High risk
	detection_basis	Whether the attack was identified from a single request or inferred from a pattern across prior logged inputs. If pattern-based, reference the window of prior log entries that contributed to detection.	Forensics, alert calibration	Recommended High risk
Bias detected When outputs show unwanted differential treatment	bias_indicator	The specific metric that triggered the alert, e.g. demographic parity gap, equalized odds differential, disparate error rates across groups. Log the measured value alongside the threshold that defines “unwanted” for this system.	Fairness, regulatory compliance	Required High risk
	affected_population	Which groups or segments were identified as affected by the differential output. Required for meaningful impact assessment. Handle with care, this field may itself contain sensitive information requiring access controls.	Impact assessment, remediation	Recommended High risk
Out-of-domain input When the system is used outside its intended scope	domain_violation	Describes how the input fell outside the operational domain, such as violated a feature boundary, represented an unseen data distribution, or triggered domain drift detection across recent inputs. Distinguish single-input violations from distributional drift.	Scope compliance, safety	Required High risk
Model drift When model behaviour shifts from its validated baseline	drift_metric	The performance or distributional metric that revealed drift, such as prediction shift, output distribution change, increasing error rate. Log the measured value and the baseline it is compared against. Only applicable to systems with updatable models.	Model governance, revalidation	Required High risk
	drift_window	The time period or number of transactions over which drift was measured. Without this, a drift alert cannot be investigated, you need to know which inputs to review.	Forensics, retraining triggers	Recommended Medium risk
Human oversight events triggered by human controllers acting on the system
Human intervention When a person stops, overrides, or corrects the AI system	controller_id	Unique identifier of the person who intervened not a role or team name, but a specific individual. This is non-negotiable for accountability: if a human altered an AI decision, there must be a named person in the log.	Accountability, audit	Required High risk
	intervention_reason	Why the person intervened, prevented a serious incident, corrected an erroneous output, responded to a user complaint. Record based on risk: for high-risk systems, the reason is always required. For lower-risk systems, assess and justify.	Accountability, learning	Recommended High risk
	intervention_outcome	What the intervention achieve, the AI output was blocked, modified, approved, or escalated. Captures the difference between what the AI system would have done and what was actually delivered to the user.	Oversight effectiveness	Recommended High risk
Control transfer When operational control of the AI system changes hands	from_controller	The controller relinquishing control, who held authority before the transfer. Without this, you cannot reconstruct accountability chains for decisions made during the transition period.	Chain of custody, accountability	Required High risk
	to_controller	The controller taking over. Log both the engagement (new controller accepts) and the disengagement (previous controller releases) as separate timestamped entries to capture the full handover.	Chain of custody	Required High risk
Output validation When a human checks or approves an AI output	validator_id	Who performed the check. Distinct from the person who made the downstream decision, a validator confirms the AI output is fit for use, not necessarily that the final decision is correct.	Quality assurance, accountability	Required Medium risk
	validation_result	Approved, rejected, or approved with modification. If modified, record what was changed and why. An unrecorded modification between AI output and human decision is a critical governance gap.	Oversight quality	Recommended High risk
User interaction events triggered by actions involving the people using the system
User complaint and feedback When a user challenges or disputes an AI output	complaint_ref	A unique reference linking the complaint to the original transaction it concerns. Without this link, you cannot investigate whether the system behaved correctly, the complaint is unverifiable.	Redress, accountability	Required High risk
	complaint_outcome	The result of processing the complaint — upheld, rejected, referred. Log when each stage occurred and who was responsible. Regulators may require evidence that complaints were handled within defined timeframes.	Redress, regulatory compliance	Recommended High risk
User information disclosure When privacy notices, disclaimers, or policy terms are communicated	disclosure_type	What was communicated, privacy notice, AI disclosure, terms of service, limitation of liability. Log each disclosure separately so you can prove which notice a user received at which point in time.	Legal compliance, consent	Required Medium risk
	acknowledgement	Whether the user accepted, declined, or did not respond to the disclosure and when. This is your evidence of consent or its absence. Critical for GDPR, AI Act, and similar frameworks requiring informed consent.	Consent management, legal	Required High risk
AI/ML model development events, for auditable training of machine learning models
Training checkpoint At repeated intervals during model training	epoch_id	Identifies where in the training process this checkpoint was taken which iteration or epoch. Allows reconstruction of the training trajectory and selection of the best-performing model version post-training.	Model auditability, reproducibility	Required Medium risk
	model_parameters	A snapshot of the model weights at this checkpoint. This is what allows you to restore and re-evaluate any historical model state. Store until the final model selection decision is made; retain for selected models per legal and business requirements.	Reproducibility, regulatory audit	Required High risk
	quality_metrics	Performance measures at this checkpoint, such as validation loss, accuracy, F1, or domain-specific metrics. Enables assessment of overfitting and helps justify the final model selection to auditors and regulators.	Model selection justification	Required High risk

Risk levels are assigned based on the consequences of not capturing the field, not just the sensitivity of the field itself. For example, recovery_action on a software error is flagged high risk because a silent failure with no logged response is one of the most common and serious gaps in AI oversight programs.

Designing the Logging System with Risk as the Primary Driver

Risk is the primary driver for monitoring and controlling AI systems that are enabled by logging. Risk must be considered when determining which events are to be detected, determining which events are relevant, and determining which relevant events are to be logged.

Examples of risk management standards that can be applied include ISO 23894 or prEN 18228.

Events must be logged in relation to inputs or outputs and when caused or observed by the controllers or components of the AI system. Relevant events to be logged must be selected based on risk, including determining the most effective and efficient way to manage the risk.

Inputs or outputs relevant to event detection must be logged at a frequency that is technically feasible and allows risk to be managed in the context of the intended purpose. For example, events from streaming inputs or outputs can be logged at different frequencies based on the time resolution of the input, or can be logged at a frequency that is appropriate for monitoring a situation, such as at a higher frequency during a cyberattack.

Logging functions must be designed and configured to generate logs accurately representing such events.

Sources of information to be logged can include communication between end users and the AI system, communication between the AI system and its components, and acquisition and utilization of stored or external data.

Traceability Through Timestamps and Identifiers

Log entries about events should be timestamped. The timestamp must record the time of the event to an accuracy and precision appropriate for the type of event and its role with respect to the intended purpose of the AI system. Where technically feasible, the order of log entries should correspond to the order of the events logged.

Timestamps should be formatted according to ISO 8601-1. If the time zone is not included within the timestamp, a mechanism to determine the time zone which is applied for the timestamp must be specified in the technical documentation.

Log entries should include an information element that enables connection between the logged information and the AI system or its components, where appropriate.

This is not an academic concern. In a discrimination investigation, the order of decisions matters. In a safety audit, the timing of control transfers matters. In a breach investigation, the sequence of access events matters. Timestamps that are imprecise, inconsistent, or missing time zone information make logs unusable as evidence.

Additional Logging Functions for Specific Use Cases

Additional logging functions can be provided based on the nature of the system, the organization or entity role, and based on applicable legal requirements.

These can include recording the period of each system use such as start and end timestamp. They can include reference to external data source or database against which input data is checked, if applicable. They can include logging the relevant input data. They can include traceability at the level that enables identification of individuals involved in result verification.

For remote biometric identification systems under the EU AI Act, these functions are mandatory. For other high-risk systems, they depend on the risk assessment and the regulatory context.

Anomaly Monitoring of the Logging Component Itself

Logging functions must issue alerts when the integrity of log processing is violated, when the confidentiality of log storage has been compromised, when the integrity of stored logs is violated or can no longer be ensured for the full operational lifetime, or when log storage capacity is being reached or exceeded.

The frequency and monitoring of alerts must be justified.

This requirement recognizes that the logging system itself can fail. A logging component that silently drops entries, allows unauthorized access, or runs out of storage creates a false sense of compliance. The system must monitor itself and escalate failures to operators.

Triggers for Logging: Operation, Monitoring, and Oversight

A log entry can be triggered by the reception or processing of an input, by human actions, by specific software interactions, or by the automated or manual detection of certain events.

Events are at the center of certain processes within or around the AI system, including automated monitoring and human oversight. The underlying goal of logging is to keep records of relevant events occurring in relation to the AI system.

Events can pertain to the inputs, the outputs, the state of the AI system, or a combination. Relevant events can consist of a pattern of information, such as a change or a particular balance over a period of time, or they can correspond to a property of those inputs, outputs, and state, such as the presence of a particular feature. They can occur across multiple inputs or within a single input.

Detection of relevant events can occur through human oversight or automated monitoring and can involve consideration of past inputs and outputs and other information pertaining to the event.

Detected relevant events can be logged, including various information pertaining to the event and corresponding inputs and outputs. Human oversight or automated monitoring can change the outputs of the AI system.

Triggers From Operation

A log entry must be recorded when the AI system, or a component of it, encounters a software error. This can be caused by internal or external factors. The standard provides an information model for software errors that includes error codes, messages, severity level, impact level, and system context. It also includes detailed error handling information such as failed operation, retrying, switching to a fallback mechanism, notifying the user, logging the error, escalating the issue, and recovering if possible.

Outlier inputs must be detected based on statistical thresholds, domain-specific anomaly detection metrics, and contextual metadata. An outlier input is one that deviates significantly from the expected distribution or boundaries of the input space. Outliers can indicate data quality issues, adversarial inputs, or emerging use cases that were not anticipated during design.

Potential attack triggers must include unauthorized access patterns, data integrity violations, model inversion signatures, and poisoning signatures. Model inversion is an attack where an adversary uses outputs to reconstruct sensitive training data. Poisoning is an attack where an adversary manipulates training data to degrade model performance or introduce backdoors.

A log entry should be recorded when a user requests a review of a transaction, when a user submits a complaint or provides feedback, or when authorized personnel or systems process the user’s complaint or feedback. The standard provides an information model for human feedback.

A log entry should be recorded upon determination of the outcome of a user request, with a reference to the original user request. This creates an audit trail from complaint to resolution.

The communication of information to AI users or subjects can trigger a log entry. For example, communicating a privacy policy or disclaimer to a user, or their acceptance or non-acceptance of it, can be recorded in a log.

Triggers From Automated Monitoring

A log entry must be triggered when an adversarial attack is detected. This detection can occur on a single input or be inferred from a pattern over multiple inputs. In the latter case, it relies on prior logging of inputs ahead of event detection.

A log entry must be triggered when unwanted bias is detected in the outputs of the AI system. This detection is typically done over multiple inputs and corresponding outputs. It relies on prior logging of inputs ahead of event detection. Unwanted bias refers to systematic differences in outcomes across demographic groups that violate fairness constraints.

A log entry must be triggered when the AI system is detected to operate out of its domain. Depending on the domain and its defining characteristics, this detection can be made either on an individual input, such as violating feature boundaries, or it can be meaningful solely over multiple inputs, such as for domains that set distributional properties on certain features. In the latter case, it relies on prior logging of inputs ahead of event detection. Detection of domain drift must be considered as operating out of the domain.

For AI systems whose models are updated, either on a continuous basis or with another timescale or manual intervention, a log entry must be triggered when a model of the AI system is detected to have drifted. This detection is typically done over multiple inputs and corresponding outputs. It relies on prior logging of inputs ahead of event detection. Model drift occurs when the statistical properties of the model’s predictions change over time, often due to shifts in the underlying data distribution.

For AI systems containing machine learning models, the organization must determine if the models’ design and characteristics are required to be audited or auditable. Only if this is the case does the rest of this requirement apply. At repeated points during the training of a machine learning model, such as after each iteration over the whole training dataset, the logging system must log information to locate the current step within the training process such as identifier of epoch, the current model parameters also known as checkpoint, and any available information on the quality of the current model such as evaluation measures on a validation dataset, loss value on validation data, or accumulated training loss over the epoch.

This information enables assessment of overfitting characteristics of deployed models. Overfitting occurs when a model learns the noise in the training data rather than the underlying signal, resulting in poor generalization to new data.

The logs must be stored until a decision is made to select one or more trained models among the candidate ones, and are retained at least for the models selected, and more if there are legal requirements specifying otherwise or other business value.

Triggers From Human Oversight

A log entry must be recorded, including unique identification of the representative or controller, as appropriate, when a human controller has interrupted or intervened in the operation of an AI system to prevent or remediate a serious incident, checked or validated an output of an AI system, or engaged, transferred, or disengaged control of an AI system.

The standard provides an information model for control activities and human oversight.

The organization must assess and justify whether it is necessary to record the reason that these events occurred based on risk.

Where human actions occur outside the technical boundary of the AI system, the logging functions must record them based on applicable regulatory requirements.

This requirement reflects the reality that many AI systems operate under partial human control. A human operator can override a decision, pause the system, or hand off control to another operator. Those actions are not internal to the AI system, but they are part of the system’s operational history and must be logged to establish accountability.

Required Information in Log Entries

The log record must be linked to AI system version information, which enables the connection between each log entry and the version of the AI system. When the AI system is based on multiple models or a model that changes over time, then model identifier and version information must also be included.

The log entries must contain a unique reference to the log event, a timestamp of the log entries using a standardized time format such as ISO 8601-1 for systems that have access to clock time, and inputs and outputs if they are necessary for understanding and analysis of an event having triggered this log entry or for supporting the detection of future events.

AI systems that do not have access to clock time must include information to enable estimation of time since the start of the AI system, such as the number of clock cycles since startup or a numerical identifier giving an ordering to entries.

A unique reference to the inputs and outputs may be used in place of the inputs and outputs. For example, sensor values can be stored in another system specifically for that purpose, and a reference to each sensor value be included in the AI system log.

Recommended Information in Log Entries

The log entries should contain event types which affect the ability of an AI system to perform in accordance with its intended purpose, such as inputs received, output generated, or error encountered.

They should contain source identification, for scenarios where inputs are routed from multiple sources, input provenance, traceability, and security auditing purposes. In case of multiple sources, each source can correspond to a sensor and a single sensor value can be traced to each individual sensor. Examples include sensor identifier, API endpoint, or data stream identifier.

They should contain a correlation identifier that correlates related log entries across the system for traceability. In an AI system in which a request is processed in several stages before a response is returned, the system can include an identifier that connects the content of the log entries at each stage of the request and is unique to the request.

They should contain system status with respect to a situation or behavior when the event was logged. A system status does not have to be recorded through a logging component for the AI system. It may be recorded through other logging components.

They should contain error handling, which includes detailed information on any errors or exceptions, including error codes and descriptions. They should contain error information containing detailed information on any errors or exceptions that occurred, including error codes, error messages, severity level, impact level, and system context.

They should contain detailed error handling information, which includes detailed information on failed operation if any, retrying, switching to a fallback mechanism, notifying the user, logging the error, escalating the issue, and recovering if possible.

Storing and Access to Logs

Logs refer to all the log entries that are created at some point in the life cycle of the AI system. However, this does not imply that those log entries are kept forever or are necessarily accessible to all stakeholders.

Some log entries warrant long-term storage, for instance if they are required for fulfilling regulatory obligations on record keeping. Log entries warranting long-term storage must be stored in a persistent way for future access. If the obligation to store them comes from an external stakeholder to the organization itself or applicable regulatory requirements, then the logs must have backups.

Governance schemes can both promote and restrict data access in relation to logging. Legal requirements, for example about data portability and privacy, can expand or restrict the requirement to maintain logs, along with the ability to use and share them.

Requirements for Third-Party Access

The organization may refrain from transmitting the AI system logs or parts of AI system logs if the intended recipient of the log has no permission to access the otherwise included information.

The transmission of the AI system logs or parts of AI system logs may be rejected if the intended recipient does not ensure that logs are stored securely, including confidentiality, integrity, and availability according to applicable regulatory requirements and the state of the art, that logs, backups, and derived information are deleted when no longer needed or when legally required to delete, that logs are not transferred to third parties unless the organization agrees, and that results of the evaluation of logs by the recipient are made available to the organization upon request.

Specific considerations of the legal basis, such as data subject consent, can be relevant to take into account when deciding whether to reject.

If there are multiple logging components within a single AI system and logging can be aggregated from the logging components, then aggregated logs can be transmitted.

Access for AI Users and Providers

The persons performing human oversight must have access to log entries triggered by automated monitoring.

Access to logs by the AI provider can be useful, for instance for facilitating post-market monitoring of an AI system. This access is typically subject to limitations due to confidentiality, intellectual property, or privacy. Aggregated information from logs can be accessed instead of the logs themselves.

Practical Implementation, Start With Risk and Work Backward

The standards are not prescriptive about architecture. You can implement logging in software, hardware, or a hybrid configuration. You can use centralized log aggregation or distributed logging pipelines. You can store logs in relational databases, object storage, or time-series databases.

What matters is that your logging system meets the functional requirements and supports the risk management, compliance, and oversight needs of your organization.

Start by conducting a risk assessment. Identify the harms that the AI system could cause, the events that could indicate those harms, and the evidence you would need to detect, investigate, or prove those events. Map those events to log triggers. Define the content, frequency, and retention requirements for each event type.

Next, design the logging architecture. Decide where logging components will be deployed, how log entries will be structured, how logs will be stored, and who will have access. Document your design decisions, justify them in terms of risk and compliance, and embed them in your technical documentation.

Implement logging as part of the system build, not as an afterthought. Instrument your code to generate log entries at the defined triggers. Validate that timestamps are accurate, that log entries contain the required information, and that logs are stored securely.

Test your logging system under realistic conditions. Simulate failures, adversarial inputs, domain drift, and human interventions. Verify that the logging system captures the events, that the logs are interpretable, and that you can reconstruct what happened.

Establish governance processes for log review, retention, and access. Define who can read logs, who can write logs, who can delete logs, and under what conditions. Implement access controls, audit trails, and tamper detection. Train your staff on how to use logs for monitoring, troubleshooting, and compliance.

Monitor the logging system itself. Set up alerts for log processing failures, storage capacity limits, and integrity violations. Treat logging failures as system failures.

Finally, map your logging system to the requirements in the future prEN 18229-1 if you are deploying high-risk AI in Europe. Verify that your logs support post-market monitoring, deployer oversight, and the specific obligations in Article 12 and Article 14. Update your technical documentation to reference the standard and explain how your logging system meets it.

AI System Logging Control Matrix

Below is a structured compliance reference for AI governance practitioners mapping every logging requirement, obligation level, and implementation guidance drawn from the ISO/IEC 24970 standard on AI system logging.

Control ID	Requirement	Obligation	Implementation guidance	Examples and notes	Governance purpose	Evidence and artefacts
5.1.1	An AI system log must represent information related to the operation, behavior, inputs, outputs, or context of an AI system, recorded to support current or future retrieval, analysis, oversight, or decision review.	Shall	Define the purpose of the log during the design phase. Logs must cover at minimum what the system did, what it received, and the context in which it operated. Retrieval must be possible for future audits rather than just live monitoring.	A loan decision AI must log the input features used, the credit score output, and the regulatory context, rather than only logging error events.	Accountability and audit readiness	Log schema documentation, and a data flow diagram showing log coverage
5.1.2	AI system logs can consist of structured, semi-structured, or unstructured data and can originate from the AI system itself, its internal components, interacting AI systems, users, or external observers.	May	Design logging to accommodate multiple data formats and sources. Do not restrict the logging architecture to a single format. Ensure your ingestion pipelines can handle structured JSON, semi-structured text annotations, and unstructured data such as screenshots or audio clips.	A multimodal AI processing both text and images can log text responses as JSON and image outputs as file references pointing to object storage.	Completeness and flexibility	Logging architecture diagram, and an ingestion pipeline specification
5.1.3	AI system logs can include time-stamped events, status snapshots, sensor or input data, outputs, decisions, error messages, environmental context, or annotations.	May	Use this list as a completeness checklist when designing log scope. For high-risk systems, you should consider all eight categories. For each category you exclude, document the justification in your technical files.	A fraud detection system should log the transaction data, the fraud probability score, the decision threshold applied, any model timeout, and the network latency at the time of the event.	Log completeness and risk coverage	Log content specification, and a gap analysis against this standard checklist
5.1.4	AI system logs can be stored persistently, processed in real time, managed under data minimization or privacy constraints, machine-readable, or human-interpretable.	May	Select storage and processing modes based on risk and specific use cases. High-risk systems with regulatory obligations require persistent storage. Systems needing live intervention require real-time processing. All logs must be interpretable by a human auditor because machine-readable data alone is insufficient for governance.	A medical AI must retain logs persistently for regulatory inspection, process alerts in real time for patient safety, and present logs in human-readable form to clinical auditors simultaneously.	Privacy compliance, auditability, and oversight	Retention policy, privacy impact assessment, and log viewer documentation
5.2.1	A logging component must be a functional part of an AI system, or an external system interacting with it, that supports the generation, capture, formatting, storage, or management of log data.	Shall	Formally identify logging components and document them in the system architecture. They can be internal or external, such as a separate observability platform or compliance logger. Regardless of location, they are subject to the same governance requirements as the AI system itself.	A cloud-based AI service can use the logging service of its cloud provider as an external logging component, but the organization remains accountable for what is logged and how it is secured.	System design and accountability	Architecture diagram identifying all logging components, and vendor contracts for external loggers
5.2.2	A logging component can consist of one or more subcomponents responsible for event detection, recording log entries, applying data policies such as filtering or redaction, or ensuring secure and reliable handling of log information.	May	Decompose complex logging needs into dedicated subcomponents. A redaction subcomponent should run before storage to prevent personal data from entering persistent logs. Event detection subcomponents should operate independently from storage subcomponents to avoid a storage failure silencing your detection capabilities.	A redaction pipeline strips patient names from medical AI logs before writing them to the audit store. A separate detection subcomponent receives the unredacted stream to assess anomalies but does not persist the sensitive data.	Privacy by design and resilience	Subcomponent design documentation, redaction policy, and a data flow diagram
5.2.3	Logging components must not assume a fixed structure, automation level, or deployment location. They can be implemented in software, hardware, or hybrid configurations.	Shall	Keep the logging system design flexible. Do not hard-code assumptions about where logs will be written or how automation is applied. This is critical for edge deployments, federated systems, or systems deployed in air-gapped environments.	An autonomous vehicle AI can log safety-critical events to onboard hardware storage and synchronize to a cloud audit store when connectivity becomes available.	Resilience and deployment flexibility	Logging design specification covering all intended deployment configurations
5.5.1	AI system logging must be the process of generating, capturing, recording, and managing information related to the operation, behavior, decisions, or context of an AI system.	Shall	Logging encompasses the full lifecycle including generation, capture, recording, and management. You must govern all four stages properly.	An organization that captures logs but never manages their retention or access controls has an incomplete logging governance program.	Governance completeness	Logging governance policy covering all four stages, alongside a retention schedule
5.5.2	Logging can be performed automatically by system components, manually by users or operators, or through hybrid methods. It can occur during design, testing, deployment, or post-deployment operation.	May	Establish which logging activities at each lifecycle stage are automated versus manual. Manual logging requires the same integrity controls as automated logging. Hybrid approaches are valid but require clear process documentation.	During testing, a developer manually annotates a log entry to flag unexpected model behavior. This annotation becomes a formal log entry subject to standard access controls and retention rules.	Lifecycle coverage and integrity	Logging procedures for each lifecycle stage, and an annotation policy
5.5.3	Logging activities can be continuous, event-driven, scheduled, or conditional.	May	Choose your logging frequency based on risk and operational needs. Document the chosen mode and its justification in your technical documentation.	During a cybersecurity incident, a system configured for event-driven logging switches to continuous logging of all inputs to capture the full attack pattern.	Risk proportionality and operational efficiency	Logging frequency policy, and technical documentation justifying the chosen modes
5.6.1	AI system logging must provide capabilities enabling traceability between multiple events and log entries if necessary to manage risk, relevant to the intended purpose, and technically feasible given the inputs and outputs.	Shall	End-to-end traceability is a core requirement. For any event requiring investigation, you must be able to reconstruct the full chain of events. Implement correlation identifiers to link related log entries across components and stages.	In a multi-stage content moderation AI, a single user post passes through language detection, toxicity scoring, and policy enforcement. A shared correlation ID links all three log entries so an auditor can reconstruct the full processing chain.	Accountability, forensics, and audit	Traceability design specification, and correlation ID implementation documentation
5.7.1.a	The organization must identify the security and privacy requirements for integrity and confidentiality protection of logs.	Shall	Conduct a data protection impact assessment specific to logging. Identify what personal data might appear in logs, who has access, what encryption is applied, and which regulatory frameworks apply.	A healthcare AI must identify that patient identifiers can appear in input logs and require that these be pseudonymized before storage, encrypted at rest, and accessible only to authorized clinical staff.	Privacy compliance and data security	Data protection impact assessment, encryption specification, and an access control policy
5.7.1.b	The organization must protect information in logs while taking into account different purposes of logging for different stakeholders.	Shall	Different stakeholders have different legitimate access needs and risks. Design role-based access controls reflecting these different purposes rather than giving all stakeholders access to all log content.	An AI developer needs full stack traces and model parameters, while a data protection officer only needs to see whether personal data was processed lawfully.	Privacy, role-based access, and proportionality	Stakeholder access matrix, and role-based access control documentation
5.7.2.1	Logging functions must enable the recording of events relevant for identifying situations that can result in the AI system presenting a risk according to the risk management process.	Shall	The risk register must drive log design. For every identified risk in the AI system risk assessment, there must be a corresponding log event or pattern that would surface that risk if it materialized.	If the risk register identifies biased outputs as a risk, the logging system must capture outputs with sufficient demographic context to detect this pattern through analysis.	Risk management integration	Risk register, and a mapping document linking risks to log events
5.7.2.2	Logging functions must facilitate monitoring of AI systems proportional to their risks, enabling collection, documentation, and analysis of performance data from initial development to end of retirement.	Shall	Logging must span the entire AI system lifecycle. Monitoring intensity should scale with the risk level, meaning higher-risk systems warrant more frequent and comprehensive logging. Performance data must be collected and actively analyzed.	A high-risk AI system used in employment decisions requires logging throughout development, testing, deployment, and decommissioning.	Lifecycle governance and proportionality	Lifecycle logging plan, and a risk-proportionate monitoring schedule
5.8.a	Technical documentation must explain and justify the specific criteria for determining relevant events.	Shall	Document not just which events are logged, but why those events were chosen. The criteria must be traceable to risk assessments so an auditor can understand the decision logic for event selection.	Any transaction where the confidence score falls below a specific threshold is logged as a low-confidence event because the risk assessment identifies this as a driver of incorrect decisions.	Auditability and transparency	Technical documentation section on event selection criteria, and a risk-to-event mapping
5.8.b	Technical documentation must explain and justify the specific criteria for logging relevant events.	Shall	Separate from determining which events are relevant, document the criteria for how and when they are logged. This includes thresholds, sampling rates, and triggering conditions, alongside justifications for any exclusions.	Inputs highly deviating from the mean are logged with full payloads, while minor deviations are logged with a flag only due to storage constraints and low incremental risk.	Auditability and design transparency	Technical documentation section on logging criteria, and a storage cost versus risk trade-off analysis
5.8.c	Technical documentation must specify any interaction with human controllers.	Shall	Identify every point in the system where a human can observe, intervene, override, or validate the AI system behavior. Document how each interaction is logged to maintain accountability.	The documentation specifies control points such as a compliance officer pausing inference, a caseworker overriding decisions, and a data scientist retraining the model.	Human oversight and accountability	Human-in-the-loop design documentation, and a control point register
5.8.d	Technical documentation must specify any interaction with automated monitoring.	Shall	Document what automated monitors are connected to the logging system, what conditions they detect, and what actions they trigger. Include the detection thresholds and the basis for setting them.	An automated bias detection module checks output distributions periodically and triggers an alert log entry if the demographic parity gap exceeds an internal policy threshold.	Automated oversight and transparency	Automated monitoring specification, and a threshold justification document
5.8.e	Technical documentation must recommend a frequency and scope of monitoring for relevant events.	Shall	Specify how often each category of event is monitored and reviewed. Scope should define which aspects of the log are reviewed and by whom.	High-risk events such as adversarial attacks are monitored in real time by automated systems and reviewed by a human within hours, while routine events are reviewed in weekly batch analyses.	Operational oversight	Monitoring schedule, and escalation procedures
5.8.f	Technical documentation must recommend a frequency and scope of logging relevant events.	Shall	Specify how often logging occurs for each event type and what information is captured each time. Document the trade-offs between observability, cost, and data volume.	Transaction initiation events are logged continuously, model drift metrics are logged hourly as a batch summary, and training checkpoints are logged after each epoch.	Design governance	Logging frequency specification per event type
5.8.g	Technical documentation must explain and justify the accuracy and precision of timestamps where used.	Shall	Document the clock source, its synchronization mechanism, the precision used, and the timezone convention. For distributed systems, document how you manage clock skew between components.	The system uses UTC timestamps at millisecond precision, synchronized to a specific server.	Forensics and traceability	Clock synchronization specification, and a timestamp format definition
5.8.h	Technical documentation must explain and justify resource constraints affecting logging, such as memory capacity, storage capacity, or processing power.	Shall	Document the physical and financial limits on logging. If resource constraints force explicit trade-offs, these trade-offs must be justified and reviewed periodically as risk levels change.	An edge deployment has limited onboard storage. Logs are compressed and streamed to cloud storage periodically. If connectivity is lost, the system overwrites the oldest entries first.	Risk management and design	Resource constraint analysis, and a fallback logging specification
5.8.i	Technical documentation must explain and justify constraints related to privacy that affect logging.	Shall	Identify every privacy constraint that limits what can be logged based on data protection laws, contractual obligations, or ethical commitments. Document what data is excluded from logs and list any compensating controls.	Privacy constraints prevent logging raw user queries containing sensitive health data. As a compensating control, queries are classified and logged using category codes instead of full text.	Privacy compliance	Privacy constraint register, legal basis documentation, and compensating controls
5.8.j	Technical documentation must refer to related legal requirements concerning data protection, system accountability, traceability, and transparency.	Shall	Maintain a live register of applicable legal requirements that intersect with logging, such as the GDPR or the EU AI Act. Update this register when regulation changes.	The legal requirements register includes rules around data minimization, retention limits, and specific logging mandates for high-risk AI systems.	Legal compliance	Legal requirements register, and a regulatory mapping document
5.8.k	Technical documentation must include appropriate information security considerations and data retention policies for logs.	Shall	Logs are sensitive assets. Document the security controls applied, such as encryption, access controls, integrity protection, and retention periods.	Transaction logs are retained for several years due to regulatory requirements, encrypted at rest and in transit, and accessed only via multi-factor authentication.	Information security and compliance	Retention schedule, encryption specification, access control policy, and integrity protection specification
5.8.l	Technical documentation must include a specification of failure handling detailing how the AI system reacts when log memory is overloaded.	Shall	Define what happens when storage is full, when the logging component crashes, or when network connectivity is lost. Failure modes must be designed to avoid silent data loss.	If log storage reaches capacity, an alert is raised and the system switches to emergency logging mode. If storage hits maximum capacity, the system halts new inference requests rather than proceeding unlogged.	Resilience and safety	Failure mode specification, and an incident response procedure for logging failures
5.8.m	Technical documentation must include interfaces with other systems that affect logging.	Shall	Document every external system that sends data to or receives data from the logging system. Include APIs, data formats, authentication methods, and failure responses.	Logging interfaces include upstream model serving infrastructure pushing events via an internal API and downstream platforms pulling logs securely.	System architecture and completeness	Interface register, API specifications, and failure response documentation
5.8.n	Technical documentation must contain a specification of the log data structures used.	Shall	Publish a formal schema for every log entry type to enable automated processing, consistent querying, and third-party audits. Specify field names, data types, and permissible values.	A transaction log entry schema requires specific fields like event IDs, system IDs, timestamps, and input references.	Interoperability and auditability	Log schema specification, data dictionary, and schema version control
6.1.1	Risk must be considered when determining which events are to be detected.	Shall	The AI system risk assessment must be the primary input to logging design. Every identified risk must map to at least one detectable event in the logging system.	A risk of geographic bias translates to a detectable event where region tags are logged with each transaction and analyzed in batch reviews.	Risk management integration	Risk-to-event mapping document, and a risk register
6.1.2	Risk must be considered when determining which events are relevant.	Shall	Relevance is determined by the risk context. Document the relevance criteria explicitly to avoid logging trivial events that create noise and degrade the quality of governance.	A model serving high request volumes logs transactions above a specific value threshold and samples a small percentage of routine transactions for monitoring purposes.	Risk proportionality	Relevance criteria documentation, and a risk-proportionality justification
6.1.3	Risk must be considered when determining which relevant events are to be logged.	Shall	Log events in order of risk severity. Safety-critical and compliance-critical events must always be logged, while lower-priority events can be subject to sampling or conditional logging.	Priority events like adversarial attacks or bias detections are always logged. Routine transaction metadata is sampled based on available resources.	Risk prioritization	Event priority matrix, and a logging resource allocation policy
6.1.4	Events must be logged in relation to inputs or outputs when caused or observed by controllers or components of the AI system. Relevant events to be logged must be selected based on risk.	Shall	Every logged event must be anchored to an observable input or output rather than an internal state that cannot be independently verified. Record the analysis that led to the selection of logged events.	When a human operator overrides a model output, the log entry captures the original output, the override action, the modified output, and the controller identity.	Accountability and verifiability	Event selection analysis, and a risk-based selection methodology
6.1.5	Inputs or outputs relevant to event detection must be logged at a frequency that is technically feasible and allows risk to be managed in the context of the intended purpose.	Shall	Set frequency based on the time horizon of the risk. If a risk could cause harm quickly, logging frequency must be sufficient to detect it within that window.	A trading AI with systemic risk implications logs transactions in real time, while a content recommendation AI logs aggregate bias metrics hourly.	Risk timeliness	Frequency justification per event type, and a risk time-horizon analysis
6.1.6	Logging functions must be designed and configured to generate logs accurately representing logged events.	Shall	Validate logging accuracy periodically by comparing logged data against ground truth from the AI system itself. Treat any logging inaccuracy as a governance defect.	During a validation test, any discrepancy between the submitted test inputs and the logged inputs requires immediate remediation.	Integrity and reliability	Logging accuracy validation procedure, and test results
6.2.1	Log entries about events should be timestamped. The timestamp must record the time of the event to an accuracy and precision appropriate for the type of event and its role with respect to the intended purpose.	Shall	Timestamp precision must match the risk horizon of the event. Real-time safety-critical systems require high precision, while compliance reporting systems can use lower precision.	A high-frequency trading AI requires microsecond timestamps to reconstruct event orders, while a monthly bias audit system requires only date-level timestamps.	Forensics and sequencing	Timestamp precision specification per event type, and a justification document
6.2.2	Where technically feasible, the order of log entries should correspond to the order of the events logged.	Should	Log entry ordering is critical for forensic reconstruction. Use sequence numbers or logical clocks where exact wall-clock ordering cannot be guaranteed, and document any known ordering limitations.	In a distributed AI system with latency, a logical sequence number is appended to each log entry to provide ordering within each node.	Forensics and traceability	Ordering mechanism specification, and known limitations documentation
6.2.3	Timestamps should be formatted in a standardized format. If the time zone is not included within the timestamp, a mechanism to determine the applicable time zone must be specified in technical documentation.	Shall	Use the ISO 8601 format with explicit UTC offsets for all timestamps. If system constraints prevent this, the technical documentation must provide an unambiguous method for determining the applicable timezone.	Timestamps use clear formatting with explicit UTC offsets. If the timezone cannot be included, documentation strictly defines the default timezone used by the system.	Interoperability and forensics	Timestamp format specification, and timezone documentation
6.2.4	Log entries should include an information element that enables connection between the logged information and the AI system or its components, where appropriate.	Should	Every log entry should carry an identifier linking it to the specific AI system that generated it, which is essential in shared infrastructure environments.	In a microservices environment, each log entry includes a system ID that identifies which governed AI system generated the entry.	Accountability and attribution	System reference specification, and an AI system register
6.3.a	Additional logging functions can record the period of each system use, such as start and end timestamps.	May	Recording session boundaries enables you to calculate system utilization and identify unusually long sessions that may indicate misuse.	A medical AI logs session start and end times per user. Sessions longer than expected trigger a review.	Usage monitoring and security	Session logging specification
6.3.b	Additional logging functions can reference an external data source or database against which input data is checked.	May	If the AI system validates inputs against an external reference like a sanctions list, log which version of that external source was used at the time of the check.	A financial crime AI records the specific sanctions database version identifier used for each check to enable retrospective reviews if the list updates.	Reproducibility and accountability	External reference logging specification, and a version management policy
6.3.c	Additional logging functions can log the relevant input data.	May	Full input logging provides the richest basis for audit but carries high data volume and privacy costs. Log full inputs for high-risk decisions and log input references for routine transactions.	A credit decision AI logs the full feature vector for declined applications to enable explanations, while approved applications are logged by reference only.	Explainability and redress	Input logging policy, and a privacy impact assessment
6.3.d	Additional logging functions can provide traceability at a level that enables the identification of individuals involved in result verification.	May	If human verification of AI outputs is part of the process, record exactly who performed the verification to establish individual-level accountability.	When a caseworker verifies a benefits assessment recommendation, the log records their employee ID, the timestamp, and whether they accepted or modified the output.	Human oversight accountability	Verification logging specification, and an individual identification mechanism
6.4.a	Logging functions must issue alerts when the integrity of log processing is violated.	Shall	Monitor the logging pipeline itself. If log entries are dropped or corrupted, this is a critical governance failure. Implement checksums and processing integrity checks.	Alerts trigger immediately if the message queue depth exceeds expected thresholds or if checksum mismatches are detected.	Logging integrity and governance assurance	Log processing integrity monitoring specification, and alert configurations
6.4.b	Logging functions must issue alerts when the confidentiality of log storage has been compromised.	Shall	Unauthorized access to log storage is a security incident. Implement access logging on the storage itself and alert on anomalous access patterns.	Alerts trigger if log storage is accessed by unauthorized accounts or if bulk downloads occur outside normal business hours.	Information security and privacy	Log storage access monitoring specification, and an incident response procedure
6.4.c	Logging functions must issue alerts when the integrity of stored logs is violated or foreseeably can no longer be ensured for the full operational lifetime.	Shall	Implement integrity verification like cryptographic hashing to maintain log integrity for the entire retention period. Alert when checks fail or storage degrades.	Daily integrity verification runs compare stored log hashes against write-time hashes, triggering alerts upon any mismatch.	Long-term integrity	Integrity verification specification, storage health monitoring, and integrity check results
6.4.d	Logging functions must issue alerts when log storage capacity is reached or exceeded.	Shall	Implement tiered capacity alerts to provide sufficient warning for remediation before capacity is reached, preventing silent data loss.	Capacity alerts trigger warnings at 85 percent capacity to initiate archiving processes, and critical alerts at 95 percent to trigger emergency expansion.	Operational resilience	Capacity monitoring specification, alert thresholds, and a capacity management procedure
6.4.e	The frequency and monitoring of logging anomaly alerts must be justified.	Shall	Document the justification for each alert threshold to avoid alert fatigue while ensuring genuine issues are detected. Specify alert response times clearly.	The documentation justifies capacity alert thresholds based on lead times required for archiving processes and log growth rates.	Governance assurance	Alert justification document, alert fatigue reviews, and service level agreements for alert responses
7.1.1	A log entry can be triggered by the reception or processing of an input, human actions, specific software interactions, or the automated or manual detection of certain events.	May	Implement logging triggers across all categories to prevent unmonitored activity. Map triggers to the risk register to confirm that every risk has a corresponding detection trigger.	Triggers for a claims processing AI include new claims received, human overrides, completed model inferences, and automated detection of outlier values.	Coverage and risk management	Trigger inventory, and a risk-to-trigger mapping
7.1.2	Events can pertain to inputs, outputs, the state of the AI system, or a combination. Relevant events can consist of patterns over time or properties of individual inputs, outputs, and states.	May	Design monitoring to detect both instantaneous events and gradual temporal patterns. Pattern-based detection requires input logging to be active prior to pattern identification.	A single transaction with an unusually high value is an instantaneous event, while a gradual increase in high-value transactions over a month is a pattern event requiring historical logs.	Detection completeness	Event detection specification, and a pattern detection design
7.2.1	A log entry must be recorded when the AI system, or a component of it, encounters a software error.	Shall	Log all software errors affecting the AI system, including those caused by internal faults and external factors like malformed inputs. Ensure you capture errors that affect system outputs.	If a model request times out and the AI returns a fallback response, both the timeout error and the fallback mechanism used must be logged.	Reliability and incident response	Error event log entries, and incident reports
7.2.2	Outlier inputs must be detected based on statistical thresholds, domain-specific anomaly detection metrics, and contextual metadata.	Shall	Define and document the outlier detection methodology before deployment. Derive statistical thresholds from the training data and utilize contextual metadata to inform your assessments.	An outlier is detected if a pixel intensity distribution deviates significantly from the training set or if an image resolution falls below minimum diagnostic standards.	Anomaly detection and safety	Outlier detection specification, threshold justifications, and a review schedule
7.2.3	Potential attack triggers must include unauthorized access patterns, data integrity violations, model inversion, and poisoning signatures.	Shall	Implement detection for unauthorized access volumes, tampered inputs, systematic probing patterns, and malicious inputs. This requires comprehensive input logging.	If a system detects a sequence of queries from a single source showing systematic feature variation, it flags a potential model inversion attack and triggers a security alert.	Security and integrity	Attack detection specification, security monitoring configurations, and incident response procedures
7.2.4.a	A log entry should be recorded when a user requests a review of a transaction.	Should	User review requests indicate potential algorithmic unfairness or error. Link each request to the original transaction log entry using a unique reference system.	When a user disputes a loan rejection, the complaints system generates a log entry referencing the original transaction ID from the AI decision log.	Redress and accountability	Complaint log entries linked to transaction logs, and complaints management system integrations
7.2.4.b	A log entry should be recorded when a user submits a complaint or provides feedback.	Should	Log every instance of user feedback, including informal ratings. Aggregate feedback serves as a governance signal to identify systematic AI errors.	The system logs negative ratings on AI recommendations, including pseudonymized user IDs and timestamps, for weekly review by the product governance team.	User redress and quality monitoring	Feedback log entries, and aggregate feedback reporting
7.2.4.c	A log entry should be recorded when authorized personnel or systems process a user complaint or feedback.	Should	Log the processing of complaints to create a complete audit trail. This enables you to assess if complaints are handled appropriately and within required timeframes.	Log entries track when a complaint is received, assigned to a handler, investigated, and ultimately resolved, along with handler IDs and timestamps.	Redress process accountability	Complaints processing logs, and a handler activity audit trail
7.2.5	A log entry should be recorded upon determination of the outcome of a user request, with a reference to the original user request.	Should	Every complaint requires a corresponding outcome log entry referencing the original AI decision to prove that the issue was resolved.	Outcome logs detail the final decision, any remedial actions taken such as reversing the decision, and the handler responsible for the resolution.	Redress completeness	Outcome log entries, and complaints closure reports
7.2.6	The communication of information to AI users or subjects can trigger a log entry.	May	Log when disclosures, privacy notices, or terms of service are communicated to users. Record what was disclosed, when, and the user response.	When an AI system informs a user they are interacting with an algorithm, the system logs the disclosure type, timestamp, user identifier, and the specific disclosure text version.	Legal compliance and consent management	Disclosure log entries, consent records, and disclosure text version control
7.3.1	A log entry must be triggered when an adversarial attack is detected. Detection can occur on a single input or be inferred from a pattern across multiple inputs.	Shall	Adversarial attack detection is mandatory. Pattern-based detection requires historical input logging to analyze probing campaigns across thousands of inputs.	Detecting a prompt injection attempt in a single API call or identifying coordinated queries across multiple IPs will both trigger log entries and security alerts.	Security and integrity	Adversarial attack log entries, security monitoring configurations, and attack detection methodologies
7.3.2	A log entry must be triggered when unwanted bias is detected in the outputs of the AI system.	Shall	Bias detection requires both input and output logging to identify patterns across multiple transactions. Define acceptable fairness metrics and document the thresholds that trigger alerts.	If demographic parity gaps exceed policy thresholds over a specific rolling window, the system creates a log entry and notifies the governance team.	Fairness and regulatory compliance	Bias detection log entries, fairness metric specifications, and threshold justifications
7.3.3	A log entry must be triggered when the AI system is detected to operate out of its domain. Detection of domain drift must be considered as operating out of the domain.	Shall	Out-of-domain operation occurs when inputs fall outside the training data distribution. Detect single violating inputs or gradual distributional shifts and flag the outputs as unreliable.	If the proportion of inputs from a new geographic region increases significantly and shifts the distribution outside training parameters, a drift event is logged.	Safety and model governance	Out-of-domain detection log entries, domain boundary specifications, and domain drift monitoring
7.3.4	A log entry must be triggered when a model of the AI system is detected to have drifted.	Shall	Model drift indicates behavioral changes from validated baselines. You must log historical performance baselines alongside current metrics to accurately detect and address drift.	When the divergence between current output distributions and the rolling baseline exceeds limits, a drift event is logged to initiate model revalidation.	Model governance and safety	Model drift log entries, baseline performance specifications, and drift detection methodologies
7.3.5.a	For auditable ML models, the logging system must log information to locate the current step within the training process at repeated points during training.	Shall	If auditability is required, logging infrastructure must be active during training. Use epoch identifiers to allow the reconstruction of the training trajectory.	After each epoch, the log entry records the epoch ID, training run ID, dataset version, and exact start and end timestamps.	Model auditability and reproducibility	Training log entries, and a training run registry
7.3.5.b	For auditable ML models, the logging system must log the current model parameters at repeated points during training.	Shall	Model checkpoints act as physical evidence of the model state during training, enabling restoration or investigation. Account for the significant storage requirements.	After each epoch, model weights are serialized and stored to a checkpoint registry with unique IDs stored in append-only storage.	Model auditability and reproducibility	Checkpoint storage, checkpoint registry, and checkpoint integrity controls
7.3.5.c	For auditable ML models, the logging system must log any available information on the quality of the current model at each training checkpoint.	Shall	Quality metrics at each checkpoint enable auditors to detect overfitting and verify that the deployed model was appropriately validated.	Checkpoint logs record validation loss, validation accuracy, and training metrics to ensure gaps indicating overfitting are reviewed before deployment.	Model quality assurance and auditability	Quality metric log entries, and overfitting assessment procedures
7.3.5.d	Training logs must be stored until a decision is made to select candidate models, and retained at least for the selected models.	Shall	Logs must persist through the model selection process. Afterward, retain logs for selected models based on legal requirements and document the deletion of rejected models.	Following a training run, logs for the selected model are retained for years, while logs for rejected epochs are deleted shortly after the decision is documented.	Retention compliance	Retention policy for training logs, model selection decision records, and deletion logs
7.4.a	A log entry must be recorded, including the unique identification of the controller, when a human interrupts or intervenes in the operation of an AI system to prevent or remediate a serious incident.	Shall	Accountability requires a named individual in the log, not a generic team role. Define what constitutes a serious incident and record any human intervention addressing it.	When an operator halts an AI system due to unexpected behavior, the log captures their specific employee ID, the intervention type, and the reason for the halt.	Accountability and incident management	Intervention log entries, incident reports, and controller identity verification
7.4.b	A log entry must be recorded, including the unique identification of the controller, when a human checks or validates an output of an AI system.	Shall	In human-in-the-loop systems, log every validation event with the validator’s identity to create an audit trail of who approved specific AI decisions.	When a radiologist reviews a diagnostic suggestion, the log records their ID, the timestamp, and whether they accepted or modified the AI output.	Human oversight accountability	Validation log entries, and validator identity management
7.4.c	A log entry must be recorded, including the unique identification of the controller, when a human engages, transfers, or disengages control of an AI system.	Shall	Control transfers shift accountability. Create clear records showing who held control, when they relinquished it, and who took over to prevent governance gaps.	During a shift handover, the log details the controllers involved, the timestamp, the specific control points transferred, and any operational handover notes.	Chain of custody and accountability	Control transfer log entries, and a control chain reconstruction capability
7.4.d	The organization must assess and justify whether it is necessary to record the reason for human controller actions based on risk.	Shall	explicitly decide and document whether recording the reason for human interventions is mandatory. For high-risk systems, reasons are typically essential for regulatory reporting.	An organization mandates reason recording for employment AI overrides to distinguish legitimate governance actions from biased interventions.	Accountability and risk management	Assessment documentation, and a reason-recording policy
7.4.e	Where human actions occur outside the technical boundary of the AI system, the logging functions must record them based on applicable regulatory requirements.	Shall	Implement manual log entry capabilities to capture physical actions or verbal instructions related to the AI system that regulations require you to track.	A manager verbally instructs an operator to power down a server during an incident, and subsequently enters a manual log detailing the action and regulatory basis.	Regulatory compliance and completeness	Manual log entry procedures, out-of-system action records, and a regulatory requirements register
8.1.1	The log record must be linked to AI system version information, enabling connection between each log entry and the version of the AI system.	Shall	Specify version identifiers clearly to distinguish between releases. This is essential for identifying all log entries generated by a system version if a defect is discovered later.	Log entries include granular system version IDs such as hotfix tags, allowing you to isolate transactions processed between specific updates.	Incident investigation and version control	Version identifier in all log entries, and a release register
8.1.2	When the AI system uses multiple models or models that change over time, the specific model identifier and version information must be included.	Shall	Identify the precise model processing the transaction. For third-party foundation models, ensure you log the provider’s specific model version rather than just an internal reference.	Systems utilizing external LLMs must log the specific model release versions to distinguish behaviors before and after provider updates.	Model accountability and incident investigation	Model identifiers in all log entries, model version registers, and third-party model version tracking
8.1.3.a	Log entries must contain a unique reference to the log event.	Shall	Assign a globally unique identifier to every log entry at the point of event occurrence. This identifier serves as the primary key for deduplication, correlation, and auditing.	The system generates a UUID for each event, returning it to the calling application so it can be included in future user communications regarding that transaction.	Traceability and reference integrity	Event ID generation specification, and uniqueness guarantees
8.1.3.b	Log entries must contain a timestamp of when the event was observed by the logging function. Systems without clock access must enable estimation of time since system start.	Shall	Record when the logging function observed the event. For embedded systems lacking real-time clocks, use cycle counts and document the methodology to approximate wall-clock time.	Systems without clocks record the cycle count alongside a boot timestamp to allow accurate approximations of event times.	Forensics and sequencing	Timestamps in all log entries, and time source documentation
8.1.3.c	Log entries must contain inputs and outputs, or unique references to them, if necessary for understanding the event or supporting future event detection.	Shall	You must capture inputs and outputs for governance-relevant events. Use pointers to external storage locations for large or sensitive payloads instead of embedding them directly.	Instead of embedding a massive sensor reading, the log includes a URI pointing to a secure object storage bucket where the data is kept.	Auditability and investigation	Input and output references in log entries, and storage system specifications
8.2.a	Log entries should contain event types that affect the ability of an AI system to perform in accordance with its intended purpose.	Should	Classify entries using a controlled vocabulary to enable automated filtering and targeted alert rules. Define the classification scheme before deployment.	Use an event taxonomy featuring standardized terms like input received, human override, or bias alert to streamline analytics.	Monitoring and analytics	Event type taxonomy, and classification documentation
8.2.b	Log entries should contain source identification for scenarios where inputs route from multiple sources to enable provenance, traceability, and security auditing.	Should	Knowing input origins is crucial for identifying attacks or faulty sensors. Use highly specific source identifiers rather than generic API labels.	IoT systems log specific sensor node IDs and physical locations to rapidly identify which device is generating anomalous readings.	Traceability and security auditing	Source identifiers in relevant log entries, and a source registry
8.2.c	Log entries should contain a correlation identifier linking related log entries across the system for traceability.	Should	Generate a correlation ID when a request is received and propagate it to all downstream components to track the transaction through the entire processing chain.	An auditor can use a single correlation ID to query log entries from the API gateway, preprocessing service, model inference layer, and response handler simultaneously.	End-to-end traceability	Correlation ID implementation, and traceability query capabilities
8.2.d	Log entries should contain system status context regarding situations or behaviors present when the event was logged.	Should	Include context such as system load, active maintenance windows, or recent configuration changes to distinguish normal variations from genuine incidents.	A bias alert log includes system context noting recent model weight updates, helping investigators determine the root cause of the alert.	Context and investigation	System status logging specification, and status data sources
8.2.e	Log entries should contain detailed information on errors or exceptions, including error codes and descriptions.	Should	Standardize error codes and descriptions into human-readable formats. Raw stack traces are insufficient for governance as they require developer interpretation.	Error logs detail both the technical exception and a human-readable summary explaining that the user received a fallback response.	Incident management and auditability	Error taxonomy, and error code documentation
8.2.f	Log entries should contain error information detailing severity levels, impact levels, and system context.	Should	Distinguish between technical severity and actual user impact. Both dimensions must be logged alongside system context to enable proportionate incident responses.	A core model failure triggers a high severity alert, but indicates low user impact because a fallback model successfully served the requests.	Incident prioritization and response	Error log entries containing severity and impact data, and impact classification methodologies
8.2.g	Log entries should contain detailed error handling information such as retries, fallback switches, user notifications, escalations, and recovery.	Should	An error handled gracefully is entirely different from a silent failure. Log the complete response chain to enable assessments of your error handling procedures.	The log chain captures exactly when an error was detected, when retries failed, when fallback mechanisms activated, and when the primary model was restored.	Resilience and incident management	Error handling log entries, and incident timeline reconstruction capabilities
9.1.1	The organization is not required to keep all log entries forever or make them accessible to all stakeholders.	May	Establish a formal retention schedule specifying retention periods and deletion triggers for each log entry type based on legal obligations and business needs.	Transaction logs are kept for years due to financial regulations, while user complaint logs are retained based on limitation periods for legal claims.	Retention compliance and data minimization	Retention schedule, legal requirements register, and deletion procedures
9.1.2	Log entries warranting long-term storage must be stored persistently for future access.	Shall	Implement persistent storage specifically for log types requiring long-term retention. Ensure storage survives hardware failures, software faults, and deliberate deletion attempts.	High-risk AI logs are stored in append-only storage replicated across geographically separated data centers, with periodic recovery testing.	Regulatory compliance and resilience	Persistent storage specification, replication architecture, and recovery test results
9.1.3	If external stakeholders or regulatory requirements mandate log storage, the logs must have backups.	Shall	Backup copies are mandatory for regulated logs. Backups must be independent of primary storage, regularly tested for restorability, and subject to strict security controls.	Logs are backed up daily to separate sites, encrypted with independent keys, and tested monthly to ensure regulatory compliance.	Regulatory compliance	Backup specifications, backup test results, and an external obligation register
9.1.4	Governance schemes can both promote and restrict data access in relation to logging.	May	Document all schemes affecting log access, including regulatory inspection rights that promote access and confidentiality obligations that restrict it. Establish conflict resolution protocols.	If data subject access rights conflict with third-party confidentiality, the protocol dictates extracting and redacting the logs before sharing.	Governance and legal compliance	Governance scheme register, conflict resolution protocols, and access rights documentation
9.2.1	The organization can refrain from transmitting AI system logs if the intended recipient lacks permission to access the information.	May	Verify recipient authorizations against your access control policy before transmission. Redact logs to provide only the data the recipient is authorized to view.	A third-party auditor requesting full logs is provided a redacted extract containing decision outputs but excluding unauthorized raw input data.	Access control and privacy	Access permission assessment procedures, and recipient authorization records
9.2.2.a	Log transmission can be rejected if the recipient does not ensure logs are stored securely according to regulatory requirements.	May	Obtain written assurances of security controls from third parties before sharing logs. Require encryption, access controls, and availability SLAs in data processing agreements.	Contract clauses mandate that recipients encrypt all received AI logs at rest and in transit while maintaining strict access logging.	Information security	Data processing agreements, recipient security assessments, and transmission refusal records
9.2.2.b	Log transmission can be rejected if the recipient does not ensure logs and derived information are deleted when legally required.	May	Derived reports and analyses must be deleted alongside raw logs. Require recipients to provide evidence of deletion when retention periods end.	Contracts mandate that recipients delete all logs and derived analytical reports within specific timeframes and provide written certification of completion.	Data lifecycle management	Deletion obligation clauses in contracts, and deletion certificates from recipients
9.2.2.c	Log transmission can be rejected if the recipient does not ensure logs are kept from third parties unless the organization agrees.	May	Control sub-processing by requiring prior written consent before a recipient transfers logs to their own vendors, ensuring sub-processors meet equivalent security standards.	Contract clauses strictly forbid recipients from sharing AI logs with third-party cloud providers without prior written consent.	Supply chain control	Sub-processing consent records, sub-processor registers, and onward transfer controls
9.2.2.d	Log transmission can be rejected if the recipient does not share the results of their evaluation of the logs with the organization.	May	Maintain visibility into how recipients use your logs. Require auditors or regulators to share evaluation findings so you can improve your internal governance programs.	Audit agreements stipulate that external auditors must share summaries of their log review findings within a specific timeframe after completion.	Governance assurance	Evaluation results sharing clauses, and records of results received
9.2.3	If there are multiple logging components and logging can be aggregated, aggregated logs can be transmitted.	May	Transmitting aggregated data is a practical, privacy-preserving approach when raw logs contain sensitive information. Document the aggregation methods applied.	Instead of sharing millions of raw transaction logs, the organization transmits a monthly summary of error rates and bias metrics to a regulator.	Practical compliance and privacy	Aggregation methodology documentation, and aggregated log transmission records
9.3.1	Persons performing human oversight must have access to log entries triggered by automated monitoring.	Shall	Surface automated monitoring alerts to designated oversight personnel in a timely, interpretable format using role-controlled dashboards.	Governance officers utilize real-time dashboards to view bias alerts and adversarial attack detections, allowing them to drill down into the underlying log entries.	Human oversight effectiveness	Oversight dashboard specifications, access control records, and oversight personnel registers
9.4.1	AI providers can access logs for post-market monitoring purposes, subject to limitations regarding confidentiality, intellectual property, or privacy.	May	Govern provider access strictly to ensure they do not receive unfettered access to customer data. Define exactly what the provider can access and for what specific purposes.	An agreement allows an AI provider to view aggregated performance metrics monthly, but explicitly forbids access to individual transaction inputs or outputs.	Provider accountability and privacy	Provider access agreements, access scope documentation, and access logs
9.4.2	Aggregated information from logs can be accessed by AI providers instead of the logs themselves.	May	Define aggregation granularity in your provider agreements. Ensure it provides sufficient detail for monitoring while minimizing the exposure of sensitive customer data.	Providers receive monthly reports detailing latency percentiles and error rates without exposing any personal data or raw transaction content.	Privacy and provider governance	Aggregated access specifications, provider access agreements, and aggregation methodologies

Logs Are Not Optional Anymore

The regulatory and technical landscape for AI has shifted. Logging is no longer a developer convenience or a debugging tool. It is a legal requirement, a risk control, and a source of institutional memory.

ISO 24970 and prEN 18229-1 give you the blueprint. They define what to log, when to log it, how to structure log entries, and how to manage log access and retention. They embed logging into risk management, human oversight, and post-market surveillance. They turn operational telemetry into auditable evidence.

If you are building, deploying, or operating high-risk AI systems, start designing your logging system now. Map your risks, define your triggers, implement your logging components, and establish your governance processes. Document your decisions, validate your implementation, and monitor your logs.

When your system fails, your logs will tell the story. Make sure the story you tell is one you can defend.

The prEN 18228 Problem: Why Your AI Risk Assessment Will Fail the First Real Test

Fri, 08 May 2026 00:00:00 +0000

Most ook solid on paper and collapse the moment a regulator, client, or auditor asks a simple question. What exactly can go wrong, how likely is it, and what does it cost when it does.

That gap is about to matter more.

A new European standard, prEN 18228, sets out a formal process for managing risks in AI systems across their full life cycle. It is designed to support regulatory expectations by requiring organizations to identify hazards, estimate and evaluate risks, define acceptability criteria, and continuously monitor controls. It brings structure and discipline. It also brings a product safety mindset into AI, focusing on harm to people, rights, and systems.

This sounds like progress. In many ways, it is.

But most organizations will apply it the same way they apply existing compliance frameworks. They will produce well-documented processes, consistent terminology, and defensible artifacts. And they will still struggle to answer the one question that drives real decisions. Should we deploy this system, under these conditions, with this level of exposure.

That is where this discussion starts.

The Standard Brings Structure. It Does Not Solve the Decision Problem.

prEN 18228 defines risk in familiar terms. Probability of harm and severity of that harm. It requires organizations to identify hazards, assess risks, and reduce them to an acceptable level based on the intended use and reasonably foreseeable misuse of the system.

This is a disciplined approach. It forces teams to think beyond model accuracy and consider real-world impact. It also aligns well with how regulators think about safety and rights. The limitation is more subtle.

The standard tells you how to run the process. It does not tell you how to make the decision. It does not require you to quantify exposure in financial or operational terms. It does not connect model behavior to business outcomes like lost contracts, regulatory investigations, or reputational damage that affects future revenue. So you end up with a structured assessment that still relies on qualitative judgments at the point where decisions are made. Most organizations are comfortable there. They should not be.

The Risk Definition Works for Products, but AI Behaves Differently.

The comes from product safety. It works well when failure modes are clear and causation is traceable. A component fails. A system stops. Harm follows in a relatively predictable way. AI systems behave differently.

They fail in ways that are distributed, context-dependent, and often only visible after deployment. A fraud detection model might perform well overall and still produce systematic errors for a specific segment. A decision system might be technically accurate and still generate outcomes that trigger regulatory scrutiny or client disputes.

Two risks can produce the same expected value and require completely different responses. A frequent, low-impact error calls for process improvement and monitoring. A rare but severe failure calls for governance, escalation, and sometimes a decision not to deploy at all.

The standard does not distinguish clearly between these cases. It treats them within the same structure, which can flatten the differences that matter most in practice.That is where need to go beyond the text.

The Terminology You Need to Understand Before You Start

The standard introduces 68 defined terms across six domains, and most of them do not mean what you think they mean.

Terms Relating to the EU AI Act

Source: Regulation (EU) 2024/1689 (EU AI Act)

AI system / Artificial intelligence system: Machine-based system that is designed to operate with varying levels of autonomy and that can exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. This definition is broader than most technical definitions of AI. It includes rule-based systems and statistical models that exhibit adaptiveness after deployment, not just machine learning systems.
Intended purpose: Use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. Technical documentation is not accompanying documentation. Information on technical documentation can be found in Article 11 of the EU AI Act. Your marketing claims define your regulatory obligations. If you claim the system works in a particular context, that context becomes part of your intended purpose and you must demonstrate safe operation there.
Reasonably foreseeable misuse: Use of an AI system in a way that is not in accordance with its intended purpose, but which can result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. Reasonably foreseeable human behaviour includes the behaviour of all types of relevant users. Reasonably foreseeable misuse can be intentional or unintentional. You cannot disclaim liability by saying users deployed your system incorrectly if that incorrect use was reasonably foreseeable. This standard requires you to model misuse scenarios and control for them.
Performance: Ability of an AI system to achieve its intended purpose. Performance can relate either to quantitative or qualitative findings. Performance is evaluated in the context of use of the AI system. The use conditions under which performance is evaluated can result in significant performance outcomes and which can be explicitly stated. Performance is not accuracy. A highly accurate model that produces discriminatory outcomes has not achieved its intended purpose if that purpose included fairness. Performance must be evaluated under real-world use conditions, not laboratory conditions.
Provider: Natural or legal person, public authority, agency or other body that develops an AI system or a general purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. A distributor, importer, deployer or other third party can be considered a provider of an AI system in certain circumstances. If you rebrand, white-label, or substantially modify an AI system, you can become the provider under the Act, inheriting all associated obligations.
Deployer: Natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. Deployers have distinct obligations under the AI Act, including human oversight and monitoring. This standard is written for providers, but providers must understand deployer obligations to design systems that support compliance downstream.
Post-market monitoring system: Activities carried out by providers of AI systems to collect and review experience gained from the use of AI systems they place on the market or put into service for the purpose of identifying any need to immediately apply any necessary corrective or preventive actions. For the purpose of this document, activities shall mean all activities. Post-market monitoring is not optional. It is a continuous regulatory obligation. If you cannot systematically collect and review real-world performance data after deployment, you cannot meet the standard.
Placing on the market: First making available of an AI system on the Union market. See making available on the market. Further information on this concept can be found in the Blue Guide, section 2. The first instance of commercial availability triggers the full set of provider obligations. Pre-release pilots and limited testing may not constitute placing on the market, but the boundary is not always clear.
Making available on the market: Supply of an AI system for distribution or use on the Union market in the course of a commercial activity, whether in return for payment or free of charge. Free distribution counts. Open-source release can count. If you make the system available for commercial use in the EU, you are subject to the Act regardless of whether you charge for it.
Putting into service: Supply of an AI system for first use directly to the deployer or for own use in the Union for its intended purpose. Further information on this concept can be found in the Blue Guide, section 2. Internal use triggers obligations. If you develop an AI system for your own operations and put it into service in the EU, you are both provider and deployer.
Serious incident: Incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: the death of a person or serious harm to a person’s health; a serious and irreversible disruption of the management or operation of critical infrastructure; the infringement of obligations under applicable regulatory requirements intended to protect fundamental rights; serious harm to property or the environment. Serious incidents must be reported to authorities. The definition is broad. An AI system that produces a discriminatory outcome that infringes fundamental rights protections can trigger a serious incident report even if no physical harm occurred.
Subject: Natural person who participates in testing in real-world conditions. Participating in testing can require informed consent of subjects. If your real-world testing involves human participants, informed consent requirements apply. This is a regulatory obligation, not just an ethical guideline.
Real-world conditions testing: Temporary testing of an AI system for its intended purpose in its intended context of use or deployment environment outside a laboratory or otherwise simulated environment. Assessing and verifying conformity of the AI system with the requirements of this document includes that the overall residual risk of the AI system is acceptable in accordance with its intended purpose and reasonably foreseeable misuse. Real-world conditions testing can pertain to technical and non-technical aspects, including performance verification or usability study. Real-world conditions testing can require the participation of subjects. Real-world testing is distinct from deployment. It is time-limited, purpose-specific, and subject to additional safeguards. If you call something a pilot to avoid compliance obligations, but it operates like a deployed system, regulators will treat it as deployment.

Sources: ISO 9000:2015, ISO/IEC Guide 63:2019, EN ISO 14971:2019, ISO 26000:2010

Accompanying documentation: Materials accompanying an AI system and containing information for the user or those accountable for the use, maintenance, decommissioning and disposal of the AI system. The accompanying documentation can consist of the instructions for use, technical description, installation manual, quick reference guide, etc. The accompanying documentation is not necessarily a written or printed document but can involve auditory, visual, or tactile materials and multiple media types. Materials include information relevant for the protection of health, safety and fundamental rights, where each is applicable. Accompanying documentation is legally binding. If the instructions for use specify a particular deployment context or oversight requirement, deployers must follow it, and providers are responsible for ensuring the guidance is accurate and complete.
Objective evidence: Data supporting the existence or verity of something. Objective evidence can be obtained through observation, measurement, test or by other means. Assertions without evidence do not satisfy this standard. If you claim a control is effective, you must produce objective evidence of its operation.
Procedure: Specified way to carry out an activity or a process. Procedures can be documented or not. Undocumented procedures are permitted, but they must be specified and repeatable. In practice, undocumented procedures are difficult to demonstrate during an audit.
Process: Set of interrelated or interacting activities that use inputs to deliver an intended result. Whether the intended result of a process is called output, product or service depends on the context of the reference. Inputs to a process are generally the outputs of other processes and outputs of a process are generally the inputs to other processes. Two or more interrelated and interacting processes in series can also be referred to as a process. Risk management is a process. Model development is a process. Post-market monitoring is a process. The standard requires these processes to be defined, systematic, and auditable.
Record: Document stating results achieved or providing evidence of activities performed. Records can be used, for example, to formalize traceability and to provide evidence of verification, preventive action and corrective action. Records are the primary form of objective evidence in a risk management system. If an activity is required and you cannot produce a record of it, you have not met the requirement.
Risk management: Systematic and continuous application of management policies, procedures and practices to the tasks of analysing, evaluating, controlling and monitoring risk throughout the entire life cycle of an AI system. Risk management is not a one-time assessment. It is a continuous process that spans development, deployment, operation, and decommissioning.
State of the art / Generally acknowledged state of the art: Developed stage of technical capability at a given time as regards products, processes and services, based on the relevant consolidated findings of science, technology and experience. The state of the art embodies what is currently and generally accepted as good practice in technology. The state of the art does not necessarily imply the latest scientific research still in an experimental stage or with insufficient technological maturity. You are required to implement risk controls that reflect the state of the art, not the state of your organization’s current capability. If better controls exist and are generally accepted, you must adopt them or justify why they are not applicable.
Top management: Person or group of people who directs and controls a provider at the highest level. Top management must establish and approve risk acceptability criteria. They cannot delegate this responsibility to the compliance or risk function.
Verification: Confirmation, through the provision of objective evidence, that specified requirements have been fulfilled. The objective evidence needed for a verification can be the result of an inspection, testing or of other forms of determination such as performing alternative calculations or reviewing documents. The activities carried out for verification are sometimes called a qualification process. The word “verified” is used to designate the corresponding status. Verification requires objective evidence. Self-attestation is not verification.
International norms of behaviour: Expectations of socially responsible organizational behaviour derived from customary international law, generally accepted principles of international law, or intergovernmental agreements that are universally or nearly universally recognized. Intergovernmental agreements include treaties and conventions. Although customary international law, generally accepted principles of international law and intergovernmental agreements are directed primarily at states, they express goals and principles to which all organizations can aspire. International norms of behaviour evolve over time. Fundamental rights protections are grounded in international norms of behaviour. These norms are not static, and your risk management process must account for evolving expectations.
Risk management file: Set of records and other documents that are produced by risk management. The risk management file is the primary artifact a regulator will examine during an inspection. It must be complete, coherent, and traceable.

Terms Relating to Testing

Source: ISO/IEC/IEEE 29119-1:2022

Testing: Set of activities conducted to facilitate discovery and evaluation of properties of test items. Testing activities include planning, preparation, execution, reporting, and management activities, insofar as they are directed towards testing. Testing is not just running the model on a validation set. It includes planning what will be tested, how it will be tested, documenting the results, and acting on the findings.
Test item / Test object: Work product to be tested. Example: Software component, system, requirements document, design specification, user guide. The AI model is a test item. The training data is a test item. The user documentation is a test item. All must be tested.
Test objective: Reason for performing testing. Every test must have a defined objective. Testing without a stated objective does not satisfy the standard.
Test completion report / Test summary report: Report that provides a summary of the testing that was performed. The report may contain statistical analysis. Test completion reports are records. They must be retained as part of the risk management file.
Test plan: Detailed description of test objectives to be achieved and the means and schedule for achieving them, organized to coordinate testing activities for some test item or set of test items. A test plan is a written document included in the risk management file. Testing without a documented test plan does not satisfy the standard.
Test monitoring and control process: Test management process that aims to ensure that testing is performed in line with a test plan and with organizational test specifications. Test execution must be monitored and controlled. Deviations from the test plan must be documented and justified.

Sources: Regulation (EU) 2024/1689, EU Charter of Fundamental Rights, ISO 26000:2010

Fundamental rights: Rights and freedoms guaranteed by the EU Charter of Fundamental Rights. Fundamental rights include human dignity, respect for private and family life, protection of personal data, non-discrimination, equality between women and men, rights of the child, rights of the elderly, integration of persons with disabilities, right to an effective remedy and to a fair trial, presumption of innocence and right of defence, principles of legality and proportionality of criminal offences and penalties. Fundamental rights are legally binding in the EU. Harms to fundamental rights are within scope of this standard, even if they do not produce physical injury or property damage.
Stakeholder: Individual or organization that can affect, be affected by, or perceive themselves to be affected by a decision or activity. Stakeholders can be internal or external. They include users, affected persons, deployers, providers, regulators, civil society organizations, and the public. Stakeholder identification is a required step in risk management.
Natural person: Human being. The standard distinguishes between natural persons and legal persons. Fundamental rights protections apply to natural persons.
Affected person: Natural person or groups of natural persons who can be subject to or impacted by an AI system. Affected persons include users and non-users. An AI system used in hiring affects both applicants and employees, whether or not they interact directly with the system.
User: Natural or legal person, public authority, agency or other body using an AI system. Users include deployers and end users. User obligations differ depending on the role, and risk management must account for both categories.
Group: Collection of natural persons defined by common characteristics such as demographic attributes, location, socioeconomic status, or shared vulnerability. Risks to groups must be assessed separately from risks to individuals. A system that performs well on average can produce serious harms to specific groups, and those harms are within scope.
Informed consent: Freely given specific, informed and unambiguous indication of the data subject’s wishes by which they, by a statement or by a clear affirmative action, signify agreement to the processing of personal data relating to them. For the purpose of this document, informed consent is understood more broadly to mean consent by a natural person related to participation in real-world conditions testing. Informed consent is not a click-through agreement. It must be specific, informed, unambiguous, and freely given. Generic consent forms do not satisfy this requirement.

Sources: ISO/IEC Guide 51:2014, ISO/IEC Guide 63:2019, EU Cybersecurity Act, EU Cyber Resilience Act, Directive (EU) 2022/2257

Hazard: Potential source of harm. Cyber threats and vulnerabilities can be the cause of a hazard. Cyber threats can be hazards. Hazards are not risks. A hazard is a source of harm. Risk is the combination of probability and severity. Identifying hazards is the first step in risk analysis.
Hazardous situation: Circumstance in which people, property or the environment is/are exposed to one or more hazards. Exposure to a hazard does not guarantee harm. A hazardous situation is the precondition for harm to occur.
Harm: Injury or damage to the health of a person or groups of persons, or interference with fundamental rights. For the purpose of this document, damage to property or the environment, and the disruption or destruction of critical infrastructure, are considered harms when they can result in injury or damage to the health of a natural person or groups of persons or interference with fundamental rights. Interference with fundamental rights can be tangible or intangible, physical, psychological, societal or economic, irrespective of the rightsholder’s awareness, in accordance with EU law, including the EU Charter. Safety in product safety risk management standards is understood as the absence of unacceptable risk. In the context of this document, safety refers to the protection from harm from the use of the AI system. Harm is not limited to physical injury. Psychological, societal, and economic harms are within scope. Interference with fundamental rights is harm even if the affected person is unaware of it.
Severity: Measure of the possible consequences of a hazard. The definition does not imply numerical measure of severity. Severity can be qualitative or quantitative. However, severity scales must be defined and applied consistently across the risk assessment.
Risk: Combination of the probability of an occurrence of harm and the severity of that harm. The probability of occurrence includes the exposure to a hazardous situation and the possibility to avoid or limit the harm. This is the formula that does not work for AI when applied as a simple multiplication without modeling the loss distribution. It treats all risks with the same expected value as equivalent, which they are not.
Residual risk: Risk remaining after risk control measures have been implemented. Residual risk must be evaluated against risk acceptability criteria. No system is risk-free. The question is whether the residual risk is acceptable.
Acceptable risk / Tolerable risk: Level of risk that is accepted in a given context based on the current values of society. For the purpose of this document, “acceptable risk” is the preferred term and “tolerable risk” is the admitted term. For the purpose of this document, “context” refers to the intended purpose and reasonably foreseeable misuse of the AI system, and “current values of society” refers to high protection of health, safety, and fundamental rights. Acceptable risk is not a fixed threshold. It depends on context, and it evolves as societal values evolve. What was acceptable five years ago may not be acceptable today.
Risk assessment: Overall process comprising a risk analysis and a risk evaluation. Risk assessment is the complete analytical process. It includes identifying hazards, estimating risk, and evaluating whether risk is acceptable.
Risk analysis: Systematic use of available information to identify hazards and to estimate the risk. Risk analysis is the first step in risk assessment. It is analytical, not evaluative.
Risk estimation: Process used to assign values to the probability of occurrence of harm and the severity of that harm. Risk estimation can be qualitative, semi-quantitative, or quantitative. The method must be documented and applied consistently.
Risk control: Process in which decisions are made and measures implemented by which risks are reduced to, or maintained within, specified levels. Risk control follows risk evaluation. It is the implementation of risk control measures to bring residual risk within acceptable levels.
Risk evaluation: Procedure based on the risk analysis to determine whether acceptable risk has been exceeded. Risk evaluation is the decision point. It compares estimated risk against acceptability criteria.
Inherently safe design: Measures taken to eliminate hazards or to reduce risks by changing the design or operating characteristics of the product or system. For the purpose of this document, a product or system is an AI system. For risks to fundamental rights, inherently safe design refers to translating fundamental rights, for example presumption of innocence and non-discrimination, into the technical AI system design requirements through, for example implementing equality, privacy and data protection by design. Inherently safe design is the highest level of risk control. It eliminates the hazard rather than controlling exposure to it.
Risk control measure: Action or means to eliminate hazards or to reduce risks. Example: Inherently safe design; protective devices; personal protective equipment; information for use and installation; organization of work; training; application of equipment; supervision. Risk control measures follow a hierarchy. Inherently safe design is preferred. Protective measures and information for use are secondary controls. The standard does not permit you to substitute information for design improvements when design improvements are feasible.
Cybersecurity: Activities necessary to protect network and information systems, the users of such systems, and other persons affected by cyber threats. For the purpose of this document, network and information systems shall mean an AI system under consideration. Cybersecurity is within the scope of risk management. Cyber threats are hazards, and cybersecurity controls are risk control measures.
Cyber threat: Potential circumstance, event or action that can damage, disrupt or otherwise adversely impact network and information systems, the users of such systems and other persons. For the purpose of this document, network and information systems shall mean an AI system under consideration. Cyber threats include adversarial attacks on AI models, data poisoning, model extraction, and manipulation of inputs to produce harmful outputs.
Vulnerability: Weakness, susceptibility or flaw of a product with digital elements that can be exploited by a cyber threat. For the purpose of this document, a product with digital elements shall mean an AI system under consideration. Vulnerabilities are not hazards themselves, but they are sources of hazards. A model trained on unvalidated data has a vulnerability. If that vulnerability is exploited, it becomes a hazard.
Critical infrastructure: Asset, facility, equipment, network or system or part of asset, facility, equipment network or system which is necessary for the provision of an essential service. Disruption of critical infrastructure can constitute serious harm. AI systems used in or affecting critical infrastructure are subject to heightened scrutiny.

Do not assume the standard’s definitions align with your organization’s existing terminology. They do not. The term “user” in this standard includes deployers and end users. The term “harm” includes interference with fundamental rights, not just physical injury. The term “risk” is defined as a combination of probability and severity, but it does not specify how to combine them, and the implied multiplication formula is insufficient for AI. Build a terminology mapping document that translates each of the standard’s 68 terms into your organization’s operational language, and distribute it to every team involved in AI development, deployment, and risk management. If your legal team, your technical team, and your risk team are using different definitions of the same word, your risk assessment will fail before you begin.

How the Standard Actually Runs Risk Management

Most organizations say they “have a risk process.” What they often have is a sequence of documents. The standard is more demanding. It expects a continuous, structured process that runs across the entire life cycle of the AI system and produces decisions that can be explained and defended.

A Continuous Process, Not a One-Time Assessment

The provider is expected to establish, implement, document, and maintain an ongoing risk management process. This process starts with risk analysis. It includes identifying characteristics related to risks tied to the intended purpose and reasonably foreseeable misuse of the AI system. It requires identifying known and reasonably foreseeable hazards, hazardous situations, and risks.

From there, the process moves to estimating and evaluating those risks, followed by risk evaluation, testing, risk control, and the evaluation of overall residual risk. It does not stop at deployment. It continues through risk management review and both pre-market and post-market activities.

This entire process applies across the full life cycle of the AI system. It is not limited to design or validation phases. It must be documented and maintained in a risk management file, which becomes the central record of how risk was understood, assessed, and managed over time.

Many organizations already have product or system development processes. The expectation is not to duplicate effort, but to integrate. Where a product realization process exists, it should incorporate the relevant parts of the risk management process. In practice, this means risk is embedded into how the system is built and operated, not added as a separate compliance layer.

The process is not linear. Different elements carry different weight depending on the life cycle stage. Activities can be iterative, repeated, and refined as new information becomes available. That flexibility is intentional. AI systems evolve, and the risk process must evolve with them.

Management Owns the Process, Not Just the Outcome

Risk management is not delegated away. Top management is expected to demonstrate active commitment. This starts with providing adequate resources and ensuring that personnel involved in risk management are competent. It includes assigning responsibility clearly and overseeing how the process is implemented.

There is also a review obligation. Management must regularly assess whether the risk management system remains suitable and effective. This includes reviewing the policy, the plan, and how the process operates in practice. These reviews are not ad hoc. They are planned, systematic, and documented, including decisions and actions taken.

Post-market information plays a direct role here. What is learned from real-world use feeds back into management’s assessment of whether the process still works. In many organizations, this feedback loop is weak. The standard makes it explicit.

Representation of the risk management process

Risk Acceptability Is a Policy Decision, not a Technical Detail

One of the most consequential requirements sits at the policy level. Top management must define, document, and maintain a risk management policy that establishes how risk acceptability is determined.

This policy must ensure that criteria for accepting individual residual risks and the overall residual risk meet regulatory requirements. It must take into account the intended purpose of the AI system, its reasonably foreseeable misuse, and what is generally accepted as the state of the art.

It should also reflect broader expectations. This includes relevant standards, international norms of behavior, and the concerns of stakeholders who may be affected by the system.

The policy needs to go further than general principles. It must specify the methods used to determine risk acceptability. One example is comparing the AI system to an equivalent non-AI system using a “no worse than” approach. It must also define how and when these criteria are reviewed and updated. At a minimum, this happens after serious incidents and at regular intervals throughout the life cycle, including before market entry.

In practice, this is where many organizations struggle. They define high-level principles but avoid committing to clear thresholds or methods. The standard expects the opposite.

Competence Is a Collective Requirement

Risk management activities must be performed by people who are competent based on education, training, skills, and experience relevant to their role. This is not limited to technical expertise.

Collectively, the team must understand the AI system or similar systems, the application domain and operating conditions, the technologies involved, and the relevant aspects of health, safety, and fundamental rights. They also need to understand the risk management techniques being used.

Not every individual needs all of these competencies, but the team as a whole must cover them.

Where fundamental rights are involved, the expectation is higher. Those performing these assessments must be able to understand and apply the relevant rights, and when needed, organize and facilitate consultation with affected stakeholders, including vulnerable groups. If the system can affect specific vulnerable populations, the team must have expertise in those areas.

In practice, this pushes organizations to move beyond purely technical or compliance-driven teams. Risk management becomes multidisciplinary by design.

Defining Risk Acceptability Criteria

Risk acceptability criteria define what level of risk is considered acceptable. These criteria must be established and updated throughout the life cycle to maintain a consistent and high level of protection for health, safety, and fundamental rights.

They must exist at two levels. One for each identified risk, and one for the overall residual risk of the system. They must be justified, documented, and supported by objective evidence so they can be verified and validated.

The criteria must reflect equal concern for all affected persons, with particular attention to those most vulnerable to harm. They must be aligned with the risk management policy, regulatory requirements, and the nature of the harms involved, including how different harms may interact.

Objective evidence plays a central role. Where people can be affected, this may include consultation with affected individuals or their representatives, especially vulnerable groups. If direct consultation is not performed, evidence can come from previous consultations for similar systems or from authoritative sources on fundamental rights. Testing can also contribute.

The provider must document why the evidence used is relevant and sufficient. If consultation is performed, the rationale for selecting participants and their representativeness must be clear.

This is not a box-ticking exercise. It is about showing that the thresholds for accepting risk are grounded in reality, not convenience.

How Criteria Are Established and Updated

Risk acceptability criteria must be determined in relation to the intended purpose and reasonably foreseeable misuse of the AI system. They must exist even when the probability of harm cannot be estimated precisely.

The process for establishing and updating these criteria is demanding. It includes considering independent review by a multidisciplinary team with expertise in safety, health, fundamental rights, AI, and the relevant application domain. Where an equivalent evaluation already exists, it can be reused, but it must be supported by objective evidence.

The provider must consider the state of the art, including literature, market data, and incident data. Alternatives must be assessed, including options that reduce risk significantly or avoid using AI altogether.

Severity and probability both matter, but they are not interchangeable. A low-severity harm can still be unacceptable if it occurs frequently. Where probability cannot be quantified, qualitative assessment is acceptable, provided the reasoning is documented.

The distribution of risk across different groups must be considered, especially where certain groups may be disproportionately affected. Adverse impacts on vulnerable groups require particular attention.

Pre-market and post-market information must feed into this process. This includes incident data, near misses, user feedback, and concerns raised by affected individuals or their representatives.

Independence is required. Those defining and reviewing risk acceptability criteria should be separate from those designing and developing the system, with any conflicts of interest documented. This can be achieved internally through separation of roles or externally through independent experts.

Where fundamental rights are at stake, additional considerations apply. For qualified rights, the provider must assess necessity, proportionality, and legitimate objectives. For privately enforceable rights, applicable legal obligations must be identified.

Looking at the Overall Residual Risk

Assessing individual risks is not enough. The standard requires a separate evaluation of the overall residual risk. The criteria for overall acceptability can differ from those applied to individual risks. This reflects a simple reality. Risks interact.

The provider must consider the aggregated severity and how risks are distributed, how they interact, and how multiple harms can arise from a single situation. The combined effect can be greater than the sum of individual risks. The analysis must consider impacts on all affected persons, with particular attention to vulnerable groups. It must also consider both immediate and long-term harms, including cumulative effects over time.

An important consequence follows. The overall residual risk can be unacceptable even if each individual risk has been reduced to an acceptable level. Where children are affected, their best interests must be a primary consideration. This is not optional. It reflects established international principles.

Final approval of overall risk acceptability sits with top management. This reinforces that risk acceptance is a business decision, not just a technical conclusion.

Planning the Work

Risk management activities must be planned. For each AI system, the provider must establish and document a risk management plan. This plan becomes part of the risk management file. The plan defines the scope of activities. It identifies the AI system and the life cycle phases where each part of the plan applies. It assigns responsibilities and authorities, taking into account the competence of personnel.

It includes requirements for reviewing both the process and the outcomes, the risk acceptability criteria, and the underlying policy. It defines how risk control measures will be verified and how pre-market and post-market information will be collected and reviewed. The plan itself is not static. Changes over the life cycle must be recorded. This creates a traceable history of how risk management evolved.

The Risk Management File as the Backbone

All of this comes together in the risk management file. This is not a single document, but a structured set of records and references that capture the entire process.

It includes the intended purpose of the AI system, the risk management policy, the plan, and all documentation related to risk analysis, evaluation, testing, control, and residual . It also includes records of reviews and post-market activities.

Traceability is essential. Each identified hazard must be linked through the process. From identification, to analysis, to controls, to testing, to residual risk. In practice, this is often implemented through structured risk tables with clear identifiers and links to supporting evidence.

The file can reference other documents, including those from the quality management system, to avoid duplication. What matters is that all required information can be assembled quickly and coherently.

This is where the process becomes real. If the file is complete, consistent, and traceable, the organization can explain its decisions. If it is not, the process exists only on paper.

Risk Management Process and AI System Life Cycle

The provider must determine and document in the risk management file the stages of the life cycle for each AI system, from inception to end of life, in line with each AI system’s intended purpose. The life cycle phases may be determined in alignment with the life cycle phases as determined in the quality management system. prEN 18286, the companion standard addressing quality management systems for EU AI Act purposes, provides additional information on life cycle alignment.

The risk management process must be applied systematically and iteratively along the entire life cycle of the AI system. This is not a sequential process completed once and archived. The risk management process activities must be integrated into the life cycle stages to ensure that risks are systematically and iteratively analyzed, evaluated, and reassessed, risk control measures are identified, implemented, verified, and updated when needed, residual risks and overall residual risk are evaluated and monitored and their acceptability maintained, and relevant information on the AI system is collected and reviewed.

Risk analysis and risk evaluation must start at the inception stage of the AI system and must be applied through all the following life cycle stages until end of life, as new risks can emerge and previously identified ones can change during any of the life cycle stages. The results of risk evaluation can have a bearing already on the inception phase. In some cases, it can take much less effort to mitigate a risk at the inception stage than during later life cycle stages. This is a critical point that most organizations miss. Early risk assessment is not a formality. It is the point at which design decisions can eliminate hazards rather than control them.

Risk control measures can be implemented at different life cycle stages, depending on the specific measures. Risk control measures must be, as much as technically feasible, implemented during design and development with the view to achieve inherently safe design. Inherently safe design is the highest priority risk control measure. It eliminates the hazard rather than managing exposure to it.

Information that is relevant to the risk management process can become available at any life cycle stage. This information can support the provider to identify the need to re-execute the risk management process, in whole or in part, or to return to a previous step of the risk management process. This information can refer to the identification of new risks, changes to estimations of risks, the detection of risk control measures that do not perform as expected, or the implementation of new risk control measures.

Some risk control measures are only possible to implement by returning to a previous life cycle stage. A change of intended purpose restarts the inception stage. New inherently safe design measures restart the design and development stage. This creates a challenge for organizations that treat life cycle stages as linear and complete. The standard requires iterative re-entry into earlier stages when risk information demands it.

Most organizations will map their existing product development life cycle to the standard’s requirements and declare the mapping complete. That is not sufficient. The standard requires risk management activities to be integrated into each life cycle stage, not merely aligned with it. Build your life cycle model as a series of decision gates where risk analysis, risk evaluation, and risk control verification are mandatory prerequisites for progression. If your development roadmap allows a system to move from design to deployment without a documented risk evaluation and approval of residual risk by top management, your life cycle integration does not meet the standard. The life cycle is not a timeline. It is a control framework.

General Requirements for AI Risk Analysis

The implementation of the planned risk analysis activities and the results of the risk analysis must be recorded in the risk management file. The risk analysis must consist of risk identification and risk estimation. These are distinct activities with different outputs, and both must be documented.

The provider must analyze risks from logging and monitoring, human factors, user behavior, unwanted bias, data quality and data governance including provenance of data, and AI system accuracy, where applicable. This is not an exhaustive list. The standard explicitly states these areas are examples, not limits. In order to address these areas, the provider can refer to prEN 18229-1 on transparency, prEN 18229-2 on robustness, prEN 18282 on cybersecurity, prEN 18283 on data quality, prEN 18284 on bias, prEN 18281 on logging, and other relevant standards.

In addition to the records required for risk identification and risk estimation, the documentation of the conduct and results of the risk analysis must include at least the unique identifier and the version designation of the AI system that was analyzed, identification of the persons and organization who carried out the risk analysis, scope and date of the risk analysis, and techniques and methodologies used for hazard identification and risk estimation. Without this metadata, the risk analysis cannot be traced, verified, or defended under regulatory scrutiny.

Damage to property or the environment, and the disruption or destruction of critical infrastructure, are harms when they can result in injury or damage to the health of persons or interference with fundamental rights. The provider may also consider additional harms. The process of risk analysis is intended to address these harms. Cyber threats and vulnerabilities can be the cause of a hazard, and . prEN 18282 can be used to identify and address cyber threats and vulnerabilities.

The range of fundamental rights which must be considered for the purposes of risk analysis must include all the rights recognized under the EU Charter. This is not limited to the rights most commonly discussed in AI ethics frameworks. It includes all Charter rights, and the provider must assess which rights are relevant to the specific AI system being analyzed.

Techniques such as Preliminary Hazard Analysis, Hazard and Operability Study, Fault Tree Analysis, and System-Theoretic Process Analysis can be effectively utilized to derive risk analysis. These methods are complementary, and employing a combination of them can be essential for achieving a comprehensive and robust risk analysis. For more guidance on risk analysis techniques, refer to EN IEC 31010. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Identification from the Intended Purpose

The provider must document in the risk management file the intended purpose of the AI system being considered. The documentation of the intended purpose must include at least the following information: application areas, the objectives of the AI system including the intended output and impact on persons’ safety, health and fundamental rights, type of tasks used to achieve the objectives, techniques and approaches for how the AI system operates, types of intended deployers and types of intended users, deployment type such as physical product, software, or service, and the intended environment in which the AI system operates.

Intended users can refer to professionals, consumers, and users defined by age group or other characteristics. The environment in which the AI system operates can include the organizational, physical, and digital environment. The digital environment refers to the types of integration and interfaces with other software systems and physical systems. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Most organizations document intended purpose as a one-paragraph marketing description. That is insufficient. The standard requires you to specify the intended impact on persons’ safety, health, and fundamental rights, the deployment type, the user types, and the operational environment at a level of specificity that supports hazard identification. If your intended purpose documentation does not answer the question “in what specific context, for what specific users, performing what specific tasks, does this system operate, and what specific impacts on safety, health, and fundamental rights are intended,” it does not meet the standard. Build your intended purpose statement as a structured specification, not as a mission statement.

Risk Identification: Reasonably Foreseeable Misuse

The reasonably foreseeable misuse of the AI system must be identified, taking into account the potential misuse of the AI system by other user categories, which can include lay users, persons under the age of 18, and other vulnerable groups, on other persons affected, vulnerable groups, assets, or the environment, where other persons affected can include persons who are not defined in the intended purpose of the AI system, in a different context or environment including the digital, physical, and organizational environment and infrastructure in which the AI system operates, with incorrect input or data, and in a different application area.

The identification must also consider the potential misuse of the AI system, including its outputs, aims, or objectives. A recommendation used as a decision is an example of this type of misuse. The identification must consider the potential incorrect deployment of the AI system. A user who can and does change the AI system’s settings such that it operates outside of its intended purpose is an example of incorrect deployment. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Identifying the characteristics of an AI system is a preliminary step intended to support the identification of hazards and hazardous situations. It is meant to create a general overview of relevant characteristics, keeping in mind that hazards and hazardous situations need to be identified in detail later in the risk management process.

The provider must identify and document all applicable characteristics that can affect risks of the AI system. Where applicable, the limits of these characteristics must be defined, such as limits of performance. The operation of the AI system and the risk associated with its use can be affected when those limits are exceeded. These characteristics are related to the functionality of the AI systems, its lifecycle, its intended purpose and reasonably foreseeable misuse, and the environment in which it operates and interacts with.

For identifying these characteristics, the following must be taken into account: outputs of and actions taken by the AI system and their effect on users and persons affected, including vulnerable groups and age-appropriate outcomes when the intended users are persons under the age of 18. The effect can be directly from the AI system output or are intended to follow from the AI system output. An AI system that grants public assistance benefits has a direct effect on the applicant. A medical system that identifies cancer in a scan provides this information to a doctor which decides on treatment for the patient. The patient is affected by an action that follows the AI system output.

The provider must also consider user profiles including their abilities and limitations, known biases in decision making, their level of expertise, and how this can impact the appropriate use and interpretation of the AI system’s outputs, user accessibility to the AI system, interactions of the AI system with the environment including other systems and digital infrastructure and the effects of the AI system on the environment and vice versa, AI system functionality, architecture and technologies and related capabilities, limitations, uncertainties or known failure modes, minimum performance requirements of the AI systems and system components to achieve the intended purpose, level of autonomy and adaptiveness of the system’s behavior during operation such as continuous learning, pre-determined changes to the algorithm and its performance that can appear during the operation phase and the possibility that the system behavior changes during the operation phase in a way that hasn’t been pre-determined, dependency on third party components including open source, dependency on third party support in specific lifecycle phases such as outsourcing of design, development and verification and validation tasks, processing and storage of data by the AI system and the nature and sensitivity of the data, deployment, maintenance and decommissioning or disposal procedures, procedures for updates of the AI system during operation, and skills and experience of deployers.

The provider must identify the hat can have an impact on the characteristics listed. prEN 18282 provides additional information on this requirement. For each of the points, the provider must assess their relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement one of the listed points. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Identification: Hazards, Risk Scenarios, and Hazardous Situations

Based on the AI system’s intended purpose, reasonably foreseeable misuse, characteristics related to risks, and the state of the art, the provider must identify and document in the risk management file known and reasonably foreseeable hazards, risk scenarios, and hazardous situations. These are distinct concepts, and the standard requires all three to be identified and documented.

Hazards can only lead to harm if a hazardous situation exists. A risk scenario clarifies under which context and conditions a hazard can result in harm by describing what leads to the hazardous situation. A risk scenario can consist of a single event, a sequence of events, a combination of events, a circumstance including all external factors to the system, including normal use and user interactions, or a state such as internal factors to the AI system. Events that form part of a risk scenario can also be referred to as hazardous events.

A hazard can lead to multiple hazardous situations, and each hazardous situation can lead to multiple harms. Hazards and hazardous situations can be technical and non-technical. AI system decisions or outputs can be hazards. Hazards can have more than one cause.

The provider must describe the risk scenarios. Risk scenario descriptions must include known and foreseeable related hazard and hazardous situation, elements affecting the probability of harm occurring, elements affecting the severity of harm, events, sequences and interactions of events, if any, that lead to the hazardous situation, any contributing factors including any relevant failure modes and their underlying potential causes, and AI system characteristics that can lead to hazardous situations.

Risk scenarios must consider at least interactions and behaviors of the system, users and persons potentially affected, contributing human factors, system errors which can include errors resulting from design, development, deployment and incorrect system operation, cyber threats and vulnerabilities, and environmental factors influencing the system, users or person affected. Sequences of events can also comprise a chronological chain of causes and effects, non-occurrence of expected events as well as combinations of concurrent events. A risk scenario can be initiated in all phases of the AI system’s life cycle.

The provider must consider all available relevant information to support the hazard and hazardous situation identification process. Relevant information includes pre-market sources, including testing, expert reviews, stakeholder consultation feedback, available data on comparable systems already on the market, incident databases, published research, regulatory reports, market surveillance findings, and other credible sources that provide insight into potential hazards or known issues. It also includes post-market sources, including incident reports, complaints, potential serious incident data, user feedback and information collected by automatic logging of the AI system. The logs can contain many events of which some can be labeled as hazardous events.

The identification of hazards must include hazards associated with each identified vulnerability and cyber threat to the AI system. prEN 18282 can be used to identify vulnerabilities and cyber threats to the AI system. Vulnerabilities and cyber threats concern the AI system itself, whereas hazards concern risks to health, safety, and fundamental rights.

The AI system provider must, as appropriate, use multiple different risk identification techniques throughout the AI system life cycle to ensure a comprehensive hazards identification process. When identifying hazards and hazardous situations not previously recognized, systematic techniques for risk identification that cover the specific situation can be used. Guidance on some available techniques is provided in relevant standards and guidelines, such as EN IEC 31010.

Top-down techniques, which are typically used in the early planning phases, are valuable for identifying high-level hazards and risk scenarios. As development progresses, diverse methods must be applied to systematically analyze specific hazards, failures and cyber threats, supporting root-cause exploration and targeted risk control measures. These risk identification techniques are complementary and it is often necessary to use multiple approaches in combination to achieve a robust and complete risk analysis.

When identifying hazards and hazardous situations, the provider must consider the specific risks and harms that the AI system can pose to persons under the age of 18 and other vulnerable groups, taking into account their evolving capacities, vulnerabilities and, where applicable, the best interests of persons under the age of 18. This should include risks related to exposure to harmful or inappropriate content, unwanted contact or interactions with adults for persons under the age of 18, privacy violations and data misuse, excessive screen time and addiction, dignity, negative impacts on physical and mental health, and exploitation and abuse.

Risk scenario identification and analysis can inform about the relevant events to be logged. The risk scenario identification and analysis can become more robust as it is iterated through at the different relevant life cycle stages. Frequently, only a first subset of the intended purpose-related hazards and hazardous situations can be identified at the inception and early design and development stages. At the early design and development stage, the obtained results, observations and feedbacks can lead to new hazard, to new risk scenario and to new hazardous situation identifications. Post-market monitoring can lead to the identification of new hazards and hazardous situations and related conditions identifications.

For each of the points in this section that must be considered, the provider must assess the relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement the point. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Hazard identification is where most risk assessments fail. Organizations identify high-level hazard categories such as “bias” or “data quality issues” and treat the identification as complete. That is not compliant with the standard. The standard requires you to identify specific hazards, describe the risk scenarios that lead from hazard to hazardous situation, and document the contributing factors and failure modes. If your hazard identification does not answer the question “what specific event, sequence, or state leads from this specific hazard to this specific hazardous situation, affecting which specific persons in which specific way,” you have not identified the hazard. You have named a category. Build your hazard identification as a structured cause-and-effect analysis, not as a list of concerns.

Risk Estimation

For each identified hazard and hazardous situation, the provider must estimate the probability of occurrence of the associated harm and the severity of that harm, based on the AI system’s intended purpose, reasonably foreseeable misuse, characteristics related to risks, and the state of the art. The provider must estimate the associated risks using available information or data. For hazards for which the probability of the occurrence of harm cannot be estimated quantitatively, the possible harms must be listed and a qualitative estimation made of their probability, for use in risk evaluation and risk control.

When identifying the possible harms resulting from hazards, and in estimating the severity of the associated harm, the groups of persons potentially affected must be determined. Specific vulnerabilities of persons potentially affected can increase the probability and severity of harm. These vulnerabilities can include characteristics and external factors that classify a person as being a member of a vulnerable group, and characteristics and external factors beyond those that classify a person as being a member of a vulnerable group.

For each hazard and hazardous situation, the severity of the associated harms must be estimated taking into account the nature of the harm, which refers to whether it concerns health, safety or fundamental right, the reversibility or irreversibility of the harm, which in relation to persons affected refers to the ability to revert fully or partially to pre-impact situation or equivalent and whether adequate remedies are made available in a timely manner, the number of persons affected in absolute terms rather than only expressed as a proportion of an affected population, the specific characteristics of persons potentially affected, the possible combination of harms, and the potential cumulative effects on persons affected.

For each hazard and hazardous situation related to fundamental rights, the estimation of the severity of the associated harms must also entail the consideration of the character of the right as being an absolute right or qualified right, the applicability of legal obligations imposed on private actors to protect that fundamental right, the level of protection accorded to the right, and the scope, significance and scale of the harm of a potential fundamental rights interference which entails consideration of how widespread its effects and adverse impacts of the hazard or hazardous situation, including whether the rights at risk are those of rights-holder groups that enjoy additional or particular protections. Rights holder vulnerable groups that enjoy additional or particular protection can refer to persons under the age of 18.

If a fundamental rights risk is likely to affect only 0.1% of users on a digital communication platform, it can initially appear negligible, but if this comprises 25% of a religious minority, then the latter indicates that the scope can be serious. This example illustrates why absolute numbers and distributional analysis matter.

An AI system’s intended purpose or reasonably foreseeable misuse can affect a number of different fundamental rights pertaining to multiple persons affected. A hazardous situation can generate risks to multiple fundamental rights that can affect more than one person potentially affected. Risks to the fundamental rights of persons affected can interact with each other, for example when risks are compounded or cumulative.

The strength of legal protection accorded to the activity protected by a fundamental right is based on whether it is an absolute right, a privately enforceable right or a qualified right or a principle in support of a right. The stronger the level of protection accorded to the right, the greater the severity of a potential interference to it.

For the estimation of the severity of fundamental rights risks and the potential effects of interaction from the AI system with persons potentially affected, the provider should consult with persons potentially affected or their proxies, at least before placing on the market or putting into service the AI system. The results of these activities must be documented.

The system used for qualitative or quantitative categorization of probability of occurrence of harm and severity of harm must be documented and recorded. If a risk chart or risk matrix is used for ranking risks for the purpose of estimating their severity and probability, or qualitative estimate of likelihood if probability cannot be estimated, then the parameters and the interpretation of the particular risk chart or risk matrix used must be explained and justified for that application and in accordance with the intended purpose and the reasonably foreseeable misuse of the AI system.

Risk estimation incorporates an analysis of the probability of occurrence of harm and the severity of the harm. Depending on the application area, only certain elements of the risk analysis process can be relevant to consider in detail. When the harm is minimal, an initial hazard and consequence analysis can be sufficient, or when insufficient information or data are available, a conservative estimate of the probability of occurrence can give some indication of the risk. Risk estimation always has a qualitative component and can additionally have a quantitative component. Methods of risk estimation are described in relevant guidelines and standards for application area specific safety risk management, such as EN IEC 31010.

Information or data for estimating risks can be obtained from information received from post-market monitoring system, civil society groups and academic literature, published standards and guidelines for AI risk management, scientific or technical investigations, field data from similar systems already in use including publicly available reports of incidents, usability tests employing typical users, performance metrics and evaluation results, results of relevant investigations or simulations, expert opinion, and external quality assessment schemes for AI systems.

The scale of a health, safety or fundamental rights risk for the purposes of evaluating its severity concerns its gravity, and entails consideration of the potential cumulative effects on persons affected in light of the intended purpose and its reasonably foreseeable misuse, which is also a product of the ease and speed with which the AI system can diffuse across multiple domains and beyond its intended application area. It also entails the assessment of whether persons affected belong to a vulnerable group, including their ability to take protective measures to safeguard their rights and interests. The protected characteristics of vulnerable groups can also be a separate parameter to demonstrate clearly these characteristics have been considered in the severity.

For the purposes of estimating its severity, a fundamental rights risk is considered irremediable if the persons affected cannot be restored to a situation at least equivalent to their situation if there had been no interference to the respective fundamental right through the use of an AI system.

The provider must ascribe the highest severity classification for risks to fundamental rights that are considered absolute rights in accordance with applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk estimation is where the standard’s formula breaks down most visibly. The requirement to estimate probability and severity does not specify how to combine them, and most organizations default to a simple multiplication that treats a 10% chance of moderate harm the same as a 1% chance of severe harm because both produce the same expected value. That approach does not satisfy the requirement to evaluate risk acceptability. Build your risk estimation process to produce not only point estimates but also distributional analysis, confidence intervals, and scenario-weighted outcomes. Document the method you use to combine probability and severity, justify why that method is appropriate for the specific AI system and application area, and present the results in a way that distinguishes between frequent low-severity risks and rare high-severity risks. If your risk estimation outputs a single number per hazard, it does not provide the information top management needs to approve residual risk acceptability.

Risk Evaluation Turns Analysis into Decisions

For each identified hazard and hazardous situation related to the AI system, the provider must evaluate the estimated risks to health, safety, and fundamental rights by comparing them against the predefined risk acceptability criteria in the risk management plan. Based on this evaluation, the provider must determine whether the risk is acceptable or not.

If the risk is acceptable, it is not required to apply risk control activities to this hazardous situation, and the estimated risk must be treated as residual risk. If the risk is not acceptable, then the provider must perform risk control activities to reduce the risk to an acceptable level.

The results of the risk evaluation activities must be recorded in the risk management file with a breakdown of the reasoning for each identified risk to health, safety, and fundamental rights. A risk evaluation that records only a conclusion without the reasoning behind it does not meet the standard. The reasoning must be documented for each risk individually.

Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk evaluation is where the gap between internal comfort and external defensibility becomes most visible. Organizations usually compare estimated risks against acceptability criteria that were defined too loosely to produce a meaningful comparison. If your acceptability criteria say “risks are acceptable when they are low,” and your risk estimation says “this risk is low,” you have performed a circular evaluation that tells a regulator nothing about how you actually made the decision. Build your risk evaluation as a documented comparison between a specific estimated risk, expressed in terms of probability and severity with supporting evidence, and a specific acceptability threshold, defined in terms that can be independently verified. Document the reasoning that connects the evidence to the conclusion. If the reasoning cannot be reproduced by someone who was not in the room when the evaluation was performed, it is not sufficient.

Testing Is Evidence, Not Validation Theater

Testing is an essential activity to support the risk management process. Testing can support the identification of hazards and associated causes, sequences of events and hazardous situations related to intended purpose and reasonably foreseeable misuse, the provision of objective evidence of residual risk and overall residual risk acceptability, and the post-market monitoring system activities, including the monitoring of continuously learning AI systems to ensure they remain within their predefined changes.

Usability testing can be used as a method for validating the effectiveness of human-machine interfaces and instructions for use. Testing performed to identify characteristics of training data sets that reveals inconsistent labelling of the data and bias in representativeness of the data in relation to the intended purpose informs about the appropriate risk control measures to be implemented.

Testing must be applied at least prior to placing on the market or putting into service to identify appropriate risk control measures and to provide objective evidence for the effectiveness of the applied risk control measures. For risk reduction of a specific risk, two different risk control measures can be applied, and the effectiveness of both methods can be tested to select the most effective measure. Verifying that hazards are eliminated through inherently safe design measures is an example of testing applied to provide objective evidence of risk control effectiveness. Testing of the AI system performance to assess creditworthiness of persons that reveals women are consistently discriminated against compared to men is an example of testing that triggers the need for further risk control.

For testing which is aimed to provide supporting objective evidence of the overall residual risk acceptability of the AI system, test acceptance criteria must specify metrics and probabilistic thresholds against which test results are evaluated. Testing that can affect persons must be performed according to applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test Plan

The test plan provides a detailed description of how the testing for the associated test management processes should be done, including how it must be monitored and controlled. The test plan is used by the test monitoring and control process as the basis for managing the testing activity.

The test plan must describe and provide a justification of the following elements, taking into account the state of the art: test objectives, where a test can have multiple objectives for related but distinct purposes, the test item and its relationship to a specific risk and the test objectives, appropriate test methods for achieving the stated test objectives, test completion criteria, resource use, allocation and independence or neutrality principles to conduct testing, collection of testing results and evaluation of testing results in accordance with acceptance criteria, and test plan updates.

In identifying and specifying the test objectives, the provider must identify, justify, and document the assumed relationship between the test objectives and the test methods including the intended test environment, and how the resulting evidence that the test is expected to generate contributes to risk control.

The definition of the test plan can be supported by the companion standards on data quality, robustness, cybersecurity, transparency, bias, and logging. Test acceptance criteria are specific to the test item and test objectives and can depend on specific considerations from other standards. The provider can refer to international standards and state of the art considerations to define the risk acceptance criteria suitable for a particular test item and test objectives.

Testing acceptance criteria must be specified in advance and documented in relation to the test objectives and test methods including the intended test environment. Specifying acceptance criteria after testing is complete defeats the purpose of the testing requirement and will not satisfy a regulator or auditor reviewing the risk management file.

For AI systems potentially impacting persons, the provider must involve relevant stakeholders, stakeholders’ proxies, or an independent cross-functional panel of experts in the design and approval of the test plan. The level of detail of stakeholder involvement can vary depending on the context. Relevant stakeholders can include established user feedback groups involved in public administration applications, patient groups, or trained proxies.

The provider must evaluate the likely impact of the test plan on vulnerable groups and make any necessary adjustments to the test plan to ensure that they are duly protected from adverse impacts. Where applicable, informed consent must be obtained from persons affected and managed in accordance with applicable regulatory requirements. A justification for the representativeness, the number of participating persons, and the duration of the testing process must be documented.

The test plan and all subsequent amendments must be prepared and documented by the provider and, when applicable, in consultation with relevant stakeholders, stakeholder representatives, independent cross-functional panels of experts, or competent authorities. The test plan, including all subsequent amendments, must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test plans are where most organizations reveal that they are testing what is convenient rather than what is required. They test aggregate model performance across the full dataset and conclude that the system performs within acceptable parameters. They do not test performance across demographic subgroups, deployment environments, edge cases, or conditions of reasonably foreseeable misuse. They do not specify acceptance criteria in advance. They do not involve stakeholders in test plan design. And they do not document the relationship between test objectives and risk control. Build your test plan as a structured document that links each test objective explicitly to a specific identified risk, specifies the acceptance criteria before testing begins, documents the rationale for the test method selected, and includes subgroup analysis and misuse scenario testing as standard components. If your test plan does not specify what result would cause you to conclude that a risk is not acceptable, it is not a test plan. It is a performance measurement exercise.

Real-World Conditions Testing

Real-world conditions testing can be used for the purpose of gathering data and as part of fulfilling the requirements of the standard. If real-world conditions testing is performed, the provider must ensure that the risks associated with the testing do not exceed risk acceptability criteria. Real-world conditions testing can be considered when there is insufficient objective evidence that the overall residual risk of the AI system is acceptable for its intended purpose and under conditions of reasonably foreseeable misuse.

If real-world conditions testing is performed, the provider must comply with applicable regulatory requirements. Testing involving persons affected can require consent management in accordance with applicable national laws and international norms of behaviour. Article 61 of the EU AI Act provides more information about informed consent.

Risk management activities with respect to the testing must be performed throughout the real-world conditions testing process. The provider must predefine or establish risk acceptability thresholds and trigger a risk assessment to determine whether actions are needed as soon as thresholds are reached or exceeded. Test monitoring and control process must be documented in the risk management file.

Data generated through real-world conditions testing must be recorded in the real-world conditions testing test completion report, ensuring all personal data are handled in accordance with applicable regulatory requirements. The real-world conditions testing test completion report must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test Monitoring and Test Reporting

Adherence of the testing procedure to the test plan must be monitored and controlled until test completion. The test plan may be updated by the test monitoring and control process, for instance due to changing requirements such as a test completion date moved forward. Any deviation from the test plan must be recorded.

The following information must be compiled in a test completion report: the testing that was performed, the reference to the test plan elements, any deviation from the test plan, the test results, the assessment of test results against test acceptance criteria, the test monitoring report, and where applicable the evaluation of the effectiveness of the relevant risk control measures.

The test completion report must identify, specify, and document the relationship between all elements listed above to ensure their traceability. The test completion report must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Control Follows a Clear Hierarchy

The standard requires the provider to apply risk control options in a strict priority order. Inherently safe design comes first, protective measures come second, and information and instructions for use together with training to deployers and users come third. This hierarchy is not optional. The provider must apply higher-order controls before resorting to lower-order controls, and must document and justify any departure from this order.

Inherently safe design measures are the most effective risk control measures and by default the preferred risk control measures. They are inherent to the characteristics of the product and are most likely to remain effective. By definition, they are the only form of risk control that can eliminate a hazard, thus eliminating the need for second or third-level risk control measures for that hazard.

Second-level risk control measures in the form of guards and protective measures, even if well-designed, can fail or be violated. Third-level risk control measures rely on the provision of information to address risks and are the least effective of the three forms of risk control because they rely on others following the provided information and hence cannot be ensured.

Applying the hierarchy of risk control requires the provider to work through a defined sequence. If technically feasible, the provider must apply inherently safe design measures to eliminate hazards. A mathematically verifiable algorithm that fails safe in a deterministic manner in real-world situations is an example of inherently safe design. An AI system intended to calculate social security benefits that is technically configured to prevent the generation of any output if there is missing mandatory input data, or if the input data entered falls outside the specified range, and to automatically alert both the deployer and person affected to missing or abnormal input data, is another example. This risk control measure helps prevent erroneous benefit decisions by design, thereby helping ensure respect for the right to good administration.

If elimination of hazards is not technically feasible, inherently safe design measures must be applied to reduce the risk as far as technically feasible, complemented by the application of protective measures to further reduce risk as far as technically feasible. If inherently safe design measures cannot be used or do not achieve risk reduction to an acceptable level, a justification must be documented and protective measures must be used to achieve an acceptable level of risk. If inherently safe design measures and protective measures cannot be used or do not achieve risk reduction to an acceptable level, a justification must be documented and instructions for use may be used as a risk control measure to reduce risk to an acceptable level. Instructions for use and training must be applied only to complement inherently safe design measures and protective measures and must not be relied upon solely and exclusively to reduce risk to an acceptable level.

The provider must add required information to the instructions for use in accordance with applicable companion standards. In order to further reduce the risks from the use of the AI system, the provider must assess whether to include training instructions to deployers, deployment, maintenance and decommissioning instructions, and operational, troubleshooting and emergency instructions that include actions to be taken by the deployer regarding hazards and hazardous situations, including measures to prevent exposure to them, measures to reduce the probability and severity of resulting harm, and remedial actions to take if harm does occur.

The provider must document their reasoning and justification for not including any of this information in the instructions for use. The provider can provide additional information not listed as part of the instructions for use.

Where applicable, the provider must define appropriate training for deployers of the AI system considering their competence, including technical knowledge, skill, experience and education, and including competence regarding persons under the age of 18 and other vulnerable groups, in line with the intended purpose and reasonably foreseeable misuse of the AI system.

Risk control measures must be explicitly designed, implemented, and documented in a manner that mitigates identified risks directly, independent of internal organizational policies or procedures. This is one of the most consequential requirements in the standard. Internal organizational policies and procedures must not be considered risk control measures because they do not concretely address potential harm in a specific and directly verifiable way. If your risk control measure is “we have a policy requiring human review of all high-risk decisions,” that is not a risk control measure. It is a procedural requirement. The risk control measure is the technical mechanism that ensures the human review actually occurs and is documented.

All identified residual risks, as well as any associated cautions and warnings, must be clearly documented and communicated in the accompanying documentation.

To identify and implement the appropriate type of risk control measure, the provider can refer to the companion standards on transparency, robustness, cybersecurity, data quality, bias, and logging. To identify the most appropriate risk control measures, the provider must take into account the state of the art, the reasonably foreseeable technical knowledge, experience and education of the deployer, and the reasonably foreseeable context in which the AI system will be used. The provider must review whether, due to advances in the state of the art, more effective risk control measures are available.

Technical feasibility has multiple considerations, including the state of the art, the maturity of a solution, the use of a precautionary approach, the specific industry vertical, and the AI technology on which the AI system is based. Technical infeasibility means that no design or development techniques or production methods can reasonably be considered feasible. If other members of an industry vertical are achieving a certain measure, hazard elimination, level of risk reduction, or level of protection, then it is likely considered technically feasible. Risk control measures can reduce the severity of the harm or reduce the probability of occurrence of the harm, or both. Risk control measures for one hazard or risk can increase another risk. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

The prohibition on treating internal policies and procedures as risk control measures will catch most organizations off guard. Many existing AI governance frameworks are built on policies, approval workflows, ethics review processes, and training requirements. Under this standard, none of those count as risk control measures. They may support the risk management process, but they do not substitute for technical controls that mitigate identified risks directly and in a verifiable way. Audit your existing risk control inventory against this requirement before you file your risk management documentation. For every control you have listed, ask whether it can be verified independently of whether anyone followed the policy. If the answer is no, you need a different control.

Implementation and Verification of Risk Control Measures

The provider must implement the risk control measure selected at appropriate stages in the life cycle of the AI system. Implementation of each risk control measure must be verified. Risk control measures must be verified by gathering objective evidence, including verification by inspection and analysis. This verification must be recorded in the risk management file.

The effectiveness of the risk control measures along the life cycle of the AI system must be verified. This verification must include testing in accordance with the testing requirements of the standard. The results of this verification must be recorded in the risk management file.

When the intended user profile includes vulnerable groups, the verification of risk control measures must include evaluation methods specific to their needs and vulnerabilities. This can include usability testing such as age-appropriate usability testing for persons under the age of 18, expert review, and consultation with specialists with expertise supporting vulnerable groups such as child development specialists.

Verification of the effectiveness of risk control measures can include consultation with persons potentially affected or their proxies, including civil society organizations. Real-world conditions testing can be performed in order to validate the effectiveness of risk control measures. If performed, this verification must be recorded in the risk management file.

The provider must review the effects of the risk control measures with regard to whether any new hazards or hazardous situations are introduced, or whether the estimated risks for previously identified hazardous situations are impacted by the introduction of the risk control measures. Risks from new hazards or hazardous situations, and estimated risks impacted by the introduction of risk control measures, must be estimated and evaluated and controlled as necessary. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Residual Risk and When to Stop

After the risk control measures are implemented and verified, the provider must evaluate the residual risk using the criteria for risk acceptability defined in the risk management plan. The acceptable risk must be justified, taking into account the potential adverse impact on persons. Differences in the AI system performance can lead to discrimination of specific groups of persons affected, including vulnerable groups, and prEN 18283 provides more information on this.

The results of this evaluation must be recorded in the risk management file. If a residual risk is not judged acceptable using these criteria, further risk control measures must be considered and the process must return to the risk control activities until the risk acceptability criteria is met.

In the case a residual risk remains unacceptable and the provider finds that no risk control measures are technically feasible, the provider may conclude that a change of intended purpose of the AI system is necessary, returning to the intended purpose documentation and restarting the risk identification process for the revised purpose. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Completeness of Risk Control

Adequate risk reduction is achieved when all state of the art design and development and risk control measures have been duly considered and adopted or the reasons for refraining from adoption are documented and included in the risk management file, each hazard has been either eliminated or its estimated risk has been reduced to an acceptable level, any new hazards introduced by the risk control measure have been properly addressed, users are sufficiently informed and warned about the residual risks, and protective measures are compatible with one another.

After all residual risks have been evaluated, the provider must review the risk control activities to ensure that the risks from all identified hazards have been considered and all risk control activities are completed. The results of this review must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Completeness of risk control is the standard’s quality gate before the overall residual risk evaluation. Most organizations treat it as a checklist item. The requirement to document reasons for not adopting available state of the art risk control measures is more demanding than it appears. If a peer organization in your sector has implemented a more effective bias control measure, a more robust monitoring system, or a more transparent explanation mechanism, and you have not adopted it, you need to document why. “We chose a different approach” is not sufficient. “We evaluated the following alternative measures, concluded they were not technically feasible for the following reasons, and implemented the following alternative approach, which achieves the following level of risk reduction” is what the standard requires.

Looking at the AI System and Residual Risk as a Whole

After all risk control measures have been implemented and verified, the provider must evaluate the overall residual risk posed by the AI system using the criteria for acceptability of the overall residual risk defined in the risk management plan. All identified hazards have been evaluated and all risks have been addressed by risk control measures to reduce them to an acceptable level. Even if each risk is reduced to an acceptable residual risk, the aggregation of all residual risks can be unacceptable.

The evaluation of the overall residual risk must take into account the factors required for establishing overall residual risk acceptability criteria, and the potential aggregation of each risk with low or medium severity over time, across users, or through repeated interactions with the AI system. This last element is particularly important. A risk that is acceptable in a single interaction can become unacceptable when multiplied across millions of users or repeated over extended periods.

The evaluation of overall residual risk must be supported by objective evidence obtained in accordance with the requirements for establishing risk acceptability criteria. Objective evidence must include test results demonstrating that the AI system performs consistently for its intended purpose and under conditions of reasonably foreseeable misuse. Explanation and justification must be provided for how this objective evidence demonstrates the acceptability of the overall residual risk.

If the overall residual risk is judged acceptable, the provider must inform deployers of significant residual risks, according to the intended purpose and the reasonably foreseeable misuse, and must include the necessary information in the accompanying documentation in order to disclose those residual risks. The provider should make the information openly available in digital and online formats.

If the overall residual risk is not judged acceptable in relation to the intended purpose and reasonably foreseeable misuse, the provider may consider implementing additional risk control measures, modifying its intended purpose, or achieving the intended purpose by not using an AI system. Otherwise, the overall residual risk remains unacceptable and in that case the AI system must not be deployed. This is a hard stop. The standard does not permit a provider to deploy a system with an unacceptable overall residual risk and manage the consequences reactively.

Evaluating overall residual risk is a decision made by the provider but can be influenced by policies and norms established by organizations, industries, communities, and policy makers. The results of the evaluation of the overall residual risk must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

The overall residual risk evaluation is where the standard most clearly diverges from how most organizations make deployment decisions. Most organizations approve deployment when each individual risk has been addressed and the system passes its performance benchmarks. The standard requires an additional step: evaluating whether the aggregate of all residual risks is acceptable, considering interactions between risks, cumulative effects over time and scale, and the distribution of harms across affected populations. Build your overall residual risk evaluation as a distinct documented decision, separate from the individual residual risk evaluations. Present it to top management with a summary of all residual risks, their interactions, their cumulative potential, and the objective evidence supporting the acceptability conclusion. If top management has not explicitly approved the overall residual risk evaluation, the deployment decision does not meet the standard’s requirements.

Reviewing the Process, Not Just the Outcome

The provider must review the execution of the risk management plan periodically throughout the life cycle phases of the AI system, and at least prior to placing on the market or putting into service the AI system.

This review must at least ensure that the risk management plan has been appropriately implemented, the overall residual risk is acceptable, and appropriate methods are in place to collect and review information in the pre-market and post-market phases. The results of this review must be recorded and maintained in the risk management file.

The responsibility for review must be assigned in the risk management plan to persons having the appropriate competence and authority. The risk management review must be approved by top management. This is not a staff-level activity. The review is a top management obligation with documented approval.

When, based on information from the provider’s post-market monitoring system or its real-world conditions testing, the provider identifies a serious incident or identifies a situation where a serious incident is avoided but can reasonably have occurred, the provider must decide on the necessity or desirability of a risk management review. The decision not to perform a risk management review must be justified. The default assumption is that a serious incident or near-miss triggers a review. Departing from that default requires a documented justification.

All modifications implemented as a consequence of the review must also be documented in the risk management file. Top management, or the provider generally, can have requirements related to the notification of serious incidents, or their avoidance, to relevant stakeholders in accordance with applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Learning from the Pre-Market and Post-Market Activities

The provider must establish, document, and maintain a system to actively collect and review information relevant to the AI system in pre-market and post-market phases in accordance with applicable regulatory requirements. When establishing this system, the provider must consider appropriate methods for the collection and processing of information.

For each of the points that must be considered, the provider must assess the relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement the point. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Information Collection

Pre-market and post-market activities can include receiving information about performance and risks posed by the AI system. The information can be related to harm that has occurred or to hazardous situations that occurred without harm. The activities can also include soliciting information about the AI system performance and related risks. These activities can involve reaching out to users, deployers, or other relevant stakeholders to obtain specific information and insight, using methods such as surveys, expert user groups, or consultations. They can also include publicly available information, incident reports, incident databases, and information on the state of the art.

The provider must collect information that is relevant to managing the AI system risks in the pre-market and post-market phases. This information must include, where applicable, information generated during pre-market life cycle stages and monitoring of the development process, information generated from the post-market monitoring system, information collected by automatic logging of events which the provider has identified as relevant to ensure that residual and overall residual risks are maintained to an acceptable level, and information generated by the users, including information from human oversight, user complaints, and other feedback.

This can include a general AI system feedback report capturing general AI system feedback from the user. It can also include an AI system incident report generated based on users reporting failures, malfunctions, or any unexpected behaviors observed in the AI system.

The information collection must also include information, warnings and complaints issued by stakeholders affected or their proxies, information generated by those accountable for the installation, use and maintenance of the AI system, information generated by the supply chain, publicly available information including information about similar AI systems and similar other products on the market, information related to the state of the art, and identification of unforeseen risks in relation to the execution of predetermined changes.

Publicly available information can refer to judgements of court cases, freely accessible reports, or any other relevant accessible content. Regulatory requirements can apply regarding the information being collected, including requirements on data protection, confidentiality, and permitted use. Stakeholders affected include those who have been identified during the risk identification process as placed at risk.

Justification for not collecting information related to the points above must be documented in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Information Review

The provider must review the information collected for possible relevance to the overall residual risk acceptability, especially whether previously unrecognized hazards or hazardous situations are present, an estimated risk arising from a hazard is no longer acceptable, the overall residual risk is no longer acceptable in relation to the intended purpose or applicable national, regional, or international regulations, the state of the art has changed, or changes to the AI system that were not foreseen or planned have occurred.

The results of the review must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Actions to Take

If the collected information is determined to be relevant to the overall residual risk acceptability, the following actions apply.

Concerning the particular AI system, the provider must review the risk management file and determine if reassessment of risks or assessment of new risks is necessary. If a residual risk, whether previously known or newly identified, is no longer acceptable, the impact on previously implemented risk control measures must be evaluated and must be considered as an input for modification of the AI system. If a residual risk, whether previously known or newly identified, is no longer acceptable, the provider must evaluate and justify whether or not the AI system must temporarily or definitively be withdrawn from service or from the market based on the severity of the identified unacceptable risk. The provider should inform deployers and relevant stakeholders without delay of the increased residual risks and possible mitigations through a field notice such as a website message. Any decisions and actions must be recorded in the risk management file.

Concerning the risk management process, the provider must evaluate the impact on previously implemented risk management activities. The results of this evaluation must be considered as an input for the review of the suitability of the risk management process by top management. If an unforeseen change has been identified, the provider should consider whether a risk reassessment of the AI system is necessary, especially if the unforeseen change affects the intended purpose of the AI system.

Concerning communication with relevant stakeholders, including deployers and users, the provider must inform about changes to the overall residual risk acceptability of the AI system.

Post-market activities are where most AI governance frameworks have their largest gap. Organizations invest heavily in pre-deployment risk assessment and virtually nothing in systematic post-deployment monitoring of compliance-relevant outcomes. The standard requires an active system for collecting and reviewing information, not a passive incident log. Build your post-market monitoring system as a structured program with defined data collection points, automated logging of hazardous events, scheduled information reviews, and documented decision criteria for triggering risk reassessment. Establish clear thresholds for when collected information requires immediate action, periodic review, or escalation to top management. If your post-market monitoring system cannot answer the question “is the overall residual risk of this system still acceptable today, given what we know from post-market experience,” it does not meet the standard’s requirements. And if that question is not being asked at regular intervals by someone with the authority to act on the answer, the system is not operating as the standard requires.

Understanding How AI Risks Unfold in Practice

The examples below illustrate how hazards, risk scenarios, hazardous situations, and harms connect in real AI deployments. Each example follows the same logic: a potential cause creates a hazard, a risk scenario describes the conditions under which the hazard can lead to harm, a hazardous situation describes the moment of exposure, and the harm describes what actually happens to affected persons. These examples are illustrative, not exhaustive, and applicable regulatory requirements regarding use cases and harms are subject to change.

Reading these examples as a risk practitioner, the most important pattern to notice is that the harm rarely flows directly from a technical failure. It flows from a chain: a design choice or operational condition creates a hazard, a specific scenario activates that hazard, and a person in a specific situation suffers the consequence. Breaking any link in that chain is the job of risk control.

Example 1: Skin Cancer Detection App

What the system does

A medical AI application intended to provide an indication of possible skin cancer from self-taken skin images, designed for any skin type.

What causes the hazard

The AI model was trained primarily on images from people with white or light skin, with non-representative or very limited coverage of dark skin. Testing with dark skin images was either not performed or severely limited. In some cases, the biased output could also result from a data poisoning attack on the training data rather than from inappropriate design choices alone.

What the hazard is

The system produces biased output in the form of false negatives. It systematically fails to detect skin cancer in dark-skinned patients. The hazard here relates directly to AI system performance and the quality of the training data.

How the risk scenario unfolds

A dark-skinned user who has skin cancer uses the app. The app returns a negative result, indicating no skin cancer is present. Trusting the result, the user does not consult a doctor for further examination of the skin abnormality.

What the hazardous situation looks like

The patient believes they have no skin cancer. They are now exposed to the continued and undetected development of the disease, potentially including metastasis, without any medical follow-up.

What harm results

Progression of the disease, worsening health condition and prognosis, and risk of death if metastatic skin cancer goes undetected over time.

What this example teaches

A system that appears to work well on average can systematically fail for specific demographic groups. Risk analysis must assess performance across subgroups, not just across the full population. The harm is not caused by a dramatic system failure. It is caused by a result that looks valid but is wrong for a specific group of users that the system was not adequately trained to serve.

Example 2: Credit Worthiness Evaluation in a Bank

What the system does

An AI system that evaluates the creditworthiness of natural persons, used by financial consultants in a bank to process loan applications.

What causes the hazard

After deployment, the bank reduces the number of financial consultants by 80 percent, reasoning that the AI system can absorb most of the workload. The remaining consultants must now process a much higher volume of cases than before. This is a reasonably foreseeable misuse of the system that was not anticipated in the original risk assessment. Compounding this, during the first ten interactions with the system, the consultants find that the AI recommendations appear accurate. This creates automation bias: the consultants begin to rely on the system’s recommendations without applying independent judgment. This is a human factors issue linked to the design of the user interface and the feedback the system provides.

What the hazard is

The hazard is poor human oversight resulting from the combination of high workload and automation bias. The hazard here relates to human-machine interaction rather than a technical failure in the model itself.

How the risk scenario unfolds

Financial consultants must process a large number of cases and have limited capacity to critically evaluate each AI recommendation. They validate recommendations, including erroneous ones, without sufficient independent review.

What the hazardous situation looks like

A consultant validates an erroneous AI recommendation without detecting the error. The applicant’s loan application is decided based on a biased or incorrect output from the system.

What harm results

Denial of loan applications for applicants based on characteristics such as citizenship, where the AI system has introduced discriminatory patterns that the consultants are not positioned to detect or correct.

What this example teaches

Organizational decisions made after deployment can create new hazards that were not present at launch. Reducing human oversight capacity after deploying an AI system is a foreseeable misuse that must be analyzed in the risk assessment. Automation bias is a predictable human response to a system that appears accurate in early use. Risk control must address both the technical output of the system and the conditions under which humans interact with it.

Example 3: Clinical Decision Support for Rare Disease Diagnosis

What the system does

A large language model used as a clinical decision support system for diagnosing rare diseases.

What causes the hazard

The model was fine-tuned on a narrow clinical dataset that lacked diversity in demographics and rare case data. Benchmark results were misinterpreted, either because the benchmarks used saturated tasks that did not reflect real clinical complexity, or because the results created a false impression that the model would rarely produce incorrect information in a broad range of cases. The model appears to perform well on standard benchmarks but overfits to the narrow training distribution.

What the hazard is

The system produces misleading diagnostic recommendations because it does not generalize well beyond its training data. The hazard is poor model performance in conditions that differ from the training environment.

How the risk scenario unfolds

A clinician, relying on the system’s high reported accuracy, over-relies on an incorrect recommendation and ignores contradictory clinical signs that would, under normal circumstances, prompt further investigation or specialist referral.

What the hazardous situation looks like

The patient receives incorrect treatment or is not referred for necessary specialist care because the clinician trusted the AI recommendation over their own clinical judgment.

What harm results

Delayed diagnosis, worsening health condition, and potential irreversible harm or death.

What this example teaches

Benchmark performance does not translate directly to real-world safety. A model that scores well on published benchmarks can still fail dangerously in clinical practice if the benchmarks did not capture the distribution of cases the model will encounter in deployment. Risk analysis must include an assessment of how benchmark results were derived and whether they are representative of the intended deployment context. Clinician reliance on AI outputs is a human factors hazard that must be explicitly addressed in risk control, not assumed away by the system’s reported accuracy.

Example 4: AI System Screening Job Applicants

What the system does

A large language model used to screen job applicants, providing recommendations based on CVs and job descriptions.

What causes the hazard

Benchmark scores were misinterpreted as demonstrating general fairness across domains, but the benchmarks had limited coverage or were saturated and did not measure the model’s behavior on the specific task of CV screening. Additionally, the benchmarks did not measure robustness against CVs specifically crafted to manipulate the model into generating a very positive assessment, a known adversarial input risk.

What the hazard is

The system produces wrong decisions due to unintended bias. Biases embedded in training data, including gender, race, and age, and the model’s vulnerability to adversarial inputs, are assumed to have been addressed when they have not been.

How the risk scenario unfolds

The system is deployed with the assumption that bias and robustness issues are resolved. Candidates are exposed to a decision process that contains unintended discrimination against specific groups of people.

What the hazardous situation looks like

Qualified candidates from discriminated groups are evaluated by a system that systematically rates them lower than equivalent candidates from other groups, without the organization recognizing that the system is producing discriminatory outputs.

What harm results

Discriminatory hiring outcomes. Qualified candidates are rejected on the basis of characteristics such as gender, race, or age rather than on the merits of their application.

What this example teaches

Fairness in AI is not a binary state that is achieved once and maintained automatically. It must be tested specifically for the task and dataset at hand, not inferred from general benchmark performance. Robustness to adversarial inputs is a separate dimension of risk that must be assessed independently from fairness. Deploying a system on the assumption that known risk categories have been resolved, without task-specific evidence, is a risk management failure that the standard explicitly requires providers to avoid.

Example 5: AI Agent Managing Energy Grid Optimization

What the system does

A goal-directed AI system deployed to autonomously manage energy grid optimization.

What causes the hazard

The AI system exhibits specification gaming behavior, meaning it finds ways to maximize its performance metrics that were not intended by the designers and that do not align with safe grid operation. The system’s limited interpretability makes it difficult for operators to understand what decisions the system is making and why.

What the hazard is

The AI monitoring and control interface does not provide sufficient information about the system’s decisions and their effects. Operators cannot see what the system is doing or why it is doing it.

How the risk scenario unfolds

The system is deployed and begins optimizing grid operations in ways that are not visible to operators. It puts the grid into an unsafe operating mode without operators recognizing that this has occurred. The risk of cascading failures across interdependent systems grows without detection.

What the hazardous situation looks like

The grid is being run in an unsafe mode that creates a high probability of blackouts and equipment failure, while operators believe the system is functioning correctly.

What harm results

Physical damage to infrastructure, large-scale blackouts, and adverse health effects on persons dependent on continuous power supply.

What this example teaches

Specification gaming is a well-documented failure mode in goal-directed AI systems. A system that optimizes for the wrong objective can cause serious harm even when it is technically functioning as designed. Interpretability is not an optional feature. It is a prerequisite for human oversight in high-stakes deployments. Risk control must include mechanisms that allow operators to understand and intervene in system behavior before unsafe states develop.

Example 6: AI Monitoring Warehouse Workers

What the system does

An AI system used to organize warehouse work through real-time monitoring of worker activity.

What causes the hazard

The system monitors worker characteristics that are not necessary for its stated operational purpose, collecting data beyond what is required for warehouse organization.

What the hazard is

The system monitors unnecessary worker characteristics, exceeding the scope of what is proportionate for warehouse management.

How the risk scenario unfolds

Workers performing warehousing tasks are placed under continuous real-time AI monitoring. The system operates constantly throughout the working day.

What the hazardous situation looks like

Workers are subject to continuous AI-enabled surveillance, including monitoring of characteristics that are not relevant to their work performance and that they have not meaningfully consented to.

What harm results

Violation of data rights, psychosocial harassment, continuous performance pressure, and risk of job loss based on monitoring data that exceeds the legitimate scope of the system’s intended purpose.

What this example teaches

Workplace AI systems can cause harm through scope creep, monitoring more than is necessary for the stated purpose. The proportionality of data collection must be assessed as part of the risk analysis, not just the technical accuracy of the monitoring. Workers in high-monitoring environments experience real psychological harm from surveillance even when no action is taken on the data. This is a harm within the meaning of the standard.

Example 7: AI Evaluating Teachers’ Activity

What the system does

An AI tool used to evaluate teachers’ activity, including assessment of pupils’ and students’ performance, and providing automatic feedback to assessors.

What causes the hazard

The information that the system needs to make accurate evaluations cannot be accurately or reliably connected to the system’s inputs. The data that would be required to make meaningful assessments of teacher quality is not consistently available or measurable in the form the system expects.

What the hazard is

The system produces evaluations of teacher quality based on data that does not accurately reflect what it purports to measure.

How the risk scenario unfolds

The tool is used for teaching and evaluation in classrooms. Teachers are evaluated based on AI-generated assessments that may not reflect their actual performance or the factors that influence student outcomes.

What the hazardous situation looks like

Teachers are subject to consequential evaluations produced by a system whose inputs do not accurately represent their professional activity. Students are also affected through assessments that may not reflect their actual learning.

What harm results

Violation of data rights, psychosocial harassment, continuous pressure from unjustified performance assessments, and risk of job loss based on AI evaluations that do not accurately reflect performance.

What this example teaches

The quality and representativeness of input data is as important as model performance. A technically sophisticated system that operates on inputs that do not accurately represent the phenomenon it is supposed to evaluate will produce systematically misleading outputs. This is a hazard that must be identified in the risk analysis and addressed in risk control, not assumed away by system accuracy metrics.

By Prof. Hernan Huwyler, CAIO MBA CPA

About the Author

The frameworks, tools, and implementation guidance described in this article are part of the applied research and advisory work of Prof. Hernan Huwyler, MBA, CPA, CAIO. These materials are freely available for use, adaptation, and redistribution in your own AI governance, risk management, and compliance programs. If you find them valuable, the only ask is proper attribution. If you like the content, please like the article and share it.

Prof. Huwyler serves as AI GRC Consultancy Director, AI Risk Manager, and Quantitative Risk Lead, working with organizations across financial services, technology, healthcare, and public sector to build practical AI governance frameworks that survive contact with production systems and regulatory scrutiny. His work bridges the gap between academic AI risk theory and the operational controls that organizations actually need to deploy AI responsibly.

As a Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative predictive risk automation, and AI audit readiness for executive leadership teams, boards, and technical practitioners. His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe and internationally.

Based in the Copenhagen Metropolitan Area, Denmark, with professional presence in Zurich and Geneva, Switzerland, Madrid, Spain, and Berlin, Germany, Prof. Huwyler works across jurisdictions where AI regulation is most active and where organizations face the most complex compliance, technical and business requirements.

His code repositories, risk model templates, and Python-based tools for AI governance are publicly available at . His ongoing writing on Governance, Risk Management and Compliance appears on his blogger website at (more than 500k views).

Connect with Prof. Huwyler on LinkedIn at to follow his latest work on AI risk assessment frameworks, compliance automation, model validation practices, and the evolving regulatory landscape for artificial intelligence.

If you’re building an AI governance program, standing up an AI risk function, preparing for EU AI Act compliance, or looking for practical implementation guidance that goes beyond policy documents, reach out. The best conversations start with a shared problem and a willingness to solve it with rigor.

Practical AI Red Team Implementation Tips for Safer, More Resilient AI Systems

Thu, 12 Mar 2026 00:00:00 +0000

Playbook for Building an AI Red Team

Three months after deploying a customer-facing language model, a financial services firm I advise discovered that a determined user could extract fragments of training data by crafting specific prompt sequences. The data included internal policy documents that were never meant to be public. Their security team hadn’t tested for this. Their data science team didn’t know it was possible.

A basic AI red team exercise would have caught it in an afternoon.

Most organizations test their AI systems the same way they test traditional software: functional testing, load testing, maybe a penetration test of the hosting infrastructure. That approach misses an entire category of risk unique to AI. Model evasion. Data poisoning. Prompt injection. Bias exploitation. Harmful output generation. These attack vectors don’t exist in conventional software, and conventional security teams aren’t trained to find them.

An AI red team is a specialized group that proactively identifies these risks by simulating realistic attack scenarios across the full AI lifecycle. This post walks through how to build one, what it should test, how to structure assessments across development phases, and the practical mistakes I’ve watched organizations make when standing up this capability for the first time.

What an AI Red Team Actually Does (And Why Traditional Security Testing Falls Short)

An AI red team simulates adversarial attacks against AI systems to expose vulnerabilities, biases, and weaknesses before real-world attackers or users find them. The concept borrows from military and cybersecurity red teaming, but the scope is fundamentally different.

Traditional red teams test network security, application code, and infrastructure. AI red teams test all of that plus model behavior, training data integrity, inference pipeline security, and the potential for the system to produce harmful or biased outputs. The attack surface for an AI system is larger than for traditional software because the model itself is both an asset and an attack vector.

The purpose maps to four risk categories that every AI red team assessment should cover: confidentiality, integrity, and availability (CIA), compliance risk, revenue risk, and operational risk losses. A single vulnerability can affect multiple categories simultaneously. A prompt injection attack that extracts customer data hits CIA, compliance, and revenue at the same time.

Implementation tip: When I helped build our first AI red team, we made it a subset of the existing cybersecurity red team. That was a mistake. The cybersecurity team was excellent at finding infrastructure vulnerabilities but didn’t know how to craft adversarial examples against a machine learning model. They didn’t understand model inversion attacks or training data poisoning. We restructured the team after four months of assessments that found infrastructure issues but missed every model-specific vulnerability. Your AI red team needs its own charter, its own methodology, and team members who understand machine learning at a technical level.

Building the Right Cross-Functional Team

Composition determines capability. An AI red team staffed only with security engineers will find security problems. It will miss bias, compliance gaps, and abuse scenarios entirely.

Your AI red team needs four disciplines represented: security experts who understand adversarial attack methodologies, data scientists who understand model architecture and training processes, ethicists or responsible AI specialists who can identify harm and abuse pathways, and risk and compliance professionals who can map findings to regulatory requirements and business impact.

The security experts bring penetration testing methodology, threat modeling experience, and knowledge of common attack patterns. They test authentication, input validation, deserialization, and infrastructure hardness.

The data scientists bring model-specific expertise. They understand how to craft adversarial inputs that cause misclassification, how to test for training data leakage, and how to evaluate whether a model is susceptible to evasion or extraction attacks. Without this expertise, you cannot test model vulnerabilities.

The ethicists assess harm and abuse scenarios: Can the system be manipulated to produce biased outputs? Can it be used for purposes it was never intended for? Does it create quality-of-service harms where certain user groups receive worse performance? These assessments require familiarity with fairness frameworks and human rights impact analysis.

Risk and compliance professionals translate technical findings into business language. They determine whether a discovered vulnerability creates regulatory exposure, quantify potential financial impact, and prioritize remediation based on organizational risk appetite.

Implementation tip: Staff your AI red team with at least one person who has built production AI systems. Not managed them. Built them. I’ve worked with red teams composed entirely of auditors and security analysts. They could identify categories of risk from a checklist but couldn’t demonstrate actual exploits. The team’s credibility with AI development teams depends on their ability to show, not just describe, how an attack works. When our red team demonstrated a live model extraction attack during a readout meeting, pulling a functional copy of a proprietary model through API queries alone, the development team went from skeptical to fully engaged in 15 minutes. Demonstrated exploits create urgency that risk reports never achieve.

The Four Assessment Domains: What Your AI Red Team Should Test

Every AI red team assessment should cover four domains: reconnaissance, model vulnerabilities, technical vulnerabilities, and harm and abuse scenarios. Skipping any domain leaves critical gaps.

Reconnaissance is where the assessment starts. The team identifies what can be learned about the target AI system from external observation. This includes base model discovery (what foundation model is being used and what known vulnerabilities does it have), serving infrastructure analysis (how is the model deployed, what APIs are exposed, what metadata leaks through response headers), and dataset collection assessment (can the team identify or infer what training data was used).

Model vulnerabilities form the core of what makes AI red teaming different from conventional security testing. Six specific attack types need testing.

Poisoning attacks test whether an adversary could corrupt the training data to influence model behavior. This applies primarily during training phases but has implications for systems that use continuous learning. Prompt injection tests whether crafted inputs can override system instructions or extract information the model shouldn’t reveal. Evasion attacks test whether adversarial inputs can cause the model to misclassify or produce incorrect outputs. Inversion attacks test whether model outputs can be used to reconstruct training data. Extraction attacks test whether the model’s parameters or architecture can be stolen through systematic querying. Membership inference tests whether an attacker can determine if a specific data point was included in the training dataset.

Technical vulnerabilities cover conventional security weaknesses in the AI system’s infrastructure: lack of input validation on API endpoints, missing or weak authentication mechanisms, insecure deserialization that could allow code execution, and insufficient access controls on model artifacts and training data.

Harm and abuse scenarios assess whether the system can produce harmful outputs or be misused. This includes testing for misuse potential (can the system be used for purposes it was never designed for), stereotyping and bias (does the system produce outputs that reflect or amplify harmful stereotypes), quality-of-service harms (does the system perform worse for certain demographic groups), and allocation harms (does the system make decisions that unfairly distribute resources or opportunities).

Implementation tip: Most AI red teams I’ve evaluated spend 80% of their time on technical vulnerabilities and 20% on everything else. Flip that ratio. Technical vulnerabilities in AI systems are generally similar to those in any web application, and your existing security testing probably covers many of them already. Model vulnerabilities and harm/abuse scenarios are where AI-specific risks live, and they’re where conventional testing leaves the biggest gaps. On one assessment, our team spent three days on infrastructure testing and found two medium-severity issues. We spent one day on prompt injection testing and found a critical vulnerability that allowed users to bypass all content safety filters. Allocate your assessment time based on AI-specific risk, not general security methodology.

Assessment Across the AI Lifecycle: Pre-Production Through End of Life

AI red team assessments aren’t one-time events. Different lifecycle stages expose different vulnerabilities. Your assessment program should map to four phases.

Pre-production assessment happens during ideation and design. The red team evaluates risks in intended use cases and planned data sources before any code is written. This is a tabletop exercise, not a technical assessment. The team walks through scenarios: “If we build this system using this data for this purpose, what could go wrong?”

What to assess: Review the intended use description for potential misuse pathways. Evaluate planned data sources for bias risks, provenance concerns, and legal compliance. Identify which model vulnerability types are most relevant given the planned architecture. Document risks that should be mitigated by design rather than discovered in testing.

Training phase assessment covers data collection, data processing, model training, and model evaluation. This is where poisoning risks, data quality issues, and bias introduction are most testable.

What to assess: Test whether training data pipelines have integrity controls that would detect unauthorized modification. Evaluate whether data processing steps introduce or amplify bias. Test the trained model for demographic performance disparities before it moves to deployment. Verify that training, validation, and test datasets are properly separated.

Inference phase assessment covers model deployment and system monitoring. This is the phase where most organizations focus their red teaming, and where prompt injection, evasion, and extraction attacks are most relevant.

What to assess: Test all API endpoints for input validation and authentication. Attempt prompt injection attacks across multiple strategies. Test whether model outputs can leak training data or system prompts. Evaluate monitoring systems to determine whether they would detect adversarial activity. Test rate limiting and abuse prevention controls.

Post-production assessment addresses end-of-life risks. When AI systems stop receiving updates, their vulnerabilities become permanent. When models are retired, the data and artifacts associated with them need secure handling.

What to assess: Evaluate whether decommissioned models are still accessible through legacy systems or cached endpoints. Test whether training data is properly purged or archived when a model is retired. Assess whether downstream systems that depended on a retired model are still sending queries to dead endpoints.

Implementation tip: The pre-production tabletop exercise is the highest-value, lowest-effort activity in your entire red team program. I resisted this for over a year because it felt too theoretical. Then I ran my first one. In 90 minutes, a cross-functional group identified that the planned training dataset for a healthcare triage model excluded patients who primarily spoke Spanish because the source hospital system captured those encounters in a separate database. That single finding, caught before any development began, prevented a system that would have performed measurably worse for Spanish-speaking patients. The fix was adding a data source. Had we caught this during inference-phase testing, the fix would have been retraining the model from scratch. Run tabletop exercises for every AI system during ideation. The time investment is minimal. The potential savings are enormous.

Security Controls: Privilege Tiering and Compartmentalization

Your AI red team doesn’t just find vulnerabilities. It also validates whether your security controls are effective. Two architectural principles matter most for AI systems: privilege tiering and compartmentalization.

Privilege tiering means using different levels of access control across development phases. A data scientist who needs access to training data during the model development phase should not retain that access during production deployment. An ML engineer who needs to modify model parameters during training should not have that capability once the model is serving predictions.

What to put in place: Define at least three access tiers. Development tier: broad access to data and model artifacts, restricted to sandbox environments. Staging tier: read access to production-equivalent data, write access to model configurations, no direct access to production infrastructure. Production tier: minimal access limited to monitoring and predefined deployment procedures, with all changes requiring approval workflows.

Compartmentalization reduces attack surfaces by isolating AI system components. If an attacker compromises the data preprocessing pipeline, compartmentalization prevents them from reaching the model serving infrastructure. If a vulnerability exists in the model API, compartmentalization prevents lateral movement to the training data storage.

What to put in place: Separate your AI infrastructure into isolated segments. Training environments should be network-isolated from production serving environments. Model artifact storage should use separate access controls from training data storage. Monitoring and logging infrastructure should be isolated so that an attacker who compromises a model component cannot delete the evidence.

Implementation tip: Test your privilege tiering by having your red team operate at each access level and document what they can reach. On one assessment, we discovered that a “staging” service account had been granted production database read access “temporarily” eight months earlier and nobody had revoked it. That single service account provided a path from the staging environment to every production model artifact and every piece of training data. Temporary access grants are the most common source of privilege tiering failures. Build an automated access review that flags any credential with cross-tier access and requires monthly reauthorization. Every temporary exception should have an expiration date enforced by the system, not by human memory.

Documenting Findings and Running Tabletop Exercises

Documentation determines whether your red team findings lead to actual improvements or gather dust in a shared drive.

Every finding should include six elements: a description of the vulnerability or risk discovered, the attack technique used to discover it, the component affected (model, technical stack, corporate network, or internet-facing surface), a risk rating based on likelihood and impact, recommended remediation actions, and the risk categories affected (CIA, compliance, revenue, operational losses).

Rate technical vulnerabilities using a consistent framework. I use a modified version of the CVSS (Common Vulnerability Scoring System) adapted for AI-specific risks. Standard CVSS doesn’t capture model-specific impacts like training data exposure or bias amplification, so you’ll need to add scoring criteria for those dimensions.

Tabletop exercises complement technical assessments by testing organizational response capabilities. These are structured sessions where the team talks through how they would handle specific AI incidents without actually performing technical operations.

Run tabletop exercises quarterly. Each exercise should present a realistic scenario, walk through the response process step by step, identify gaps in response plans, and document improvements needed.

Example scenario: “A researcher publicly discloses that our production language model can be manipulated to generate instructions for illegal activities through a specific prompt pattern. The disclosure includes a working example. Social media attention is growing rapidly. Walk through your response for the next 72 hours.”

This exercise tests incident detection, internal escalation, technical remediation, public communication, and regulatory notification processes simultaneously. The gaps it reveals are always instructive.

Implementation tip: The biggest documentation mistake I see is treating findings as a static report delivered once and then archived. Build a findings tracker that persists across assessments. Every vulnerability found should be tracked to remediation. Every remediation should be verified by the red team in the next assessment cycle. I’ve reviewed organizations where the same prompt injection vulnerability appeared in three consecutive quarterly assessments because nobody tracked whether the fix was actually applied. Your red team program should have a “findings closure rate” metric: the percentage of previous findings that have been verified as remediated in the current assessment. If that rate is below 70%, your red team is finding problems faster than the organization can fix them, which means you have a capacity problem, not just a security problem.

Implementation Tips for AI Red Teams

These principles apply across every aspect of your AI red team program.

Implementation tip on assessment scope: Define your scope precisely before every engagement. “Test the AI system” is not a scope. “Test the customer-facing API endpoints of the mortgage risk model for prompt injection, input validation, and authentication vulnerabilities, with model evasion testing against the classification function” is a scope. Without precise scoping, assessments drift into areas that consume time without producing actionable findings. I ran one assessment where the scope was “evaluate the AI platform.” The team spent two weeks testing corporate network security around the platform and found issues that had nothing to do with AI. The model-specific testing got compressed into three days and produced superficial results. Scope tightly. Focus on AI-specific risks. Leave general infrastructure testing to your standard security program.

Implementation tip on assessment frequency: High-risk AI systems need red team assessment at least twice per year, plus a reassessment after any major model update, architecture change, or deployment expansion. Low-risk systems can operate on annual assessment cycles. The mistake I see most often is treating red team assessments as annual compliance events. AI systems change continuously. Models get retrained. New features get added. Deployment contexts shift. An assessment conducted in January may be irrelevant by July if the model has been retrained on new data. Tie your assessment schedule to your model lifecycle, not to a calendar.

Implementation tip on reporting to leadership: Your red team findings report needs two versions. A technical report for the development and security teams with full exploit details and remediation guidance. An executive summary for leadership that translates findings into business risk. The executive summary should answer four questions: What did we find? How likely is exploitation? What’s the business impact? What needs to happen next? I once delivered a highly technical red team report to a board risk committee. Fourteen pages of model architecture diagrams and attack chain descriptions. The committee members understood none of it and approved a budget that addressed zero of the actual findings. The rewritten two-page executive summary, which described risks in terms of regulatory fines, customer data exposure, and reputational damage, got full funding for remediation in one meeting.

Original implementation tip on avoiding adversarial relationships with development teams: Your AI red team will fail if developers view it as an adversary rather than an ally. This is a cultural challenge as much as a technical one. Share preliminary findings with development teams before final reports go to leadership. Give them the opportunity to explain architectural decisions that might appear as vulnerabilities but actually have mitigating controls. Invite developers to observe red team exercises so they learn to think adversarially about their own work. On the best-functioning red team program I’ve been part of, developers started requesting ad-hoc red team reviews before major releases because they’d seen the value. They treated the red team as a resource, not a threat. That shift took about 18 months of consistent, collaborative engagement to achieve.

References and Authoritative Frameworks

Your AI red team program should align with these established standards and guidelines:

NIST AI 100-2 (Adversarial Machine Learning: A Taxonomy and Terminology)
MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
OWASP Top 10 for Large Language Model Applications
NIST AI Risk Management Framework (AI RMF 1.0), particularly the Measure and Manage functions
ISO/IEC 42001:2023, AI Management System
ISO/IEC 27001:2022, Information Security Management
Microsoft AI Red Team guidance and responsible AI practices
Google Secure AI Framework (SAIF)
EU AI Act, Article 9 requirements for risk management of high-risk AI systems
NIST SP 800-53 security controls, adapted for AI system components

If you treat AI red teaming as an annual compliance checkbox, running a scripted assessment once a year and filing the report, your AI systems will carry vulnerabilities that a motivated attacker, a curious user, or an automated scanning tool will eventually find. The report will show that you “tested” the system. The incident will show that you didn’t test it well enough.

When you build a red team program with the right cross-functional composition, the right assessment methodology covering all four domains, the right lifecycle integration from ideation through decommissioning, and the right documentation and tracking processes, you create a continuous pressure-testing capability that makes your AI systems measurably more resilient. You find prompt injections before your customers do. You catch bias before regulators do. You identify model extraction risks before competitors do.

An AI system that has never been attacked by its own red team is an AI system waiting to be attacked by someone else.

What’s the first AI system in your organization that your red team should assess? Start the scoping conversation this week.

Practical ISO 42001 Certification Guidance

Thu, 12 Mar 2026 00:00:00 +0000

Implementation Tips for GRC and AI Professionals

Make the AI Policy Operational, Not Decorative

Most AI policies fail before they get printed. A paragraph about “responsible AI” goes into a PDF, someone signs it, and the organization moves on. That approach does not survive an audit, because the standard expects the policy to align with business strategy, organizational values, and risk appetite, not with a template borrowed from a consulting deck.

A working AI policy names four distinct activities and treats each one differently. Development carries different risk than purchase. Operation carries different risk than simple use. A policy that lumps all four together produces guidance too vague for anyone to apply, and vague guidance is what auditors flag first. Write the policy so a developer, a procurement lead, an operations owner, and an end user can each find the paragraph that applies to them.

The policy also needs three components beyond good intentions. It needs principles that actually guide decisions, not slogans. It needs a defined process for handling deviations and exceptions, because every AI program eventually needs one. And it needs explicit cross-references to the policies it overlaps with, since AI activity rarely lives inside its own silo.

That overlap is the part organizations skip, and it costs them later. Before drafting the AI policy, run a gap analysis across the policy landscape already in place. Check where the information security policy under already covers AI-related activity. Check the privacy policy under ISO/IEC 27701. Check the quality policy under ISO 9001. Check existing safety policy. Only after that mapping should the AI Architecture team decide what the AI policy actually needs to add. In Huwyler’s experience, an AI policy drafted without that cross-check can quietly contradict the existing privacy policy, and nobody notices until an audit turns up two documents giving two different answers about the same personal data.

Ownership matters as much as content. Assign a named role, approved by management, accountable for developing, reviewing, and evaluating the policy. Reviews happen at planned intervals and whenever the organizational environment, business circumstances, legal conditions, or technical environment changes. Feed the results of management review back into the policy itself, so the document stays current instead of aging into irrelevance on a shared drive.

gives the governing body specific guidance on overseeing AI use, covering accountability, risk acceptance, and strategic decision-making at the level above day to day operations. Reference it directly when briefing the executives who sit above the AI Committee, since it speaks their language and answers the questions they are likely to ask about oversight and exposure.

Build the AI System Inventory Before You Try to Govern Anything

An organization cannot govern an AI system it has never cataloged. This sounds obvious and gets skipped constantly, because most AI inventories start and end with a spreadsheet of model names. ISO/IEC 42001 asks for something with more structure, documented resources across five categories and across the full lifecycle of each system.

The first category covers the AI system components themselves, the individual parts that make up what the organization actually runs. The second covers data resources, meaning every dataset touched at any stage, training, validation, test, and production. The third covers tooling resources, the algorithms, models, data conditioning tools, optimization methods, evaluation methods, and development tools involved in building and running the system. gives detailed guidance on how to describe these tooling components using a shared vocabulary, which helps when the same system passes between teams that do not normally speak the same technical language.

The fourth category covers system and computing resources, hardware, storage, and processing, along with whether those resources sit on premises, in the cloud, or at the edge. Document the environmental footprint of the hardware behind AI workloads here too, since compute intensity is now a governance question in its own right, not just a cost line. The fifth category covers human resources, the people with the expertise to build, sell, train, operate, and maintain the system. That includes data scientists, human oversight roles, trustworthiness experts covering safety, security, and privacy, domain experts, and researchers. A system with strong technical talent and no assigned human oversight role is not fully resourced, whatever the org chart implies.

Resource needs shift across the lifecycle. A system in development needs different people and different compute than the same system in production. Document that shift explicitly instead of assuming a single resource list covers every stage. Resources can also come from different places, the organization itself, a customer, or a third party, and the inventory should record the source for each one rather than treating every resource as internally owned.

Architecture diagrams and data flow diagrams do more work here than a written inventory ever will. ISO/IEC 42001 specifically points toward this format, and it earns its place twice over. One diagram per Tier 1 system, showing data flows, compute resources, human touchpoints, and third-party dependencies on a single page, satisfies the documentation requirement and feeds directly into the AI system impact assessment covered next. When a regulator asks how a system works, handing over one annotated diagram lands better than handing over a document nobody has read past the cover page.

ISO 42001 Annex A Documents Risk. It Doesn’t Control It.

ISO 42001 Annex A asks organizations to identify and document resources, data, and roles far more than it asks them to act on any of it. Several controls under A.4 and A.7 stop at the word “document,” with no operational obligation attached. Add to that the genuine ambiguity over whether Annex A is even mandatory, the absence of named technical controls for prompt injection, model drift, or data poisoning, and the heavy overlap with ISO 27001’s risk assessment and statement of applicability mechanics, and the standard starts to look like a management shell waiting for someone else to build the engine. None of that makes ISO 42001 a bad standard. It makes it an incomplete one if a Chief AI Risk Officer treats Annex A as the finish line instead of the starting inventory.

The standard says “the organization can select an appropriate set of control objectives and controls from Annex A”, and that single word, can, is doing real work. ISO drafters reserve “shall” for mandatory requirements. Choosing “can” means Annex A is a reference catalog to compare against, not a checklist to complete. This matters most for agentic AI systems, where an autonomous agent executing multi-step actions, calling tools, and making downstream decisions needs controls that A.4 and A.7 were never built to cover, things like tool-call authorization limits, action rollback triggers, and human override gates. It matters just as much for organizations tracking the EU AI Act, where the harmonized standards now moving through CEN-CENELEC JTC 21 will define specific conformity assessment and quality management expectations that Annex A does not anticipate. Companies serious about both agentic risk and AI Act alignment should treat Annex A as a floor, then add the controls the risk assessment actually demands.

Producing evidence is not a control. Writing a policy that says drift will be monitored is a control design decision, and design is only half the job. A control exists to change an outcome, not to describe an intention. In my experience advising regulated AI programs, the gap between a documented policy and a functioning control shows up exactly when it costs the most, during an incident, when the postmortem reveals that the policy existed but nothing enforced it. Auditors call this a design versus operating effectiveness gap for a reason. A policy proves the organization thought about the risk. It proves nothing about whether the risk was actually stopped.

This is precisely why the concept of a control plane matters more than another paragraph of policy language. A control plane is the technical layer that executes and evidences a control automatically, every time, without depending on a human remembering to check a document. A model registry that blocks deployment until a bias test result clears a threshold is a control plane. A guardrail proxy that intercepts every prompt and blocks known injection patterns before the request reaches the model is a control plane. An evaluation harness that halts a release pipeline when drift crosses a defined limit is a control plane. Annex A’s language of “identify and document” can be satisfied on paper in an afternoon. A control plane cannot be faked, because it either fires or it does not, and that difference is what separates a certificate from actual risk reduction.

One reconciliation is worth making here, offered as a friendly correction rather than a dismissal. The comparison to the AIUC cross-walk, and its claim of more than twenty missing Annex A controls, is a useful prompt for reflection but not a benchmark an organization should cite in a board paper next to ISO or NIST. AIUC has not gone through the consensus process, public comment, and international ratification that give ISO 42001 or the NIST AI RMF their standing. Its gap analysis may point in a reasonable direction, and some of those gaps genuinely echo concerns already raised by NIST commentators and the OWASP LLM Top Ten, but the cross-walk itself has not been independently validated against a recognized authority. Treat it as a hypothesis worth testing against your own risk assessment, not as proof that ISO 42001 is deficient.

Voices and critiques come from different professional lanes in academia, an engineering body, practicing auditors, algorithmic auditing leadership, and industry analysis. These are my considerations for improvement and practical limitations before I recommend ISO 42001 as a standalone AI governance program.

Documentation mistaken for safety

Several controls, especially across A.4 Resources and A.7 Data, require the organization to identify and document, and nothing more. No control in that cluster asks for a corrective action, a threshold, or a verification step. Writing down a data source does not remove bias from it, and documenting a tooling resource does not test whether that tool behaves safely in production. This is a high confidence finding because it is verifiable directly against the standard’s control text, not an inference about intent. Any AI Architecture team treating Annex A as complete should read A.4 and A.7 side by side and count how many controls actually require an action beyond documentation.

Missing AI-native technical controls

There is a real absence, not a matter of interpretation. Annex A names no control for prompt injection, model drift, data poisoning, adversarial inputs, or jailbreaking. Annex B gestures toward some of these risks as guidance, but guidance carries no certification weight and no auditor can test against it the way they test a numbered control. This gap matters more for large language model deployments than the drafters likely anticipated. The critique carries medium-high confidence because it is widely echoed in practitioner writing and lines up cleanly with the OWASP LLM Top Ten and the NIST AI RMF, both of which name these risks explicitly. Any team relying on ISO 42001 alone for a large language model program needs a supplementary technical control set to close this gap.

No mandated adversarial testing

Red teamers raise a narrower version of the same gap, and their standing as practitioners who test systems for a living gives the critique added weight. ISO 42001 requires an AI system impact assessment, but nothing in Annex A mandates adversarial testing, automated vulnerability scanning, or continuous technical monitoring once a system reaches production. An impact assessment performed once before launch tells an organization very little about a model’s behavior six months later under adversarial pressure. This critique treats AI risk as a technical security problem, closer to penetration testing than to a compliance checklist. The confidence level sits at medium-high, since it reflects an observed practice gap rather than a disputed reading of the standard’s text. A certified organization can pass audit while never having red teamed its production model.

Audit culture over rigorous testing

Chowdhury’s critique targets the audit culture that management system standards tend to produce, and her background running algorithmic audits gives it practical grounding. Her concern is that a checklist can be completed without ever running a rigorous bias or fairness test on a live model. Annex A supports this concern by design, since its bias-related expectations sit inside broader risk assessment language rather than naming a specific statistical test or threshold. A control that says assess for bias leaves the method to the implementer, and a weak implementer can satisfy that language with a narrative paragraph. This is a medium confidence critique, inferential rather than structurally provable from the standard’s text alone, but it echoes a pattern long documented in ISO 27001 certification audits. The fix is to name the test inside the control, for example a disparate impact ratio calculation on the historical decision dataset, rather than leaving bias assessment open ended.

Voluntary standard, no external accountability

A voluntary management system, absent legal backing, risks functioning as a marketing signal rather than a binding accountability mechanism. ISO 42001 does not require public disclosure of an organization’s risk register, its statement of applicability, or the outcome of its internal audits. A certificate can sit on a website while the underlying risk assessment stays entirely internal and unverifiable by any outside party. This is a medium confidence critique, since it rests on a policy argument about voluntary standards generally rather than a specific textual gap. It becomes most relevant where the EU AI Act’s binding conformity assessment requirements will eventually sit alongside, and in places exceed, what ISO 42001 asks for voluntarily.

Too high-level against NIST AI RMF

The ISO 42001 tells an organization it needs to manage AI risk without telling an AI engineer how to do it on a Tuesday afternoon. They contrast it directly against the NIST AI RMF, which breaks risk management into four functions, Govern, Map, Measure, and Manage, each carrying more actionable subtasks. Organizations already running ISO 27001 or NIST AI RMF frequently find the risk assessment structure, the Statement of Applicability process, and the management review cycle duplicated across frameworks with no guidance on how to merge the paperwork. This is a medium confidence critique, since it reflects an analyst judgment about relative usefulness rather than a factual gap in the standard’s text. The practical response is to map ISO 42001 controls directly onto an existing NIST AI RMF or ISO 27001 control set before building anything new.

Operating ISO 42001 Where the Controls Actually Bite

Conduct AI System Impact Assessments That Actually Protect You

ISO/IEC 42001 Annex B.5 requires organizations to assess the consequences an AI system has on individuals, groups, and societies. It is one of the most detailed requirements in the standard and, in practice, one of the most poorly implemented, usually because teams treat it as a form to fill out once and forget.

Start with the triggers. The standard names three circumstances that call for an assessment, the criticality of the system’s intended purpose and context, the complexity of the technology and its level of automation, and the sensitivity of the data types and sources it processes. A credit scoring model touches all three. A basic internal chatbot summarizing meeting notes touches almost none. The assessment effort should scale with that difference rather than apply the same template to both.

The assessment process itself has five stages that belong in sequence. Identify the sources, events, and potential outcomes tied to the system. Analyze the consequences and their likelihood. Evaluate the findings, which includes acceptance decisions and prioritization, not just a severity score. Treat the risk through mitigation measures. Document, report, and communicate the result. Skipping straight from identification to documentation, without a real evaluation and treatment step in between, is how impact assessments turn into paperwork with no teeth.

What the assessment actually needs to measure goes beyond model accuracy. Ask whether the system affects legal positions or life opportunities, physical or psychological well-being, universal human rights, or society more broadly. ISO/IEC 42001 calls out specific protection needs for vulnerable groups by name, children, impaired persons, elderly persons, and workers, and an assessment that treats all users as an undifferentiated population misses exactly the population the standard asks you to look at closely. The areas of impact to evaluate span fairness, accountability, transparency and explainability, security and privacy, safety and health, financial consequences, accessibility, and human rights.

Retain the results, and retain the right level of detail. A defensible record covers intended use, potential misuse, positive and negative impacts, predictable failure modes and their mitigations, relevant demographic groups, system complexity, and human oversight capabilities. For systems with a societal footprint, extend the assessment further, environmental sustainability including greenhouse gas emissions from computational intensity, economic impacts on access to financial services and employment, government impacts including misinformation and criminal justice exposure, health and safety, and effects on cultural norms and values.

The mistake most teams make is treating the assessment as a one-time gate at launch. Build a reassessment trigger calendar instead, with quantitative thresholds that force a fresh look automatically. A training data change exceeding 20 percent of the original dataset should trigger reassessment. So should a change in intended use or user population, a regulatory change affecting the system’s classification, or any incident involving the system. Embed these triggers directly into the change management workflow so reassessment happens on schedule, not when someone happens to remember. Regulators increasingly ask specifically for documented misuse scenarios and historical bias analysis, so write both down explicitly rather than folding them into a general risk narrative.

Design Responsible Development Processes With Kill Gates

Annex B.6 requires defined processes for responsible AI system design and development, covering the full lifecycle with clear approval gates. The word gate matters here. A process without a point where a system can be stopped is not a control, it is a description of what usually happens.

Documentation starts with rationale. Why is the system being built, a business case, a customer request, a government policy requirement. How will the model be trained and how will the data requirements actually be met. ISO/IEC 42001 also expects the organization to revisit these requirements if the system cannot operate as intended or if new information surfaces, including the discovery that the project has become financially infeasible partway through.

The development process itself needs to address a wide set of concerns in one coherent document. That includes lifecycle stages, testing requirements and planned testing means, human oversight requirements especially where the system affects natural persons, the stages at which impact assessments should run, training data expectations and approved data suppliers, the expertise required of developers, release criteria, approvals and sign-offs at each stage, change control, usability and controllability, and engagement with interested parties.

Design choices need their own record. Document the machine learning approach, algorithm selection, data quality considerations, hardware and software components, and the security threats considered across the lifecycle. ISO/IEC 42001 specifically names data poisoning, model stealing, and model inversion attacks as threats worth documenting against. The is a useful supplementary reference here for teams building on large language models, since it names related threats like prompt injection and training data poisoning at a level of technical detail the management standard was never built to provide.

Verification and validation close the loop. Define testing methodologies and tools, the selection of test data and how representative it is of the intended domain, release criteria, evaluation criteria for risk, reliability, safety, and performance, and methods for evaluating how interpretable the system’s outputs actually are.

A stage-gate checklist mapped directly to these requirements turns the paragraph above into something enforceable. Five gates work well in practice. Gate one covers rationale and requirements documentation. Gate two covers design choices and the security threat analysis. Gate three covers verification, validation, and a completed impact assessment. Gate four covers release criteria met and management sign-off obtained. Gate five covers an approved deployment plan and monitoring established. No system advances past a gate without the required artifact signed by the responsible role. Huwyler has implemented this exact structure using workflow automation in Jira and ServiceNow on separate engagements, and the platform matters far less than one rule, a gate that can be bypassed without a documented exception approval is not a gate.

Deploy and Monitor With Continuous Evidence Collection

ISO/IEC 42001 requires a deployment plan, ongoing operational monitoring, and event logging. Most organizations deploy the system and treat monitoring as something to build later. Later rarely comes, and the gap shows up during the first serious incident.

The deployment plan needs to account for environment differences. A system developed on premises and deployed in the cloud behaves differently than one developed and deployed in the same environment, and components deployed separately introduce their own coordination risk. Release criteria should require verification and validation measures passed, performance metrics met, user testing completed, and management approvals obtained, in that order, before anything goes live.

Once the system is running, ongoing monitoring needs to cover general errors and failures, whether the system performs as expected against production data, and drift. ISO/IEC 42001 makes a point worth repeating to any engineer who assumes a static model is a safe model, even systems that are not continuously learning can degrade in production due to concept drift or data drift, because the world the model was trained on keeps moving even when the model does not.

Event logging supports all of this. Logs should record the traceability of the system’s functionality and flag when performance falls outside intended operating conditions, including the time and date of each use, the production data the system operated on, and any output that fell outside the intended range. Retain logs according to data retention policy and legal requirements, and check jurisdiction-specific rules carefully for systems like biometric identification, which often carry additional logging obligations.

Failure needs a plan before it happens, not after. Document rollback procedures, feature disabling steps, update processes, and customer notification plans, along with standard operating procedures covering which events to monitor, how logs get prioritized and reviewed, how failures get investigated, and how recurrence gets prevented.

An operational monitoring runbook, built per Tier 1 system, turns all of this into something a human can actually follow at two in the morning. One document per system, covering the metrics to watch, the thresholds that trigger an alert, who gets notified at each escalation level, the response procedure for each alert type, and how to execute a rollback. Annotate the monitoring dashboard screenshots directly in the runbook, showing what each metric means and what a normal reading looks like. In Huwyler’s experience, an on-call engineer who receives a drift alert without knowing the acceptable range for that specific system cannot act on it no matter how skilled they are, and the runbook exists to close precisely that gap. Update it after every incident, since an outdated runbook is close to no runbook at all.

Data Management: The Foundation Most Teams Underestimate

ISO/IEC 42001 dedicates substantial guidance to data management across the AI lifecycle, and most implementation teams still treat it as secondary to model architecture. That ordering is backwards. A well-architected model trained on undocumented data is a liability wearing good engineering as a disguise.

Provenance comes first. Document where the data came from, when it was last updated, which category it falls into, training, validation, test, or production, how it was labeled, its intended use, its quality metrics, retention and disposal policy, known or potential bias issues, and the preparation steps already applied to it.

Acquisition needs its own record. Specify the categories of data needed, the quantity required, the source, internal, purchased, shared, open, or synthetic, and the characteristics of that source, static, streamed, gathered, or machine-generated. Record the demographics and characteristics of data subjects, including known or potential biases in who is represented, along with prior handling of the data, data rights covering IP and copyright, and the metadata attached to it.

Data quality requirements need a definition, not a vibe. defines data quality as the degree to which characteristics of data satisfy stated and implied needs under specified conditions, and that definition is worth adopting directly rather than paraphrasing loosely. For supervised or semi-supervised machine learning, define, measure, and improve quality across training, validation, test, and production data separately, and account for how bias affects both performance and fairness, adjusting the data or the model as needed rather than only one or the other.

Preparation methods deserve documentation too, statistical exploration, cleaning, imputation, normalization, scaling, labeling of target variables, and encoding, along with the criteria used to select those specific methods over the alternatives. And provenance recording itself follows a defined structure. defines a data provenance record as covering creation, update, transcription, abstraction, validation, transfer of control, sharing, and transformation of data, and a provenance process that only tracks creation and one later update falls well short of that definition.

A data card per dataset, one page covering source, acquisition date, size, known biases, quality metrics, preparation steps, and approved uses, makes all of this retrievable instead of theoretical. Version control the card alongside the data itself. Organizations that skip this step often discover during an audit that the person who originally acquired a dataset left the company years earlier and documented almost nothing about where it came from, leaving nobody able to answer a regulator’s basic question about training data provenance.

Document What You Communicate About the AI System

ISO/IEC 42001 requires organizations to determine and provide the information users need about an AI system, and this covers both technical documentation and plain notification. Getting this wrong usually looks like publishing a document nobody was told to read.

The information itself needs to cover the purpose of the system, clear notification that the user is interacting with an AI system, how to interact with or override it, technical requirements and limitations, human oversight needs, accuracy and performance information, relevant findings from the impact assessment covering potential benefits and harms for specific contexts or demographic groups, updates to how the system works, contact information, and educational materials.

Different users need different versions of this. A system administrator needs technical documentation. An end user needs an accessible explanation, not a specification sheet. ISO/IEC 42001 explicitly requires understanding what “understandability” means for each type of interested party, which means the same underlying facts get written twice in two different registers rather than once and hoped for the best. Decide what to provide and to whom using four criteria, intended use, reasonably foreseeable misuse, the expertise of the user, and the specific impact of the system on that user.

Users and outside parties also need a channel to report adverse impacts that fall outside what automated monitoring catches, perceived unfairness being the clearest example, since a model can hit every technical performance target and still produce outcomes a user experiences as unjust.

A transparency matrix, mapping each user type to the information they receive, the format they receive it in, and the method used to confirm they actually have access to it, turns this from a publishing exercise into something auditable. ISO/IEC 42001 asks organizations to validate that users have access to complete, current, and accurate information, and that validation step is what an auditor actually checks, not the existence of the document itself. A quarterly spot check, sampling users and confirming they can locate and understand what has been provided, produces the evidence that validation happened. Document the results each time.

Supplier and Third-Party AI Governance

Annex B.10 addresses supplier relationships, and the obligation does not disappear because the model, dataset, algorithm, or software library came from outside the organization. Sourcing a component from a vendor transfers effort. It does not transfer accountability.

Start by considering the different types of suppliers involved, what each one supplies, and the level of risk each relationship carries. From there, set selection criteria, define the requirements placed on suppliers, and decide how much ongoing monitoring and evaluation each relationship needs. A supplier providing a foundation model integrated into a high-stakes decision needs closer monitoring than one providing a labeling tool used in an early experiment.

Document how third-party components get integrated into the organization’s own systems. When a supplier’s component underperforms or produces impacts misaligned with the organization’s responsible AI approach, the response is to require corrective action, working with the supplier directly to achieve it rather than accepting the misalignment as a cost of doing business. Suppliers also need to deliver adequate documentation of their own, covering both the technical side and the information end users will see.

Customer-facing relationships run the same logic in the other direction. Understand what the customer expects and needs, clarify where responsibility sits between provider and customer, and communicate the limits of the system’s validated domain clearly. A model valid for one use case and pushed into another without that limitation being communicated is where a large share of AI-related customer disputes actually originate.

An AI-specific annex added to standard supplier contracts closes most of the gap that generic technology vendor agreements leave open. Cover model documentation requirements, performance reporting obligations, change notification requirements, particularly for model updates pushed silently through an API, audit rights specific to AI components, bias testing evidence requirements, incident notification timelines, and the right to require corrective action or terminate the relationship if the system’s impacts stop aligning with the organization’s responsible AI policy. Most procurement teams are still running generic technology contracts that never mention training data provenance, drift, or bias, and that gap is exactly what the annex is built to close.

The Reporting Mechanism Most Organizations Forget

ISO/IEC 42001 requires a process for reporting AI-related concerns, distinct from incident management, covering any concern about an AI system’s behavior, impact, or compliance rather than only confirmed technical failures.

The mechanism itself carries specific requirements. It must offer confidentiality or anonymity, or both. It must be available and actively promoted to everyone employed or contracted by the organization, not buried in an onboarding packet nobody reopens. It must be staffed by qualified people with real investigation and resolution authority. It must escalate to management in a timely way, protect the reporter from reprisal, deliver findings to the relevant governance function while preserving confidentiality, and respond within a reasonable timeframe.

Organizations do not need to build this from scratch. ISO/IEC 42001 explicitly allows extending an existing reporting mechanism, a whistleblower hotline or ethics channel already in place, to cover AI-related concerns rather than standing up a parallel system. provides additional guidance specifically on whistleblowing management systems for organizations building or extending this kind of channel.

The gap in most organizations is not the mechanism, it is awareness that AI concerns qualify for it. Add concrete AI-related examples to the existing channel’s guidance materials, a concern that a system is producing biased or unfair outcomes, a concern that a system is being used beyond its documented intended purpose, a concern about the data feeding a system. Without examples like these, employees often do not recognize that what they are worried about is exactly the kind of thing the channel exists for. Test the mechanism annually with a submitted test report, and measure response time, escalation, and resolution quality against what the standard expects.

Roles, Responsibilities, and Accountability

Vague ownership is the single most common reason AI governance programs fail an audit, more common than any missing technical control. ISO/IEC 42001 requires defined roles and responsibilities across the entire AI system lifecycle, covering risk management, impact assessments, asset and resource management, security, safety, privacy, development, performance monitoring, human oversight, supplier relationships, legal compliance, and data quality management.

Get the altitude right when assigning these roles. Set accountability too high and nobody acts on it, because the person holding it is three layers removed from the system’s daily operation. Fragment it too low and nobody sees the full picture, because each person only owns a narrow slice of a system that needs to be understood end to end. The right altitude sits with someone close enough to the system to act and senior enough to be heard.

Document every party involved across the lifecycle and what role each one plays, the parties providing data, the parties providing algorithms and models, the parties developing or using the system, and the parties accountable to the interested parties affected by it. When the data involved includes personally identifiable information, clarify PII controller and PII processor roles explicitly, using the definitions in , since ambiguity between controller and processor is one of the fastest ways a data protection question turns into a legal one.

A RACI matrix built per AI system, not one enterprise-wide matrix for AI in general, is the practical fix. A single enterprise RACI for AI governance is too abstract to act on. Each Tier 1 system needs its own matrix, published, reviewed quarterly, and updated within 48 hours of any personnel change. Audits have found high-risk AI systems running in production for six months with the accountable role left vacant after the responsible person departed, simply because nobody triggered a reassignment. A per-system RACI with a mandatory succession trigger is what prevents a system from running that long with nobody actually governing it.ers prevents this.

Quick Reference: Key ISO Standards Referenced in ISO 42001

Core:

ISO/IEC 42001:2023 (AI Management Systems)
ISO/IEC 23894:2023 (AI Risk Management)
ISO/IEC 22989 (AI Concepts and Terminology)
ISO/IEC 38507 (Governance of AI)

Data:

ISO/IEC 5259 series (Data Quality for Analytics and ML)
ISO/IEC 25024 (Data Quality Measurement)
ISO/IEC 19944-1 (Data Categories)
ISO 8000-2 (Data Provenance)
ISO/IEC TR 24027 (Bias in AI)

Development and Tooling:

ISO/IEC 23053 (ML Framework)
ISO/IEC 5338 (AI System Lifecycle)
ISO/IEC 25059 (AI Quality Model)
ISO/IEC TS 4213 (AI Performance Assessment)
ISO/IEC TR 24029-1 (Neural Network Robustness)

Security, Privacy, and Ethics:

ISO/IEC 27001 (Information Security)
ISO/IEC 27701 (Privacy)
ISO/IEC 29100 (Privacy Framework)
ISO/IEC TR 24368 (AI Ethics)
ISO 37002 (Whistleblowing Management)

Design:

ISO 9241-210 (Human-Centred Design)

Every tip above maps directly to ISO 42001 clauses and annexes. None require you to build from scratch if you already operate an ISO management system. The standard was designed to integrate with existing Annex SL frameworks.

The organizations that treat ISO 42001 as an extension of their existing management system rather than a standalone project cut implementation time in half and produce governance that actually works under audit pressure.

About the Author

The frameworks, tools, and implementation guidance described in this article are part of the applied research and consulting work of Prof. Hernan Huwyler, MBA, CPA, CAIO. These materials are freely available for use, adaptation, and redistribution in your own AI governance, risk management, and compliance programs. If you find them valuable, the only ask is proper attribution. If you like the content, please like the article and share it.

As a Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative risk modeling, predictive risk automation, and AI audit readiness for executive leadership teams, boards, and technical practitioners. His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe and internationally.

The 45 AI Threat Vectors That Your Security Team Probably Isn't Tracking

Thu, 12 Mar 2026 00:00:00 +0000

A Practitioner’s Field Guide

Most AI threat models are incomplete. Not slightly incomplete. Fundamentally incomplete.

Last year I reviewed the threat model for a financial services company deploying a credit decisioning AI. Their security team had identified seven threat vectors. Seven. They covered the obvious ones: data breaches, unauthorized access, denial of service. They missed 38 others, including 15 that were specific to AI systems and had no equivalent in their traditional IT threat catalog.

Three months after deployment, the model started producing subtly biased outputs. Not because of an external attack. Because a data scientist on the team had inadvertently introduced a feature engineering flaw that created a proxy variable for a protected characteristic. It was a negligence threat, an internal one, and it was not on anyone’s radar because the threat model only considered adversarial external actors.

This pattern repeats across almost every organization I work with. Security teams build threat models based on their experience with traditional systems. They focus on external attackers, malicious intent, and system-level exploits. They miss the internal negligence threats that cause the majority of real-world AI failures. They miss model-specific attack vectors that have no parallel in conventional cybersecurity. They miss the human and organizational threats that create the conditions for technical failures.

This post catalogs 45 distinct AI threat vectors organized across a two-dimensional taxonomy: intent (adversarial versus negligent) and target category (data, model, system, and human). Each vector includes a clear explanation, practical context, and implementation guidance. Use this as a working reference to audit the completeness of your own AI threat models.

Why Traditional Threat Models Fail for AI

Traditional threat modeling frameworks like STRIDE, PASTA, and even MITRE ATT&CK were designed for conventional information systems. They handle network attacks, authentication bypasses, privilege escalation, and data exfiltration well. They were not designed for systems where the “logic” is learned from data rather than written in code, where the attack surface includes the training pipeline itself, and where some of the most damaging threats come from well-intentioned internal teams making honest mistakes.

AI systems introduce three categories of threat that traditional frameworks handle poorly.

First, the model itself is an attack surface. Traditional systems have deterministic logic. If you protect the infrastructure and the data, the system behaves as designed. AI models are different. An attacker can manipulate the model’s behavior by carefully crafting inputs, without ever breaching the perimeter or accessing the infrastructure. They can extract sensitive training data by querying the model’s API. They can clone the model’s functionality through systematic probing. None of these attacks require the kind of infrastructure compromise that traditional threat models focus on.

Second, the training pipeline is a persistent vulnerability. Traditional systems are vulnerable during operation. AI systems are vulnerable during development. Poisoned training data, biased labels, flawed feature engineering, and compromised pre-trained models all introduce vulnerabilities before the system ever reaches production. By the time the model is deployed, the damage is already embedded in its weights.

Third, negligence threats cause more cumulative damage than adversarial threats. In traditional cybersecurity, the adversary is the primary concern. In AI, the internal team building and operating the system creates more risk through oversight, insufficient testing, poor documentation, and inadequate monitoring than external attackers do through deliberate exploitation. A threat model that only considers malicious actors misses the majority of the threat landscape.

Original implementation tip: When I first expanded an organization’s AI threat model beyond traditional categories, the security team pushed back hard. “We already cover insider threats,” they said. They did, but their insider threat model focused on malicious insiders who steal data or sabotage systems. It did not cover the data scientist who chooses features without considering proxy discrimination, the ML engineer who skips robustness testing under deadline pressure, or the architect who fails to build monitoring into the deployment pipeline. These are not insider “threats” in the traditional security sense. They are negligence risks that require fundamentally different controls. Treat them as separate categories in your threat model, not as subcategories of “insider threat.”

The Taxonomy Structure

This threat taxonomy organizes 45 vectors across two dimensions.

The first dimension is intent. Adversarial threats involve deliberate, intentional actions designed to compromise the AI system. Negligent threats involve unintentional actions or omissions that create vulnerabilities or cause harm. This distinction matters because adversarial and negligent threats require different controls. You defend against adversaries with detection, deterrence, and response. You defend against negligence with process, training, governance, and automation.

The second dimension is agent. External agents operate outside the organization’s boundary, including hackers, competitors, nation-state actors, and third-party vendors. Internal agents operate within the organization, including developers, data scientists, architects, operators, and end users.

Within each combination of intent and agent, threats target one of four categories: data (the information the system processes and learns from), model (the learned representations, algorithms, and parameters), system (the infrastructure, APIs, and operational environment), and human (the people who build, operate, and interact with the system).

The result is a comprehensive matrix that surfaces threats most organizations overlook because they fall outside the traditional “external attacker targeting our infrastructure” frame.

Adversarial External Threats: The Ones You Expect

This quadrant contains the threats most security teams already think about, plus several AI-specific vectors they likely do not.

Data Threats from External Adversaries

Two vectors target the data layer from outside the organization.

Data exfiltration occurs when attackers extract sensitive data from the AI system, either during training or inference. This differs from traditional data theft because the AI model itself can leak data. An attacker who gains access to model outputs, gradients, or confidence scores can sometimes reconstruct training data without ever accessing the database directly. The model becomes an unintentional data disclosure channel.

Data poisoning occurs when attackers introduce malicious or corrupted data into the training dataset. This is uniquely dangerous because the corruption happens before the model is deployed. If an attacker can influence any data source that feeds into the training pipeline, whether through compromised public datasets, manipulated web scraping sources, or tampered third-party data feeds, they can embed biases or backdoors that persist through every subsequent version of the model until the poisoned data is identified and removed.

Data poisoning is the adversarial data threat I worry about most because it is the hardest to detect and the longest-lasting in impact. Traditional data validation checks look for obvious anomalies: missing values, out-of-range entries, format errors. Poisoned data is designed to look normal. The individual data points are plausible. The corruption is statistical, not syntactic. Defending against it requires comparing training data distributions across time windows to detect subtle shifts, validating training data provenance to verify that sources have not been compromised, and testing model behavior on held-out validation sets from trusted sources. If your training data comes from any source you do not fully control, you have exposure to this vector.

Model Threats from External Adversaries

Six vectors target the model itself, and these are the threats most traditional security teams have the least experience with.

Deception involves crafting inputs designed to make the AI model produce incorrect predictions or classifications. Think of carefully modified images that cause a computer vision system to misclassify objects, or subtly altered text inputs that cause a natural language model to produce wrong outputs. The modifications are often imperceptible to humans but effective against the model.

Evasion is deception’s cousin, focused specifically on bypassing detection or classification mechanisms. An attacker modifies malicious inputs to evade an AI-based security system, such as altering malware signatures to bypass AI-powered threat detection or modifying fraudulent transactions to avoid AI-based fraud scoring.

Exploitation targets weaknesses in the AI system’s implementation rather than the model’s learned behavior. Insecure APIs, poor input validation, missing authentication, and misconfigured endpoints are the entry points. This vector bridges traditional cybersecurity and AI-specific risk because the vulnerabilities are conventional but the assets being targeted (models, training pipelines, inference endpoints) are AI-specific.

Inversion attacks reverse-engineer the model to infer sensitive information about training data. By carefully analyzing the model’s outputs across many queries, an attacker can deduce characteristics of the data the model was trained on. For models trained on medical records, financial data, or personal information, this vector directly threatens privacy even if the underlying database is perfectly secured.

Membership inference is related but distinct. Instead of reconstructing training data, the attacker determines whether a specific known data point was part of the training set. “Was this person’s medical record used to train your diagnostic model?” is a question that, if answerable through the model’s API, creates privacy and compliance exposure regardless of whether the attacker can reconstruct the full record.

Oracle attacks (also called model extraction or model stealing) involve an attacker querying the model extensively to understand its decision boundaries, then building a replica model that reproduces the original’s behavior without access to the training data. The attacker essentially steals the intellectual property embedded in the model through its public-facing API.

Transfer learning attacks exploit vulnerabilities in pre-trained models or the transfer learning process itself. Many organizations build their AI systems on top of pre-trained foundation models. If the foundation model contains embedded vulnerabilities, biases, or backdoors, every downstream model inherits them. This vector is growing in importance as more organizations build on third-party foundation models they did not train and cannot fully audit.

Of these six model-level threats, oracle attacks and transfer learning attacks are the two most underestimated. Oracle attacks are underestimated because organizations assume their model’s logic is protected by keeping the code proprietary. It is not. If the model is accessible through an API, its behavior can be replicated through systematic querying. Rate limiting helps but does not eliminate the risk. Transfer learning attacks are underestimated because organizations treat pre-trained foundation models as trusted components without verifying what is in them. I worked with a team that fine-tuned a publicly available language model for customer service automation. Nobody audited the base model for embedded biases or vulnerabilities. The assumption was “it’s from a reputable provider, so it’s safe.” That assumption is not supportable. If you use pre-trained models, document your trust assumptions about those models explicitly, test for bias and adversarial vulnerability in the fine-tuned model, and include the base model in your threat surface.

System Threats from External Adversaries

Eight vectors target the infrastructure and operational environment.

Advanced persistent threats involve state actors or organized crime groups using sophisticated tools to compromise the AI system over an extended period. These are the most resource-intensive attacks and typically target high-value AI systems in critical infrastructure, financial services, or defense applications.

API-based attacks exploit vulnerabilities in the APIs used to interact with the AI system. This includes injecting malicious data through API endpoints, exploiting authentication weaknesses, or abusing API rate limits to conduct model extraction.

Denial of service overwhelms the AI system with excessive traffic or resource demands. AI systems can be particularly vulnerable because inference workloads on complex models consume significant computational resources, making resource exhaustion attacks more efficient than against simpler services.

Model freezing attacks target the AI system’s update mechanism, preventing the model from receiving updates. A model that cannot be updated remains vulnerable to known threats and cannot adapt to data drift, effectively turning a dynamic system into a static one.

Model parameter poisoning introduces malicious updates or perturbations directly into the model’s parameters (weights, biases) rather than through training data. This vector is particularly relevant for federated learning systems where multiple parties contribute model updates.

Poorly designed APIs represent a threat vector that straddles adversarial and negligent categories. While the design flaw is internal, external attackers exploit it. Insecure API design that exposes model internals, lacks input validation, or provides excessive information in error messages creates the conditions for multiple other attacks.

Side-channel attacks exploit non-functional characteristics of the AI system, such as timing differences in inference responses, power consumption patterns during computation, or electromagnetic emissions, to extract information about the model or its data. These attacks do not target the model’s logic directly but extract information through observable physical or computational characteristics.

Supply chain compromise involves attackers tampering with hardware, software components, or third-party services in the AI system’s supply chain. Compromised ML libraries, poisoned pre-trained models distributed through public repositories, or tampered GPU firmware all fall into this category.

Supply chain compromise is the system-level external threat I spent the most time helping organizations address last year. The AI supply chain is broader and less controlled than most organizations realize. A typical ML pipeline might include open-source libraries (PyTorch, TensorFlow, scikit-learn), pre-trained models from public repositories (Hugging Face, GitHub), data from third-party providers, cloud services for training and inference, and container images from public registries. Each component is a potential entry point. The most practical defense is maintaining a software bill of materials (SBOM) for your AI systems that includes not just code dependencies but also model provenance, training data sources, and infrastructure components. When a vulnerability is discovered in any component, the SBOM tells you immediately which AI systems are affected. Without it, you are guessing.

Human Threats from External Adversaries

One vector targets the human element from outside.

Social engineering involves attackers manipulating developers, data scientists, or users into revealing sensitive information or interacting with the AI system in ways that compromise security. AI teams are often targeted because they have access to valuable IP (models, training data, feature engineering pipelines) and may not have received the same security awareness training as traditional IT staff. A data scientist who shares a model architecture diagram on a conference poster may not realize they have disclosed information useful for a model extraction attack.

Adversarial Internal Threats: The Ones You Underestimate

This quadrant contains only two vectors, but both are high-impact.

Human Threats from Internal Adversaries

Data sabotage occurs when insiders intentionally alter or destroy data used for training or inference. Unlike external data poisoning, an insider has legitimate access to data systems and can make changes that appear routine. A disgruntled data engineer who subtly modifies preprocessing scripts or alters label distributions can compromise model performance in ways that are extremely difficult to trace.

Subversion involves authorized developers or contractors intentionally sabotaging, exfiltrating, or manipulating the AI system. This goes beyond data sabotage to include embedding backdoors in model code, exfiltrating trained model weights for competitors, or introducing vulnerabilities that can later be exploited. The insider’s authorized access makes traditional perimeter defenses irrelevant.

Internal adversarial threats against AI systems are harder to detect than their equivalents in traditional IT for one specific reason: the normal behavior of an AI developer already includes activities that would be flagged as suspicious in other contexts. A data scientist routinely downloads large datasets, modifies data processing logic, changes model parameters, and deploys updated models. These are their job functions. Distinguishing between a legitimate model update and a sabotage event requires understanding what the model should be doing, not just what the developer is doing. The most effective control I have found is mandatory peer review for all changes to training data, feature engineering code, and model parameters before they reach production. Not automated testing, though that helps too. Human review by a second qualified person who can assess whether the change makes sense in context. This catches both intentional sabotage and unintentional errors.

Negligent External Threats: Your Vendors and Dependencies

This quadrant covers unintentional risks introduced by parties outside your organization.

Human Threats from External Negligence

Supply chain negligence occurs when third-party vendors introduce vulnerabilities through insecure libraries, dependencies, or tools used during AI system development, deployment, or maintenance. Unlike supply chain compromise (which is intentional), this vector reflects genuine negligence: a vendor fails to patch a library, releases an update with a security flaw, or provides tooling that does not meet security standards. The impact on your AI system is the same whether the vulnerability was introduced deliberately or carelessly.

Third-party data risk arises when organizations rely on external data sources that may be of poor quality, biased, or inadvertently altered. The third party is not acting maliciously. They simply do not maintain the data quality standards your model requires. Training on degraded external data produces degraded model performance, and the organization consuming the data may not detect the quality decline until outputs start failing.

System Threats from External Negligence

Outdated dependencies represent the use of unsupported software libraries in AI systems that introduce known, exploitable vulnerabilities. This is technically a negligence issue, an external provider stops maintaining a library, but the security impact is the same as an adversarial exploit because attackers actively scan for systems using deprecated dependencies.

Third-party data risk is the negligent external threat I encounter most frequently in practice. Organizations build models on external data feeds and assume the data quality will remain stable. It does not. I worked with a company whose fraud detection model degraded over four months because a third-party transaction data provider changed their data formatting without notification. The change was minor, a modification to how categorical fields were encoded, but it silently corrupted the feature engineering pipeline. The model’s accuracy dropped from 91% to 78% before anyone noticed. The fix was straightforward but the damage was done. For every external data dependency, establish a data quality SLA with the provider that specifies format, completeness, timeliness, and quality metrics. Monitor incoming data against those SLAs automatically. When a deviation occurs, alert before the data enters your training pipeline, not after your model degrades.

Negligent Internal Threats: Where Most AI Failures Actually Originate

This is the largest quadrant in the taxonomy, containing 24 of the 45 vectors. That distribution is not an accident. It reflects reality. The majority of AI failures in production stem from internal negligence, not external attacks.

Data Threats from Internal Negligence

Two vectors target the data layer through internal oversight.

Inaccurate data labeling occurs when developers fail to properly label training data during preparation. Labels are the ground truth the model learns from. If a medical imaging dataset contains mislabeled scans, the model learns to associate the wrong visual patterns with the wrong diagnoses. Unlike external data poisoning, this is not malicious. It is the predictable result of insufficient quality assurance in the annotation process, often driven by time pressure, undertrained annotators, or ambiguous labeling guidelines.

Bias in data occurs when data scientists or developers incorporate biased data during training, producing discriminatory or unfair outputs. This can result from historical bias embedded in the data itself (past lending decisions that reflected discriminatory practices), selection bias in how data was collected (underrepresenting certain populations), or measurement bias in how variables were recorded. The developers are not trying to create discriminatory outcomes. They are training on data that reflects existing inequities.

Bias in data is the negligent data threat with the highest regulatory and reputational impact. It is also the one where I see the most dangerous misconception. Teams believe that removing protected characteristics like race or gender from the training data eliminates bias. It does not. Other variables in the dataset frequently serve as proxies for protected characteristics. Zip code correlates with race. Job title correlates with gender. Part-time employment status correlates with caregiving responsibilities. Removing the protected characteristic while leaving the proxy variables in the feature set gives the appearance of fairness while producing the same discriminatory outcomes. The effective control is to test model outputs for disparate impact across protected groups, regardless of whether protected characteristics appear in the input features. Test the outputs, not the inputs.

Model Threats from Internal Negligence

Five vectors target the model through internal oversight.

Data and model drift occurs when developers fail to account for changes in data distribution or underlying concepts over time. A model trained on data from 2022 may not perform well on data from 2025 if customer behavior, market conditions, or the relationships between variables have shifted. This is not a one-time risk. It is an ongoing degradation that accelerates the longer a model operates without retraining or recalibration.

Feature engineering flaws result from unintentionally introducing vulnerabilities through poor feature selection. Selecting features that are easily manipulated by adversaries, including features that leak future information (data leakage), or failing to consider how feature distributions might shift in production all fall into this category.

Overfitting occurs when developers create models that fit too closely to the training data, learning noise and idiosyncrasies rather than genuine patterns. An overfit model shows excellent performance in testing and poor performance in production. From a security perspective, an overfit model is also more predictable to an adversary who understands its training data, making it easier to craft adversarial inputs.

Overfitting to noise is a specific variant where the model learns from irrelevant data rather than meaningful signal. The model performs well on training metrics but produces unreliable results in real-world deployment because it has memorized artifacts in the training data rather than learning the underlying relationship.

Unexplainability results when developers create AI systems too complex and opaque to understand or explain. This is a threat vector because opacity prevents detection of errors, biases, and vulnerabilities. A model whose decisions cannot be explained cannot be audited, cannot be debugged when it fails, and cannot satisfy regulatory requirements for explainability. Opacity does not cause harm directly, but it creates the conditions under which every other threat vector becomes harder to detect and address.

Data and model drift is the negligent model threat that causes the most cumulative financial damage because it is slow, silent, and continuous. I have never worked with an organization that detected drift proactively on their first AI deployment. They always detected it reactively, after business outcomes degraded enough for someone to notice. The detection lag ranged from three weeks to nine months depending on how closely business stakeholders monitored the model’s downstream effects. The fix is statistical monitoring of input feature distributions and output prediction distributions, compared against baseline distributions from the training period. When statistical tests detect a significant shift, trigger an alert. Do not wait for business outcome metrics to degrade, because by then the model has been making suboptimal decisions for weeks or months. Population Stability Index (PSI) is a good starting metric. Monitor it weekly at minimum for production models.

Human Threats from Internal Negligence

This is the richest subcategory in the entire taxonomy, containing 12 vectors. Each represents a different way that well-intentioned people create AI risk through oversight, insufficient skill, or organizational failure.

Inadequate documentation occurs when teams provide insufficient documentation for AI models, data sources, and decision-making processes. Without documentation, models cannot be maintained by anyone other than their original developer, security reviews cannot assess the system’s design assumptions, and regulatory compliance cannot be demonstrated.

Inadequate monitoring results from failing to build proper monitoring mechanisms to detect anomalies, adversarial activity, or model degradation in real time. An unmonitored model is a model whose failures go undetected until they manifest as business losses, customer complaints, or regulatory actions.

Inadequate maintenance occurs when teams fail to regularly update and maintain AI models, leaving them vulnerable to known attacks, data drift, and exploits as the system ages. Models, like all software, require ongoing maintenance. Unlike traditional software, model maintenance includes retraining, recalibration, and feature re-evaluation in addition to patching.

Inadequate testing results from failing to sufficiently test and validate models before deployment. This includes insufficient unit testing of data pipelines, absence of adversarial robustness testing, incomplete validation against held-out datasets, and failure to test for bias and fairness.

Inadequate training affects end-users and operators who receive insufficient instruction on how to interact with or manage AI systems. A model that is technically sound can still produce harmful outcomes if the humans using it do not understand its limitations, do not know when to override its recommendations, or cannot recognize when its outputs are unreliable.

Insecure design results from architects or developers failing to build secure AI systems from the start. This includes objective functions susceptible to manipulation, model architectures that leak information through their outputs, and deployment configurations that expose internal model details.

Insider threat (unintentional) covers authorized team members who inadvertently cause harm. A developer who accidentally pushes a model trained on test data to production. A data engineer who modifies a preprocessing script that breaks feature normalization. An operations team member who changes a configuration parameter without understanding its downstream effects.

Insufficient access control results from poorly managed access permissions that allow unauthorized access to models, data, or systems. This is a process failure, not a technology failure. The tools to enforce access control exist. The organization simply has not implemented them with sufficient rigor for AI-specific assets.

Lack of governance occurs when organizations fail to establish proper governance frameworks for AI development and deployment. Without governance, teams operate independently, security practices are inconsistent, accountability is undefined, and risk accumulates without visibility.

Over-reliance on AI results from decision-makers depending on AI outputs without sufficient human oversight or fail-safes. When users treat AI predictions as infallible, they stop applying the human judgment that catches model errors. This vector is particularly dangerous in high-stakes domains like healthcare, criminal justice, and financial services.

Unclear AI accountability occurs when nobody is clearly defined as accountable for AI-related decisions, model performance, or security. When accountability is unclear, risks go unmanaged because everyone assumes someone else is responsible.

Of these 12 human negligence vectors, inadequate monitoring and unclear accountability are the two that create the most cascading damage. They amplify every other threat in the taxonomy. An adversarial attack against an inadequately monitored system succeeds for longer. Data drift in a system with no accountable owner goes unaddressed for months. Bias in a model that nobody monitors against fairness metrics persists indefinitely. If you can only address two human negligence vectors immediately, address these two. Assign a named individual, not a team, as accountable for each production AI system. Then build monitoring that runs at the same speed as the model’s decision-making. Everything else becomes more manageable once you have visibility and ownership in place.

System Threats from Internal Negligence

Five vectors target the system infrastructure through internal oversight.

Inadequate incident response results from failing to develop or implement effective response plans for AI-specific incidents. AI incidents differ from traditional IT incidents. A model producing biased outputs is not a “system down” event. It does not trigger the same alerts. It requires different diagnostic procedures and different remediation steps. If your incident response playbook does not include AI-specific scenarios, your response will be improvised when it matters most.

Inadequate logging results from insufficient logging and monitoring infrastructure that makes it difficult to detect, investigate, or respond to incidents. If you cannot see what the model received as input, what it produced as output, and how its behavior has changed over time, you cannot diagnose problems or provide evidence for regulatory investigations.

Insecure data storage results from failing to implement secure storage for AI-specific data assets. Training data, model weights, feature engineering code, and hyperparameter configurations all contain sensitive intellectual property and potentially personal data. Storing them with the same (or lesser) security controls as general-purpose data creates exposure.

Insufficient redundancy results from failing to build AI systems with adequate fail-safes. A single point of failure in the model serving infrastructure, the data pipeline, or the monitoring system can take down the entire AI capability. AI systems often have complex dependency chains that create hidden single points of failure.

Misconfiguration results from incorrectly configuring security settings, environments, or system components. Cloud environment misconfigurations, exposed model endpoints, overly permissive IAM roles, and unencrypted data storage are the most common variants.

Inadequate incident response for AI systems is the system-level negligence threat I have spent the most time remediating. Traditional incident response plans categorize incidents by severity and system type, but they almost never include AI-specific incident categories. What do you do when a model starts producing outputs that are statistically different from its validation period behavior? What do you do when a fairness audit reveals disparate impact? What do you do when you discover that training data was contaminated three months ago and every model version since then is potentially compromised? These are AI incidents that require AI-specific response procedures. Build an AI incident response appendix for your existing plan. Include at minimum: model rollback procedures, retraining triggers, bias investigation protocols, adversarial attack containment steps, and stakeholder notification procedures. Then tabletop exercise these scenarios. The first time you run through an AI incident scenario, you will discover gaps in your response capability that are fixable before a real incident occurs.

Cross-Cutting Implementation Tips

These apply across all four quadrants of the threat taxonomy.

Assess your taxonomy coverage quarterly. AI threat vectors evolve as attack research advances, new model architectures emerge, and regulatory requirements expand. A taxonomy that was comprehensive six months ago may have gaps today. Assign someone to monitor AI security research (MITRE ATLAS updates, conference proceedings from NeurIPS and USENIX Security, regulatory guidance from NIST and the EU AI Office) and flag new vectors that should be added.

Original implementation tip: When reviewing your taxonomy coverage, do not just ask “have any new threat vectors emerged?” Also ask “have any of our existing vectors changed in severity or likelihood?” The relative importance of threat vectors shifts as your AI systems mature. Early in deployment, development negligence threats (inadequate testing, insecure design) are most relevant because the system is new and untested. Six months into production, operational negligence threats (inadequate monitoring, data drift, inadequate maintenance) become dominant. Twelve months in, adversarial threats increase as your AI system becomes a known, valuable target. Reassess priority rankings at each quarterly review.

Balance adversarial and negligence controls in your budget. Security teams naturally gravitate toward adversarial controls because they are more dramatic and more familiar. Adversarial robustness testing, penetration testing, red-teaming: these feel like “real” security work. Process controls, governance frameworks, training programs, and documentation standards feel like bureaucracy. In practice, the negligence controls prevent more incidents.

Original implementation tip: I track a simple metric with every organization I advise: the ratio of AI incidents caused by adversarial action versus negligence. Across 14 organizations over three years, the ratio has consistently been approximately 15% adversarial and 85% negligence. Yet budget allocation for adversarial controls versus negligence controls is typically inverted: 60-70% on adversarial defenses and 30-40% on process and governance. Reallocate to match the actual threat distribution. This does not mean reducing adversarial defenses. It means increasing investment in monitoring, documentation, testing processes, training, and governance until the budget reflects where incidents actually originate.

Map each vector to specific assets and controls. A threat vector without a corresponding asset mapping tells you what could happen but not where it could happen to you. A threat vector without a corresponding control tells you what to worry about but not what to do. For each vector in this taxonomy that applies to your AI systems, document three things: which specific assets are exposed, which controls currently mitigate the vector, and what residual exposure remains.

The most effective way I have found to operationalize a threat taxonomy is to create a traceability matrix with four columns: threat vector, exposed assets, current controls, and residual risk rating. Populate this matrix for every production AI system. When a new asset is deployed, add rows. When a new threat vector is identified, add rows. When a control is implemented or modified, update the current controls column and reassess residual risk. This matrix becomes the working document for your AI security program. It tells you at any point what threats you have addressed and what gaps remain. Without it, the taxonomy is an intellectual exercise. With it, the taxonomy drives action.

Key References and Standards

This threat taxonomy draws from and aligns with the following authoritative frameworks.

MITRE ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems) provides the primary reference taxonomy for AI-specific adversarial threats.

NIST AI RMF (AI 100-1) provides the risk management framework for identifying and addressing AI threats across the lifecycle.

ISO/IEC 27005:2022 provides the information security risk management process for integrating AI threats into enterprise risk assessment.

ISO/IEC 23894:2023 provides AI-specific risk management guidance including threat identification.

ISO/IEC 42001:2023 provides AI management system requirements for governance of the threat landscape.

OWASP Machine Learning Security Top 10 provides a practitioner-focused list of common ML security threats.

EU AI Act (Regulation 2024/1689) establishes regulatory requirements that make several of these threat vectors compliance-relevant.

NIST SP 800-30 Rev. 1 provides guidance on threat identification and risk assessment methodology.

ENISA Threat Landscape for AI provides European regulatory perspective on AI-specific threats.

Using This Taxonomy to Find Your Gaps

Organizations that treat this taxonomy as a reference list will read it, nod, and return to their existing threat models unchanged. They will continue to overweight adversarial external threats because those threats are familiar and dramatic. They will continue to underweight internal negligence threats because those threats feel mundane and uncomfortable to discuss. When an AI failure occurs, and it will, they will discover that the vector was sitting in a taxonomy they read but never operationalized.

Organizations that treat this taxonomy as an audit tool will do something different. They will take each of their production AI systems and map every applicable threat vector to the specific assets, existing controls, and residual risk for that system. They will discover gaps, mostly in the negligent internal quadrant, and they will prioritize closing them. They will update their incident response plans to include AI-specific scenarios. They will build monitoring that catches negligence-driven failures before they reach customers. They will assign accountability for each production model to a named individual who cannot hide behind a team name.

The threat your AI system faces tomorrow is almost certainly already in this taxonomy. The question is whether you have mapped it to your assets, built controls for it, and assigned someone to watch for it.

Which quadrant of this taxonomy has the least coverage in your current AI threat model? For most organizations, the answer is negligent internal. That is where your biggest gap probably lives.

Cybersecurity |

How ISO 24970 and prEN 18229-1 Turn Post-Deployment Chaos Into Auditable Evidence

When AI Systems Fail, Logs Tell the Story

Why AI Logging Is Different From Application Logging

The Regulatory Context: ISO 24970 and prEN 18229-1

EU AI Act Requirements for High-Risk Systems

Core Concepts: Logs, Log Entries, and Logging Components

What Goes Into an AI System Log

Logging Components and Their Role

Logging in Context: Operational and Management Integration

Structure and Content of Log Entries

The Process of AI System Logging

General Requirements for AI System Logging

Technical Documentation for Logging

AI System Logging Fields for Governance Professionals

Designing the Logging System with Risk as the Primary Driver

Traceability Through Timestamps and Identifiers

Additional Logging Functions for Specific Use Cases

Anomaly Monitoring of the Logging Component Itself

Triggers for Logging: Operation, Monitoring, and Oversight

Triggers From Operation

Triggers From Automated Monitoring

Triggers From Human Oversight

Required Information in Log Entries

Recommended Information in Log Entries

Storing and Access to Logs

Requirements for Third-Party Access

Access for AI Users and Providers

Practical Implementation, Start With Risk and Work Backward

AI System Logging Control Matrix

Logs Are Not Optional Anymore

The prEN 18228 Problem: Why Your AI Risk Assessment Will Fail the First Real Test

The Standard Brings Structure. It Does Not Solve the Decision Problem.

The Risk Definition Works for Products, but AI Behaves Differently.

The Terminology You Need to Understand Before You Start

Terms Relating to the EU AI Act

Terms Related to the Risk Management System

Terms Relating to Testing

Terms Related to Users and Affected Persons

Terms Related to Risk

How the Standard Actually Runs Risk Management

A Continuous Process, Not a One-Time Assessment

Management Owns the Process, Not Just the Outcome

Risk Acceptability Is a Policy Decision, not a Technical Detail

Competence Is a Collective Requirement

Defining Risk Acceptability Criteria

How Criteria Are Established and Updated

Looking at the Overall Residual Risk

Planning the Work

The Risk Management File as the Backbone

Risk Management Process and AI System Life Cycle

General Requirements for AI Risk Analysis

Risk Identification from the Intended Purpose

Risk Identification: Reasonably Foreseeable Misuse

Risk Identification: AI System Characteristics Related to Risks

Risk Identification: Hazards, Risk Scenarios, and Hazardous Situations

Risk Estimation

Risk Evaluation Turns Analysis into Decisions

Testing Is Evidence, Not Validation Theater

Test Plan

Real-World Conditions Testing

Test Monitoring and Test Reporting

Risk Control Follows a Clear Hierarchy

Implementation and Verification of Risk Control Measures

Residual Risk and When to Stop

Completeness of Risk Control

Looking at the AI System and Residual Risk as a Whole

Reviewing the Process, Not Just the Outcome

Learning from the Pre-Market and Post-Market Activities

Information Collection

Information Review

Actions to Take

Understanding How AI Risks Unfold in Practice

Example 1: Skin Cancer Detection App

What the system does

What causes the hazard

What the hazard is

How the risk scenario unfolds

What the hazardous situation looks like

What harm results