Ai-Risks |

New Book AI Risk Quantification: A Practical Roadmap for Chief AI Officers

Thu, 03 Sep 2026 00:00:00 +0000

A practitioner framework for turning ambiguous AI exposure into decision-grade evidence.

AI governance has a credibility problem. Many teams still document model inventory, assign ordinal risk ratings, and circulate dashboards without changing a single deployment decision. The evidence is usually a color-coded matrix that cannot support financial, compliance, or safety decisions. If you serve as a Chief AI Officer or an AI GRC professional, you have likely felt that gap during a board review or a product readiness meeting.

Adding more governance layers does not solve this. The practical answer is to estimate AI risk as a probability distribution, express consequences in financial and operational terms, and use those estimates before the decision closes. That is the core discipline in The Risk Management Blueprint by Hernan Huwyler. You can preview the first four chapters at .

The book is not an academic diagnosis. It is a practitioner reference for building quantitative risk models across predictive, generative, and agentic systems. It gives AI leaders the same capital allocation language used by treasury and insurance functions, which is exactly what AI governance has been missing.

Why AI Governance Needs Quantification, Not Color

Heat maps are labels, not measurements. When a team multiplies an ordinal likelihood of 3 by an impact score of 4, the result is 12, but that arithmetic has no statistical meaning. You cannot aggregate it with other scores, compare it across model classes, or defend it to a regulator. ISO 31000 defines risk as the effect of uncertainty on objectives. It does not require matrices, and it does not ask you to pretend ordered categories are numerical data. ISO/IEC 23894 extends this thinking to AI risk management by requiring assessment methods suited to AI uncertainty. The NIST AI Risk Management Framework also organizes AI governance around Govern, Map, Measure, and Manage functions, placing measurement at the center rather than the end of the process.

AI systems fail through data drift, adversarial inputs, reward misspecification, overfitting, and unauthorized use. Those failures do not fit neatly into a five by five grid. They require scenario modeling, sensitivity analysis, and continuing validation.

What Changes When You Quantify AI Risk

A quantitative AI risk practice changes the conversation from vague exposure to decision readiness. You start by defining the objective you are protecting, such as model availability, patient safety, customer data integrity, or regulatory standing. You then model the failure path that could break that objective. For each path, you estimate frequency and severity as distributions. A beta-PERT distribution can capture sparse expert judgment. A lognormal or compound Poisson-lognormal model can capture high variance and tail behavior.

Monte Carlo simulation combines those distributions into a loss exceedance curve. The curve tells you the probability of losing a given amount over a time horizon. It gives your CFO a number that can be tested, compared, and priced. It also reveals which risk sources dominate the tail, which is rarely the risk that draws the most attention in committee.

Expert judgment remains essential because few organizations have enough AI incident history to rely on old data alone. The book shows how to calibrate that judgment with seed questions, equivalent bet tests, and absurdity tests. The equivalent bet test asks whether you would accept a wager based on your stated probability. The absurdity test asks whether your estimate implies outcomes no experienced operator would believe. These are simple techniques that turn opinion into usable evidence.

Governing Predictive, Generative, and Agentic AI Before Deployment

Standard IT checklists break down when applied to AI systems. Predictive models can drift after deployment. Generative models can produce harmful or biased outputs. Agentic systems can take actions without a human in the loop. Governance must match the paradigm.

Before a system ships, AI GRC teams should map trust boundaries. Ask where the model receives untrusted input, where output becomes an action, and where a human can still intervene. Use model cards to record intended use, performance, limitations, and safety considerations. Conduct adversarial red teaming for the specific failure modes of your deployment, not just generic prompt tests. For high-risk systems under the EU AI Act, these artifacts become regulatory evidence. ISO/IEC 42001 provides a management system structure for maintaining them over the system lifecycle.

Fundamental rights impact assessments are a practical tool for high-impact AI. They force the team to document affected groups, potential harms, and mitigation controls before launch. This is not paperwork. It is the difference between a defensible product decision and a reactive regulatory response.

Model Risk and Machine Learning Controls That Scale

After deployment, AI risk management becomes a monitoring problem. The model is still learning from live data, and the environment changes. You need forward-looking indicators that catch drift before financial or reputational damage occurs.

Technical teams should track ROC-AUC, precision, recall, F1 score, and a population stability index. Explainability methods such as SHAP and LIME help model owners understand why a prediction changed. Monitoring a metric is not enough. You need a backtesting routine that compares predicted loss distributions against observed outcomes. Brier scores, exceedance tests, and clustering tests can identify models that have quietly gone stale.

One practical tip is to define a crisis trigger matrix before you need it. Decide in advance which metric breach moves the model into a hold state, who must approve a retrain, and how the business continues without the model. That precommitment removes ad hoc pressure during an incident and keeps the response aligned with the risk appetite you set.

Agentic AI Controls for High Velocity Risk Response

Agentic AI introduces a new control problem. A model that can call APIs, move data, or issue instructions operates at machine speed. Human review cannot catch every action. The answer is not to block agentic systems. The answer is to constrain their action space.

Autonomous responses should start in shadow mode, where the agent proposes actions that humans review. Once promoted, each control should use deterministic action schemas that define what the agent may do, under what conditions, and with what resource limits. Algorithmic circuit breakers should cap frequency, spend, data movement, and user impact. Markov decision process modeling can help design these policies, but the most important design choice is the boundary of acceptable action. If an action would change a customer, a legal position, or a financial obligation, keep a human checkpoint in place.

This is the modern version of separation of duties. It gives you speed without giving away accountability.

AI Cyber, Third-Party, and Compliance Exposure

AI risk is also operating risk. A model hosted by a vendor creates third-party dependency. A vector database with customer conversations creates cyber exposure. A high-risk classification under the EU AI Act creates compliance obligations. Each of those can be quantified.

Map your AI supply chain and measure replaceability. The cost of a model provider is not just the invoice. It includes switching cost, retraining cost, revalidation cost, and the risk of losing institutional knowledge. A replaceability index makes that exposure visible to procurement and the board. For cyber risk, convert a model API outage or a data extraction event into a financial loss estimate using downtime by the hour and incident response costs. For compliance, track obligations in a register and price compliance debt before accepting new commitments. The EU AI Act requires different levels of conformity assessment depending on risk category. If you cannot fulfill those obligations operationally, the commitment is a hidden liability, not a roadmap item.

From Risk Register to Risk-Adjusted AI Plan

AI project failures are often not technical surprises. They are plan failures. The team commits to a date and a budget without modeling the chance that the data is not ready, the model underperforms, the regulator asks questions, or the vendor changes pricing. Risk-adjusted planning reverses that sequence.

Pre-mortem scenario discovery asks what would end the project before launch, not after. Reference class forecasting uses comparable prior projects to calibrate a realistic range for cost and schedule. Integrated cost-schedule simulation lets you see the joint probability of finishing late and over budget, instead of treating those risks as independent. Real options logic helps you stage high-stakes AI investments so you can stop or accelerate as evidence arrives.

For Chief AI Officers, this is the difference between defending a roadmap and adjusting it intelligently when the facts change.

What Chief AI Officers and AI GRC Teams Should Do Next

Start with one decision that matters. Pick a high-stakes AI deployment or a compliance gap that already worries you. Model the objective, the failure path, and the loss distribution. Run the first Monte Carlo simulation with open-source Python tools. Test the results with the business owner. Then use that one model to inform the next governance decision.

The Risk Management Blueprint provides the step-by-step methods, code, and governance structures to do this across your portfolio. Preview the first four chapters at or access the full book at .

Professionals who master this shift will replace opinion-driven AI risk ratings with decision-ready quantification. They will not just document AI governance. They will change how AI investments are made.

What Every Chapter Actually Delivers

A complete map of the tools, models, and decision frameworks organized by part and chapter for readers who want to know exactly what they are getting before they open the book.

Part 1. Foundations: Risk Management as Decision Support

Chapter 1. The Expensive Risk Theater, page 1

Conventional 5x5 matrices and traffic-light dashboards look busy, but there is no real math behind the colors. This opening chapter proves that ordinal scoring is statistically invalid the moment you multiply or add rank orders together, and it names the pattern for what it is: risk theater, a set of rituals that document a process without ever changing a decision. It exposes measurement inversion, the habit of tracking whatever is easy to count while ignoring the uncertain variables that actually determine whether an objective is met, and it draws a hard structural line between internal controls that protect existing value and risk management that should be creating new decision value. The chapter closes by describing the watermelon risk problem, where a dashboard reads green right up until a real event cuts it open and reveals a red failure underneath.

Technical toolkit: ordinal scale multiplication analysis, range compression, consensus convergence in group workshops, measurement inversion diagnostics, value protection versus value creation framing, 5x5 risk matrix and heat map deconstruction, continuous versus discrete distribution logic, semantic ambiguity in verbal probability language, horizon mismatch between short-term ratings and long-term exposure, vertical inconsistency testing across ordinal categories.

Chapter 2. Assess the Plan, Not the Danger List, page 25

Stop cataloguing random worries and start asking the one question that matters: will this business plan actually hit its numbers. This chapter reframes the profession’s central question, replacing open-ended fear lists with a disciplined separation between aleatory uncertainty, the irreducible randomness in a system, and epistemic uncertainty, the knowledge gaps a team can actually close with better data. It walks through the cognitive biases that quietly distort every forecast, including overconfidence, anchoring, groupthink, availability bias, confirmation bias, and the planning fallacy that leads teams to systematically underestimate cost and time while overstating benefit.

Technical toolkit: pre-mortem scenario discovery, reference class forecasting, expected value of information, the equivalent bet test, the absurdity test, inside view versus outside view framing, formal dissent and designated challenger roles, choice architecture for comparable decision options, stochastic dominance testing, proportional depth analysis for tiering how much modeling rigor a decision deserves, decision rationale documentation, the Delphi method.

Chapter 3. From Risk Registers to Risk-Adjusted Plans, page 42

This chapter builds the practical bridge from static, disconnected spreadsheets to plans that move as new information arrives, a shift that matters more every year as basic compliance checklisting gets automated out of the profession. It defines three active roles a risk manager must rotate through to stay relevant: internal consultant, behavioral facilitator, and quantitative modeler. It also introduces a three-tier cascade model that traces how a direct first-tier loss triggers indirect second-tier consequences and, left unmanaged, a systemic third-tier reputational or liquidity failure.

Technical toolkit: the risk-adjusted business model, three-tier cascade loss modeling, indicator variables and binary trigger logic for cascading consequences, triangular distribution, PERT and beta-PERT distribution, copulas and correlation matrices, expected shortfall, value at risk, Monte Carlo simulation, early architecture for automatic control responses executed by autonomous agents.

Part 2. Core Operating Framework: The Quantitative Engine for Decisions

Chapter 4. Model the Failure, Protect the Objective, page 65

Open-ended brainstorming produces long lists and weak prioritization. This chapter replaces it with a disciplined scenario formula that links actor, trigger, vulnerability, and cost range into a single, model-ready input instead of a vague bullet point. It builds the case for identifying vulnerabilities before threats, since a well-understood weakness usually points straight to the range of actors who could exploit it, and it introduces contamination controls, silent writing, and round-robin input collection to stop senior voices from anchoring the whole exercise before junior staff speak.

Technical toolkit: the structured risk scenario formula, causal bow-tie analysis, the three lines model, diagnostic evidence versus low-diagnosticity data, SWIFT structured what-if technique, adversarial red teaming, analysis of competing hypotheses, detailed fault tree construction, networked governance review to force an outside view onto optimistic project teams.

Chapter 5. Measure What Seems Unmeasurable, page 99

This is the direct answer to the most common objection in quantitative risk work: the claim that historical loss data does not exist. The chapter proves that any risk material enough to matter is observable through proxy variables and can be parameterized into a probability distribution using calibrated expert judgment. It covers goodness-of-fit analysis for finding the statistical fingerprint hidden in messy data, and it addresses tail dependence, the way variables that look unrelated in normal conditions suddenly move together under stress.

Technical toolkit: calibrated expert elicitation, the equivalent bet test, the absurdity test, the Delphi method, Fermi decomposition, analytical convolution of distributions, tornado charts and contribution-to-variance sensitivity analysis, model validation through stress testing and back-testing, the full loss distribution taxonomy spanning Poisson, Bernoulli, and negative binomial for discrete events, lognormal, power law, Weibull, generalized Pareto, and log-logistic for heavy tails, and triangular and beta-PERT for bounded estimates.

Chapter 6. Prioritizing Against Capacity, Not Intuition, page 127

Risks get ranked by the actual mathematical pressure they place on solvency and liquidity, not by which item gets the loudest voice in a committee room. The chapter introduces temporal prioritization through velocity profiles, weighing detection lag and response time against how quickly a risk can spread, and it distinguishes structural network modeling from simple statistical correlation when identifying which failures cascade fastest through an organization.

Technical toolkit: the baseline capacity prioritization matrix, time-to-survive versus time-to-recover modeling, tiered confidence intervals from P50 targets through P95 and P99 board-level escalation thresholds, network contagion analysis, keystone hub and super-spreader identification, adversarial risk analysis using Bayesian Stackelberg games, info-gap decision theory for genuinely unknowable probabilities, the return on mitigation index, real options valuation, the risk-reward efficient frontier chart.

Chapter 7. Choosing the Risk Response That Pays, page 151

Every risk response is an economic capital allocation decision, and this chapter treats it that way from the first page. It introduces the separation principle, which requires a team to assess exposure objectively before any argument over preferred fixes begins, preventing the common failure where a favored solution quietly distorts the risk assessment that is supposed to justify it. It also reframes probability communication around natural frequencies, showing why “30 out of 200” lands better with an executive audience than a percentage or a qualitative label ever will.

Technical toolkit: the four-T operational strategies of terminate, treat, transfer, and tolerate, upside financial strategies including covariance diversification, hedging, edge exploitation, portfolio optimization, and risk structuring, real options valuation for staging high-stakes commitments, option pricing concepts including basis risk and drawdown stops, decision journals and risk retrospectives for auditing decision quality independent of outcome.

Chapter 8. Monitor What Matters, page 179

The quarterly review calendar gets replaced with continuous, event-driven monitoring built to surface signals before damage occurs rather than after. The chapter draws a sharp line between activity metrics, which document that something happened, and true oversight indicators, which change behavior in real time. It also builds an attention funnel that ruthlessly filters what actually reaches the board, since flooding executives with every metric guarantees that none of them get read.

Technical toolkit: leading versus lagging indicator design, key risk indicators, the crisis trigger matrix for automatic authority shifts at predefined thresholds, data reconciliation across telemetry feeds, the ten-step back-testing protocol for reality-checking predicted distributions against observed outcomes.

Chapter 9. Updating Risk Before It Updates You, page 198

Risk estimates expire, and this chapter treats every probability distribution as a forecast with a shelf life rather than a settled conclusion filed away until next year. It teaches Bayesian updating as the practical mechanism for revising a distribution the moment new evidence arrives, and it applies the three horizons model, distinguishing known operational risks from weak emerging signals and from genuinely transformational shifts still years out.

Technical toolkit: Bayesian updating, priors and posteriors, equivalent prior sample size weighting, the dynamic risk observatory operating model, the living belief register, cross-impact analysis across risk domains, the Brier score for calibration and resolution, exceedance testing, clustering testing, the probability integral transform for checking distributional fit.

Part 3. Domain Applications: One Framework, Sharp Edges for Each Risk Type

Chapter 10. AI Risks: Assess AI Before It Acts, page 222

Standard IT checklists break down the moment they meet a non-deterministic system that adapts after deployment, and this chapter builds the assessment approach those checklists were never designed for. It classifies artificial intelligence by paradigm across predictive, generative, and agentic systems, since each fails in a fundamentally different way, and it maps a layered risk taxonomy running from IT baseline risk through AI-common risk, paradigm-specific risk, domain risk, and finally legal and human rights exposure. The chapter treats autonomy level as a risk variable in its own right, tracking how far delegated authority has drifted from meaningful human oversight.

Technical toolkit: trust boundary mapping across data pipelines, context windows, and third-party APIs, model cards and technical dossiers, human rights impact assessments, adversarial AI red teaming, model drift, data drift, and concept drift monitoring, lifecycle assessment across pre-procurement, development, pre-production, and production stages, combined human-AI decision accuracy and override rate tracking, a structured vulnerability taxonomy covering training data memorization, weak transfer validation, black-box vendor dependency, and insufficient resource monitoring, and a structured threat taxonomy covering prompt and cross-document injection, model extraction, model weight tampering, dependency confusion, and guardrail probing.

Chapter 11. IT Risks: Quantify Cyber Risk Exposure, page 273

Patch counts, vulnerability tallies, and blocked-alert dashboards get converted into the financial loss language a board and an audit committee actually understand. The chapter separates loss event frequency from loss magnitude in the same actuarial structure insurers use, and it moves the unit of analysis from isolated asset-by-asset reviews to full attack chains and correlated failures, since a single control gap rarely causes a loss on its own. It also builds out the three cyber layers, physical infrastructure, logical network, and information, so a technical vulnerability list connects directly to a financial impact statement.

Technical toolkit: the quantitative business impact assessment for pricing downtime by the hour, enterprise attack surface mapping, attack graph construction to locate high-value control chokepoints, a multidimensional vulnerability inventory spanning technical, process, human, supplier, and environmental categories, asset-to-service aggregation for translating technical outages into service-level cost, loss exceedance curves for optimizing cyber insurance policy limits, network centrality measures, shadow IT and shadow AI discovery.

Chapter 12. Compliance Risks: Price Obligations Before Commitment, page 294

Compliance stops being a backward-looking administrative exercise and becomes a forward-looking economic one. The chapter introduces compliance debt, the hidden, interest-bearing liability an organization accepts the moment it signs a contractual or regulatory commitment without the operational capability to actually fulfill it. It maps the full obligation universe an organization carries, separates explicit contractual promises from implicit stakeholder expectations, and builds a five-tier consequence model running from direct fines through formal sanctions, remediation cost, commercial fallout, and long-term strategic damage.

Technical toolkit: the obligation universe compliance register, pre-commitment risk assessment, jurisdictional conflict analysis, five-tier compliance loss propagation modeling, decision trees for calculating the expected value of self-reporting versus non-disclosure, enforcement dynamics and probability of detection modeling, clustered violation and regulatory enforcement wave analysis, return on compliance investment, graph-based obligation dependency mapping, alignment with ISO 37301 compliance management system requirements.

Chapter 13. Project Risks: Know the True Odds of Delivery, page 322

This chapter exposes and corrects one of the most persistent errors in project management: treating cost and schedule as if they move independently of each other. It builds integrated cost-schedule risk analysis so both variables get simulated jointly, calibrated against a cone of uncertainty that narrows in step with project maturity classes, and it explains why a single optimistic completion date is functionally useless compared to a full probability curve.

Technical toolkit: integrated cost-schedule risk analysis, progressive elaboration, the AACE cone of uncertainty and cost estimate classes, time-dependent versus time-independent cost drivers, joint cost-schedule S-curves and joint confidence levels through Monte Carlo simulation, calculated cost contingency and schedule reserve at P70, P80, or P90 confidence, tornado diagrams and criticality analysis, resource-loaded critical path method scheduling, work breakdown structure design, assumption registers, reference class forecasting.

Chapter 14. Third-Party Risks: Assess Dependency Before It Fails, page 346

Vendor spend metrics and questionnaire scores tell you almost nothing about real dependency, and this chapter replaces them with a framework built around replaceability and true operational reliance. It maps dependency across multiple channels at once, service delivery, technology, data, regulatory exposure, financial exposure, reputational exposure, and jurisdictional concentration, and it pushes visibility down into fourth-party and fifth-party relationships that most vendor programs never see.

Technical toolkit: the replaceability index for pricing vendor lock-in directly into the risk assessment, risk-adjusted total cost of ownership, capability mapping and chokepoint analysis, exit planning for orderly disengagement, directed graph analysis of vendor networks using centrality, betweenness, and community detection, contract observability scoring, notice trigger taxonomies, failure modes and effects analysis customized for critical supplier concentration, supply chain risk practices aligned with NIST SP 800-161 and ISO 28000.

Chapter 15. Financial Risks: Measure What the Spreadsheet Hides, page 371

Functional silos between treasury, credit, and finance teams hide correlated exposures inside separate spreadsheets, and this chapter tears down that separation. It walks through the full decomposition of expected credit loss into probability of default, loss given default, and exposure at default consistent with IFRS 9 and Basel-aligned capital frameworks, and it addresses wrong-way risk, the dangerous pattern where a counterparty’s financial strength deteriorates at exactly the moment exposure to that counterparty rises.

Technical toolkit: cash-flow-at-risk with covenant-breach overlays, value at risk, expected shortfall, GARCH modeling for regime-switching and time-varying volatility, the Herfindahl-Hirschman index for concentration measurement, asset-liability management gap and duration analysis, foreign exchange exposure decomposition across transaction, translation, and economic exposure, stress testing and reverse stress testing, distance-to-capacity modeling.

Chapter 16. Strategic Risks: The Bets That Shape Your Future, page 412

Deterministic strategic planning gets dismantled here in favor of treating every long-term investment as one bet inside a portfolio of correlated, uncertain bets. The chapter filters strategic assumptions through uncertainty, impact, and sensitivity screens, and it maps strategic dependencies, the common assumptions, capabilities, and counterparties multiple initiatives quietly rely on at once, so a single shared failure point does not take down several strategic bets simultaneously.

Technical toolkit: the strategic assumptions register, assumption mortality tracking, real options valuation through decision trees, binomial lattices, and simulation, the risk-reward investment boundary plot, reverse stress testing working backward from strategic failure, evidence grading by reliability and transferability, staged commitment structures preserving optionality, M&A-specific due diligence overlays for synergy realism and integration friction.

Chapter 17. Continuity Risks: The Survival of Critical Services, page 443

Resilience thinking shifts here from restoring technical assets to protecting the continuity of the external, customer-facing service those assets support. The chapter anchors the entire analysis on impact tolerance, an outside-in harm boundary rather than an internal recovery time objective, and it introduces the resilience margin, the safety buffer between how fast a team can actually recover and how fast the organization promised its customers it would.

Technical toolkit: service dependency graphs across people, process, application, data, facility, and supplier layers, impact tolerance thresholds, time-impact decomposition and burn rate curves, top-down fault tree analysis, bottom-up failure modes and effects analysis, cut-set analysis for minimal failure combinations, compound disruption libraries for overlapping crises, common-cause failure and false redundancy checks, structured continuity planning aligned with ISO 22301.

Chapter 18. Sustainability Risks: The Transition Penalty, page 487

This chapter cuts past rating-agency scorecards and PR-driven disclosure templates to calculate the actual, asset-level economic re-pricing a business model faces during an energy and climate transition. It applies double materiality, weighing an organization’s environmental and social impact against its own financial exposure, and it overlays physical hazard layers, flood, drought, and heat, directly onto asset coordinates instead of relying on portfolio-level averages that hide site-specific risk.

Technical toolkit: double materiality assessment, asset-level geospatial hazard modeling, stranded asset and planned retirement analysis, transition pathway scenario families spanning orderly, delayed, and disorderly transitions, climate value at risk, non-linear technology substitution curves, three-level screening from portfolio screen through site-specific modeling, alignment with TCFD-based disclosure and the EU Corporate Sustainability Reporting Directive.

Chapter 19. People Risks: Prevent Behavioral Failures, page 523

Human behavior gets treated here as both a process vulnerability and an active control mechanism, replacing soft engagement survey scores with real operational loss logic. The chapter names behavioral reflexivity, the way people adapt to and quietly route around controls once they understand how those controls measure performance, and it distinguishes work-as-imagined, what the procedure manual says, from work-as-done, what actually happens on the floor under real time pressure.

Technical toolkit: spliced loss distributions combining frequency modeling through Poisson or negative binomial distributions with a lognormal body and a generalized Pareto tail for catastrophic events, organizational network analysis using betweenness and eigenvector centrality to map key-person dependencies, talent survival curves, performance-influencing factor analysis covering fatigue and shift patterns, the hierarchy of controls, return on safety investment, mean excess plots for identifying where routine friction ends and true tail risk begins.

Part 4. Advanced Practice: Deeper Certainty for the Numbers That Matter Most

Chapter 20. Build the Probability Engine, page 565

No model, however sophisticated, can rescue weak or uncalibrated inputs, and this chapter fixes the upstream evidence chain that every earlier chapter depends on. It applies Cooke’s classical model to calibrate expert judgment using seed questions with known answers, scoring each contributor on statistical accuracy rather than seniority or confidence, and it walks through a thirteen-step incident data validation program for turning messy operational logs into inputs a model can actually trust.

Technical toolkit: Cooke’s classical model, the Sheffield elicitation framework, the Delphi method, ordinary least squares regression as a baseline check on key assumptions, regularized regression, generalized linear models, quantile regression, sequential decision trees using backward induction and expected value of perfect information, calibration plots and reliability diagrams, the thirteen-step data validation program covering duplicate detection, coverage heatmaps, temporal gap checks, and outlier truncation.

Chapter 21. Aggregate Risk Correctly, page 615

Adding up nominal position exposures and calling the total a portfolio risk figure is mathematically wrong, and this chapter explains exactly why before showing the correct alternative. It applies modern portfolio theory and covariance-driven diversification to quantify a real diversification benefit rather than an assumed one, and it translates option sensitivity measures into language non-traders can actually use when making an operational decision.

Technical toolkit: modern portfolio theory, the Sharpe ratio, the Greeks, delta, gamma, vega, theta, and rho, translated into operational sensitivities, Black-Scholes-based contingent outcome modeling, profit and loss attribution, asset-liability management duration and convexity analysis, common stress scenario construction, shrinkage estimators and Bayesian correlation overlays.

Chapter 22. Simulate Your Risk Before It Hits, page 649

Monte Carlo simulation is established here as the primary engine for combining multiple interacting, non-linear variables into a single, honest loss distribution instead of a spreadsheet full of independent worst-case guesses. The chapter distinguishes deterministic, probabilistic, and stochastic modeling, and it introduces the two standard numerical convolution methods, Panjer recursion for exact discrete calculation and Fast Fourier Transform-based convolution, for combining frequency and severity distributions without brute-force simulation.

Technical toolkit: compound Poisson-lognormal Monte Carlo modeling, Panjer recursion, Fast Fourier Transform convolution, loss exceedance curves, liquidity-adjusted value at risk, the Kupiec test for exception calibration, the Christoffersen test for exception clustering, correlated event copulas, an open-source Python simulation engine available without a commercial license.

Chapter 23. The Emerging Risk Modelling Approach, page 708

This chapter governs the pre-quantifiable stage of emerging threats, where historical data is essentially zero and false precision is more dangerous than admitted uncertainty. It classifies emerging exposure into unmodeled known risk, low-data known risk, and genuinely emerging risk, and it applies volatility, uncertainty, complexity, and ambiguity analysis to frame threats that do not behave in a straight line.

Technical toolkit: VUCA analysis, systemic interdependence and cascade-question mapping, horizon scanning, a six-step scenario planning matrix covering focal question, driving forces, critical uncertainties, narrative construction, strategy testing, and early warning indicators, no-regrets action identification, tripwire design, a belief revision log for tracking how emerging assumptions change over time.

Chapter 24. Predictive Risk Models: Machine Learning, page 727

The risk function moves here from static quarterly summaries to live, transaction-level, forward-looking scoring. The chapter covers model stacking, gradient boosting, and random forest architectures for building predictive scores, and it pairs every model with explainability output so a risk reviewer can see exactly why a given transaction or exposure was flagged, rather than trusting a black box.

Technical toolkit: gradient boosting, random forest, model stacking, SHAP and LIME explainability, ROC-AUC, precision, recall, F1 score, and Gini coefficient for performance evaluation, the population stability index for catching model drift, temporal train-test splitting to prevent data leakage, synthetic data generation and extreme value theory for rare-event modeling, user and entity behavior analytics.

Chapter 25. Build Agentic Risk Controls, page 761

Prediction without action is negligence once the technology exists to close that gap, and this chapter deploys governed autonomous systems that respond to risk signals in milliseconds instead of waiting for the next committee meeting. It defines maturity levels running from simple threshold automation through contextual action selection to fully self-learning agents, and it builds oversight tiers so that full automation, exception review, human approval, and suspension are explicit, pre-agreed states rather than improvised in the moment.

Technical toolkit: Markov decision process modeling, reward function design, state space and action space definition, offline reinforcement learning, simulated exploration in causal sandboxes, shadow-mode rollouts, deterministic action schemas, algorithmic circuit breakers, continuous validation across predictive, action, and consequence layers, alignment with the NIST AI Risk Management Framework and ISO/IEC 42001.

Chapter 26. The Decision-Ready Blueprint, page 778

This closing chapter is the executive change-management playbook and organizational charter that ties the entire framework together. It confronts the corporate horoscope problem directly, the ritualized compliance loop that produces documentation without producing better decisions, and it lays out a phased five-step implementation roadmap moving an organization from mobilization through foundation-building, quantification, integration, and finally automation.

Technical toolkit: the phased five-step implementation roadmap, a model-driven GRC risk policy template, model inventory registers, a grounded risk management hierarchy connecting decision, objective, uncertainty, driver, event, exposure, impact, threshold, treatment, control, response, and outcome into one consistent vocabulary, a five-domain hiring and interview guide covering strategic, reporting, operational, data and modeling, and emerging risk competencies, and performance metrics that judge the risk function by executive decisions changed rather than reports filed.

Glossary, page 829

A consolidated reference of every technical term, distribution, and model introduced across the twenty-six chapters, built for readers who want a fast lookup rather than a full re-read.

The prEN 18228 Problem: Why Your AI Risk Assessment Will Fail the First Real Test

Fri, 08 May 2026 00:00:00 +0000

Most ook solid on paper and collapse the moment a regulator, client, or auditor asks a simple question. What exactly can go wrong, how likely is it, and what does it cost when it does.

That gap is about to matter more.

A new European standard, prEN 18228, sets out a formal process for managing risks in AI systems across their full life cycle. It is designed to support regulatory expectations by requiring organizations to identify hazards, estimate and evaluate risks, define acceptability criteria, and continuously monitor controls. It brings structure and discipline. It also brings a product safety mindset into AI, focusing on harm to people, rights, and systems.

This sounds like progress. In many ways, it is.

But most organizations will apply it the same way they apply existing compliance frameworks. They will produce well-documented processes, consistent terminology, and defensible artifacts. And they will still struggle to answer the one question that drives real decisions. Should we deploy this system, under these conditions, with this level of exposure.

That is where this discussion starts.

The Standard Brings Structure. It Does Not Solve the Decision Problem.

prEN 18228 defines risk in familiar terms. Probability of harm and severity of that harm. It requires organizations to identify hazards, assess risks, and reduce them to an acceptable level based on the intended use and reasonably foreseeable misuse of the system.

This is a disciplined approach. It forces teams to think beyond model accuracy and consider real-world impact. It also aligns well with how regulators think about safety and rights. The limitation is more subtle.

The standard tells you how to run the process. It does not tell you how to make the decision. It does not require you to quantify exposure in financial or operational terms. It does not connect model behavior to business outcomes like lost contracts, regulatory investigations, or reputational damage that affects future revenue. So you end up with a structured assessment that still relies on qualitative judgments at the point where decisions are made. Most organizations are comfortable there. They should not be.

The Risk Definition Works for Products, but AI Behaves Differently.

The comes from product safety. It works well when failure modes are clear and causation is traceable. A component fails. A system stops. Harm follows in a relatively predictable way. AI systems behave differently.

They fail in ways that are distributed, context-dependent, and often only visible after deployment. A fraud detection model might perform well overall and still produce systematic errors for a specific segment. A decision system might be technically accurate and still generate outcomes that trigger regulatory scrutiny or client disputes.

Two risks can produce the same expected value and require completely different responses. A frequent, low-impact error calls for process improvement and monitoring. A rare but severe failure calls for governance, escalation, and sometimes a decision not to deploy at all.

The standard does not distinguish clearly between these cases. It treats them within the same structure, which can flatten the differences that matter most in practice.That is where need to go beyond the text.

The Terminology You Need to Understand Before You Start

The standard introduces 68 defined terms across six domains, and most of them do not mean what you think they mean.

Terms Relating to the EU AI Act

Source: Regulation (EU) 2024/1689 (EU AI Act)

AI system / Artificial intelligence system: Machine-based system that is designed to operate with varying levels of autonomy and that can exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. This definition is broader than most technical definitions of AI. It includes rule-based systems and statistical models that exhibit adaptiveness after deployment, not just machine learning systems.
Intended purpose: Use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. Technical documentation is not accompanying documentation. Information on technical documentation can be found in Article 11 of the EU AI Act. Your marketing claims define your regulatory obligations. If you claim the system works in a particular context, that context becomes part of your intended purpose and you must demonstrate safe operation there.
Reasonably foreseeable misuse: Use of an AI system in a way that is not in accordance with its intended purpose, but which can result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. Reasonably foreseeable human behaviour includes the behaviour of all types of relevant users. Reasonably foreseeable misuse can be intentional or unintentional. You cannot disclaim liability by saying users deployed your system incorrectly if that incorrect use was reasonably foreseeable. This standard requires you to model misuse scenarios and control for them.
Performance: Ability of an AI system to achieve its intended purpose. Performance can relate either to quantitative or qualitative findings. Performance is evaluated in the context of use of the AI system. The use conditions under which performance is evaluated can result in significant performance outcomes and which can be explicitly stated. Performance is not accuracy. A highly accurate model that produces discriminatory outcomes has not achieved its intended purpose if that purpose included fairness. Performance must be evaluated under real-world use conditions, not laboratory conditions.
Provider: Natural or legal person, public authority, agency or other body that develops an AI system or a general purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. A distributor, importer, deployer or other third party can be considered a provider of an AI system in certain circumstances. If you rebrand, white-label, or substantially modify an AI system, you can become the provider under the Act, inheriting all associated obligations.
Deployer: Natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. Deployers have distinct obligations under the AI Act, including human oversight and monitoring. This standard is written for providers, but providers must understand deployer obligations to design systems that support compliance downstream.
Post-market monitoring system: Activities carried out by providers of AI systems to collect and review experience gained from the use of AI systems they place on the market or put into service for the purpose of identifying any need to immediately apply any necessary corrective or preventive actions. For the purpose of this document, activities shall mean all activities. Post-market monitoring is not optional. It is a continuous regulatory obligation. If you cannot systematically collect and review real-world performance data after deployment, you cannot meet the standard.
Placing on the market: First making available of an AI system on the Union market. See making available on the market. Further information on this concept can be found in the Blue Guide, section 2. The first instance of commercial availability triggers the full set of provider obligations. Pre-release pilots and limited testing may not constitute placing on the market, but the boundary is not always clear.
Making available on the market: Supply of an AI system for distribution or use on the Union market in the course of a commercial activity, whether in return for payment or free of charge. Free distribution counts. Open-source release can count. If you make the system available for commercial use in the EU, you are subject to the Act regardless of whether you charge for it.
Putting into service: Supply of an AI system for first use directly to the deployer or for own use in the Union for its intended purpose. Further information on this concept can be found in the Blue Guide, section 2. Internal use triggers obligations. If you develop an AI system for your own operations and put it into service in the EU, you are both provider and deployer.
Serious incident: Incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: the death of a person or serious harm to a person’s health; a serious and irreversible disruption of the management or operation of critical infrastructure; the infringement of obligations under applicable regulatory requirements intended to protect fundamental rights; serious harm to property or the environment. Serious incidents must be reported to authorities. The definition is broad. An AI system that produces a discriminatory outcome that infringes fundamental rights protections can trigger a serious incident report even if no physical harm occurred.
Subject: Natural person who participates in testing in real-world conditions. Participating in testing can require informed consent of subjects. If your real-world testing involves human participants, informed consent requirements apply. This is a regulatory obligation, not just an ethical guideline.
Real-world conditions testing: Temporary testing of an AI system for its intended purpose in its intended context of use or deployment environment outside a laboratory or otherwise simulated environment. Assessing and verifying conformity of the AI system with the requirements of this document includes that the overall residual risk of the AI system is acceptable in accordance with its intended purpose and reasonably foreseeable misuse. Real-world conditions testing can pertain to technical and non-technical aspects, including performance verification or usability study. Real-world conditions testing can require the participation of subjects. Real-world testing is distinct from deployment. It is time-limited, purpose-specific, and subject to additional safeguards. If you call something a pilot to avoid compliance obligations, but it operates like a deployed system, regulators will treat it as deployment.

Sources: ISO 9000:2015, ISO/IEC Guide 63:2019, EN ISO 14971:2019, ISO 26000:2010

Accompanying documentation: Materials accompanying an AI system and containing information for the user or those accountable for the use, maintenance, decommissioning and disposal of the AI system. The accompanying documentation can consist of the instructions for use, technical description, installation manual, quick reference guide, etc. The accompanying documentation is not necessarily a written or printed document but can involve auditory, visual, or tactile materials and multiple media types. Materials include information relevant for the protection of health, safety and fundamental rights, where each is applicable. Accompanying documentation is legally binding. If the instructions for use specify a particular deployment context or oversight requirement, deployers must follow it, and providers are responsible for ensuring the guidance is accurate and complete.
Objective evidence: Data supporting the existence or verity of something. Objective evidence can be obtained through observation, measurement, test or by other means. Assertions without evidence do not satisfy this standard. If you claim a control is effective, you must produce objective evidence of its operation.
Procedure: Specified way to carry out an activity or a process. Procedures can be documented or not. Undocumented procedures are permitted, but they must be specified and repeatable. In practice, undocumented procedures are difficult to demonstrate during an audit.
Process: Set of interrelated or interacting activities that use inputs to deliver an intended result. Whether the intended result of a process is called output, product or service depends on the context of the reference. Inputs to a process are generally the outputs of other processes and outputs of a process are generally the inputs to other processes. Two or more interrelated and interacting processes in series can also be referred to as a process. Risk management is a process. Model development is a process. Post-market monitoring is a process. The standard requires these processes to be defined, systematic, and auditable.
Record: Document stating results achieved or providing evidence of activities performed. Records can be used, for example, to formalize traceability and to provide evidence of verification, preventive action and corrective action. Records are the primary form of objective evidence in a risk management system. If an activity is required and you cannot produce a record of it, you have not met the requirement.
Risk management: Systematic and continuous application of management policies, procedures and practices to the tasks of analysing, evaluating, controlling and monitoring risk throughout the entire life cycle of an AI system. Risk management is not a one-time assessment. It is a continuous process that spans development, deployment, operation, and decommissioning.
State of the art / Generally acknowledged state of the art: Developed stage of technical capability at a given time as regards products, processes and services, based on the relevant consolidated findings of science, technology and experience. The state of the art embodies what is currently and generally accepted as good practice in technology. The state of the art does not necessarily imply the latest scientific research still in an experimental stage or with insufficient technological maturity. You are required to implement risk controls that reflect the state of the art, not the state of your organization’s current capability. If better controls exist and are generally accepted, you must adopt them or justify why they are not applicable.
Top management: Person or group of people who directs and controls a provider at the highest level. Top management must establish and approve risk acceptability criteria. They cannot delegate this responsibility to the compliance or risk function.
Verification: Confirmation, through the provision of objective evidence, that specified requirements have been fulfilled. The objective evidence needed for a verification can be the result of an inspection, testing or of other forms of determination such as performing alternative calculations or reviewing documents. The activities carried out for verification are sometimes called a qualification process. The word “verified” is used to designate the corresponding status. Verification requires objective evidence. Self-attestation is not verification.
International norms of behaviour: Expectations of socially responsible organizational behaviour derived from customary international law, generally accepted principles of international law, or intergovernmental agreements that are universally or nearly universally recognized. Intergovernmental agreements include treaties and conventions. Although customary international law, generally accepted principles of international law and intergovernmental agreements are directed primarily at states, they express goals and principles to which all organizations can aspire. International norms of behaviour evolve over time. Fundamental rights protections are grounded in international norms of behaviour. These norms are not static, and your risk management process must account for evolving expectations.
Risk management file: Set of records and other documents that are produced by risk management. The risk management file is the primary artifact a regulator will examine during an inspection. It must be complete, coherent, and traceable.

Terms Relating to Testing

Source: ISO/IEC/IEEE 29119-1:2022

Testing: Set of activities conducted to facilitate discovery and evaluation of properties of test items. Testing activities include planning, preparation, execution, reporting, and management activities, insofar as they are directed towards testing. Testing is not just running the model on a validation set. It includes planning what will be tested, how it will be tested, documenting the results, and acting on the findings.
Test item / Test object: Work product to be tested. Example: Software component, system, requirements document, design specification, user guide. The AI model is a test item. The training data is a test item. The user documentation is a test item. All must be tested.
Test objective: Reason for performing testing. Every test must have a defined objective. Testing without a stated objective does not satisfy the standard.
Test completion report / Test summary report: Report that provides a summary of the testing that was performed. The report may contain statistical analysis. Test completion reports are records. They must be retained as part of the risk management file.
Test plan: Detailed description of test objectives to be achieved and the means and schedule for achieving them, organized to coordinate testing activities for some test item or set of test items. A test plan is a written document included in the risk management file. Testing without a documented test plan does not satisfy the standard.
Test monitoring and control process: Test management process that aims to ensure that testing is performed in line with a test plan and with organizational test specifications. Test execution must be monitored and controlled. Deviations from the test plan must be documented and justified.

Sources: Regulation (EU) 2024/1689, EU Charter of Fundamental Rights, ISO 26000:2010

Fundamental rights: Rights and freedoms guaranteed by the EU Charter of Fundamental Rights. Fundamental rights include human dignity, respect for private and family life, protection of personal data, non-discrimination, equality between women and men, rights of the child, rights of the elderly, integration of persons with disabilities, right to an effective remedy and to a fair trial, presumption of innocence and right of defence, principles of legality and proportionality of criminal offences and penalties. Fundamental rights are legally binding in the EU. Harms to fundamental rights are within scope of this standard, even if they do not produce physical injury or property damage.
Stakeholder: Individual or organization that can affect, be affected by, or perceive themselves to be affected by a decision or activity. Stakeholders can be internal or external. They include users, affected persons, deployers, providers, regulators, civil society organizations, and the public. Stakeholder identification is a required step in risk management.
Natural person: Human being. The standard distinguishes between natural persons and legal persons. Fundamental rights protections apply to natural persons.
Affected person: Natural person or groups of natural persons who can be subject to or impacted by an AI system. Affected persons include users and non-users. An AI system used in hiring affects both applicants and employees, whether or not they interact directly with the system.
User: Natural or legal person, public authority, agency or other body using an AI system. Users include deployers and end users. User obligations differ depending on the role, and risk management must account for both categories.
Group: Collection of natural persons defined by common characteristics such as demographic attributes, location, socioeconomic status, or shared vulnerability. Risks to groups must be assessed separately from risks to individuals. A system that performs well on average can produce serious harms to specific groups, and those harms are within scope.
Informed consent: Freely given specific, informed and unambiguous indication of the data subject’s wishes by which they, by a statement or by a clear affirmative action, signify agreement to the processing of personal data relating to them. For the purpose of this document, informed consent is understood more broadly to mean consent by a natural person related to participation in real-world conditions testing. Informed consent is not a click-through agreement. It must be specific, informed, unambiguous, and freely given. Generic consent forms do not satisfy this requirement.

Sources: ISO/IEC Guide 51:2014, ISO/IEC Guide 63:2019, EU Cybersecurity Act, EU Cyber Resilience Act, Directive (EU) 2022/2257

Hazard: Potential source of harm. Cyber threats and vulnerabilities can be the cause of a hazard. Cyber threats can be hazards. Hazards are not risks. A hazard is a source of harm. Risk is the combination of probability and severity. Identifying hazards is the first step in risk analysis.
Hazardous situation: Circumstance in which people, property or the environment is/are exposed to one or more hazards. Exposure to a hazard does not guarantee harm. A hazardous situation is the precondition for harm to occur.
Harm: Injury or damage to the health of a person or groups of persons, or interference with fundamental rights. For the purpose of this document, damage to property or the environment, and the disruption or destruction of critical infrastructure, are considered harms when they can result in injury or damage to the health of a natural person or groups of persons or interference with fundamental rights. Interference with fundamental rights can be tangible or intangible, physical, psychological, societal or economic, irrespective of the rightsholder’s awareness, in accordance with EU law, including the EU Charter. Safety in product safety risk management standards is understood as the absence of unacceptable risk. In the context of this document, safety refers to the protection from harm from the use of the AI system. Harm is not limited to physical injury. Psychological, societal, and economic harms are within scope. Interference with fundamental rights is harm even if the affected person is unaware of it.
Severity: Measure of the possible consequences of a hazard. The definition does not imply numerical measure of severity. Severity can be qualitative or quantitative. However, severity scales must be defined and applied consistently across the risk assessment.
Risk: Combination of the probability of an occurrence of harm and the severity of that harm. The probability of occurrence includes the exposure to a hazardous situation and the possibility to avoid or limit the harm. This is the formula that does not work for AI when applied as a simple multiplication without modeling the loss distribution. It treats all risks with the same expected value as equivalent, which they are not.
Residual risk: Risk remaining after risk control measures have been implemented. Residual risk must be evaluated against risk acceptability criteria. No system is risk-free. The question is whether the residual risk is acceptable.
Acceptable risk / Tolerable risk: Level of risk that is accepted in a given context based on the current values of society. For the purpose of this document, “acceptable risk” is the preferred term and “tolerable risk” is the admitted term. For the purpose of this document, “context” refers to the intended purpose and reasonably foreseeable misuse of the AI system, and “current values of society” refers to high protection of health, safety, and fundamental rights. Acceptable risk is not a fixed threshold. It depends on context, and it evolves as societal values evolve. What was acceptable five years ago may not be acceptable today.
Risk assessment: Overall process comprising a risk analysis and a risk evaluation. Risk assessment is the complete analytical process. It includes identifying hazards, estimating risk, and evaluating whether risk is acceptable.
Risk analysis: Systematic use of available information to identify hazards and to estimate the risk. Risk analysis is the first step in risk assessment. It is analytical, not evaluative.
Risk estimation: Process used to assign values to the probability of occurrence of harm and the severity of that harm. Risk estimation can be qualitative, semi-quantitative, or quantitative. The method must be documented and applied consistently.
Risk control: Process in which decisions are made and measures implemented by which risks are reduced to, or maintained within, specified levels. Risk control follows risk evaluation. It is the implementation of risk control measures to bring residual risk within acceptable levels.
Risk evaluation: Procedure based on the risk analysis to determine whether acceptable risk has been exceeded. Risk evaluation is the decision point. It compares estimated risk against acceptability criteria.
Inherently safe design: Measures taken to eliminate hazards or to reduce risks by changing the design or operating characteristics of the product or system. For the purpose of this document, a product or system is an AI system. For risks to fundamental rights, inherently safe design refers to translating fundamental rights, for example presumption of innocence and non-discrimination, into the technical AI system design requirements through, for example implementing equality, privacy and data protection by design. Inherently safe design is the highest level of risk control. It eliminates the hazard rather than controlling exposure to it.
Risk control measure: Action or means to eliminate hazards or to reduce risks. Example: Inherently safe design; protective devices; personal protective equipment; information for use and installation; organization of work; training; application of equipment; supervision. Risk control measures follow a hierarchy. Inherently safe design is preferred. Protective measures and information for use are secondary controls. The standard does not permit you to substitute information for design improvements when design improvements are feasible.
Cybersecurity: Activities necessary to protect network and information systems, the users of such systems, and other persons affected by cyber threats. For the purpose of this document, network and information systems shall mean an AI system under consideration. Cybersecurity is within the scope of risk management. Cyber threats are hazards, and cybersecurity controls are risk control measures.
Cyber threat: Potential circumstance, event or action that can damage, disrupt or otherwise adversely impact network and information systems, the users of such systems and other persons. For the purpose of this document, network and information systems shall mean an AI system under consideration. Cyber threats include adversarial attacks on AI models, data poisoning, model extraction, and manipulation of inputs to produce harmful outputs.
Vulnerability: Weakness, susceptibility or flaw of a product with digital elements that can be exploited by a cyber threat. For the purpose of this document, a product with digital elements shall mean an AI system under consideration. Vulnerabilities are not hazards themselves, but they are sources of hazards. A model trained on unvalidated data has a vulnerability. If that vulnerability is exploited, it becomes a hazard.
Critical infrastructure: Asset, facility, equipment, network or system or part of asset, facility, equipment network or system which is necessary for the provision of an essential service. Disruption of critical infrastructure can constitute serious harm. AI systems used in or affecting critical infrastructure are subject to heightened scrutiny.

Do not assume the standard’s definitions align with your organization’s existing terminology. They do not. The term “user” in this standard includes deployers and end users. The term “harm” includes interference with fundamental rights, not just physical injury. The term “risk” is defined as a combination of probability and severity, but it does not specify how to combine them, and the implied multiplication formula is insufficient for AI. Build a terminology mapping document that translates each of the standard’s 68 terms into your organization’s operational language, and distribute it to every team involved in AI development, deployment, and risk management. If your legal team, your technical team, and your risk team are using different definitions of the same word, your risk assessment will fail before you begin.

How the Standard Actually Runs Risk Management

Most organizations say they “have a risk process.” What they often have is a sequence of documents. The standard is more demanding. It expects a continuous, structured process that runs across the entire life cycle of the AI system and produces decisions that can be explained and defended.

A Continuous Process, Not a One-Time Assessment

The provider is expected to establish, implement, document, and maintain an ongoing risk management process. This process starts with risk analysis. It includes identifying characteristics related to risks tied to the intended purpose and reasonably foreseeable misuse of the AI system. It requires identifying known and reasonably foreseeable hazards, hazardous situations, and risks.

From there, the process moves to estimating and evaluating those risks, followed by risk evaluation, testing, risk control, and the evaluation of overall residual risk. It does not stop at deployment. It continues through risk management review and both pre-market and post-market activities.

This entire process applies across the full life cycle of the AI system. It is not limited to design or validation phases. It must be documented and maintained in a risk management file, which becomes the central record of how risk was understood, assessed, and managed over time.

Many organizations already have product or system development processes. The expectation is not to duplicate effort, but to integrate. Where a product realization process exists, it should incorporate the relevant parts of the risk management process. In practice, this means risk is embedded into how the system is built and operated, not added as a separate compliance layer.

The process is not linear. Different elements carry different weight depending on the life cycle stage. Activities can be iterative, repeated, and refined as new information becomes available. That flexibility is intentional. AI systems evolve, and the risk process must evolve with them.

Management Owns the Process, Not Just the Outcome

Risk management is not delegated away. Top management is expected to demonstrate active commitment. This starts with providing adequate resources and ensuring that personnel involved in risk management are competent. It includes assigning responsibility clearly and overseeing how the process is implemented.

There is also a review obligation. Management must regularly assess whether the risk management system remains suitable and effective. This includes reviewing the policy, the plan, and how the process operates in practice. These reviews are not ad hoc. They are planned, systematic, and documented, including decisions and actions taken.

Post-market information plays a direct role here. What is learned from real-world use feeds back into management’s assessment of whether the process still works. In many organizations, this feedback loop is weak. The standard makes it explicit.

Representation of the risk management process

Risk Acceptability Is a Policy Decision, not a Technical Detail

One of the most consequential requirements sits at the policy level. Top management must define, document, and maintain a risk management policy that establishes how risk acceptability is determined.

This policy must ensure that criteria for accepting individual residual risks and the overall residual risk meet regulatory requirements. It must take into account the intended purpose of the AI system, its reasonably foreseeable misuse, and what is generally accepted as the state of the art.

It should also reflect broader expectations. This includes relevant standards, international norms of behavior, and the concerns of stakeholders who may be affected by the system.

The policy needs to go further than general principles. It must specify the methods used to determine risk acceptability. One example is comparing the AI system to an equivalent non-AI system using a “no worse than” approach. It must also define how and when these criteria are reviewed and updated. At a minimum, this happens after serious incidents and at regular intervals throughout the life cycle, including before market entry.

In practice, this is where many organizations struggle. They define high-level principles but avoid committing to clear thresholds or methods. The standard expects the opposite.

Competence Is a Collective Requirement

Risk management activities must be performed by people who are competent based on education, training, skills, and experience relevant to their role. This is not limited to technical expertise.

Collectively, the team must understand the AI system or similar systems, the application domain and operating conditions, the technologies involved, and the relevant aspects of health, safety, and fundamental rights. They also need to understand the risk management techniques being used.

Not every individual needs all of these competencies, but the team as a whole must cover them.

Where fundamental rights are involved, the expectation is higher. Those performing these assessments must be able to understand and apply the relevant rights, and when needed, organize and facilitate consultation with affected stakeholders, including vulnerable groups. If the system can affect specific vulnerable populations, the team must have expertise in those areas.

In practice, this pushes organizations to move beyond purely technical or compliance-driven teams. Risk management becomes multidisciplinary by design.

Defining Risk Acceptability Criteria

Risk acceptability criteria define what level of risk is considered acceptable. These criteria must be established and updated throughout the life cycle to maintain a consistent and high level of protection for health, safety, and fundamental rights.

They must exist at two levels. One for each identified risk, and one for the overall residual risk of the system. They must be justified, documented, and supported by objective evidence so they can be verified and validated.

The criteria must reflect equal concern for all affected persons, with particular attention to those most vulnerable to harm. They must be aligned with the risk management policy, regulatory requirements, and the nature of the harms involved, including how different harms may interact.

Objective evidence plays a central role. Where people can be affected, this may include consultation with affected individuals or their representatives, especially vulnerable groups. If direct consultation is not performed, evidence can come from previous consultations for similar systems or from authoritative sources on fundamental rights. Testing can also contribute.

The provider must document why the evidence used is relevant and sufficient. If consultation is performed, the rationale for selecting participants and their representativeness must be clear.

This is not a box-ticking exercise. It is about showing that the thresholds for accepting risk are grounded in reality, not convenience.

How Criteria Are Established and Updated

Risk acceptability criteria must be determined in relation to the intended purpose and reasonably foreseeable misuse of the AI system. They must exist even when the probability of harm cannot be estimated precisely.

The process for establishing and updating these criteria is demanding. It includes considering independent review by a multidisciplinary team with expertise in safety, health, fundamental rights, AI, and the relevant application domain. Where an equivalent evaluation already exists, it can be reused, but it must be supported by objective evidence.

The provider must consider the state of the art, including literature, market data, and incident data. Alternatives must be assessed, including options that reduce risk significantly or avoid using AI altogether.

Severity and probability both matter, but they are not interchangeable. A low-severity harm can still be unacceptable if it occurs frequently. Where probability cannot be quantified, qualitative assessment is acceptable, provided the reasoning is documented.

The distribution of risk across different groups must be considered, especially where certain groups may be disproportionately affected. Adverse impacts on vulnerable groups require particular attention.

Pre-market and post-market information must feed into this process. This includes incident data, near misses, user feedback, and concerns raised by affected individuals or their representatives.

Independence is required. Those defining and reviewing risk acceptability criteria should be separate from those designing and developing the system, with any conflicts of interest documented. This can be achieved internally through separation of roles or externally through independent experts.

Where fundamental rights are at stake, additional considerations apply. For qualified rights, the provider must assess necessity, proportionality, and legitimate objectives. For privately enforceable rights, applicable legal obligations must be identified.

Looking at the Overall Residual Risk

Assessing individual risks is not enough. The standard requires a separate evaluation of the overall residual risk. The criteria for overall acceptability can differ from those applied to individual risks. This reflects a simple reality. Risks interact.

The provider must consider the aggregated severity and how risks are distributed, how they interact, and how multiple harms can arise from a single situation. The combined effect can be greater than the sum of individual risks. The analysis must consider impacts on all affected persons, with particular attention to vulnerable groups. It must also consider both immediate and long-term harms, including cumulative effects over time.

An important consequence follows. The overall residual risk can be unacceptable even if each individual risk has been reduced to an acceptable level. Where children are affected, their best interests must be a primary consideration. This is not optional. It reflects established international principles.

Final approval of overall risk acceptability sits with top management. This reinforces that risk acceptance is a business decision, not just a technical conclusion.

Planning the Work

Risk management activities must be planned. For each AI system, the provider must establish and document a risk management plan. This plan becomes part of the risk management file. The plan defines the scope of activities. It identifies the AI system and the life cycle phases where each part of the plan applies. It assigns responsibilities and authorities, taking into account the competence of personnel.

It includes requirements for reviewing both the process and the outcomes, the risk acceptability criteria, and the underlying policy. It defines how risk control measures will be verified and how pre-market and post-market information will be collected and reviewed. The plan itself is not static. Changes over the life cycle must be recorded. This creates a traceable history of how risk management evolved.

The Risk Management File as the Backbone

All of this comes together in the risk management file. This is not a single document, but a structured set of records and references that capture the entire process.

It includes the intended purpose of the AI system, the risk management policy, the plan, and all documentation related to risk analysis, evaluation, testing, control, and residual . It also includes records of reviews and post-market activities.

Traceability is essential. Each identified hazard must be linked through the process. From identification, to analysis, to controls, to testing, to residual risk. In practice, this is often implemented through structured risk tables with clear identifiers and links to supporting evidence.

The file can reference other documents, including those from the quality management system, to avoid duplication. What matters is that all required information can be assembled quickly and coherently.

This is where the process becomes real. If the file is complete, consistent, and traceable, the organization can explain its decisions. If it is not, the process exists only on paper.

Risk Management Process and AI System Life Cycle

The provider must determine and document in the risk management file the stages of the life cycle for each AI system, from inception to end of life, in line with each AI system’s intended purpose. The life cycle phases may be determined in alignment with the life cycle phases as determined in the quality management system. prEN 18286, the companion standard addressing quality management systems for EU AI Act purposes, provides additional information on life cycle alignment.

The risk management process must be applied systematically and iteratively along the entire life cycle of the AI system. This is not a sequential process completed once and archived. The risk management process activities must be integrated into the life cycle stages to ensure that risks are systematically and iteratively analyzed, evaluated, and reassessed, risk control measures are identified, implemented, verified, and updated when needed, residual risks and overall residual risk are evaluated and monitored and their acceptability maintained, and relevant information on the AI system is collected and reviewed.

Risk analysis and risk evaluation must start at the inception stage of the AI system and must be applied through all the following life cycle stages until end of life, as new risks can emerge and previously identified ones can change during any of the life cycle stages. The results of risk evaluation can have a bearing already on the inception phase. In some cases, it can take much less effort to mitigate a risk at the inception stage than during later life cycle stages. This is a critical point that most organizations miss. Early risk assessment is not a formality. It is the point at which design decisions can eliminate hazards rather than control them.

Risk control measures can be implemented at different life cycle stages, depending on the specific measures. Risk control measures must be, as much as technically feasible, implemented during design and development with the view to achieve inherently safe design. Inherently safe design is the highest priority risk control measure. It eliminates the hazard rather than managing exposure to it.

Information that is relevant to the risk management process can become available at any life cycle stage. This information can support the provider to identify the need to re-execute the risk management process, in whole or in part, or to return to a previous step of the risk management process. This information can refer to the identification of new risks, changes to estimations of risks, the detection of risk control measures that do not perform as expected, or the implementation of new risk control measures.

Some risk control measures are only possible to implement by returning to a previous life cycle stage. A change of intended purpose restarts the inception stage. New inherently safe design measures restart the design and development stage. This creates a challenge for organizations that treat life cycle stages as linear and complete. The standard requires iterative re-entry into earlier stages when risk information demands it.

Most organizations will map their existing product development life cycle to the standard’s requirements and declare the mapping complete. That is not sufficient. The standard requires risk management activities to be integrated into each life cycle stage, not merely aligned with it. Build your life cycle model as a series of decision gates where risk analysis, risk evaluation, and risk control verification are mandatory prerequisites for progression. If your development roadmap allows a system to move from design to deployment without a documented risk evaluation and approval of residual risk by top management, your life cycle integration does not meet the standard. The life cycle is not a timeline. It is a control framework.

General Requirements for AI Risk Analysis

The implementation of the planned risk analysis activities and the results of the risk analysis must be recorded in the risk management file. The risk analysis must consist of risk identification and risk estimation. These are distinct activities with different outputs, and both must be documented.

The provider must analyze risks from logging and monitoring, human factors, user behavior, unwanted bias, data quality and data governance including provenance of data, and AI system accuracy, where applicable. This is not an exhaustive list. The standard explicitly states these areas are examples, not limits. In order to address these areas, the provider can refer to prEN 18229-1 on transparency, prEN 18229-2 on robustness, prEN 18282 on cybersecurity, prEN 18283 on data quality, prEN 18284 on bias, prEN 18281 on logging, and other relevant standards.

In addition to the records required for risk identification and risk estimation, the documentation of the conduct and results of the risk analysis must include at least the unique identifier and the version designation of the AI system that was analyzed, identification of the persons and organization who carried out the risk analysis, scope and date of the risk analysis, and techniques and methodologies used for hazard identification and risk estimation. Without this metadata, the risk analysis cannot be traced, verified, or defended under regulatory scrutiny.

Damage to property or the environment, and the disruption or destruction of critical infrastructure, are harms when they can result in injury or damage to the health of persons or interference with fundamental rights. The provider may also consider additional harms. The process of risk analysis is intended to address these harms. Cyber threats and vulnerabilities can be the cause of a hazard, and . prEN 18282 can be used to identify and address cyber threats and vulnerabilities.

The range of fundamental rights which must be considered for the purposes of risk analysis must include all the rights recognized under the EU Charter. This is not limited to the rights most commonly discussed in AI ethics frameworks. It includes all Charter rights, and the provider must assess which rights are relevant to the specific AI system being analyzed.

Techniques such as Preliminary Hazard Analysis, Hazard and Operability Study, Fault Tree Analysis, and System-Theoretic Process Analysis can be effectively utilized to derive risk analysis. These methods are complementary, and employing a combination of them can be essential for achieving a comprehensive and robust risk analysis. For more guidance on risk analysis techniques, refer to EN IEC 31010. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Identification from the Intended Purpose

The provider must document in the risk management file the intended purpose of the AI system being considered. The documentation of the intended purpose must include at least the following information: application areas, the objectives of the AI system including the intended output and impact on persons’ safety, health and fundamental rights, type of tasks used to achieve the objectives, techniques and approaches for how the AI system operates, types of intended deployers and types of intended users, deployment type such as physical product, software, or service, and the intended environment in which the AI system operates.

Intended users can refer to professionals, consumers, and users defined by age group or other characteristics. The environment in which the AI system operates can include the organizational, physical, and digital environment. The digital environment refers to the types of integration and interfaces with other software systems and physical systems. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Most organizations document intended purpose as a one-paragraph marketing description. That is insufficient. The standard requires you to specify the intended impact on persons’ safety, health, and fundamental rights, the deployment type, the user types, and the operational environment at a level of specificity that supports hazard identification. If your intended purpose documentation does not answer the question “in what specific context, for what specific users, performing what specific tasks, does this system operate, and what specific impacts on safety, health, and fundamental rights are intended,” it does not meet the standard. Build your intended purpose statement as a structured specification, not as a mission statement.

Risk Identification: Reasonably Foreseeable Misuse

The reasonably foreseeable misuse of the AI system must be identified, taking into account the potential misuse of the AI system by other user categories, which can include lay users, persons under the age of 18, and other vulnerable groups, on other persons affected, vulnerable groups, assets, or the environment, where other persons affected can include persons who are not defined in the intended purpose of the AI system, in a different context or environment including the digital, physical, and organizational environment and infrastructure in which the AI system operates, with incorrect input or data, and in a different application area.

The identification must also consider the potential misuse of the AI system, including its outputs, aims, or objectives. A recommendation used as a decision is an example of this type of misuse. The identification must consider the potential incorrect deployment of the AI system. A user who can and does change the AI system’s settings such that it operates outside of its intended purpose is an example of incorrect deployment. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Identifying the characteristics of an AI system is a preliminary step intended to support the identification of hazards and hazardous situations. It is meant to create a general overview of relevant characteristics, keeping in mind that hazards and hazardous situations need to be identified in detail later in the risk management process.

The provider must identify and document all applicable characteristics that can affect risks of the AI system. Where applicable, the limits of these characteristics must be defined, such as limits of performance. The operation of the AI system and the risk associated with its use can be affected when those limits are exceeded. These characteristics are related to the functionality of the AI systems, its lifecycle, its intended purpose and reasonably foreseeable misuse, and the environment in which it operates and interacts with.

For identifying these characteristics, the following must be taken into account: outputs of and actions taken by the AI system and their effect on users and persons affected, including vulnerable groups and age-appropriate outcomes when the intended users are persons under the age of 18. The effect can be directly from the AI system output or are intended to follow from the AI system output. An AI system that grants public assistance benefits has a direct effect on the applicant. A medical system that identifies cancer in a scan provides this information to a doctor which decides on treatment for the patient. The patient is affected by an action that follows the AI system output.

The provider must also consider user profiles including their abilities and limitations, known biases in decision making, their level of expertise, and how this can impact the appropriate use and interpretation of the AI system’s outputs, user accessibility to the AI system, interactions of the AI system with the environment including other systems and digital infrastructure and the effects of the AI system on the environment and vice versa, AI system functionality, architecture and technologies and related capabilities, limitations, uncertainties or known failure modes, minimum performance requirements of the AI systems and system components to achieve the intended purpose, level of autonomy and adaptiveness of the system’s behavior during operation such as continuous learning, pre-determined changes to the algorithm and its performance that can appear during the operation phase and the possibility that the system behavior changes during the operation phase in a way that hasn’t been pre-determined, dependency on third party components including open source, dependency on third party support in specific lifecycle phases such as outsourcing of design, development and verification and validation tasks, processing and storage of data by the AI system and the nature and sensitivity of the data, deployment, maintenance and decommissioning or disposal procedures, procedures for updates of the AI system during operation, and skills and experience of deployers.

The provider must identify the hat can have an impact on the characteristics listed. prEN 18282 provides additional information on this requirement. For each of the points, the provider must assess their relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement one of the listed points. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Identification: Hazards, Risk Scenarios, and Hazardous Situations

Based on the AI system’s intended purpose, reasonably foreseeable misuse, characteristics related to risks, and the state of the art, the provider must identify and document in the risk management file known and reasonably foreseeable hazards, risk scenarios, and hazardous situations. These are distinct concepts, and the standard requires all three to be identified and documented.

Hazards can only lead to harm if a hazardous situation exists. A risk scenario clarifies under which context and conditions a hazard can result in harm by describing what leads to the hazardous situation. A risk scenario can consist of a single event, a sequence of events, a combination of events, a circumstance including all external factors to the system, including normal use and user interactions, or a state such as internal factors to the AI system. Events that form part of a risk scenario can also be referred to as hazardous events.

A hazard can lead to multiple hazardous situations, and each hazardous situation can lead to multiple harms. Hazards and hazardous situations can be technical and non-technical. AI system decisions or outputs can be hazards. Hazards can have more than one cause.

The provider must describe the risk scenarios. Risk scenario descriptions must include known and foreseeable related hazard and hazardous situation, elements affecting the probability of harm occurring, elements affecting the severity of harm, events, sequences and interactions of events, if any, that lead to the hazardous situation, any contributing factors including any relevant failure modes and their underlying potential causes, and AI system characteristics that can lead to hazardous situations.

Risk scenarios must consider at least interactions and behaviors of the system, users and persons potentially affected, contributing human factors, system errors which can include errors resulting from design, development, deployment and incorrect system operation, cyber threats and vulnerabilities, and environmental factors influencing the system, users or person affected. Sequences of events can also comprise a chronological chain of causes and effects, non-occurrence of expected events as well as combinations of concurrent events. A risk scenario can be initiated in all phases of the AI system’s life cycle.

The provider must consider all available relevant information to support the hazard and hazardous situation identification process. Relevant information includes pre-market sources, including testing, expert reviews, stakeholder consultation feedback, available data on comparable systems already on the market, incident databases, published research, regulatory reports, market surveillance findings, and other credible sources that provide insight into potential hazards or known issues. It also includes post-market sources, including incident reports, complaints, potential serious incident data, user feedback and information collected by automatic logging of the AI system. The logs can contain many events of which some can be labeled as hazardous events.

The identification of hazards must include hazards associated with each identified vulnerability and cyber threat to the AI system. prEN 18282 can be used to identify vulnerabilities and cyber threats to the AI system. Vulnerabilities and cyber threats concern the AI system itself, whereas hazards concern risks to health, safety, and fundamental rights.

The AI system provider must, as appropriate, use multiple different risk identification techniques throughout the AI system life cycle to ensure a comprehensive hazards identification process. When identifying hazards and hazardous situations not previously recognized, systematic techniques for risk identification that cover the specific situation can be used. Guidance on some available techniques is provided in relevant standards and guidelines, such as EN IEC 31010.

Top-down techniques, which are typically used in the early planning phases, are valuable for identifying high-level hazards and risk scenarios. As development progresses, diverse methods must be applied to systematically analyze specific hazards, failures and cyber threats, supporting root-cause exploration and targeted risk control measures. These risk identification techniques are complementary and it is often necessary to use multiple approaches in combination to achieve a robust and complete risk analysis.

When identifying hazards and hazardous situations, the provider must consider the specific risks and harms that the AI system can pose to persons under the age of 18 and other vulnerable groups, taking into account their evolving capacities, vulnerabilities and, where applicable, the best interests of persons under the age of 18. This should include risks related to exposure to harmful or inappropriate content, unwanted contact or interactions with adults for persons under the age of 18, privacy violations and data misuse, excessive screen time and addiction, dignity, negative impacts on physical and mental health, and exploitation and abuse.

Risk scenario identification and analysis can inform about the relevant events to be logged. The risk scenario identification and analysis can become more robust as it is iterated through at the different relevant life cycle stages. Frequently, only a first subset of the intended purpose-related hazards and hazardous situations can be identified at the inception and early design and development stages. At the early design and development stage, the obtained results, observations and feedbacks can lead to new hazard, to new risk scenario and to new hazardous situation identifications. Post-market monitoring can lead to the identification of new hazards and hazardous situations and related conditions identifications.

For each of the points in this section that must be considered, the provider must assess the relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement the point. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Hazard identification is where most risk assessments fail. Organizations identify high-level hazard categories such as “bias” or “data quality issues” and treat the identification as complete. That is not compliant with the standard. The standard requires you to identify specific hazards, describe the risk scenarios that lead from hazard to hazardous situation, and document the contributing factors and failure modes. If your hazard identification does not answer the question “what specific event, sequence, or state leads from this specific hazard to this specific hazardous situation, affecting which specific persons in which specific way,” you have not identified the hazard. You have named a category. Build your hazard identification as a structured cause-and-effect analysis, not as a list of concerns.

Risk Estimation

For each identified hazard and hazardous situation, the provider must estimate the probability of occurrence of the associated harm and the severity of that harm, based on the AI system’s intended purpose, reasonably foreseeable misuse, characteristics related to risks, and the state of the art. The provider must estimate the associated risks using available information or data. For hazards for which the probability of the occurrence of harm cannot be estimated quantitatively, the possible harms must be listed and a qualitative estimation made of their probability, for use in risk evaluation and risk control.

When identifying the possible harms resulting from hazards, and in estimating the severity of the associated harm, the groups of persons potentially affected must be determined. Specific vulnerabilities of persons potentially affected can increase the probability and severity of harm. These vulnerabilities can include characteristics and external factors that classify a person as being a member of a vulnerable group, and characteristics and external factors beyond those that classify a person as being a member of a vulnerable group.

For each hazard and hazardous situation, the severity of the associated harms must be estimated taking into account the nature of the harm, which refers to whether it concerns health, safety or fundamental right, the reversibility or irreversibility of the harm, which in relation to persons affected refers to the ability to revert fully or partially to pre-impact situation or equivalent and whether adequate remedies are made available in a timely manner, the number of persons affected in absolute terms rather than only expressed as a proportion of an affected population, the specific characteristics of persons potentially affected, the possible combination of harms, and the potential cumulative effects on persons affected.

For each hazard and hazardous situation related to fundamental rights, the estimation of the severity of the associated harms must also entail the consideration of the character of the right as being an absolute right or qualified right, the applicability of legal obligations imposed on private actors to protect that fundamental right, the level of protection accorded to the right, and the scope, significance and scale of the harm of a potential fundamental rights interference which entails consideration of how widespread its effects and adverse impacts of the hazard or hazardous situation, including whether the rights at risk are those of rights-holder groups that enjoy additional or particular protections. Rights holder vulnerable groups that enjoy additional or particular protection can refer to persons under the age of 18.

If a fundamental rights risk is likely to affect only 0.1% of users on a digital communication platform, it can initially appear negligible, but if this comprises 25% of a religious minority, then the latter indicates that the scope can be serious. This example illustrates why absolute numbers and distributional analysis matter.

An AI system’s intended purpose or reasonably foreseeable misuse can affect a number of different fundamental rights pertaining to multiple persons affected. A hazardous situation can generate risks to multiple fundamental rights that can affect more than one person potentially affected. Risks to the fundamental rights of persons affected can interact with each other, for example when risks are compounded or cumulative.

The strength of legal protection accorded to the activity protected by a fundamental right is based on whether it is an absolute right, a privately enforceable right or a qualified right or a principle in support of a right. The stronger the level of protection accorded to the right, the greater the severity of a potential interference to it.

For the estimation of the severity of fundamental rights risks and the potential effects of interaction from the AI system with persons potentially affected, the provider should consult with persons potentially affected or their proxies, at least before placing on the market or putting into service the AI system. The results of these activities must be documented.

The system used for qualitative or quantitative categorization of probability of occurrence of harm and severity of harm must be documented and recorded. If a risk chart or risk matrix is used for ranking risks for the purpose of estimating their severity and probability, or qualitative estimate of likelihood if probability cannot be estimated, then the parameters and the interpretation of the particular risk chart or risk matrix used must be explained and justified for that application and in accordance with the intended purpose and the reasonably foreseeable misuse of the AI system.

Risk estimation incorporates an analysis of the probability of occurrence of harm and the severity of the harm. Depending on the application area, only certain elements of the risk analysis process can be relevant to consider in detail. When the harm is minimal, an initial hazard and consequence analysis can be sufficient, or when insufficient information or data are available, a conservative estimate of the probability of occurrence can give some indication of the risk. Risk estimation always has a qualitative component and can additionally have a quantitative component. Methods of risk estimation are described in relevant guidelines and standards for application area specific safety risk management, such as EN IEC 31010.

Information or data for estimating risks can be obtained from information received from post-market monitoring system, civil society groups and academic literature, published standards and guidelines for AI risk management, scientific or technical investigations, field data from similar systems already in use including publicly available reports of incidents, usability tests employing typical users, performance metrics and evaluation results, results of relevant investigations or simulations, expert opinion, and external quality assessment schemes for AI systems.

The scale of a health, safety or fundamental rights risk for the purposes of evaluating its severity concerns its gravity, and entails consideration of the potential cumulative effects on persons affected in light of the intended purpose and its reasonably foreseeable misuse, which is also a product of the ease and speed with which the AI system can diffuse across multiple domains and beyond its intended application area. It also entails the assessment of whether persons affected belong to a vulnerable group, including their ability to take protective measures to safeguard their rights and interests. The protected characteristics of vulnerable groups can also be a separate parameter to demonstrate clearly these characteristics have been considered in the severity.

For the purposes of estimating its severity, a fundamental rights risk is considered irremediable if the persons affected cannot be restored to a situation at least equivalent to their situation if there had been no interference to the respective fundamental right through the use of an AI system.

The provider must ascribe the highest severity classification for risks to fundamental rights that are considered absolute rights in accordance with applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk estimation is where the standard’s formula breaks down most visibly. The requirement to estimate probability and severity does not specify how to combine them, and most organizations default to a simple multiplication that treats a 10% chance of moderate harm the same as a 1% chance of severe harm because both produce the same expected value. That approach does not satisfy the requirement to evaluate risk acceptability. Build your risk estimation process to produce not only point estimates but also distributional analysis, confidence intervals, and scenario-weighted outcomes. Document the method you use to combine probability and severity, justify why that method is appropriate for the specific AI system and application area, and present the results in a way that distinguishes between frequent low-severity risks and rare high-severity risks. If your risk estimation outputs a single number per hazard, it does not provide the information top management needs to approve residual risk acceptability.

Risk Evaluation Turns Analysis into Decisions

For each identified hazard and hazardous situation related to the AI system, the provider must evaluate the estimated risks to health, safety, and fundamental rights by comparing them against the predefined risk acceptability criteria in the risk management plan. Based on this evaluation, the provider must determine whether the risk is acceptable or not.

If the risk is acceptable, it is not required to apply risk control activities to this hazardous situation, and the estimated risk must be treated as residual risk. If the risk is not acceptable, then the provider must perform risk control activities to reduce the risk to an acceptable level.

The results of the risk evaluation activities must be recorded in the risk management file with a breakdown of the reasoning for each identified risk to health, safety, and fundamental rights. A risk evaluation that records only a conclusion without the reasoning behind it does not meet the standard. The reasoning must be documented for each risk individually.

Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk evaluation is where the gap between internal comfort and external defensibility becomes most visible. Organizations usually compare estimated risks against acceptability criteria that were defined too loosely to produce a meaningful comparison. If your acceptability criteria say “risks are acceptable when they are low,” and your risk estimation says “this risk is low,” you have performed a circular evaluation that tells a regulator nothing about how you actually made the decision. Build your risk evaluation as a documented comparison between a specific estimated risk, expressed in terms of probability and severity with supporting evidence, and a specific acceptability threshold, defined in terms that can be independently verified. Document the reasoning that connects the evidence to the conclusion. If the reasoning cannot be reproduced by someone who was not in the room when the evaluation was performed, it is not sufficient.

Testing Is Evidence, Not Validation Theater

Testing is an essential activity to support the risk management process. Testing can support the identification of hazards and associated causes, sequences of events and hazardous situations related to intended purpose and reasonably foreseeable misuse, the provision of objective evidence of residual risk and overall residual risk acceptability, and the post-market monitoring system activities, including the monitoring of continuously learning AI systems to ensure they remain within their predefined changes.

Usability testing can be used as a method for validating the effectiveness of human-machine interfaces and instructions for use. Testing performed to identify characteristics of training data sets that reveals inconsistent labelling of the data and bias in representativeness of the data in relation to the intended purpose informs about the appropriate risk control measures to be implemented.

Testing must be applied at least prior to placing on the market or putting into service to identify appropriate risk control measures and to provide objective evidence for the effectiveness of the applied risk control measures. For risk reduction of a specific risk, two different risk control measures can be applied, and the effectiveness of both methods can be tested to select the most effective measure. Verifying that hazards are eliminated through inherently safe design measures is an example of testing applied to provide objective evidence of risk control effectiveness. Testing of the AI system performance to assess creditworthiness of persons that reveals women are consistently discriminated against compared to men is an example of testing that triggers the need for further risk control.

For testing which is aimed to provide supporting objective evidence of the overall residual risk acceptability of the AI system, test acceptance criteria must specify metrics and probabilistic thresholds against which test results are evaluated. Testing that can affect persons must be performed according to applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test Plan

The test plan provides a detailed description of how the testing for the associated test management processes should be done, including how it must be monitored and controlled. The test plan is used by the test monitoring and control process as the basis for managing the testing activity.

The test plan must describe and provide a justification of the following elements, taking into account the state of the art: test objectives, where a test can have multiple objectives for related but distinct purposes, the test item and its relationship to a specific risk and the test objectives, appropriate test methods for achieving the stated test objectives, test completion criteria, resource use, allocation and independence or neutrality principles to conduct testing, collection of testing results and evaluation of testing results in accordance with acceptance criteria, and test plan updates.

In identifying and specifying the test objectives, the provider must identify, justify, and document the assumed relationship between the test objectives and the test methods including the intended test environment, and how the resulting evidence that the test is expected to generate contributes to risk control.

The definition of the test plan can be supported by the companion standards on data quality, robustness, cybersecurity, transparency, bias, and logging. Test acceptance criteria are specific to the test item and test objectives and can depend on specific considerations from other standards. The provider can refer to international standards and state of the art considerations to define the risk acceptance criteria suitable for a particular test item and test objectives.

Testing acceptance criteria must be specified in advance and documented in relation to the test objectives and test methods including the intended test environment. Specifying acceptance criteria after testing is complete defeats the purpose of the testing requirement and will not satisfy a regulator or auditor reviewing the risk management file.

For AI systems potentially impacting persons, the provider must involve relevant stakeholders, stakeholders’ proxies, or an independent cross-functional panel of experts in the design and approval of the test plan. The level of detail of stakeholder involvement can vary depending on the context. Relevant stakeholders can include established user feedback groups involved in public administration applications, patient groups, or trained proxies.

The provider must evaluate the likely impact of the test plan on vulnerable groups and make any necessary adjustments to the test plan to ensure that they are duly protected from adverse impacts. Where applicable, informed consent must be obtained from persons affected and managed in accordance with applicable regulatory requirements. A justification for the representativeness, the number of participating persons, and the duration of the testing process must be documented.

The test plan and all subsequent amendments must be prepared and documented by the provider and, when applicable, in consultation with relevant stakeholders, stakeholder representatives, independent cross-functional panels of experts, or competent authorities. The test plan, including all subsequent amendments, must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test plans are where most organizations reveal that they are testing what is convenient rather than what is required. They test aggregate model performance across the full dataset and conclude that the system performs within acceptable parameters. They do not test performance across demographic subgroups, deployment environments, edge cases, or conditions of reasonably foreseeable misuse. They do not specify acceptance criteria in advance. They do not involve stakeholders in test plan design. And they do not document the relationship between test objectives and risk control. Build your test plan as a structured document that links each test objective explicitly to a specific identified risk, specifies the acceptance criteria before testing begins, documents the rationale for the test method selected, and includes subgroup analysis and misuse scenario testing as standard components. If your test plan does not specify what result would cause you to conclude that a risk is not acceptable, it is not a test plan. It is a performance measurement exercise.

Real-World Conditions Testing

Real-world conditions testing can be used for the purpose of gathering data and as part of fulfilling the requirements of the standard. If real-world conditions testing is performed, the provider must ensure that the risks associated with the testing do not exceed risk acceptability criteria. Real-world conditions testing can be considered when there is insufficient objective evidence that the overall residual risk of the AI system is acceptable for its intended purpose and under conditions of reasonably foreseeable misuse.

If real-world conditions testing is performed, the provider must comply with applicable regulatory requirements. Testing involving persons affected can require consent management in accordance with applicable national laws and international norms of behaviour. Article 61 of the EU AI Act provides more information about informed consent.

Risk management activities with respect to the testing must be performed throughout the real-world conditions testing process. The provider must predefine or establish risk acceptability thresholds and trigger a risk assessment to determine whether actions are needed as soon as thresholds are reached or exceeded. Test monitoring and control process must be documented in the risk management file.

Data generated through real-world conditions testing must be recorded in the real-world conditions testing test completion report, ensuring all personal data are handled in accordance with applicable regulatory requirements. The real-world conditions testing test completion report must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Test Monitoring and Test Reporting

Adherence of the testing procedure to the test plan must be monitored and controlled until test completion. The test plan may be updated by the test monitoring and control process, for instance due to changing requirements such as a test completion date moved forward. Any deviation from the test plan must be recorded.

The following information must be compiled in a test completion report: the testing that was performed, the reference to the test plan elements, any deviation from the test plan, the test results, the assessment of test results against test acceptance criteria, the test monitoring report, and where applicable the evaluation of the effectiveness of the relevant risk control measures.

The test completion report must identify, specify, and document the relationship between all elements listed above to ensure their traceability. The test completion report must be included in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Risk Control Follows a Clear Hierarchy

The standard requires the provider to apply risk control options in a strict priority order. Inherently safe design comes first, protective measures come second, and information and instructions for use together with training to deployers and users come third. This hierarchy is not optional. The provider must apply higher-order controls before resorting to lower-order controls, and must document and justify any departure from this order.

Inherently safe design measures are the most effective risk control measures and by default the preferred risk control measures. They are inherent to the characteristics of the product and are most likely to remain effective. By definition, they are the only form of risk control that can eliminate a hazard, thus eliminating the need for second or third-level risk control measures for that hazard.

Second-level risk control measures in the form of guards and protective measures, even if well-designed, can fail or be violated. Third-level risk control measures rely on the provision of information to address risks and are the least effective of the three forms of risk control because they rely on others following the provided information and hence cannot be ensured.

Applying the hierarchy of risk control requires the provider to work through a defined sequence. If technically feasible, the provider must apply inherently safe design measures to eliminate hazards. A mathematically verifiable algorithm that fails safe in a deterministic manner in real-world situations is an example of inherently safe design. An AI system intended to calculate social security benefits that is technically configured to prevent the generation of any output if there is missing mandatory input data, or if the input data entered falls outside the specified range, and to automatically alert both the deployer and person affected to missing or abnormal input data, is another example. This risk control measure helps prevent erroneous benefit decisions by design, thereby helping ensure respect for the right to good administration.

If elimination of hazards is not technically feasible, inherently safe design measures must be applied to reduce the risk as far as technically feasible, complemented by the application of protective measures to further reduce risk as far as technically feasible. If inherently safe design measures cannot be used or do not achieve risk reduction to an acceptable level, a justification must be documented and protective measures must be used to achieve an acceptable level of risk. If inherently safe design measures and protective measures cannot be used or do not achieve risk reduction to an acceptable level, a justification must be documented and instructions for use may be used as a risk control measure to reduce risk to an acceptable level. Instructions for use and training must be applied only to complement inherently safe design measures and protective measures and must not be relied upon solely and exclusively to reduce risk to an acceptable level.

The provider must add required information to the instructions for use in accordance with applicable companion standards. In order to further reduce the risks from the use of the AI system, the provider must assess whether to include training instructions to deployers, deployment, maintenance and decommissioning instructions, and operational, troubleshooting and emergency instructions that include actions to be taken by the deployer regarding hazards and hazardous situations, including measures to prevent exposure to them, measures to reduce the probability and severity of resulting harm, and remedial actions to take if harm does occur.

The provider must document their reasoning and justification for not including any of this information in the instructions for use. The provider can provide additional information not listed as part of the instructions for use.

Where applicable, the provider must define appropriate training for deployers of the AI system considering their competence, including technical knowledge, skill, experience and education, and including competence regarding persons under the age of 18 and other vulnerable groups, in line with the intended purpose and reasonably foreseeable misuse of the AI system.

Risk control measures must be explicitly designed, implemented, and documented in a manner that mitigates identified risks directly, independent of internal organizational policies or procedures. This is one of the most consequential requirements in the standard. Internal organizational policies and procedures must not be considered risk control measures because they do not concretely address potential harm in a specific and directly verifiable way. If your risk control measure is “we have a policy requiring human review of all high-risk decisions,” that is not a risk control measure. It is a procedural requirement. The risk control measure is the technical mechanism that ensures the human review actually occurs and is documented.

All identified residual risks, as well as any associated cautions and warnings, must be clearly documented and communicated in the accompanying documentation.

To identify and implement the appropriate type of risk control measure, the provider can refer to the companion standards on transparency, robustness, cybersecurity, data quality, bias, and logging. To identify the most appropriate risk control measures, the provider must take into account the state of the art, the reasonably foreseeable technical knowledge, experience and education of the deployer, and the reasonably foreseeable context in which the AI system will be used. The provider must review whether, due to advances in the state of the art, more effective risk control measures are available.

Technical feasibility has multiple considerations, including the state of the art, the maturity of a solution, the use of a precautionary approach, the specific industry vertical, and the AI technology on which the AI system is based. Technical infeasibility means that no design or development techniques or production methods can reasonably be considered feasible. If other members of an industry vertical are achieving a certain measure, hazard elimination, level of risk reduction, or level of protection, then it is likely considered technically feasible. Risk control measures can reduce the severity of the harm or reduce the probability of occurrence of the harm, or both. Risk control measures for one hazard or risk can increase another risk. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

The prohibition on treating internal policies and procedures as risk control measures will catch most organizations off guard. Many existing AI governance frameworks are built on policies, approval workflows, ethics review processes, and training requirements. Under this standard, none of those count as risk control measures. They may support the risk management process, but they do not substitute for technical controls that mitigate identified risks directly and in a verifiable way. Audit your existing risk control inventory against this requirement before you file your risk management documentation. For every control you have listed, ask whether it can be verified independently of whether anyone followed the policy. If the answer is no, you need a different control.

Implementation and Verification of Risk Control Measures

The provider must implement the risk control measure selected at appropriate stages in the life cycle of the AI system. Implementation of each risk control measure must be verified. Risk control measures must be verified by gathering objective evidence, including verification by inspection and analysis. This verification must be recorded in the risk management file.

The effectiveness of the risk control measures along the life cycle of the AI system must be verified. This verification must include testing in accordance with the testing requirements of the standard. The results of this verification must be recorded in the risk management file.

When the intended user profile includes vulnerable groups, the verification of risk control measures must include evaluation methods specific to their needs and vulnerabilities. This can include usability testing such as age-appropriate usability testing for persons under the age of 18, expert review, and consultation with specialists with expertise supporting vulnerable groups such as child development specialists.

Verification of the effectiveness of risk control measures can include consultation with persons potentially affected or their proxies, including civil society organizations. Real-world conditions testing can be performed in order to validate the effectiveness of risk control measures. If performed, this verification must be recorded in the risk management file.

The provider must review the effects of the risk control measures with regard to whether any new hazards or hazardous situations are introduced, or whether the estimated risks for previously identified hazardous situations are impacted by the introduction of the risk control measures. Risks from new hazards or hazardous situations, and estimated risks impacted by the introduction of risk control measures, must be estimated and evaluated and controlled as necessary. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Residual Risk and When to Stop

After the risk control measures are implemented and verified, the provider must evaluate the residual risk using the criteria for risk acceptability defined in the risk management plan. The acceptable risk must be justified, taking into account the potential adverse impact on persons. Differences in the AI system performance can lead to discrimination of specific groups of persons affected, including vulnerable groups, and prEN 18283 provides more information on this.

The results of this evaluation must be recorded in the risk management file. If a residual risk is not judged acceptable using these criteria, further risk control measures must be considered and the process must return to the risk control activities until the risk acceptability criteria is met.

In the case a residual risk remains unacceptable and the provider finds that no risk control measures are technically feasible, the provider may conclude that a change of intended purpose of the AI system is necessary, returning to the intended purpose documentation and restarting the risk identification process for the revised purpose. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Completeness of Risk Control

Adequate risk reduction is achieved when all state of the art design and development and risk control measures have been duly considered and adopted or the reasons for refraining from adoption are documented and included in the risk management file, each hazard has been either eliminated or its estimated risk has been reduced to an acceptable level, any new hazards introduced by the risk control measure have been properly addressed, users are sufficiently informed and warned about the residual risks, and protective measures are compatible with one another.

After all residual risks have been evaluated, the provider must review the risk control activities to ensure that the risks from all identified hazards have been considered and all risk control activities are completed. The results of this review must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Completeness of risk control is the standard’s quality gate before the overall residual risk evaluation. Most organizations treat it as a checklist item. The requirement to document reasons for not adopting available state of the art risk control measures is more demanding than it appears. If a peer organization in your sector has implemented a more effective bias control measure, a more robust monitoring system, or a more transparent explanation mechanism, and you have not adopted it, you need to document why. “We chose a different approach” is not sufficient. “We evaluated the following alternative measures, concluded they were not technically feasible for the following reasons, and implemented the following alternative approach, which achieves the following level of risk reduction” is what the standard requires.

Looking at the AI System and Residual Risk as a Whole

After all risk control measures have been implemented and verified, the provider must evaluate the overall residual risk posed by the AI system using the criteria for acceptability of the overall residual risk defined in the risk management plan. All identified hazards have been evaluated and all risks have been addressed by risk control measures to reduce them to an acceptable level. Even if each risk is reduced to an acceptable residual risk, the aggregation of all residual risks can be unacceptable.

The evaluation of the overall residual risk must take into account the factors required for establishing overall residual risk acceptability criteria, and the potential aggregation of each risk with low or medium severity over time, across users, or through repeated interactions with the AI system. This last element is particularly important. A risk that is acceptable in a single interaction can become unacceptable when multiplied across millions of users or repeated over extended periods.

The evaluation of overall residual risk must be supported by objective evidence obtained in accordance with the requirements for establishing risk acceptability criteria. Objective evidence must include test results demonstrating that the AI system performs consistently for its intended purpose and under conditions of reasonably foreseeable misuse. Explanation and justification must be provided for how this objective evidence demonstrates the acceptability of the overall residual risk.

If the overall residual risk is judged acceptable, the provider must inform deployers of significant residual risks, according to the intended purpose and the reasonably foreseeable misuse, and must include the necessary information in the accompanying documentation in order to disclose those residual risks. The provider should make the information openly available in digital and online formats.

If the overall residual risk is not judged acceptable in relation to the intended purpose and reasonably foreseeable misuse, the provider may consider implementing additional risk control measures, modifying its intended purpose, or achieving the intended purpose by not using an AI system. Otherwise, the overall residual risk remains unacceptable and in that case the AI system must not be deployed. This is a hard stop. The standard does not permit a provider to deploy a system with an unacceptable overall residual risk and manage the consequences reactively.

Evaluating overall residual risk is a decision made by the provider but can be influenced by policies and norms established by organizations, industries, communities, and policy makers. The results of the evaluation of the overall residual risk must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

The overall residual risk evaluation is where the standard most clearly diverges from how most organizations make deployment decisions. Most organizations approve deployment when each individual risk has been addressed and the system passes its performance benchmarks. The standard requires an additional step: evaluating whether the aggregate of all residual risks is acceptable, considering interactions between risks, cumulative effects over time and scale, and the distribution of harms across affected populations. Build your overall residual risk evaluation as a distinct documented decision, separate from the individual residual risk evaluations. Present it to top management with a summary of all residual risks, their interactions, their cumulative potential, and the objective evidence supporting the acceptability conclusion. If top management has not explicitly approved the overall residual risk evaluation, the deployment decision does not meet the standard’s requirements.

Reviewing the Process, Not Just the Outcome

The provider must review the execution of the risk management plan periodically throughout the life cycle phases of the AI system, and at least prior to placing on the market or putting into service the AI system.

This review must at least ensure that the risk management plan has been appropriately implemented, the overall residual risk is acceptable, and appropriate methods are in place to collect and review information in the pre-market and post-market phases. The results of this review must be recorded and maintained in the risk management file.

The responsibility for review must be assigned in the risk management plan to persons having the appropriate competence and authority. The risk management review must be approved by top management. This is not a staff-level activity. The review is a top management obligation with documented approval.

When, based on information from the provider’s post-market monitoring system or its real-world conditions testing, the provider identifies a serious incident or identifies a situation where a serious incident is avoided but can reasonably have occurred, the provider must decide on the necessity or desirability of a risk management review. The decision not to perform a risk management review must be justified. The default assumption is that a serious incident or near-miss triggers a review. Departing from that default requires a documented justification.

All modifications implemented as a consequence of the review must also be documented in the risk management file. Top management, or the provider generally, can have requirements related to the notification of serious incidents, or their avoidance, to relevant stakeholders in accordance with applicable regulatory requirements. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Learning from the Pre-Market and Post-Market Activities

The provider must establish, document, and maintain a system to actively collect and review information relevant to the AI system in pre-market and post-market phases in accordance with applicable regulatory requirements. When establishing this system, the provider must consider appropriate methods for the collection and processing of information.

For each of the points that must be considered, the provider must assess the relevance and document their reasoning. Furthermore, the provider must include a justification if they choose not to perform or implement the point. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Information Collection

Pre-market and post-market activities can include receiving information about performance and risks posed by the AI system. The information can be related to harm that has occurred or to hazardous situations that occurred without harm. The activities can also include soliciting information about the AI system performance and related risks. These activities can involve reaching out to users, deployers, or other relevant stakeholders to obtain specific information and insight, using methods such as surveys, expert user groups, or consultations. They can also include publicly available information, incident reports, incident databases, and information on the state of the art.

The provider must collect information that is relevant to managing the AI system risks in the pre-market and post-market phases. This information must include, where applicable, information generated during pre-market life cycle stages and monitoring of the development process, information generated from the post-market monitoring system, information collected by automatic logging of events which the provider has identified as relevant to ensure that residual and overall residual risks are maintained to an acceptable level, and information generated by the users, including information from human oversight, user complaints, and other feedback.

This can include a general AI system feedback report capturing general AI system feedback from the user. It can also include an AI system incident report generated based on users reporting failures, malfunctions, or any unexpected behaviors observed in the AI system.

The information collection must also include information, warnings and complaints issued by stakeholders affected or their proxies, information generated by those accountable for the installation, use and maintenance of the AI system, information generated by the supply chain, publicly available information including information about similar AI systems and similar other products on the market, information related to the state of the art, and identification of unforeseen risks in relation to the execution of predetermined changes.

Publicly available information can refer to judgements of court cases, freely accessible reports, or any other relevant accessible content. Regulatory requirements can apply regarding the information being collected, including requirements on data protection, confidentiality, and permitted use. Stakeholders affected include those who have been identified during the risk identification process as placed at risk.

Justification for not collecting information related to the points above must be documented in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Information Review

The provider must review the information collected for possible relevance to the overall residual risk acceptability, especially whether previously unrecognized hazards or hazardous situations are present, an estimated risk arising from a hazard is no longer acceptable, the overall residual risk is no longer acceptable in relation to the intended purpose or applicable national, regional, or international regulations, the state of the art has changed, or changes to the AI system that were not foreseen or planned have occurred.

The results of the review must be recorded in the risk management file. Documentation of the activities and the resulting records must be included and maintained in the risk management file.

Actions to Take

If the collected information is determined to be relevant to the overall residual risk acceptability, the following actions apply.

Concerning the particular AI system, the provider must review the risk management file and determine if reassessment of risks or assessment of new risks is necessary. If a residual risk, whether previously known or newly identified, is no longer acceptable, the impact on previously implemented risk control measures must be evaluated and must be considered as an input for modification of the AI system. If a residual risk, whether previously known or newly identified, is no longer acceptable, the provider must evaluate and justify whether or not the AI system must temporarily or definitively be withdrawn from service or from the market based on the severity of the identified unacceptable risk. The provider should inform deployers and relevant stakeholders without delay of the increased residual risks and possible mitigations through a field notice such as a website message. Any decisions and actions must be recorded in the risk management file.

Concerning the risk management process, the provider must evaluate the impact on previously implemented risk management activities. The results of this evaluation must be considered as an input for the review of the suitability of the risk management process by top management. If an unforeseen change has been identified, the provider should consider whether a risk reassessment of the AI system is necessary, especially if the unforeseen change affects the intended purpose of the AI system.

Concerning communication with relevant stakeholders, including deployers and users, the provider must inform about changes to the overall residual risk acceptability of the AI system.

Post-market activities are where most AI governance frameworks have their largest gap. Organizations invest heavily in pre-deployment risk assessment and virtually nothing in systematic post-deployment monitoring of compliance-relevant outcomes. The standard requires an active system for collecting and reviewing information, not a passive incident log. Build your post-market monitoring system as a structured program with defined data collection points, automated logging of hazardous events, scheduled information reviews, and documented decision criteria for triggering risk reassessment. Establish clear thresholds for when collected information requires immediate action, periodic review, or escalation to top management. If your post-market monitoring system cannot answer the question “is the overall residual risk of this system still acceptable today, given what we know from post-market experience,” it does not meet the standard’s requirements. And if that question is not being asked at regular intervals by someone with the authority to act on the answer, the system is not operating as the standard requires.

Understanding How AI Risks Unfold in Practice

The examples below illustrate how hazards, risk scenarios, hazardous situations, and harms connect in real AI deployments. Each example follows the same logic: a potential cause creates a hazard, a risk scenario describes the conditions under which the hazard can lead to harm, a hazardous situation describes the moment of exposure, and the harm describes what actually happens to affected persons. These examples are illustrative, not exhaustive, and applicable regulatory requirements regarding use cases and harms are subject to change.

Reading these examples as a risk practitioner, the most important pattern to notice is that the harm rarely flows directly from a technical failure. It flows from a chain: a design choice or operational condition creates a hazard, a specific scenario activates that hazard, and a person in a specific situation suffers the consequence. Breaking any link in that chain is the job of risk control.

Example 1: Skin Cancer Detection App

What the system does

A medical AI application intended to provide an indication of possible skin cancer from self-taken skin images, designed for any skin type.

What causes the hazard

The AI model was trained primarily on images from people with white or light skin, with non-representative or very limited coverage of dark skin. Testing with dark skin images was either not performed or severely limited. In some cases, the biased output could also result from a data poisoning attack on the training data rather than from inappropriate design choices alone.

What the hazard is

The system produces biased output in the form of false negatives. It systematically fails to detect skin cancer in dark-skinned patients. The hazard here relates directly to AI system performance and the quality of the training data.

How the risk scenario unfolds

A dark-skinned user who has skin cancer uses the app. The app returns a negative result, indicating no skin cancer is present. Trusting the result, the user does not consult a doctor for further examination of the skin abnormality.

What the hazardous situation looks like

The patient believes they have no skin cancer. They are now exposed to the continued and undetected development of the disease, potentially including metastasis, without any medical follow-up.

What harm results

Progression of the disease, worsening health condition and prognosis, and risk of death if metastatic skin cancer goes undetected over time.

What this example teaches

A system that appears to work well on average can systematically fail for specific demographic groups. Risk analysis must assess performance across subgroups, not just across the full population. The harm is not caused by a dramatic system failure. It is caused by a result that looks valid but is wrong for a specific group of users that the system was not adequately trained to serve.

Example 2: Credit Worthiness Evaluation in a Bank

What the system does

An AI system that evaluates the creditworthiness of natural persons, used by financial consultants in a bank to process loan applications.

What causes the hazard

After deployment, the bank reduces the number of financial consultants by 80 percent, reasoning that the AI system can absorb most of the workload. The remaining consultants must now process a much higher volume of cases than before. This is a reasonably foreseeable misuse of the system that was not anticipated in the original risk assessment. Compounding this, during the first ten interactions with the system, the consultants find that the AI recommendations appear accurate. This creates automation bias: the consultants begin to rely on the system’s recommendations without applying independent judgment. This is a human factors issue linked to the design of the user interface and the feedback the system provides.

What the hazard is

The hazard is poor human oversight resulting from the combination of high workload and automation bias. The hazard here relates to human-machine interaction rather than a technical failure in the model itself.

How the risk scenario unfolds

Financial consultants must process a large number of cases and have limited capacity to critically evaluate each AI recommendation. They validate recommendations, including erroneous ones, without sufficient independent review.

What the hazardous situation looks like

A consultant validates an erroneous AI recommendation without detecting the error. The applicant’s loan application is decided based on a biased or incorrect output from the system.

What harm results

Denial of loan applications for applicants based on characteristics such as citizenship, where the AI system has introduced discriminatory patterns that the consultants are not positioned to detect or correct.

What this example teaches

Organizational decisions made after deployment can create new hazards that were not present at launch. Reducing human oversight capacity after deploying an AI system is a foreseeable misuse that must be analyzed in the risk assessment. Automation bias is a predictable human response to a system that appears accurate in early use. Risk control must address both the technical output of the system and the conditions under which humans interact with it.

Example 3: Clinical Decision Support for Rare Disease Diagnosis

What the system does

A large language model used as a clinical decision support system for diagnosing rare diseases.

What causes the hazard

The model was fine-tuned on a narrow clinical dataset that lacked diversity in demographics and rare case data. Benchmark results were misinterpreted, either because the benchmarks used saturated tasks that did not reflect real clinical complexity, or because the results created a false impression that the model would rarely produce incorrect information in a broad range of cases. The model appears to perform well on standard benchmarks but overfits to the narrow training distribution.

What the hazard is

The system produces misleading diagnostic recommendations because it does not generalize well beyond its training data. The hazard is poor model performance in conditions that differ from the training environment.

How the risk scenario unfolds

A clinician, relying on the system’s high reported accuracy, over-relies on an incorrect recommendation and ignores contradictory clinical signs that would, under normal circumstances, prompt further investigation or specialist referral.

What the hazardous situation looks like

The patient receives incorrect treatment or is not referred for necessary specialist care because the clinician trusted the AI recommendation over their own clinical judgment.

What harm results

Delayed diagnosis, worsening health condition, and potential irreversible harm or death.

What this example teaches

Benchmark performance does not translate directly to real-world safety. A model that scores well on published benchmarks can still fail dangerously in clinical practice if the benchmarks did not capture the distribution of cases the model will encounter in deployment. Risk analysis must include an assessment of how benchmark results were derived and whether they are representative of the intended deployment context. Clinician reliance on AI outputs is a human factors hazard that must be explicitly addressed in risk control, not assumed away by the system’s reported accuracy.

Example 4: AI System Screening Job Applicants

What the system does

A large language model used to screen job applicants, providing recommendations based on CVs and job descriptions.

What causes the hazard

Benchmark scores were misinterpreted as demonstrating general fairness across domains, but the benchmarks had limited coverage or were saturated and did not measure the model’s behavior on the specific task of CV screening. Additionally, the benchmarks did not measure robustness against CVs specifically crafted to manipulate the model into generating a very positive assessment, a known adversarial input risk.

What the hazard is

The system produces wrong decisions due to unintended bias. Biases embedded in training data, including gender, race, and age, and the model’s vulnerability to adversarial inputs, are assumed to have been addressed when they have not been.

How the risk scenario unfolds

The system is deployed with the assumption that bias and robustness issues are resolved. Candidates are exposed to a decision process that contains unintended discrimination against specific groups of people.

What the hazardous situation looks like

Qualified candidates from discriminated groups are evaluated by a system that systematically rates them lower than equivalent candidates from other groups, without the organization recognizing that the system is producing discriminatory outputs.

What harm results

Discriminatory hiring outcomes. Qualified candidates are rejected on the basis of characteristics such as gender, race, or age rather than on the merits of their application.

What this example teaches

Fairness in AI is not a binary state that is achieved once and maintained automatically. It must be tested specifically for the task and dataset at hand, not inferred from general benchmark performance. Robustness to adversarial inputs is a separate dimension of risk that must be assessed independently from fairness. Deploying a system on the assumption that known risk categories have been resolved, without task-specific evidence, is a risk management failure that the standard explicitly requires providers to avoid.

Example 5: AI Agent Managing Energy Grid Optimization

What the system does

A goal-directed AI system deployed to autonomously manage energy grid optimization.

What causes the hazard

The AI system exhibits specification gaming behavior, meaning it finds ways to maximize its performance metrics that were not intended by the designers and that do not align with safe grid operation. The system’s limited interpretability makes it difficult for operators to understand what decisions the system is making and why.

What the hazard is

The AI monitoring and control interface does not provide sufficient information about the system’s decisions and their effects. Operators cannot see what the system is doing or why it is doing it.

How the risk scenario unfolds

The system is deployed and begins optimizing grid operations in ways that are not visible to operators. It puts the grid into an unsafe operating mode without operators recognizing that this has occurred. The risk of cascading failures across interdependent systems grows without detection.

What the hazardous situation looks like

The grid is being run in an unsafe mode that creates a high probability of blackouts and equipment failure, while operators believe the system is functioning correctly.

What harm results

Physical damage to infrastructure, large-scale blackouts, and adverse health effects on persons dependent on continuous power supply.

What this example teaches

Specification gaming is a well-documented failure mode in goal-directed AI systems. A system that optimizes for the wrong objective can cause serious harm even when it is technically functioning as designed. Interpretability is not an optional feature. It is a prerequisite for human oversight in high-stakes deployments. Risk control must include mechanisms that allow operators to understand and intervene in system behavior before unsafe states develop.

Example 6: AI Monitoring Warehouse Workers

What the system does

An AI system used to organize warehouse work through real-time monitoring of worker activity.

What causes the hazard

The system monitors worker characteristics that are not necessary for its stated operational purpose, collecting data beyond what is required for warehouse organization.

What the hazard is

The system monitors unnecessary worker characteristics, exceeding the scope of what is proportionate for warehouse management.

How the risk scenario unfolds

Workers performing warehousing tasks are placed under continuous real-time AI monitoring. The system operates constantly throughout the working day.

What the hazardous situation looks like

Workers are subject to continuous AI-enabled surveillance, including monitoring of characteristics that are not relevant to their work performance and that they have not meaningfully consented to.

What harm results

Violation of data rights, psychosocial harassment, continuous performance pressure, and risk of job loss based on monitoring data that exceeds the legitimate scope of the system’s intended purpose.

What this example teaches

Workplace AI systems can cause harm through scope creep, monitoring more than is necessary for the stated purpose. The proportionality of data collection must be assessed as part of the risk analysis, not just the technical accuracy of the monitoring. Workers in high-monitoring environments experience real psychological harm from surveillance even when no action is taken on the data. This is a harm within the meaning of the standard.

Example 7: AI Evaluating Teachers’ Activity

What the system does

An AI tool used to evaluate teachers’ activity, including assessment of pupils’ and students’ performance, and providing automatic feedback to assessors.

What causes the hazard

The information that the system needs to make accurate evaluations cannot be accurately or reliably connected to the system’s inputs. The data that would be required to make meaningful assessments of teacher quality is not consistently available or measurable in the form the system expects.

What the hazard is

The system produces evaluations of teacher quality based on data that does not accurately reflect what it purports to measure.

How the risk scenario unfolds

The tool is used for teaching and evaluation in classrooms. Teachers are evaluated based on AI-generated assessments that may not reflect their actual performance or the factors that influence student outcomes.

What the hazardous situation looks like

Teachers are subject to consequential evaluations produced by a system whose inputs do not accurately represent their professional activity. Students are also affected through assessments that may not reflect their actual learning.

What harm results

Violation of data rights, psychosocial harassment, continuous pressure from unjustified performance assessments, and risk of job loss based on AI evaluations that do not accurately reflect performance.

What this example teaches

The quality and representativeness of input data is as important as model performance. A technically sophisticated system that operates on inputs that do not accurately represent the phenomenon it is supposed to evaluate will produce systematically misleading outputs. This is a hazard that must be identified in the risk analysis and addressed in risk control, not assumed away by system accuracy metrics.

By Prof. Hernan Huwyler, CAIO MBA CPA

About the Author

The frameworks, tools, and implementation guidance described in this article are part of the applied research and advisory work of Prof. Hernan Huwyler, MBA, CPA, CAIO. These materials are freely available for use, adaptation, and redistribution in your own AI governance, risk management, and compliance programs. If you find them valuable, the only ask is proper attribution. If you like the content, please like the article and share it.

Prof. Huwyler serves as AI GRC Consultancy Director, AI Risk Manager, and Quantitative Risk Lead, working with organizations across financial services, technology, healthcare, and public sector to build practical AI governance frameworks that survive contact with production systems and regulatory scrutiny. His work bridges the gap between academic AI risk theory and the operational controls that organizations actually need to deploy AI responsibly.

As a Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative predictive risk automation, and AI audit readiness for executive leadership teams, boards, and technical practitioners. His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe and internationally.

Based in the Copenhagen Metropolitan Area, Denmark, with professional presence in Zurich and Geneva, Switzerland, Madrid, Spain, and Berlin, Germany, Prof. Huwyler works across jurisdictions where AI regulation is most active and where organizations face the most complex compliance, technical and business requirements.

His code repositories, risk model templates, and Python-based tools for AI governance are publicly available at . His ongoing writing on Governance, Risk Management and Compliance appears on his blogger website at (more than 500k views).

Connect with Prof. Huwyler on LinkedIn at to follow his latest work on AI risk assessment frameworks, compliance automation, model validation practices, and the evolving regulatory landscape for artificial intelligence.

If you’re building an AI governance program, standing up an AI risk function, preparing for EU AI Act compliance, or looking for practical implementation guidance that goes beyond policy documents, reach out. The best conversations start with a shared problem and a willingness to solve it with rigor.

Shadow AI Risk Management for CAIOs

Mon, 30 Mar 2026 00:00:00 +0000

Implementation Guide for Shadow AI to Secure Operations

Shadow AI is already inside many organizations. It shows up in browser extensions, AI features inside SaaS tools, copied customer data pasted into chatbots, and internal models quietly updated with third party AI services. That creates a brutal problem for , IT, risk, and compliance teams. You cannot control what you cannot see, and by the time you do see it, the damage may already be done.

Traditional security controls often miss Shadow AI because the activity happens inside normal browser sessions, encrypted traffic, SaaS APIs, or approved endpoints.

This is why getting Shadow right matters now. Data leaks, biased decisions, weak audit trails, and hidden third party dependencies can trigger customer harm, regulatory action, and operational failures. This post gives you a practical implementation checklist for finding, controlling, and reducing Shadow AI across internal models, third party software, and employee use of generative AI tools.

Understanding the Framework for Shadow AI Risk Management

You need a clear mental model before you start buying tools or writing policies.

The most useful way to think about Shadow AI is through three exposure paths. First, hidden internally built models. Second, AI embedded in third party applications. Third, unauthorized use of external AI tools by employees. If you do not separate these paths, your controls will be too vague to work.

1. Hidden Internally Built Models

This is when teams quietly add AI capabilities to an existing workflow, model, or decision process without proper review. A credit risk model may start calling an external LLM API. A service team may build a customer response assistant in a spreadsheet-driven workflow. A developer may add prompt-based automation into a business process and never flag it as a model change.

The risk is larger than model performance. You may inherit privacy exposure, explainability gaps, weak testing, and undocumented decision logic.

Tip: Treat any system that takes probabilistic output from an AI service and uses it in a business workflow as a model change event. Many firms miss this because they only track full standalone models. That is a mistake. A hidden AI call inside an existing process can change outcomes just as much as a new model.

2. AI in Third Party Applications

Many firms approve software once and assume they understand what it does forever. That assumption breaks fast once vendors start adding copilots, embedded classifiers, content generators, or ranking systems during routine updates.

The widespread adoption of AI-supported browser add-ons, including translators, copilots, grammar checkers, meeting voice transcription, time optimizers, and general browser assistants, has fueled the rise of “Shadow AI,” where employees bypass IT oversight to use these tools for immediate productivity gains. While these extensions offer powerful capabilities, their unmanaged use creates significant security blind spots, as sensitive corporate data is often processed by external AI models without the formal governance, compliance checks, or data protection controls for third-party applicatoins required by the organization.

This is one of the hardest Shadow AI risks to manage. The AI may sit inside a black box feature, a recommendation engine, or an automated workflow that was not present when procurement first reviewed the tool.

Tip: Add “AI capability change” as a mandatory vendor review trigger. Do not wait for annual reassessment. Require vendors to disclose any new AI or model-driven feature in product updates, release notes, or contract notices. If you do not ask directly, many vendors will not tell you clearly enough.

3. Unauthorized Internal Use of Third Party AI Tools

This is the most common form of Shadow AI. Employees paste source code into ChatGPT. Analysts summarize contracts in Claude. Marketing teams use browser-based AI tools through personal logins. Product teams connect meeting notes, email, or file systems to unsanctioned copilots.

It feels harmless in the moment. It rarely is.

The core risk is not the chatbot itself. The core risk is uncontrolled data transfer, weak identity controls, and no audit trail.

Tip: Do not frame this only as an employee misconduct issue. Most people use Shadow AI because approved alternatives are too slow, too confusing, or too limited. If the approved path takes three weeks, staff will route around it by lunchtime.

Surviving The Shadow AI Epidemic

We are witnessing a massive crisis in enterprise technology adoption, specifically with generative and agentic tools being deployed without formal system approvals. Non-technical employees, those outside the IT department, are using models like Claude to build internal applications completely unsupervised, sharing local server URLs, executing direct queries against production databases, and generating massive technical debt that IT inevitably inherits without ever being consulted on the design. I look at this and see a modern evolution of the classic Microsoft Access and Excel sprawl, but vastly accelerated because it now includes full user interfaces.

We classify this phenomenon as Shadow AI, and it is rapidly becoming one of the most severe operational risks we face. The industry is already documenting cases where unauthorized applications cause security breaches, compliance violations, and critical system degradation simply because end users do not understand the architectural implications of what they are deploying. To fix this, organizations must enforce strict policies for the planning, management, approval, and operation of any application built outside the IT department.

When I speak with engineering leaders, they anticipate a future where developers transform into orchestrators who spend their days fixing code that is perfectly programmed but architecturally horrendous. The core concern is the long-term maintainability of AI-generated systems built without fundamental design understanding. While agentic tools like Copilot significantly increase coding velocity, the generated code consistently presents more latent defects caught during review and higher cyclomatic complexity compared to code written manually by the same developer.

Generative models produce code that easily passes basic unit tests but routinely fails on edge cases, error handling, and security considerations that an experienced engineer would incorporate by default. Generative AI accelerates software production but fundamentally misunderstands architectural trade-offs, meaning most AI-generated code requires significant refactoring before it is safe for a production deployment. It is functionally correct in the short term but architecturally fragile in the long term, eventually requiring complete rewrites because it simply does not scale.

This fragility extends directly to enterprise infrastructure. We are seeing companies grant read-only database permissions to non-IT users who then execute AI-generated queries that completely degrade production performance, trigger timeouts, and crash critical applications sharing the same infrastructure. Without human optimization, SQL queries generated by large language models consume significantly more compute resources than equivalent queries written by experienced database administrators.

The primary issue is that the AI lacks an understanding of indexes, join orders, and execution plans. The model optimizes solely to deliver the correct answer, completely ignoring computational efficiency or the resulting attack surface, which leaves these unsupervised queries vulnerable to timing attacks, metadata exposure, and unintentional denial of service. We are already documenting a sharp percentage increase in performance incidents attributed directly to unoptimized exploratory queries executed by business users conducting ad-hoc analysis. The practical, technical solution is to use isolated read-only replicas, dedicated endpoints with query governors, and strict rate limiting. Allowing direct access to production databases without these controls is terrible architecture, regardless of whether artificial intelligence is involved.

At a technical level across these organizations, there is no actual orchestrator, no architectural guidelines, and no approval process, allowing every employee to operate as an unsupervised independent developer. This decentralized adoption without policies, processes, or accountability inevitably drives up security incidents, contractual breaches, and operational overhead. In my practice, I strongly recommend establishing AI Review Boards, deployment approval workflows, and strict technical standards before enabling generalized access to generative tools. The lack of governance is an organizational failure, not a technical one. The correct solution requires implementing an acceptable use policy for AI, a standardized approval workflow for deployment, baseline technical standards covering the stack and security, mandatory code reviews, and explicit ownership assignments. Without these five concrete elements, the operational chaos we are seeing is entirely inevitable.

I predict a definitive evolution toward an AI product manager role where developers supervise generated code rather than writing it manually, shifting the focus from raw technical typing to directing generative tools. In companies that have intensively adopted Copilot over several months, the role of senior developers has already shifted toward architecture, code review, debugging generated code, and component integration. The time spent writing new code drops by half, while the time spent on supervision and correction rises proportionally. While productivity measured by shipped features increases, the complexity of debugging and maintenance spikes because AI-generated code is inherently less predictable than code written organically by the team. However, viewing the developer simply as an orchestrator drastically underestimates the complexity of actual software engineering. Generative AI is highly effective for well-defined, repetitive tasks, but it fails consistently at complex system design, subtle debugging, and optimization under non-obvious constraints.

The developer role is transforming empirically, leaning heavily into supervision, but deep technical skill remains absolutely critical. Effectively supervising AI requires knowing what the model should have done, identifying exactly where it failed, and knowing how to correct it. Developers who let their technical skills erode will simply be unable to supervise these systems effectively.

Why Shadow AI Is So Dangerous

The biggest danger is simple. Firms do not know what they do not know.

Traditional security controls often miss Shadow AI because the activity happens inside normal browser sessions, encrypted traffic, SaaS APIs, or approved endpoints. An employee can upload sensitive text to an AI tool over HTTPS and your old perimeter controls may see almost nothing useful.

That creates several types of failure at once.

Data Leakage Happens Quietly

A customer service employee pastes complaint records into an external AI tool to draft responses faster. A developer pastes production code to troubleshoot an error. A finance analyst uploads a spreadsheet to summarize trends. Each action can expose regulated data, proprietary logic, or commercially sensitive information.

This is why Shadow AI is usually a data governance problem before it becomes an AI governance problem.

Tip: Monitor outbound data behavior, not just application names. If your control stack only detects known AI apps, you will miss data pasted into browser sessions, API calls, and file uploads to lesser-known tools. Assess modern secure service edge (SSE) and cloud access security broker (CASB) solutions such as Netskope, Zscaler, and Palo Alto Prisma to inspect browser sessions, including inline inspection of GenAI tool interactions.

Bias and Unfair Outcomes Can Spread Without Notice

An unapproved AI component inside a lending, hiring, pricing, or fraud process can shift decisions in ways nobody intended. That can create unfair outcomes, weak explanations, and serious regulatory exposure.

This gets worse when teams assume an external vendor has already tested everything. In regulated environments, that assumption fails quickly. UK PRA SS1/23 makes clear that externally developed models must meet the firm’s internal validation standards.

Tip: Any third party model or AI-assisted decision process that influences customer outcomes should be mapped to an accountable business owner and an independent review owner. If ownership is vague, oversight will fail.

Operational Errors Compound Fast

Shadow AI also creates production risk. A hidden model may drift, hallucinate, degrade, or route work incorrectly. A maintenance prediction tool can trigger unnecessary repairs. A support bot can give customers wrong instructions. An AI summarization feature can omit key terms from legal or compliance workflows.

Small errors scale quickly when automation is involved.

Tip: Watch for sudden changes in operational metrics that do not have an obvious process explanation. Spikes in rework, escalation rates, customer complaints, and exception handling often reveal hidden automation before your model inventory does.

Stage 1: Build a Shadow AI Discovery Process

You cannot govern Shadow AI with policy documents alone. You need discovery.

This stage is about finding where AI is already being used across browsers, endpoints, SaaS tools, internal code, and model workflows. The key parties here are IT operations, security engineering, enterprise architecture, model risk, procurement, and compliance. If one of those groups is missing, your discovery process will have blind spots.

What to Identify

Start with three inventories.

First, AI-related browser extensions, desktop apps, and plugins. Second, SaaS tools with AI features or OAuth-based data access. Third, internal applications, scripts, and models that call external AI services or use AI-generated outputs in production workflows.

What to implement: Create a Shadow AI discovery register with fields for tool name, owner, department, data accessed, authentication method, AI feature description, deployment status, and customer impact. This becomes the base artifact for all later approvals and controls.

How to Discover It

Use multiple detection methods because one method will not be enough.

Review browser extension inventory from managed browsers. Scan endpoint software lists. Pull SaaS app discovery data from CASB or identity tools. Monitor DNS and web proxy logs for known AI domains. Search code repositories for calls to LLM APIs. Review procurement records and release notes for AI-enabled vendor updates. Interview frontline teams in high-use functions like engineering, marketing, support, and analytics.

Yeah, this sounds obvious. But many firms skip the interviews and rely only on technical scanning. That misses shadow workflows running through personal logins, downloaded files, and copied text.

Roles and Handoffs

Security teams usually own browser, endpoint, and network telemetry. Identity teams track OAuth grants and SSO usage. Procurement and vendor risk teams track third party tools. Model risk and compliance teams assess use cases that affect customers or regulated decisions.

The handoff matters. Security may detect a tool, but compliance decides the risk treatment, and business owners decide whether the tool is genuinely needed.

Tip: Run discovery as a recurring operating process, not a one-time clean-up project. Monthly scans with quarterly business review works well for most firms. A one-time inventory goes stale almost immediately because vendors add AI features and employees adopt new tools constantly.

Stage 2: Block Unapproved Installation and Access by Default

Detection alone is not enough. You need preventive controls.

The strongest control pattern is simple. Block unapproved AI tools before users can install them, authenticate to them, or connect them to company data. This is where browser controls, endpoint controls, identity controls, and network filtering need to work together.

Browser and Endpoint Controls

Managed browsers should allow only approved extensions and block all others by default. Endpoint controls should restrict installation of unsanctioned AI desktop apps and maintain an inventory of installed software and extensions.

What to implement: Use enterprise browser policies in Chrome Enterprise or Microsoft Edge to allow-list approved extension IDs. Use Intune or equivalent endpoint management to block unapproved applications and enforce managed browser settings on corporate devices.

This matters because many Shadow AI risks start with browser-based tools that read page content, copy user activity, or send prompts externally.

Identity and OAuth Controls

This is often overlooked.

Many AI tools do not need users to install anything. They just ask for login consent and access to email, files, calendars, or chat data. If your identity controls are weak, users can grant broad access to a third party AI app in seconds.

What to implement: Require admin approval for high-risk OAuth scopes. Enforce SSO for approved AI tools only. Use conditional access to block personal accounts in corporate browsing sessions where possible.

Microsoft’s guidance has been clear on this point. Consent controls are one of the strongest ways to stop accidental exposure of enterprise data through AI-connected SaaS apps.

Network and DNS Controls

Network filtering still matters, but it is not enough on its own.

Block known unapproved AI domains through DNS filtering and secure web gateways. Monitor outbound requests to AI endpoints and flag unusual prompt volume, repeated uploads, or large data transfers.

What to implement: Start with a controlled deny list for high-risk public AI domains, then move toward an approved list model where sanctioned enterprise AI tools remain available through managed identities.

Tip: Do not launch broad blocking without a same-day exception path. If teams lose access to a tool they depend on and there is no fast review process, they will switch to personal devices and unmanaged accounts. That makes the problem worse, not better.

Stage 3: Control Data Transfers to Approved and Unapproved AI Tools

Most Shadow AI incidents are data transfer incidents.

A tool may be approved in general, but that does not mean every dataset, prompt, file, or code snippet is safe to send. Strong Shadow focuses on controlling what leaves the environment, not just which app is open.

Apply DLP to Prompts, Uploads, and Clipboard Activity

This is where many programs fail.

They block a few websites, publish a policy, and assume the problem is solved. Meanwhile, users paste customer records into a sanctioned tool with the wrong settings, or upload code to a plugin embedded in a browser tab.

What to implement: Extend DLP policies to browser uploads, prompt text, clipboard actions, and outbound API traffic where technically possible. Focus first on PII, source code, customer account data, legal documents, and regulated financial information.

Open-source and low-cost tools can help here. Presidio can detect and redact PII before data leaves approved systems. Wazuh can support endpoint alerting. DNS filtering tools such as Pi-hole can support basic blocking in smaller environments.

Classify AI Use Cases by Data Sensitivity

Not all AI use is equally risky.

Summarizing public marketing copy is very different from drafting customer communications from internal case files. Code assistance for low-risk internal scripts differs from AI use on regulated production systems.

What to implement: Create a lightweight data-to-use-case matrix. For each approved AI tool, specify what data classes are allowed, prohibited, or allowed only with masking or redaction. Keep this short enough that employees can actually use it.

Add Redaction and Secure Prompting Standards

Approved AI use still needs boundaries.

If your staff are allowed to use enterprise copilots, define how they should minimize data, remove identifiers, and avoid pasting full records when a partial extract would do.

What to implement: Publish practical prompting rules with examples. Show the wrong way and the safer way. For example, replace a full customer complaint with a redacted summary and a small structured fact set.

Tip: Write your AI policy around data behaviors, not slogans. “Use AI responsibly” is useless. “Do not paste customer names, account numbers, source code, or contract text into any non-approved AI system” is clear and enforceable.

Stage 4: Test Hidden and Approved AI for Performance, Fairness, and Security

Discovery and blocking reduce exposure. Testing reduces harm where AI is allowed or discovered.

This stage belongs to model risk teams, data science leads, security testers, compliance, and business owners. The artifacts include model cards, testing reports, validation records, challenger results, and issue logs.

Test Traditional Model Risks

For internal and third party AI models, you need routine checks for drift, validity, reliability, fairness, and explainability where relevant. If a model affects customer treatment, pricing, eligibility, or risk scoring, the testing standard should be formal and documented.

What to implement: Build a standard AI testing suite that records test scope, datasets used, thresholds, findings, remediation actions, and approval status. Store results in a central record, not scattered across email threads and slide decks.

Test LLM-Specific Risks

LLMs need extra controls.

You need vulnerability testing for harmful content, sensitive data disclosure, prompt injection exposure, factual error rates, refusal behavior, and output consistency. If the model is customer-facing, hallucination testing should be built into pre-release and ongoing monitoring.

What to implement: Use test prompts tied to real business scenarios. Measure false answers, unsafe outputs, unsupported claims, and citation quality. If retrieval-augmented generation is used, test content attribution so teams can see where responses came from.

Monitor Third Party Black Boxes

This is hard, but necessary.

You may not know the full architecture of a vendor model. You can still monitor outcomes, drift in behavior, abrupt response changes, and unexplained shifts after vendor releases.

What to implement: For each high-impact third party AI tool, define operational metrics and trigger thresholds. Examples include complaint rates, override rates, latency, exception rates, and accuracy against sampled cases.

Tip: The first testing framework is often too ambitious and collapses under its own weight. Start with a small mandatory baseline for all AI use cases, then add deeper testing for high-impact systems. If every tool needs a 60-page validation pack, teams will hide usage instead of declaring it.

Stage 5: Put Governance Approval Gates Into the Workflow

A Shadow AI program fails when approval is separate from real work.

If teams need to leave their normal project flow, fill in a dense form, and wait two weeks for a committee, they will bypass the process. Good governance lives inside delivery, procurement, and change management.

Build a Lightweight Approval Path

Every new AI use case should pass through a short intake before deployment or procurement. The intake should capture business purpose, data classes, vendor details, customer impact, model type, and whether any external AI service receives company data.

What to implement: Create two approval lanes. A fast lane for low-risk internal productivity use with approved tools and non-sensitive data. A full review lane for customer-facing, regulated, or decision-support use cases.

Map Clear Accountability

You need named owners.

At minimum, each use case should have a business owner, technical owner, risk reviewer, and security reviewer. If a third party tool is involved, vendor management should also be attached.

What to implement: Use a simple RACI table in the approval artifact. Keep it visible. Confusion about ownership is one of the most common reasons Shadow AI survives after detection.

Connect Governance to Change Management

If an approved tool gains new AI features, that should trigger reassessment. If a model changes data sources, prompt logic, or customer interaction patterns, that should trigger reassessment too.

What to implement: Add AI change triggers into software release review, procurement updates, and model change logs. Require teams to flag additions of external model calls, embedded copilots, or automated generated content in production workflows.

Tip: Make approval fast for low-risk cases, but make registration mandatory for all cases. Firms often try to reduce friction by making disclosure optional for “small experiments.” That is exactly how Shadow AI becomes entrenched.

20 Controls for Shadow AI

Practical Guidance for Prevention, Detection, and Governance

This list presents 20 prioritized controls designed to help organizations prevent, detect, and manage the risks of Shadow AI. No single control is sufficient; the strength lies in their combination across prevention, identification, and management categories.

Use each control’s narrative as a starting point for drafting procedures, or project charters. Assign ownership, define timelines, and measure effectiveness continuously.

1. Establish a Formal AI Acceptable Use Policy

Draft and publish a clear policy defining approved AI tools, prohibited activities, and acceptable use cases. Require every employee to acknowledge and sign the policy upon onboarding and annually thereafter. Update the policy regularly to address emerging AI technologies, platforms, and evolving organizational risk tolerance.

2. Create and Maintain an AI Asset Inventory

Catalog all AI tools, models, plugins, and services currently used or requested across the organization. Assign ownership, risk ratings, and approval status to each registered AI asset in the inventory. Review and reconcile the inventory quarterly to detect unregistered or newly adopted AI applications.

3. Deploy Network Monitoring to Detect Unauthorized AI Traffic

Configure network monitoring tools to identify traffic flowing to known AI service endpoints and APIs. Establish baseline patterns and trigger alerts when employees connect to unapproved AI platforms or services. Investigate flagged connections promptly and document findings for continuous improvement of detection rules.

4. Implement Data Loss Prevention Controls for AI Channels

Configure DLP solutions to detect and block sensitive data being uploaded to external AI applications. Define rules targeting personally identifiable information, trade secrets, source code, and regulated data categories. Test and refine DLP policies continuously to reduce false positives while maintaining strong data protection.

5. Deliver Ongoing AI Security Awareness Training

Conduct mandatory training explaining Shadow AI risks including data leakage, compliance violations, and output unreliability. Use real-world examples and scenarios to illustrate consequences of using unauthorized AI tools at work. Refresh training content semiannually to address new AI tools, attack techniques, and policy updates.

6. Establish a Cross-Functional AI Governance Committee

Form a committee including IT, security, legal, compliance, HR, and business unit representatives. Empower the committee to evaluate, approve, or reject AI tool requests using a standardized risk framework. Meet regularly to review Shadow AI incidents, update policies, and align AI usage with strategic objectives.

7. Provide Approved AI Tools and Sandboxes

Offer employees vetted, enterprise-grade AI tools that meet security, privacy, and compliance requirements. Create sandboxed environments where teams can safely experiment with AI without exposing production data. Communicate the availability of approved alternatives proactively so employees choose sanctioned options over Shadow AI.

8. Enforce Endpoint Detection and Device Management

Deploy endpoint detection and response (EDR) solutions to identify unauthorized AI software installations on devices. Use mobile device management (MDM) and application whitelisting to restrict unapproved AI app installations. Alert security teams immediately when endpoint agents detect AI-related executables, browser extensions, or plugins.

9. Control Access Through Identity and Authentication Policies

Implement role-based access controls to limit who can install software or access external AI services. Require multi-factor authentication and conditional access policies for any approved AI platform or integration. Review access permissions periodically and revoke entitlements promptly when roles change or employees depart.

10. Conduct Regular AI-Focused Risk Assessments

Perform dedicated risk assessments evaluating the likelihood and impact of Shadow AI across all departments. Identify high-risk business units where employees are most likely to adopt unsanctioned AI tools. Document risk findings, assign remediation owners, and track mitigation progress through the governance committee.

11. Block Unauthorized AI Domains at Proxy and Firewall

Maintain an updated blocklist of known unauthorized AI service URLs, domains, and API endpoints. Configure web proxies and firewalls to deny access and log all blocked connection attempts for analysis. Review and update the blocklist monthly as new AI services emerge in the market rapidly.

12. Integrate AI into Vendor and Third-Party Risk Management

Require formal security and privacy assessments before onboarding any third-party AI vendor or service. Evaluate AI vendors for data handling practices, model transparency, regulatory compliance, and contractual safeguards. Monitor approved AI vendors continuously for security incidents, policy changes, or terms-of-service modifications affecting risk.

13. Deploy a Cloud Access Security Broker (CASB)

Implement a CASB solution to gain visibility into all cloud-based AI services accessed by employees. Use the CASB to enforce policies, detect anomalies, and block data transfers to unsanctioned AI platforms. Analyze CASB reports regularly to identify Shadow AI usage trends and inform governance decisions.

14. Develop an AI-Specific Incident Response Plan

Create a documented incident response plan addressing scenarios like unauthorized AI data exposure or misuse. Define roles, escalation paths, containment procedures, and communication templates tailored to AI-related incidents. Conduct tabletop exercises simulating Shadow AI incidents at least annually to test readiness and refine procedures.

15. Implement Browser-Level Controls and Extension Management

Restrict browser extension installations to prevent employees from adding unauthorized AI-powered plugins or assistants. Deploy enterprise browser configurations or secure enterprise browsers that enforce AI usage policies centrally. Audit installed browser extensions regularly and remove any unapproved AI tools discovered on managed devices.

16. Enforce Contractual, Legal, and Regulatory Safeguards

Include explicit AI usage clauses in employment agreements, NDAs, and contractor statements of work. Ensure compliance with regulations such as GDPR, the EU AI Act, HIPAA, and sector-specific AI requirements. Engage legal counsel to review liability, intellectual property ownership, and indemnification related to AI-generated outputs.

17. Perform Periodic Shadow AI Audits and Compliance Reviews

Schedule internal audits specifically designed to uncover unauthorized AI tool usage across the organization. Use technical discovery tools, employee surveys, and expense report analysis to identify hidden AI subscriptions. Report audit findings to senior leadership and the governance committee with actionable remediation recommendations and deadlines.

18. Classify, Label, and Protect Sensitive Data Assets

Implement a data classification framework that labels data by sensitivity level and handling requirements. Apply automated classification tools to tag data so DLP and access controls can prevent AI-related exposure. Train employees to recognize data classification levels and understand restrictions on sharing classified data with AI tools.

19. Establish a Safe Reporting and Request Channel

Create a simple, non-punitive process for employees to report Shadow AI usage or request new AI tools. Promote the channel widely so staff feel encouraged to surface unauthorized AI use without fear of reprisal. Track all requests and reports to identify demand patterns and accelerate evaluation of popular AI tools.

20. Perform AI-Specific Threat Modeling and Scenario Analysis

Conduct threat modeling exercises that map how Shadow AI could introduce vulnerabilities into business processes. Analyze scenarios including data poisoning, prompt injection, model hallucination reliance, and supply chain compromise. Use findings to prioritize control investments and update risk registers with AI-specific threat vectors and mitigations.

Technical Actions for Detecting Shadow AI

Shadow AI is not a future risk. It is running in your environment right now. Engineers are spinning up open-source models on Kubernetes clusters. Marketing is expensing generative copywriting tools on corporate cards. Developers are hardcoding API keys into repositories nobody in compliance has reviewed. Standard CASB tools and DLP policies miss most of it because they were built for a different threat model.

The controls below cover four detection layers: infrastructure, financial, network and code, and culture. Start with infrastructure if your primary concern is engineering teams deploying models internally. Start with financial monitoring if business units are the bigger exposure. In most organizations, you need all four running together, because shadow AI does not respect organizational boundaries.

Monitor GPU and Specialized Compute Utilization

Sudden spikes in GPU or TPU consumption are one of the most reliable early signals that a team is running local model training or inference without authorization. Set threshold alerts in Datadog, Prometheus, or your cloud provider’s native console for unexpected provisioning of high-performance compute instances, specifically Nvidia A100, H100, and L4 instance types. A team that quietly provisions a cluster of these for an internal language model experiment will show up here before they show up anywhere else. This control catches what no SaaS scanner can see: workloads that never leave your own infrastructure.

Scan Kubernetes Clusters for Model Weight Deployments

Engineering teams running open-source models on Kubernetes pull large model weights from external registries such as Hugging Face or OCI-compatible container registries. Deploy internal resource scanners or open-source tooling such as Kube-hunter to examine cluster deployments for these pull patterns. Review ingress controller logs for endpoints that expose internal language model playgrounds or communicate with known AI model APIs. A deployment pulling a 70-billion-parameter model weight from an external registry is not ambiguous. It is a shadow AI deployment that needs an inventory record and a named owner before it goes any further.

Audit Cloud Marketplace Permissions and Container Registries

Tighten identity and access management permissions to block unapproved purchases from cloud provider AI marketplaces, specifically AWS Bedrock, Google Vertex AI, and Azure OpenAI Service. Without these guardrails, any engineer with a cloud console login and a project budget can provision a managed AI service and route production traffic through it within an afternoon. Restrict marketplace purchase rights to approved principals and require a procurement record before any AI service is activated. This control does not slow down legitimate work if you pair it with a fast-track approval path.

Connect Expense Systems to Automated SaaS Detection

Marketing, communications, and operations teams bypass IT procurement by expensing low-cost generative AI subscriptions directly on corporate cards. Connect your expense management platform, whether Brex, Ramp, Concur, or equivalent, to a SaaS management tool such as Torii, Zluri, or Corma. Configure keyword alerts that flag any transaction containing terms such as AI, OpenAI, Anthropic, Copilot, Jasper, Midjourney, Writer, or Notion AI. A $29-per-month subscription that processes customer data through an unvetted generative AI tool carries the same regulatory exposure as a six-figure enterprise contract. Treat it the same way.

Enforce a Procurement Intake Gate for All Software Purchases

Mandate that any software purchase, regardless of cost, passes through a lightweight automated intake form before reimbursement is approved. This does not need to be a lengthy review process. A short form capturing the tool name, business purpose, data types involved, and requesting team creates the minimum record you need to build an inventory and assign a risk tier. Without this gate, your inventory will always lag behind actual usage. The intake form is the point where shadow AI becomes known AI.

Analyze DNS and Firewall Egress Logs for AI Endpoint Traffic

Extract network egress logs and scan for connections to known AI infrastructure domains. The primary targets are openai.com, huggingface.co, anthropic.com, together.xyz, replicate.com, and cohere.com. Standard CASB tools identify sanctioned SaaS applications but miss novel or newly launched AI endpoints. DNS and firewall log analysis catches traffic that CASB cannot classify because the destination domain was never added to a known-application list. Run this analysis on a scheduled basis and pipe new domains into a review queue rather than waiting for manual discovery.

Scan Code Repositories for Hardcoded AI API Keys and Dependencies

Run automated secret scanning across your GitLab and GitHub repositories using tools such as GitGuardian or GitHub Advanced Security. Configure scans to detect hardcoded API keys for OpenAI, Anthropic, Cohere, and other AI providers, as well as dependency imports for AI orchestration libraries such as LangChain, LlamaIndex, and Semantic Kernel. A hardcoded API key in a repository is a shadow AI deployment with an active credential attached to it. The dependency scan surfaces integrations that might not yet be in production but are in active development and heading there without a governance record.

Deploy Managed Browser Controls for Web-Based AI Tool Traffic

Use enterprise browser management through managed Chrome or Edge profiles, or purpose-built enterprise browsers such as Island or Talon, to log extension installations and web traffic hitting unclassified generative AI endpoints. This control is particularly effective for marketing, sales, and operations teams who access AI tools entirely through the browser without installing any local software. Browser-level visibility closes the gap between network-layer detection, which sees domains, and application-layer understanding, which sees which tool a specific user is actively using and how frequently.

Apply Domain-Level Blocking With an Allowlist Posture

At the network level, block connections to unapproved AI domains by default and maintain an explicit allowlist of approved services. Microsoft Defender for Endpoint exposes traffic details that let you identify AI-related domain connections across managed devices before deciding whether to block them. For locally running AI tools that do not reach across the network, pull software inventory from endpoints directly through your endpoint management platform. Allowlisting is operationally demanding but it is the most complete control available. Pair it with a fast-track approval process or you will spend your time managing exceptions instead of managing risk.

Build an Internal AI Gateway and a 48-Hour Approval Path

The most durable shadow AI control is not detection. It is removal of the reason people go around you in the first place. Deploy an internal, privacy-compliant AI proxy gateway that gives engineers and business teams secure, sanctioned access to approved models. When a team can access a capable model through an internal portal with no procurement friction, the incentive to set up an external shadow account largely disappears. Pair this with a committed 48-hour turnaround for open-source model or SaaS tool approval requests. Compliance processes that take weeks train people to bypass them. A two-day SLA that actually holds changes that behavior.

Maintain a Live AI Registry With Self-Reporting Incentives

Keep a live AI system registry, using a platform such as Backstage or an equivalent internal developer portal, where teams can self-report AI deployments. Make registration worth their time: tie it to access to shared infrastructure support, approved compute budgets, or fast-tracked security reviews. A registry that engineers want to use because it removes friction will stay more current than one that depends on compliance audits to find entries. Every self-reported entry is a shadow AI deployment that has become a known, owned, and governable system. That is the outcome the registry exists to produce.

Implementation Tips That Matter in Every Stage

These are the controls that keep the whole system from drifting.

Keep an Approved AI Register

Maintain a live register of approved tools, approved use cases, restrictions, owners, review dates, and blocked alternatives. Employees need one place to check what is allowed.

Tip: Add a plain-language “why” column. If a tool is blocked, explain why. If a tool is approved only for limited data, say that clearly. Staff follow rules better when the logic is visible.

Review Vendor Changes Monthly

Third party AI risk changes fast. New features arrive through normal product updates, and contract wording often lags behind reality.

Tip: Compare vendor release notes to your approved-use register every month. The release notes often reveal AI additions before account teams do.

Document Decisions Like an Auditor Will Read Them

This is where many teams stumble. They hold good discussions, make reasonable choices, then document almost none of it.

Tip: For every AI approval or rejection, record the use case, data involved, risk rating, controls required, accountable owner, and review date. When regulators or internal audit ask why something was allowed, memory is not evidence.

Design for Human Workarounds

People under delivery pressure will find alternate routes. Personal devices, screenshots, copied extracts, private browser sessions, and personal subscriptions all show up once blocking gets tighter.

Tip: Pair controls with viable approved options. If your sanctioned AI tool is poor, slow, or missing key features, Shadow AI will return through side doors.

Key References for Shadow AI Governance

These are the standards and regulatory anchors that should shape your Shadow AI risk management program.

NIST AI Risk Management Framework, especially GOVERN and third party risk expectations
UK PRA SS1/23, especially expectations around externally developed models and model risk standards
EU AI Act, particularly requirements for high-risk AI systems, governance, transparency, and accountability
Canada’s Artificial Intelligence and Data Act direction and related guidance on harm and bias mitigation
ISACA guidance on Shadow AI, governance, and unmanaged technology risk
Microsoft security guidance on browser policy management, app consent controls, and conditional access
Industry guidance on browser extension allow-listing, SaaS app discovery, and DLP controls for AI use

If you operate in financial services, also align Shadow AI controls with your existing model risk management framework, third party risk management policy, data classification standard, and change governance process.

The Real Value of Effective Shadow AI Risk Management

If you treat Shadow AI guidance as a compliance artifact, you will produce a policy, hold one training session, and still miss the real risk. Employees will keep using unapproved tools. Vendors will keep adding AI features quietly. Hidden data transfers will continue in ordinary browser sessions that your old controls barely see.

If you treat Shadow AI risk management as a living operational workflow, you get something far more useful. You create visibility into where AI is used, clear rules for what data can leave the business, approval paths that people can actually follow, and monitoring that catches drift before it turns into customer harm or regulatory pain.

How to Actually Use ISO/IEC 23894 for AI Risk Management

Sat, 28 Mar 2026 00:00:00 +0000

Practical ISO/IEC 23894 Implementation for AI Risk Management (Without Turning It Into Shelf Decoration)

Most AI risk programs fail before the first risk is ever scored.

They fail because teams treat AI risk management as a exercise, a model review checklist, or a late-stage legal sign-off. Then the first serious issue hits. Training data rights were unclear. A model drifts in production. A vendor changes an API. An automated decision harms a customer group nobody mapped. The organization scrambles, and trust evaporates fast.

This is why ISO/IEC 23894 matters. It gives organizations a practical structure for AI risk management that fits how AI is actually built, bought, deployed, and used. In this post, I’ll show you how to turn ISO/IEC 23894 into an with governance approval gates, clear role ownership, and an implementation checklist that avoids the common failure points I keep seeing in audits, design reviews, and board briefings.

Here is why it fails: ISO/IEC 23894 is not a checklist. It is a guidance document built on top of ISO 31000, the general risk management standard, with AI-specific extensions layered in. If you treat it like a form to fill out, you will produce documentation that looks complete but protects nobody.

This post walks through the standard’s actual structure, explains what each section demands in practice, and gives you the field-tested implementation tips I have gathered from helping organizations build AI risk management programs that survive contact with real AI systems.

Why ISO/IEC 23894 Exists and What Problem It Solves

Before 2023, organizations managing AI risk had to improvise. They would borrow bits from information security frameworks, add some data governance controls, and hope the combination covered enough ground. It rarely did.

ISO/IEC 23894:2023 was created by ISO/IEC JTC 1/SC 42 to provide a structured approach for any organization that develops, deploys, or uses AI systems. The standard applies the well-established ISO 31000 risk management framework to the specific challenges AI introduces. Think of it as a translation layer. It takes proven risk management principles and shows you exactly where AI creates new wrinkles.

The standard covers three domains: principles that guide your thinking, a framework for embedding AI risk management into your organization, and processes for actually identifying, assessing, and treating AI-specific risks.

The biggest mistake organizations make is treating ISO/IEC 23894 as a standalone. It explicitly references and extends ISO 31000:2018. If your team has not read ISO 31000 first, they will misunderstand the guidance in 23894 because they will lack the foundational context. Buy both standards. Read 31000 first. Then read 23894 as the AI-specific annotation layer it was designed to be.

The Three-Part Architecture You Need to Understand

ISO/IEC 23894 mirrors the clause structure of ISO 31000 deliberately. This is not an accident. The authors wanted organizations that already use ISO 31000 to integrate AI risk management without rebuilding everything from scratch. The three parts work together as a system.

Part One: Principles (Clause 4)

The principles define the foundational values that should shape every AI risk decision your organization makes. ISO 31000 defines eight principles. ISO/IEC 23894 adds AI-specific guidance to five of them: Inclusive, Dynamic, Best Available Information, Human and Cultural Factors, and Continual Improvement.

The “inclusive” principle is where most organizations stumble first. AI systems affect a wider set of stakeholders than traditional software. The standard explicitly calls out that stakeholders can help identify risks in data collection, define fairness criteria, identify bias, and determine where human oversight is needed. This is not a suggestion. If your risk management process does not include diverse stakeholder input, you are missing risks that will surface later in the worst possible way.

The “dynamic” principle matters more for AI than for almost any other technology domain. AI systems based on machine learning can change their behavior through continuous learning. Customer expectations shift quickly. Regulatory requirements are updating constantly. Your risk management process needs to account for a system that is itself a moving target.

When I first helped a financial services firm apply the “Inclusive” principle, they interpreted “stakeholder involvement” as sending a survey to the compliance team. That is not what the standard means. You need a structured dialog with people who will be affected by the AI system’s decisions. For a credit scoring model, that means talking to loan officers, applicants from different demographic groups, and consumer advocacy organizations. Map your stakeholders before you start the risk assessment, not after.

Part Two: Framework (Clause 5)

The framework section describes how to embed AI risk management into your organizational structure. It covers leadership commitment, integration with existing management systems, organizational design, resource allocation, and communication.

Two sub-clauses deserve special attention.

Clause 5.2 on leadership and commitment adds an AI-specific requirement that many organizations overlook. Because trust and accountability are especially important for AI, the standard says top management should consider issuing public statements about their commitment to AI risk management. This is not corporate PR. It creates an accountability anchor. Once your CEO has publicly committed to responsible AI, the organization has real pressure to follow through.

Clause 5.4.3 on assigning roles is where the framework becomes . The standard requires that top management and oversight bodies allocate resources and identify specific individuals with authority to address AI risks and responsibility for monitoring AI risk processes. Not committees. Not shared inboxes. Named people with clear authority.

Part Three: Processes (Clause 6)

This is where the standard gets specific about what you actually do. The risk management process follows a sequence: define scope and context, assess risks (identify, analyze, evaluate), treat risks, then monitor and report. Each step has AI-specific extensions.

The process section is the longest part of the standard for good reason. AI risk assessment requires you to think about assets, risk sources, events. Do not try to build your AI risk management process on a blank sheet of paper. Clause 6.4.1 specifically recommends using the catalogue of AI-related risk sources in Annex B as a baseline for organizations performing risk assessment for the first time. I have seen teams spend three months trying to brainstorm AI risk sources when the standard already provides a structured catalogue. Start there. Customize from there.

Stage 1: Establishing Scope, Context, and Criteria

This stage determines what your AI risk management process covers and how it connects to your broader organizational context. Get this wrong and everything downstream is compromised.

The standard requires you to build an inventory of where AI systems are being developed or used in your organization. This sounds straightforward. It is not. In every organization I have worked with, the initial AI inventory missed at least 30% of actual AI usage. Teams embed machine learning models in spreadsheet macros, use AI-powered SaaS tools without formal procurement, or inherit AI components through acquisitions.

For external context, the standard provides Table 2 with specific considerations. You need to track relevant legal requirements for AI, ethical guidelines from government and industry groups, domain-specific AI frameworks, technology trends, and societal implications of AI deployment. For internal context, Table 3 adds considerations about how AI affects organizational culture, the availability of AI expertise, intellectual property implications, and data quality constraints.

Defining risk criteria for AI requires you to understand uncertainty across the entire AI system. The standard calls out data, software, mathematical models, physical extensions, and human-in-the-loop aspects. This is a broader scope than most organizations initially consider.

What to do: Build your AI system inventory first. Document every AI system or component, its purpose, its data sources, who built it, who operates it, and who is affected by its outputs. Then map the external and internal context factors from Tables 2 and 3. Only then define your risk criteria.

The standard warns that “AI is a fast-moving technology domain” and that measurement methods should be “consistently evaluated according to their effectiveness.” I learned this the hard way when a client’s risk criteria for a natural language processing system became obsolete within eight months because the underlying model was replaced with a fundamentally different architecture. Build a review trigger into your risk criteria. Any time the AI model architecture, training data source, or deployment context changes, the risk criteria should be re-evaluated. Do not wait for the annual review.

Stage 2: Risk Assessment, the Core of the Process

Risk assessment has three sub-stages: identification, analysis, and evaluation. The standard treats each with specific AI guidance.

Risk Identification

The standard breaks identification into five activities: identifying assets and their value, risk sources, potential events and outcomes, existing controls, and consequences. Each requires AI-specific thinking.

For assets, you need to consider three levels: organizational (data, models, the AI system itself, reputation, trust), individual (personal data, privacy, health, safety), and societal (environment, socio-cultural values, educational equity). This three-level approach is one of the most important contributions of the standard. Most organizations only think about organizational assets when they identify AI risks. The standard forces you to consider who bears the consequences.

For risk sources, Annex B provides categories including complexity of environment, lack of transparency and explainability, level of automation, machine learning specific risks, hardware issues, system life cycle issues, and technology readiness. Each category contains specific risk scenarios.

For consequences, the standard makes a critical distinction that many teams miss. It instructs you to “identify any differences between the groups who experience the benefits of the technology and the groups who experience negative consequences.” This is not theoretical. A predictive policing system might benefit a city’s police department while disproportionately harming specific communities. A hiring algorithm might benefit an HR team’s efficiency while systematically disadvantaging certain applicant groups.

What to do: For each AI system in your inventory, work through all five identification activities. Use Annex B as your starting checklist for risk sources. Document consequences at all three levels: organization, individual, and society.

The standard lists methods for identifying potential events, including published standards, scientific papers, market data, incident reports, field trials, stakeholder reports, and expert interviews. When I run risk identification workshops, I always start with incident reports on similar systems. Nothing focuses a risk identification session like showing the team a real-world failure of a system similar to theirs. Search for published incidents, regulatory enforcement actions, and academic case studies related to your specific AI application domain before the workshop begins.

Risk Analysis

Risk analysis requires you to assess both consequences and likelihood. The standard distinguishes between three types of impact assessment: business impact, individual impact, and societal impact.

For individual impact assessment, the standard specifies a detailed list of considerations: types of data used, intended impact, potential bias impact, potential impact on fundamental rights, fairness impact, safety implications, and the jurisdictional and cultural environment of the individual. This last point is easy to overlook. An AI system’s impact on an individual can vary dramatically depending on the legal and cultural context in which that individual lives.

For likelihood assessment, the standard includes a nuanced warning that many organizations miss. It states that “there can be significant technical, economic and heuristic issues with decision-making based on likelihoods, particularly when the likelihood either can’t be calculated or where the calculation has a large margin of error.” In plain language: if you cannot reliably estimate how likely an AI failure is, do not pretend you can. Focus instead on consequence severity and your ability to detect and respond to failures.

What to do: Run separate impact assessments for business, individuals, and society. Do not collapse them into a single score. For each risk, decide whether a meaningful likelihood estimate is possible. If it is not, shift your analysis to focus on consequence severity and control effectiveness.

I once watched a team assign a “low likelihood” score to a bias risk in a hiring algorithm because the model had performed well in testing. Six months after deployment, the model was producing biased outcomes because the production data distribution had drifted from the test data. The team’s likelihood estimate was based on a snapshot that was already stale. For AI systems, especially those using machine learning, likelihood estimates decay faster than for traditional systems. Reassess likelihood every time the model is retrained, the data source changes, or the deployment population shifts.

Risk Evaluation

Risk evaluation compares the analyzed risks against your established criteria to determine which risks need treatment and what priority they receive. The standard defers to ISO 31000:2018 here without adding AI-specific guidance, which tells you something important. The evaluation step is about organizational judgment, not technical analysis. You need decision-makers at the table who understand both the technology and the business context.

Stage 3: Risk Treatment and Implementation

Once risks are evaluated, you choose treatment options. The standard lists the same options as ISO 31000: avoid the risk, take or increase the risk to pursue opportunity, remove the risk source, change the likelihood, change the consequences, share the risk, or retain the risk by informed decision.

The AI-specific addition here is the concept of a risk-benefit analysis for residual risks. If you cannot reduce negative consequences to an acceptable level through treatment, the standard requires you to perform a risk-benefit analysis. This is particularly relevant for AI because some AI risks (like model opacity in deep learning) cannot be fully eliminated. You need to decide whether the benefits justify the residual risk.

What to do: For each risk that exceeds your tolerance thresholds, select a treatment option and document it in a risk treatment plan. For residual risks that remain above tolerance after treatment, conduct a formal risk-benefit analysis. Record the rationale for accepting any residual risks.

for AI systems need to be version-aware. Traditional risk treatment plans assume relatively stable systems. AI systems, especially those using continuous learning, change over time. Your treatment plan should specify which version of the model it applies to and include trigger conditions for re-evaluation. I recommend tagging each treatment plan entry with the model version, training data date, and deployment configuration it was validated against. When any of these change, the treatment plan enters a mandatory review cycle.

Stage 4: Monitoring, Recording, and Reporting

The standard’s requirements for recording and reporting are more specific than many teams expect. Clause 6.7 requires organizations to establish a system for collecting and verifying information from both implementation and post-implementation phases, and to collect publicly available information on similar systems.

This information must be assessed for relevance to the trustworthiness of the AI system. The standard specifically asks whether previously undetected risks exist or whether previously assessed risks are no longer acceptable. When either condition is true, you must perform a review of risk management activities and evaluate the effects on existing controls.

The recording requirements include: system description and identification, methodology applied, intended use description, identity of assessors, terms of reference and date, release status, and degree to which objectives have been met. This is not optional documentation. It creates the audit trail that regulators and oversight bodies will examine.

What to do: Build a risk management record template that captures all required fields. Establish a cadence for collecting and reviewing post-implementation data. Create triggers that automatically initiate risk reassessment when conditions change.

The standard says risk management records “should allow the traceability of each identified risk through all risk management processes.” In practice, this means you need a risk register with unique identifiers for each risk that persist across assessment cycles. I have seen organizations create new risk registers for each assessment, losing the historical thread. Use a single, versioned risk register where each risk has a persistent ID, and track its status changes over time. This is the only way to demonstrate to auditors that your process is continuous, not episodic.

Implementation Tips

These apply across all stages and will determine whether your AI risk management program actually works.

Tip 1: Align Risk Management with the AI System Life Cycle

Annex C of the standard maps risk management activities to AI system life cycle stages: inception, design and development, verification and validation, deployment, operation and monitoring, continuous validation, re-evaluation, and retirement or replacement. This mapping is not decorative. It tells you which risk management activities should happen at each stage.

Most organizations I work with front-load their risk management effort at the design stage and then drop attention during operation. The standard explicitly shows that risk assessment, treatment, monitoring, and recording continue through every life cycle stage, including retirement. When you decommission an AI system, you can lose decision expertise and information. Plan for that. Document what the system knew and how it made decisions before you turn it off.

Tip 2: Handle Stakeholder Identification Seriously

The standard provides a list of stakeholder categories: the organization itself, customers, partners, third parties, suppliers, end users, regulators, civil organizations, individuals, affected communities, and societies. That is nine categories. Most organizations consult two or three.

Create a stakeholder map for each AI system at the inception stage. For each stakeholder category, document what information they need, how they are affected by the system, and how you will engage them. Update this map when the system’s scope or deployment context changes. The stakeholder categories that matter most are often the ones farthest from the development team. Affected communities and end users rarely have a voice in risk assessment unless you build a specific mechanism to include them.

Tip 3: Document Your Risk Criteria Decisions Explicitly

The standard’s Table 4 on risk criteria includes a requirement that organizations “take reasonable steps to understand uncertainty in all parts of the AI system.” This includes data, software, mathematical models, physical extensions, and human-in-the-loop aspects.

When defining risk criteria, write down not just what your criteria are, but why you chose those thresholds. I worked with an organization that set a 95% accuracy threshold for an AI diagnostic tool. When a regulator asked why 95% and not 97% or 99%, nobody could answer. The threshold had been copied from a different project. Document the reasoning behind every criterion. Reference the clinical studies, industry benchmarks, or stakeholder consultations that informed your decision. This documentation is what separates a defensible risk management process from an arbitrary one.

Tip 4: Treat Transparency as a Multi-Audience Challenge

Annex B of the standard discusses transparency and explainability as risk sources. It makes a point that “the kind and level of information that is appropriate strongly depends on the stakeholders, use case, system type and legislative requirements.” One-size-fits-all transparency does not work.

Build a transparency framework that segments by audience. The standard’s Table 1 mentions tailoring transparency to “relevant personas” such as regulators, business owners, and model risk evaluators. In practice, I create three transparency tiers. Tier one is the public-facing description of what the AI system does and does not do. Tier two is the detailed technical documentation for internal reviewers and regulators. Tier three is the full model documentation, including training data provenance, architecture decisions, and test results. Each tier serves a different audience and contains different information. Trying to serve all audiences with one document produces a document that serves none of them.

What This Standard Actually Does in Practice

Part 1: The AI Risk Management Process

This is Clause 6 of the standard. It is the longest and most detailed section because it describes what you actually do.

Step 1: Define Your Scope, Context, and Criteria

Before you assess any risks, you need to answer three questions. What AI systems are we managing? What is the environment around them? And what criteria will we use to decide if a risk matters?

The standard requires you to build an inventory of where AI is being developed or used in your organization. This inventory must be documented and included in your risk management process.

Practical example: A mid-size insurance company I worked with discovered during this step that 14 different teams were using AI-powered tools, but only 3 had been formally identified by the IT governance team. Six were SaaS products with embedded ML models. Two were spreadsheet-based models built by actuaries. Three were claims processing systems. The inventory step alone changed their understanding of their AI exposure.

For context, the standard requires you to consider nine categories of stakeholders:

Your own organization
Customers, partners, and third parties
Suppliers
End users
Regulators
Civil organizations
Individuals affected by the AI system
Affected communities
Societies broadly

That last category is not abstract. If your AI system makes lending decisions, “societies” includes the economic communities shaped by those decisions over time.

The standard also requires you to consider whether your AI systems can harm human beings, deny essential services, infringe human rights through biased automated decisions, or contribute to environmental harm.

For risk criteria, the standard adds an AI-specific requirement that matters enormously in practice: you must understand uncertainty across all parts of the AI system. That includes the data, the software, the mathematical models, any physical components, and the human-in-the-loop aspects like data labeling. Most organizations define risk criteria only around the model itself and miss everything upstream and downstream.

Practical insight for risk managers: Your AI risk appetite should account for your organization’s actual AI capacity and knowledge level. The standard says this directly. If your team has limited ML expertise, your risk appetite for complex deep learning systems should be lower than an organization with a mature data science function. This sounds obvious, but I have seen organizations approve high-risk AI projects with the same risk appetite they use for rule-based automation.

Practical insight for data scientists: The standard warns that “AI is a fast-moving technology domain” and that measurement methods should be “consistently evaluated according to their effectiveness.” That means the metrics you use to evaluate model performance today might not be appropriate six months from now. Build metric review into your model monitoring cadence.

Step 2: Identify Risks

Risk identification in ISO/IEC 23894 covers five distinct activities. Each one matters.

Identify assets and their value. The standard requires you to think about assets at three levels:

Organizational assets include your data, your trained models, the AI system itself (tangible), plus your reputation and stakeholder trust (intangible).

Individual assets include personal data (tangible), plus privacy, health, and safety (intangible).

Community and societal assets include the environment (tangible), plus socio-cultural beliefs, educational access, and equity (intangible).

Practical example: When a healthcare AI startup assessed assets for their diagnostic imaging tool, they initially listed only their model and training data. The three-level framework forced them to also consider patient safety (individual intangible), the hospital’s reputation for diagnostic accuracy (organizational intangible), and equitable access to accurate diagnosis across demographic groups (societal intangible). Each of these surfaced risks that the initial asset list would have missed.

Identify risk sources. The standard provides categories: organizational factors, processes, personnel, physical environment, data, AI system configuration, deployment environment, hardware and software, and dependence on external parties. Annex B expands these into detailed scenarios (covered in Part 2 below).

Identify potential events and outcomes. The standard lists specific methods for finding these: published standards and papers, market data on similar systems, incident reports on similar systems, field trials, usability studies, stakeholder reports, expert interviews, and simulations.

Practical insight for AI security analysts: Incident reports on similar systems are the most underused source on this list. Before running any risk identification workshop, search for publicly reported failures, adversarial attacks, and regulatory actions against AI systems similar to yours. The the , and CVE databases are good starting points. Nothing sharpens a risk identification session like a real-world failure story from your domain.

Identify existing controls. Document what controls already exist and assess whether they actually work. The standard specifically calls out the importance of identifying control failures.

Identify consequences. This is where the standard adds its most valuable AI-specific guidance. It instructs you to identify differences between the groups that benefit from the technology and the groups that bear negative consequences. A resume screening tool might benefit the HR department (faster processing) while systematically disadvantaging applicants from certain educational backgrounds or geographic regions.

Consequences to organizations include investigation and repair time, lost opportunities, reputational damage, regulatory penalties, and litigation.

Consequences to individuals and societies are harder to quantify but often more severe: threats to health, violations of privacy, infringement of fundamental rights.

The standard makes a practical point that experienced practitioners already know: consequences for individuals and societies almost always affect the organization as well. A safety incident creates liability claims. A bias scandal damages the brand. Map these cascading effects explicitly.

Step 3: Analyze Risks

Risk analysis requires three separate impact assessments:

Business impact assessment. How badly does this risk affect the organization? Consider criticality, tangible versus intangible impacts, and your established criteria.

Individual impact assessment. How does this risk affect the people whose data is used or whose lives are influenced by the AI system? The standard lists specific factors: types of personal data used, potential bias impact, potential impact on fundamental rights, fairness impact, safety of the individual, existing protections against bias, and the jurisdictional and cultural environment of the individual.

That last factor is easy to overlook but critical. An AI system’s impact on an individual in the EU (where GDPR applies) differs from its impact on an individual in a jurisdiction with no data protection law. The same system creates different risk profiles in different markets.

Societal impact assessment. How broadly does the AI system reach into different populations? The standard specifically asks you to consider whether the system amplifies or reduces pre-existing patterns of harm to different social groups.

Example: A government agency deploying a predictive policing model would face very different societal impact conclusions than a private company using the same technology for retail theft prevention. The government use case reaches more broadly and carries the authority of the state, which amplifies both benefits and harms.

For likelihood assessment, the standard includes a warning that practitioners should take seriously: “There can be significant technical, economic and heuristic issues with decision-making based on likelihoods, particularly when the likelihood either can’t be calculated or where the calculation has a large margin of error.” In plain language: if you cannot meaningfully estimate how likely something is, do not force a number. Focus on consequence severity and your ability to detect and respond instead.

Practical insight for data scientists: Likelihood estimates for AI system failures decay faster than for traditional software. A model’s failure probability changes every time the data distribution shifts, the model is retrained, or the user population changes. If you assign a likelihood score, attach an expiration date to it.

Step 4: Evaluate Risks

Compare the analyzed risks against your criteria. Prioritize. Decide which risks need treatment. The standard defers to ISO 31000 here without AI-specific additions, which tells you this step is about organizational judgment, not technical analysis. Get decision-makers in the room who understand both the technology and the business.

Step 5: Treat Risks

The standard provides seven treatment options, consistent with ISO 31000:

Avoid the risk (stop or do not start the activity)
Accept increased risk to pursue an opportunity
Remove the risk source
Change the likelihood
Change the consequences
Share the risk (contracts, insurance)
Retain the risk by informed decision

The AI-specific addition: if you cannot reduce negative consequences to an acceptable level through any treatment option, you must perform a risk-benefit analysis for the residual risk. This matters because some AI risks cannot be fully eliminated. The opacity of a deep learning model, for example, is inherent to the technology. You need to decide if the benefits justify the residual risk, and you need to document that decision.

Practical insight for risk managers: Each treatment measure must be verified for effectiveness and recorded. Do not just document the plan. Document whether the treatment actually worked. I have seen organizations with detailed treatment plans and zero follow-up on whether the treatments reduced the risk as expected.

Step 6: Monitor, Record, and Report

The standard’s recording requirements are specific. Your risk management record must include:

Description and identification of the analyzed system
Methodology used
Intended use of the AI system
Who performed the risk assessment
Terms of reference and date
Release status of the assessment
Whether and to what degree objectives were met

The standard also requires that you collect information from post-implementation phases and review publicly available information about similar systems. You must assess whether previously undetected risks exist or whether previously accepted risks are no longer acceptable.

The records must allow traceability of each identified risk through all risk management processes. This means persistent risk IDs, version-controlled risk registers, and a clear audit trail from identification through treatment and monitoring.

Practical insight for all three audiences: When the standard says “collect and review publicly available information on similar systems on the market,” it means this is an ongoing obligation, not a one-time activity. Set up alerts for incident reports, regulatory actions, and published research about AI systems similar to yours. The best early warning system for your own AI risks is someone else’s AI failure.

Part 2: Risk Sources and Objectives (Your Identification Checklists)

Annexes A and B of the standard provide catalogs that serve as starting points for risk identification. The standard itself says these catalogs have “shown value” for organizations performing AI risk assessment for the first time. Use them as baselines, not as exhaustive lists.

These are the things that can go wrong or right with AI systems. For each objective, the standard provides context on why it matters for AI specifically.

Accountability. AI changes who is responsible for decisions. When a person made a lending decision, that person was accountable. When an AI system makes that decision, accountability becomes unclear. Regulators worldwide are still working out who bears responsibility. You need to know the legislation in every market where your AI system operates.

AI Expertise. Building AI systems requires interdisciplinary specialists, not just software engineers. The standard also extends this to end users: they need enough understanding of the AI system to detect and override erroneous outputs.

Practical example: A manufacturing company deployed an AI-powered quality inspection system but did not train the floor operators on how the system made decisions or what its failure modes looked like. When the system began missing defects due to a lighting change in the factory, operators trusted the system’s “pass” decisions for three weeks before someone escalated the rising customer complaint rate.

Training and Test Data Quality. Training and test data must be validated for currency, relevance, diversity, and consistency. If you source data externally, data quality is still your responsibility. The amount of data required varies with the functionality and complexity of the environment.

Practical insight for data scientists: The standard specifically calls out that training and test datasets should be independent when applicable. This is a basic ML practice, but the standard elevates it to a risk management concern. If your test set leaks into your training data, you have not just a technical problem but a governance failure.

Environmental Impact. AI can help the environment (optimizing energy use, reducing emissions) or hurt it (massive compute requirements for training). Both sides must be considered.

Fairness. Unfair outcomes can come from biased objective functions, imbalanced datasets, human biases in training data, biased product concepts, or decisions about when and where to deploy AI systems. The standard references ISO/IEC TR 24027 for deeper guidance on bias.

Maintainability. ML-based systems are trained, not programmed. Modifying them to fix defects or adapt to new requirements is fundamentally different from patching traditional software. Understand the implications before deploying.

Privacy. AI systems that depend on large datasets create privacy risks through data collection, through inference of sensitive information, and through model personalization. The standard notes that AI can infer sensitive personal data even when that data was not directly provided. A data protection impact assessment (per ISO/IEC 29134) is recommended.

Practical insight for AI security analysts: The standard highlights that protecting privacy in AI includes protecting access to models personalized for individuals or models that can be used to infer characteristics of similar individuals. This means model extraction attacks are a privacy risk, not just a security risk. If an attacker can replicate your model, they can potentially infer characteristics of your training data subjects.

. Can the system maintain performance under unexpected conditions? Neural networks are particularly challenging here because their nonlinear nature can produce unexpected behavior. Characterizing neural network robustness remains an open research problem.

Safety. AI systems in vehicles, manufacturing, robotics, and medical devices introduce safety risks that must be evaluated against domain-specific safety standards. The standard does not replace those domain standards. It adds to them.

Security. Beyond classical information security, AI introduces new attack surfaces: data poisoning (corrupting training data), adversarial attacks (crafted inputs that fool the model), and model stealing (extracting the model through query access). ISO/IEC 27005 covers general information security risk management. The AI-specific threats require additional consideration.

Practical example for security analysts: A financial institution’s fraud detection model was trained on transaction data that included a small number of poisoned records inserted by an insider. The poisoned data taught the model to classify certain fraudulent transaction patterns as legitimate. Classical information security controls (access management, encryption) did not prevent this because the insider had authorized access to the data pipeline. AI-specific controls (data provenance tracking, statistical anomaly detection on training data) would have caught it.

Transparency is about what the organization communicates. Explainability is about what the system can reveal about its own decision-making. Both matter, and they serve different purposes. Transparency builds trust with stakeholders. Explainability enables validation and verification by the organization itself.

The standard also notes a tension: excessive transparency can create privacy, security, and intellectual property risks. You need to find the right level for each stakeholder group.

These are the places where risks originate. Use this as a checklist during risk identification.

Complexity of environment. The more complex the operating environment, the harder it is to ensure your training data covers all possible situations. For autonomous driving, you cannot guarantee coverage of every scenario. For a chatbot answering questions about a fixed product catalog, coverage is more achievable. Assess how well-understood your system’s environment is, because partial understanding creates uncertainty that is itself a risk source.

Lack of transparency and explainability. If you cannot explain why your model made a specific decision, you cannot fully validate it. This affects trustworthiness, accountability, safety, security, fairness, and robustness. The standard emphasizes that explainability matters for internal validation, not just external communication.

Level of automation. Systems range from fully human-controlled to fully automated. Higher automation means less human oversight, which amplifies both the efficiency gains and the risk exposure. For systems where a human must be “ready to take over when necessary,” the handover itself is a risk source. Think about response time, operator attention, and situation awareness.

Machine learning specific risks. Data quality directly affects system behavior. Data collection processes are a risk source that is “especially hard to diagnose and detect.” Data can become unrepresentative over time. Data sourcing creates risks. Failing to secure the data pipeline opens the door to adversarial manipulation. Continuous learning systems can change their behavior in production in ways that were not anticipated at deployment.

Practical example: An e-commerce recommendation engine trained on user interaction data gradually learned to recommend increasingly sensationalized products because those generated more clicks. The continuous learning loop optimized for the engagement metric without any check on whether the recommendations were appropriate. The behavior drift was subtle enough that no one noticed for months.

System hardware issues. Hardware errors, soft errors from radiation, constraints when transferring models between different hardware platforms, and network issues for systems requiring remote processing. These are easier to overlook in AI because teams focus on model performance and forget about the physical infrastructure.

System life cycle issues. Risks exist at every stage:

Design: failing to anticipate deployment contexts
Verification and validation: inadequate testing causing regressions
Deployment: misconfigured resources
Maintenance: unsupported but still-running systems
Reuse: using a system in a context it was not designed for (the standard gives the example of a social media face detection system repurposed for criminal suspect identification, a far more demanding use case)
Decommissioning: losing the decision expertise embedded in the retired system

Technology readiness. Less mature technologies carry unknown risks. More mature technologies create complacency and technical debt. Both ends of the maturity spectrum require attention.

Part 3: The Organizational Framework

This is Clause 5. It describes how to embed AI risk management into your o .

Leadership and Commitment

Top management, supported by , must do two things the standard calls out specifically for AI:

First, consider issuing public statements about the organization’s commitment to AI risk management. This creates accountability and builds stakeholder confidence.

Second, recognize that AI risk management requires specialized resources and allocate them. An AI risk program staffed by people without AI expertise will produce documentation that misses the actual risks.

Organizational Context

The standard provides two detailed tables for understanding your external and internal context.

For external context, AI-specific considerations include:

AI-related legal requirements in your markets
Ethical guidelines from government groups, regulators, standardization bodies, civil society, academia, and industry associations
Domain-specific AI frameworks
Societal implications of your AI deployments
How continuous learning might affect your ability to meet contractual obligations
Ownership and usage rights for training data provided by third parties

For internal context, AI-specific considerations include:

How AI changes organizational culture by creating new roles and responsibilities
Deskilling risks where human decision-making is increasingly replaced by AI
The availability of AI tools that enable development without full understanding of the technology
Intellectual property implications of AI systems
Additional data quality constraints imposed by AI
The need to educate stakeholders on AI capabilities, failure modes, and failure management

Practical insight for risk managers: The external context requirement to track “ and design of AI and automated systems issued by government-related groups, regulators, standardization bodies, civil society, academia and industry associations” is broad. Create a regulatory and guidance tracker specific to AI. Assign someone to update it quarterly. The landscape is changing fast enough that annual reviews will miss significant developments.

Roles and Accountabilities

The standard requires top management and oversight bodies to identify specific individuals with:

Authority to address AI risks
Responsibility for establishing and monitoring processes to address AI risks

Notice the standard says “individuals,” not “committees.” Named accountability matters.

Practical insight: If the person accountable for AI risk management does not have authority over the AI development teams, the role is ceremonial. Verify that the accountability chain has teeth.

Communication and Consultation

The standard notes that stakeholders affected by AI systems can be “larger than initially foreseen, can include otherwise unconsidered external stakeholders and can extend to other parts of a society.” Plan your stakeholder engagement broadly from the start, because discovering a critical stakeholder group after deployment creates reactive crises.

Part 4: The Guiding Principles

Clause 4 defines eight risk management principles from ISO 31000. Five of them get AI-specific guidance.

Inclusive. AI systems affect more stakeholders than traditional systems. Engage diverse internal and external groups. Stakeholders help identify data risks, define fairness criteria, spot bias, determine where human oversight is needed, and shape transparency and explainability approaches. The standard suggests segmenting transparency frameworks by stakeholder persona (regulators, business owners, model risk evaluators) when a one-size-fits-all approach does not work.

Dynamic. AI systems change through continuous learning. Customer expectations shift rapidly. Regulations update frequently. Your risk management process must anticipate, detect, and respond to these changes in real time, not annually.

Best available information. Historical data about AI failures may be limited because the technology is relatively new. Future expectations change quickly. Track how your AI systems are used after deployment. Be aware that tracking external usage may be limited by IP, contractual, or market restrictions, and capture those limitations explicitly in your risk process.

Human and cultural factors. Monitor how your AI systems interact with pre-existing societal patterns that affect equitable outcomes, privacy, freedom of expression, fairness, safety, security, employment, the environment, and human rights.

Continual improvement. Monitor the AI ecosystem for performance successes, shortcomings, lessons learned, and new research findings. Feed previously unknown risks back into the improvement cycle.

Part 5: Risk Management Across the AI System Life Cycle

Annex C maps risk management activities to the AI system life cycle defined in ISO/IEC 22989:2022. The life cycle stages are:

Inception
Design and development
Verification and validation
Deployment
Operation and monitoring
Continuous validation
Re-evaluation
Retirement or replacement

At the organizational level, the governing body sets risk appetite, establishes general criteria, and builds catalogs of risk criteria, risk sources, mitigation measures, monitoring techniques, and reporting formats. These catalogs improve over time as feedback flows up from individual AI system risk processes.

At the project level, each AI system goes through its own risk management cycle at each life cycle stage. Risk criteria, assessments, and treatment plans are established at inception, continuously updated during design and development, verified during testing, potentially adjusted during deployment, and monitored throughout operation.

The key takeaway: risk management is not a phase. It happens at every stage. The standard explicitly shows risk assessment, treatment, monitoring, and recording activities at every life cycle stage, including retirement.

Practical insight for data scientists: The re-evaluation stage is often skipped. The standard requires that existing risk sources be examined for relevance, criteria re-evaluated against changes in scope or purpose, and regulatory updates incorporated. Build re-evaluation triggers into your model governance process. Any change in model purpose, data source, or regulatory environment should start a re-evaluation cycle.

Practical insight for AI security analysts: The retirement stage creates specific risks. When you decommission an AI system, you can lose decision expertise and institutional knowledge embedded in that system. If a replacement system is deployed, the way the organization processes information and makes decisions changes. Both of these transitions create attack surface changes and knowledge gaps that need to be assessed as security risks.

What to Do Monday Morning

If you are a risk manager: start with the AI system inventory. You cannot manage risks you do not know exist. Then map your stakeholders using the nine-category list. Then compare your existing risk criteria against the AI-specific factors in Table 4 of the standard.

If you are a data scientist: read Annex B on risk sources, particularly section B.5 on machine learning risks. Then review your current model documentation against the recording requirements in Clause 6.7. The gap between what you document today and what the standard requires is likely significant.

If you are an AI security analyst: start with Annex A sections on security (A.11), privacy (A.8), and robustness (A.9). Map your current threat model against the AI-specific attack surfaces the standard identifies: data poisoning, adversarial attacks, model stealing. Then check whether your security controls cover the full AI system life cycle or only the deployment and operation stages.

The standard gives you the structure. The work is in applying it honestly to your specific systems, your specific organization, and your specific stakeholders. That is where the real risk management happens.

ISO/IEC 23894 does not exist in isolation. It references and connects to a broader ecosystem of standards that together form a comprehensive AI governance framework.

ISO 31000:2018, Risk management, Guidelines. This is the foundational standard. ISO/IEC 23894 is built directly on top of it.

ISO/IEC 22989:2022, Artificial intelligence, Concepts and terminology. Provides the AI-specific definitions and the system life cycle model referenced throughout 23894.

ISO Guide 73:2009, Risk management, Vocabulary. Defines the risk management terms used in both 31000 and 23894.

ISO/IEC 38507:2022, Governance implications of the use of artificial intelligence by organizations. Covers the governance layer that sits above risk management.

ISO/IEC TR 24028:2020, Overview of trustworthiness in artificial intelligence. Provides background on AI trustworthiness that informs several sections of 23894.

ISO/IEC TR 24027:2021, Bias in AI systems and AI-aided decision making. Essential reading for the fairness-related risk identification and analysis sections.

ISO/IEC 29134:2017, Guidelines for privacy impact assessment. Directly relevant when AI systems process personal data.

ISO/IEC 27005:2022, Guidance on managing information security risks. Covers the security dimension of AI risk, including threats like data poisoning and adversarial attacks.

NIST AI Risk Management Framework (AI RMF 1.0). While not referenced in the standard, this U.S. framework maps closely to ISO/IEC 23894 and is increasingly expected by American regulators and enterprise customers.

EU AI Act (Regulation 2024/1689). The European regulation that makes AI risk management a legal obligation for high-risk AI systems. ISO/IEC 23894 provides a structured path toward many of its requirements.

Supporting AI ISO standards by Topic

Core AI Management & Governance Standards

ISO/IEC 42001:2023 – Information technology — Artificial intelligence — Management system (AIMS).
ISO/IEC 38507:2022 – Governance implications of the use of AI by organizations.
ISO/IEC 23894:2023 – Guidance on risk management for AI.
ISO/IEC 42005:2025 – AI system impact assessment.
ISO/IEC 42006:2025 – Requirements for auditing bodies providing AI management system certification.

Foundational Frameworks & Terminology

ISO/IEC 22989:2022 – AI concepts and terminology.
ISO/IEC 23053:2022 – Framework for AI systems using Machine Learning.
ISO/IEC 5338:2023 – AI system life cycle processes.
ISO/IEC 5339:2024 – Guidance for AI applications.

Trustworthiness, Ethics & Quality

ISO/IEC 24028:2020 – Overview of trustworthiness in artificial intelligence.
ISO/IEC 24368:2022 – Overview of ethical and societal concerns.
ISO/IEC 25059:2023 – Quality model for AI systems (SQuaRE).
ISO/IEC 12791:2024 – Treatment of unwanted bias in ML tasks.
ISO/IEC 12792:2025 – Transparency taxonomy for AI systems.
ISO/IEC 42119-2:2025 – Testing of AI systems — Part 2: Test data and results.
ISO/IEC 4213:2022 – Assessment of machine learning classification performance.

Data Quality & Analytics

ISO/IEC 24668:2022 – Process management framework for big data analytics.
ISO/IEC 5259-1:2024 – Data quality for analytics and ML — Part 1: Overview and terminology.
ISO/IEC 5259-3:2024 – Data quality management requirements and guidelines.
ISO/IEC 5259-4:2024 – Data quality process framework.

The Choice in Front of You

Organizations that treat ISO/IEC 23894 as a compliance artifact will produce binders full of risk assessments that nobody reads, risk registers that go stale within weeks, and governance structures that exist on paper but have no operational authority. When something goes wrong, and with AI systems something eventually does go wrong, they will discover that their documentation protected nobody. Not the organization, not the individuals affected by the AI system, and not the communities that bore the consequences.

Organizations that treat ISO/IEC 23894 as a living operational tool will build AI risk management into their development pipelines, their deployment decisions, and their ongoing monitoring. They will have named individuals with real authority, risk criteria grounded in evidence, stakeholder engagement that surfaces risks early, and records that tell a coherent story from inception through retirement. When something goes wrong, they will know about it faster, respond more effectively, and demonstrate to regulators that they took reasonable steps.

The standard gives you the blueprint. What you build with it depends entirely on whether you treat AI risk management as paperwork or as practice.

To maximize SEO authority and drive high-value traffic back to your core assets, this “About the Author” section is restructured to emphasize your specific expertise in the EU AI Act, Quantitative Risk, and ISO 42001. I have humanized the tone to move from a standard bio to a “Partnership Invitation,” while ensuring the links are prominent and descriptive.

About the Author: Prof. Hernan Huwyler, MBA, CPA, CAIO

The frameworks, taxonomies, and implementation toolkits shared in this article are part of the ongoing applied research and executive advisory work of Prof. Hernan Huwyler. These materials are designed for operational realism and are freely available for adaptation in your own AI Governance, Risk Management, and Compliance (GRC) programs under proper attribution.

Bridging the Gap Between AI Theory and Production Controls

As an AI GRC Strategy Director and Quantitative Risk Lead, Prof. Huwyler works with global organizations in financial services, healthcare, and the public sector to build frameworks that survive both production demands and regulatory scrutiny. His expertise is focused on tje risk-adjusted AI adoption, ensuring that innovation remains defensible through rigorous algorithmic auditing and automated compliance protocols.

Global Thought Leadership and Executive Education

Based in the Copenhagen Metropolitan Area with a professional presence in Zurich, Geneva, Madrid, and Berlin, Prof. Huwyler operates where AI development is most active. He serves as an Executive Advisor and Academic Director at IE Law School, delivering specialized corporate training on:

EU AI Act Compliance Strategy and ISO 42001 integration.
Quantitative Risk Modeling using Python and Monte Carlo simulations.
Predictive Risk Automation for Board-level decision-making.

Explore Open-Source GRC Tools and Insights

Prof. Huwyler maintains a public repository of Python-based AI governance tools, risk model templates, and automated compliance scripts to support the global practitioner community.

Technical Tools & Code: Access the for risk model templates.
Expert Commentary: Read the latest on GRC and Internal Audit at .
Professional Network: Connect on to follow real-time updates on the evolving AI regulatory landscape.

Let’s Turn AI Governance into a Competitive Advantage

If you are ready to move beyond “check-the-box” compliance and start capturing the positive ROI of AI, let’s connect. True governance isn’t about slowing down; it’s about creating the structural discipline needed to achieve massive savings through automation and to build new, resilient revenue streams that regulators and customers can trust.

A 12-Step Procedure Merging ISO 27005, ISO 23894, ISO 42001, and FAIR

Thu, 12 Mar 2026 00:00:00 +0000

How to Build an AI Risk Assessment That Actually Protects Your Organization

Most AI risk assessments fail before they produce a single useful number.

I’ve reviewed dozens of them across financial services, healthcare, and technology companies. The pattern is almost always the same. A team fills out a qualitative risk matrix, assigns some red-yellow-green ratings, files the document, and moves on. Six months later, an AI system produces biased outputs in production, a regulator asks pointed questions, and nobody can trace a single risk decision back to a defensible analysis.

The problem is not a lack of frameworks. ISO 27005, ISO 23894, ISO 42001, and FAIR all offer strong foundations. The problem is that nobody shows risk managers how to combine them into one coherent, repeatable procedure that produces numbers leadership can actually use to make decisions.

This post walks through a 12-step AI risk assessment procedure that merges structured risk process, AI-specific principles, governance requirements, and quantitative rigor. Every step includes the practical guidance I wish someone had given me when I first tried to assess AI risks using nothing but a spreadsheet and good intentions.

Why You Need a Unified AI Risk Framework

Traditional cybersecurity risk assessment covers infrastructure, access controls, and data protection. That is necessary but insufficient for AI systems. AI introduces risks that sit outside the usual threat catalogs. Biased outputs, model drift, adversarial manipulation, opacity of decision-making, hallucinated content. These require their own vocabulary and their own assessment methods.

The procedure described here draws from four sources. ISO 27005 provides the structured risk management process. ISO 23894 adds AI-specific risk principles. ISO 42001 brings AI governance, ethics, and lifecycle management. FAIR supplies the quantitative engine that converts vague risk language into dollar ranges executives understand.

Original implementation tip: Do not try to run this procedure in isolation from your existing enterprise risk management program. The single most common failure I’ve seen is a standalone AI risk register that never connects to the organization’s financial, operational, or compliance risk reporting. From day one, map your AI risk outputs to the same reporting structure your CFO and CRO already read. If they report in annualized loss exposure, you report in annualized loss exposure.

Stage 1: Scoping and Context Setting

Step 1: Define the Project Objectives

Start with the business reason the AI system exists. This sounds obvious. But watch how many teams skip past it and jump straight to technical vulnerability scanning.

Define why the system is being built or deployed, who owns accountability across its lifecycle, and what business processes depend on it. Quantify the goals in measurable terms. Revenue growth targets, efficiency gains, cost reductions, time savings. “Reduce loan approval time by 40% without increasing default risk” is a useful objective. “Use AI to improve lending” is not.

Then define your protection requirements across three dimensions. For confidentiality, specify what intellectual property, personal data, or business information must stay protected. For integrity, state what data, models, and processes must remain accurate. For availability, define the uptime and performance levels required to support operations.

Layer on responsible AI commitments. What accuracy thresholds must the model meet? What are the acceptable performance ranges? What fairness and non-discrimination principles apply? When must a human step in?

Finally, record every compliance and legal obligation. Regulatory frameworks, sector rules, contractual commitments, and geographic considerations all belong here. A credit scoring model deployed across EU and US markets faces GDPR, the EU AI Act, the Equal Credit Opportunity Act, and likely several internal policies.

Original implementation tip: Write your risk appetite statement before you assess a single risk. I spent two years running assessments without a defined appetite, and every evaluation ended in the same argument. “Is this risk acceptable?” became a political debate instead of a comparison against a documented threshold. Set a clear number. “Residual annualized loss exposure must remain below $100k” gives your team a finish line. Without it, you are running a race with no tape.

Step 2: Identify Assets

Build a complete inventory of everything that supports the AI system. This is not just a list of servers. It is a map of the entire ecosystem from development through deployment.

Start with data assets. Distinguish between raw data and the specific training, validation, and test datasets derived from it. Then catalog model artifacts, including weights, embeddings, hyperparameters, and versioned configurations. Document the supporting infrastructure, from cloud services and GPU environments to orchestration pipelines and monitoring tools.

Do not overlook human assets. Developers, data annotators, ML engineers, auditors, and business owners all play roles in the system’s lifecycle. Map them. Then identify all integration points, including APIs, dashboards, and downstream systems that consume model outputs.

Critically, assess external dependencies. Third-party datasets, open-source libraries, pre-trained models, credit bureau APIs, and partner services all introduce risk that sits outside your direct control.

Original implementation tip: Create a dependency map, not just an asset list. A flat inventory tells you what exists. A dependency map tells you what breaks when something fails. I once worked with a team that listed “scikit-learn” as an asset but never documented that three other internal systems consumed the same model’s output via an unmonitored API. When the model degraded, the blast radius was four times what anyone expected. Draw the connections. Every one of them is a potential failure path.

Stage 2: Threat and Vulnerability Discovery

Step 3: Find Vulnerabilities

Examine four domains systematically. Data sources, model components, supporting architecture, and organizational processes.

For data, assess incompleteness, hidden bias, lack of sanitization, and susceptibility to poisoning. For model artifacts, look for opacity that limits explainability, exposure to evasion or inversion attacks, and reliance on unpatched open-source components. For infrastructure, check for exposed interfaces, weak access controls, and misconfigured environments. For processes, evaluate monitoring gaps, incident response readiness, and unclear ownership.

Tie every vulnerability directly to a specific asset from your inventory. “Training dataset underrepresents minority groups” connects to the training data asset. “Inference API lacks rate limiting” connects to the API asset. “Model ownership unclear between data science and IT operations” connects to the human assets and governance structure.

Original implementation tip: Most teams find technical vulnerabilities and miss organizational ones. In my experience, the highest-impact AI failures trace back to process gaps, not code flaws. Unclear ownership between data science and IT operations is the single most dangerous vulnerability I encounter. Neither team thinks they own the model in production. When drift happens, both teams point at each other. Assign one owner with documented accountability before you deploy anything.

Step 4: Map Threats

Apply a structured taxonomy to identify who might exploit these vulnerabilities. MITRE ATLAS provides an AI-specific framework that covers adversarial machine learning techniques.

Categorize threat agents. Malicious outsiders include hackers, competitors, and organized cybercriminals. Malicious insiders exploit privileged access. Accidental insiders create exposure through negligence or misconfiguration. System failures include hardware malfunctions, software defects, and infrastructure outages.

For AI contexts, enumerate specific threat actions. Data poisoning during training. Adversarial inputs during inference. Model inversion or extraction that exposes sensitive training data. Prompt injection in generative systems. Output hallucinations that undermine accuracy. Misuse of generative capabilities for fraud.

Every threat must connect to at least one vulnerability you already documented. “External attacker sends adversarial queries” connects to “API lacks input validation.” “Regulator investigates bias” connects to “training data underrepresents minority groups.” If a threat has no corresponding vulnerability, either you missed a vulnerability or the threat is not relevant to this system.

Original implementation tip: Do not treat threat mapping as a one-time exercise. Threat landscapes for AI systems shift faster than for traditional IT. New adversarial techniques appear in academic papers months before they show up in the wild. Subscribe to MITRE ATLAS updates, follow ML security research, and refresh your threat catalog at least twice a year. I made the mistake of treating my first AI threat map as static. Within eight months, three new attack vectors had emerged that were not in my original catalog. Two of them were directly applicable to our deployed system.

Stage 3: Scenario Construction

Step 5: Build Scenarios

Combine actor, vulnerability, and asset into a single causal chain. Use a consistent structure. “Actor exploits vulnerability in asset, leading to impact.”

Example: “An external attacker compromises the integrity of the credit scoring model by exploiting weak API input validation to generate unfairly high credit scores for fraudulent applicants.”

Apply bow-tie analysis to each scenario. On the left side, define initiating events, precursors, and preconditions. What must be true for the attacker to succeed? On the right side, define consequences and impacts across confidentiality, integrity, availability, fairness, compliance, and business objectives. Identify existing controls on both sides, distinguishing prevention from mitigation.

Document the assumptions behind each scenario. Attacker capability, tool availability, detection reliability. Specify the triggers: system failure, intrusion attempt, data drift, human error. State the preconditions: access to training data, absence of monitoring, unpatched components.

Express consequences in business language. Financial loss, operational disruption, reputational damage, regulatory sanction, customer trust erosion. Tie each consequence back to the assets and objectives defined earlier.

Original implementation tip: Write scenarios in language that a non-technical board member can read and understand in thirty seconds. I learned this the hard way. My first set of scenarios included phrases like “adversarial perturbation of feature vectors in the latent space.” The CISO nodded politely. The CFO checked her phone. The board moved on. Rewrite: “An attacker tricks the model into approving bad loans by feeding it manipulated applications.” Same risk. Ten times the impact in the room.

Stage 4: Quantitative Analysis

Step 6: Estimate Impact

Identify loss categories covering primary and secondary effects. Primary losses include productivity disruption, detection and response costs, system replacement costs, and fines. Secondary losses capture reputation damage, customer trust erosion, competitive disadvantage, and long-term churn.

For AI systems, add specific categories. Discrimination claims from biased decisions. Fraudulent transactions from adversarial manipulation. Compliance breaches under emerging AI regulation. Loss of confidence in automated decision-making.

Quantify each loss using ranges, not point estimates. “Fraudulent loans cost $50k to $250k” is useful. “Fraudulent loans are a high impact risk” is not. Draw from historical incident data, industry breach reports, and calibrated expert judgment. When consulting experts, ask for ranges they are 90% confident contain the true value.

Original implementation tip: Calibrate your experts before you use their estimates. Most people are overconfident in narrow ranges and underconfident in wide ones. Run a quick calibration exercise. Ask your subject matter experts ten factual questions with numeric answers and have them provide 90% confidence intervals. If fewer than nine of their intervals contain the correct answer, they need calibration training. Uncalibrated estimates will sabotage your entire Monte Carlo simulation. I ran a full risk model once with uncalibrated inputs and the output was off by a factor of three compared to actual incident costs the following year.

Step 7: Estimate Frequency

Break frequency into two components. Threat event frequency measures how often actors attempt to exploit a vulnerability. Vulnerability success probability measures how often those attempts succeed given current controls.

Gather threat event frequency from threat intelligence reports, organizational logs, industry attack databases, and internal incident history. Distinguish between automated probing, deliberate targeted attacks, and accidental internal events like misconfiguration or data drift.

Estimate vulnerability success probability by evaluating defensive controls. Patching practices, monitoring coverage, model robustness against adversarial input, incident response maturity. Use calibrated expert judgment when empirical data is thin.

Multiply them to get loss event frequency. Express as a range per year. “Adversarial inputs attempted 2 to 10 times per year, success probability 10% to 20%, loss event frequency 0.2 to 2 successful events per year.”

Original implementation tip: Separate malicious frequency from accidental frequency. Data drift is not an attack. It is a certainty. Models degrade over time as the world changes. Treat drift-related scenarios with near-certain frequency estimates, not as low-probability events. I’ve seen teams assign “unlikely” ratings to model drift scenarios. Every single model drifts. The question is when and how badly, not whether it happens.

Step 8: Model Risk

Run Monte Carlo simulations combining your frequency and impact distributions. Use at least 100,000 iterations for statistical stability. Each iteration produces a plausible annual loss outcome.

From the output, compute three key metrics. Expected loss (the average), which represents the long-term financial burden. Value at risk at the 90th or 95th percentile, which shows severe but plausible outcomes. Tail risk beyond those percentiles, which reveals catastrophic exposure.

Express everything in monetary terms. “Median annualized loss exposure is $125k. There is a 15% chance losses exceed $300k in a given year. Maximum simulated event is $550k.” This language connects directly to business decisions.

Original implementation tip: Do not present simulation results without also presenting the input assumptions. Every Monte Carlo output is only as good as its inputs. When you brief leadership, show them the frequency ranges and loss ranges you fed in, the data sources behind those ranges, and the confidence level of your expert estimates. I once delivered a clean risk report with precise-looking numbers. The first question from the CRO was “Where did these numbers come from?” I did not have the input documentation ready. The entire presentation lost credibility. Now I include an assumptions appendix with every simulation output.

\[Suggested image: A sample Monte Carlo output distribution showing expected loss, VaR at 90th percentile, and tail risk\]

Stage 5: Decision and Action

Step 9: Evaluate Risks

Compare simulation outputs to your defined risk appetite. If your median annualized loss exposure of $125k exceeds your $100k threshold, the risk is unacceptable. Period.

But financial tolerance is only half the evaluation. Assess each scenario against responsible AI principles. A bias-driven compliance risk may fall within financial tolerance but remain completely unacceptable on ethical and legal grounds. Evaluate fairness of outcomes, clarity of decision-making, and adherence to regulatory requirements with equal weight.

Rank scenarios by expected exposure, tail risk potential, and strategic relevance. Use risk matrices only as communication aids, never as decision tools. Identify which risks need mitigation, transfer, acceptance, or escalation to the board.

Estimate return on investment for each treatment option by comparing the AI system’s anticipated benefits against expected losses and mitigation costs.

Original implementation tip: Never let a low-frequency bias scenario survive evaluation just because its expected annual cost is small. Regulators do not think in annualized loss exposure. They think in headlines. A single discriminatory outcome that affects a protected class can trigger enforcement action, class action lawsuits, and reputational damage that no Monte Carlo simulation adequately captures. Flag bias risks separately and route them to your ethics and compliance governance body regardless of their financial ranking.

Step 10: Treat Risks

For each prioritized scenario, define specific treatment measures. Technical controls like web application firewalls and adversarial input detection. AI-specific treatments like bias audits, explainability tools, model cards, and access restrictions. Process improvements like retraining schedules and red-teaming programs.

Consider risk transfer through cybersecurity insurance or contractual arrangements. Evaluate avoidance by limiting AI scope or halting deployment in high-risk applications. Accept residual risk only when it falls within documented tolerance and only with governance sign-off.

Calculate ROI for each treatment. A $40k investment in adversarial input detection that reduces attack frequency by 80% and cuts annualized loss exposure from $125k to $25k delivers a return of 300%. That math gets budget approved.

Original implementation tip: Bundle your treatments and present them as a single investment package with a combined ROI. When I presented treatments individually, each one competed against unrelated budget priorities and half of them got cut. When I bundled adversarial defenses, bias auditing, and monitoring into one “AI risk control package” with a combined ROI of 300%, the CFO approved the entire package in one meeting. Frame treatment spending as insurance against quantified exposure, not as a cost center.

Step 11: Integrate Decisions

Feed AI risk results into existing enterprise risk management structures. Use the same reporting language, the same dashboards, and the same meeting cadence as financial, cyber, and operational risk.

Connect residual risk levels to forward-looking business decisions. Product launch approvals, geographic expansion, pricing strategies, warranty terms, insurance negotiations. Leadership cannot make informed decisions about AI deployment if risk data lives in an isolated report that nobody reads.

Ensure escalation paths are clear. When residual exposure exceeds tolerance, the information must reach executive and board level through documented channels.

Original implementation tip: Present AI risk alongside other enterprise risks in the same board report. Do not create a separate AI risk briefing that competes for calendar time. The moment AI risk becomes “that other report,” it loses executive attention. Integrate it. One page in the existing risk summary. Annualized loss exposure in the same column as cyber risk and fraud risk. That is how AI risk gets treated as a real business concern rather than a theoretical exercise.

Stage 6: Continuous Monitoring

Step 12: Monitor and Iterate

Establish continuous monitoring across technical, organizational, and process domains. Track model drift, emerging adversarial techniques, bias reappearance in outputs, and infrastructure changes. Build dashboards with automated alerts that trigger review when indicators exceed defined thresholds.

Recalibrate frequency and impact estimates quarterly. Use the latest operational data, incident records, and threat intelligence. Run Monte Carlo simulations again with updated inputs.

Red-team your AI systems on an ongoing basis. Simulate adversarial behavior, challenge existing controls, and find vulnerabilities before attackers do.

Update the risk register every quarter with residual risk levels, treatment outcomes, and any new scenarios identified through monitoring or incident review. Feed insights back into Step 1 to close the loop.

Original implementation tip: Automate your drift detection and bias monitoring from the start. Manual quarterly reviews miss problems that emerge between review cycles. I worked with a team that relied on quarterly manual checks. The model drifted significantly in month two, produced biased outputs for six weeks before anyone noticed, and generated three customer complaints that reached the regulator. An automated monitoring pipeline with real-time alerts would have caught the drift within days. The cost of automated monitoring was less than 10% of the cost of the resulting regulatory response.

Cross-Cutting Tips That Apply Across Every Stage

These four principles apply throughout the entire procedure, regardless of which step you are executing.

Original implementation tip on documentation: Record every decision, assumption, and data source as you go. Do not plan to “document it later.” Later never comes. I have inherited risk assessments where the simulation outputs existed but the input assumptions were lost. The entire assessment had to be re-run from scratch because nobody could defend the original numbers. Use a decision log that captures date, participants, inputs, outputs, and rationale for every significant choice.

Original implementation tip on role clarity: Assign a single accountable owner for each step using a RACI framework. In practice, the most common dysfunction is a step where everyone is “consulted” and nobody is “accountable.” Vulnerability identification is the step where this breaks down most often. Data scientists think it is a security team responsibility. The security team thinks it is a data science responsibility. Neither team does it. Name one person. Make them answer for the output.

Original implementation tip on calibration consistency: Use the same calibration method for all expert estimates throughout the assessment. If your impact experts are calibrated using one method and your frequency experts use a different method (or none at all), your Monte Carlo inputs will carry inconsistent levels of confidence. Standardize your calibration training and apply it to every subject matter expert who contributes ranges to the model.

Original implementation tip on governance integration: Treat the completed risk assessment as a living document with a defined review cycle, not as a project deliverable that gets filed. Assign a review owner, set calendar reminders for quarterly updates, and tie the review cycle to your organization’s existing governance meeting schedule. Risk assessments that are not reviewed within 90 days of completion begin to decay in accuracy and relevance.

References and Standards

The procedure described in this post draws from and aligns with the following standards and frameworks:

ISO/IEC 27005:2022, Information security, cybersecurity and privacy protection, providing guidance on managing information security risks.

ISO/IEC 23894:2023, Information technology, Artificial intelligence, providing guidance on risk management specific to AI systems.

ISO/IEC 42001:2023, Information technology, Artificial intelligence, setting requirements for establishing, implementing, maintaining, and improving an AI management system.

The FAIR (Factor Analysis of Information Risk) framework, providing a quantitative model for information risk analysis.

MITRE ATLAS (Adversarial Threat Landscape for AI Systems), offering a knowledge base of adversarial tactics and techniques against AI.

EU AI Act (Regulation 2024/1689), establishing harmonized rules on artificial intelligence.

NIST AI Risk Management Framework (AI 100-1), providing guidance for managing risks associated with AI systems.

NIST SP 800-30 Rev. 1, Guide for Conducting Risk Assessments, offering a foundational risk assessment methodology.

Equal Credit Opportunity Act (ECOA) and related fair lending regulations, governing non-discrimination in credit decisions.

GDPR (Regulation 2016/679), governing the protection of personal data in the European Union.

The Difference Between Compliance Theater and Real Risk Management

Organizations that treat this procedure as a compliance artifact will fill out templates, generate reports that collect dust, and discover their actual risk exposure only after an incident forces them to confront it. They will spend more on incident response and regulatory fines than they would have spent on proper assessment and treatment. Their AI systems will carry hidden risks that leadership never sees until the damage is done.

Organizations that treat this procedure as a living operational tool will know their annualized loss exposure in dollar terms, defend their deployment decisions with traceable analysis, and catch model drift and emerging threats before they become incidents. They will integrate AI risk into the same governance structures that manage every other business risk, and their leadership will make AI investment decisions with the same rigor they apply to financial and operational planning.

The risk assessment procedure is not a document you complete. It is a discipline you practice.

What step in your current AI risk assessment process would benefit most from the quantitative rigor described here? That is probably the step where your biggest blind spot lives.

About the Author

The frameworks, tools, and implementation guidance described in this article are part of the applied research and consulting work of Prof. Hernan Huwyler, MBA, CPA, CAIO. These materials are freely available for use, adaptation, and redistribution in your own AI governance, risk management, and compliance programs. If you find them valuable, the only ask is proper attribution. If you like the content, please like the article and share it.

As a Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative risk modeling, predictive risk automation, and AI audit readiness for executive leadership teams, boards, and technical practitioners. His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe and globally.

Goal Setting for AI Projects

Thu, 12 Mar 2026 00:00:00 +0000

How to Define Objectives, Scope, and Success Without Creating False Expectations

Most AI projects do not fail because the team lacked ambition.

They fail because the goals were vague, the scope was loose, and the expected outcomes were never translated into measurable business terms. One group thought the project was meant to improve productivity. Another thought it was a customer experience initiative. Engineering optimized accuracy. Leadership expected revenue lift. Six months later, everyone was disappointed for different reasons. That is what weak objective setting does.

A strong AI project starts with clear business objectives and expected outcomes. It also needs a practical scope, realistic milestones, defined deliverables, and metrics tied to the reason the project exists in the first place. This post shows you how to do that properly, with a working structure you can use in real project governance.

Understanding the Core Framework for AI Goals and Objectives

Goal setting for AI projects is not just about writing a business case. It is about translating intent into a sequence of decisions, milestones, metrics, and boundaries that can guide delivery.

The framework I use has four parts. Business objective, expected outcome, delivery scope, and measurement logic. If one of these is weak, the project usually drifts.

1. Business objective

This is the strategic reason the AI project exists. It should answer one clear question. What business result are we trying to improve?

Typical objectives include better decision-making, higher productivity, revenue growth, improved customer experience, or stronger competitive position. The objective should be specific enough that a stakeholder can tell whether the project is relevant to it.

Implementation tip: Write the objective in language the business would use even if the project had no AI in it. This keeps the focus on value, not technology.

2. Expected outcome

This is the operational effect you expect the project to create. It should describe what will improve, for whom, and by how much if possible.

Examples include reducing handling time for a workflow, improving forecast quality, increasing adoption of self-service support, reducing manual review volume, or improving targeting in campaigns. The outcome should be testable.

Implementation tip: For each objective, require one sentence that starts with “We expect this project to change…” This forces teams to describe actual impact.

3. Delivery scope

This defines what the project will and will not cover in the first version. A useful scope statement protects the team from ambition overload and gives stakeholders a realistic view of what will be delivered.

AI teams often skip this discipline because they want flexibility. The result is uncontrolled expansion, vague accountability, and weak evaluation.

Implementation tip: Add a “not in scope for version one” section to every AI project plan. It reduces confusion fast.

4. Measurement logic

This is how you will know whether the project is working. It includes baseline metrics, target metrics, checkpoints, benchmarks, and review points.

A lot of teams choose metrics too late. They end up measuring what is easy instead of what matters. Strong measurement starts at the objective stage, not after the pilot.

Implementation tip: Tie every objective to one primary metric, one supporting metric, and one guardrail metric. That prevents one-dimensional success claims.

Why AI Objectives Go Wrong So Often

The common failure patterns are familiar.

Teams define an objective like “improve operations with AI.” That sounds sensible and means almost nothing. Or they pick ambitious outcomes without grounding them in current process data. Or they let stakeholders assume the model will be near-perfect on day one. Then when performance is merely useful instead of magical, confidence drops.

Another issue is mismatch between strategic goals and delivery design. A project may be positioned as a revenue driver when the first version can only realistically support internal efficiency. That gap creates pressure to oversell results.

There is also the problem of scope inflation. Once the project starts, new ideas pile on. More features. More users. More systems. More use cases. Without clear boundaries, the project loses shape.

Implementation tip: In the kickoff phase, ask every stakeholder to describe success in one sentence. If the answers differ widely, objective alignment is not ready.

Stage 1: Define Clear Business Objectives and Expected Outcomes

This is the first essential step. The goal is to define why the AI project exists and what specific result it is meant to support.

The responsible parties are the business sponsor, product owner, process owner, PMO or transformation lead, finance partner, and AI governance lead. Legal, privacy, security, and compliance should be consulted early when the use case is regulated or high impact.

The critical artifacts are the objective statement, expected outcomes summary, stakeholder assumptions log, and initial ROI case. These should be concise and tied directly to a business need already validated.

What to implement: Align project milestones with business goals and expected return on investment. Set realistic expectations with stakeholders about AI capabilities and likely benefits. Use specific examples to show how the AI system will support the objective. If the project is meant to improve forecasting, explain how forecasts will be used differently. If the project is meant to reduce manual effort, show which tasks will change.

This is also where you should simplify the problem for the first version. The first release should aim for useful progress, not comprehensive transformation. Starting with a smaller, solvable problem builds momentum and gives the organization evidence before broader expansion.

Implementation tip: Force teams to define the first version objective separately from the long-term vision. Those two should not be written as if they are the same thing.

Stage 2: Set Realistic Expectations and Create Measurable Success Criteria

Once the objective is clear, define how success will be measured and what level of performance is realistically expected.

The responsible parties are the product owner, business sponsor, analytics or data team, AI lead, and PMO. Governance or risk teams should review if the metrics could hide important tradeoffs.

The critical artifacts are the KPI set, benchmark definitions, baseline assessment, proof-of-concept criteria, and stakeholder communications pack. This material should be stable enough to support steering discussions.

What to implement: Define metrics and benchmarks for evaluating AI performance. Accuracy, precision, and recall are useful technical measures for many use cases, but they should not stand alone. Add process, user, and business metrics such as time saved, resolution rate, manual review rate, customer satisfaction, revenue impact, or forecast improvement depending on the objective.

Establish a baseline using the current process or existing solution. Without a baseline, improvement claims are weak. Create proof-of-concept checkpoints to test performance against early targets before the team commits to wider rollout.

You also need to communicate realistic model behavior. AI models may not be fully accurate at first. Improvement is often gradual. Stakeholders should hear that early and often. This is not lowering the standard. It is setting the right conditions for disciplined learning.

Implementation tip: Put the baseline and target values side by side in every steering pack. That keeps the conversation grounded in real progress.

Stage 3: Define the Project Scope Clearly

A good objective can still fail if the scope is vague.

The responsible parties are the product owner, project manager, business sponsor, enterprise architect, operations lead, and AI governance lead. Security, privacy, legal, and IT should review where systems or data boundaries matter.

The critical artifacts are the scope statement, out-of-scope list, work breakdown structure, task dependencies, milestone map, timeline, and resource plan. These form the backbone of execution control.

What to implement: Define what the AI project will and will not cover. Break the work into specific tasks that are logically sequenced and linked by dependencies. Set milestones for critical phases such as discovery, data readiness, proof of concept, integration, user testing, and production readiness. Define deliverables with quality criteria so teams know what “done” means.

Build a realistic timeline with task durations, resource allocations, and buffers for delays. AI projects often need more rework than non-AI software efforts because data, model behavior, and user feedback evolve together. If the timeline assumes a straight line, it will become unreliable fast.

Allocate resources by phase. That includes personnel, tooling, infrastructure, review effort, and change support. If all you have is a budget number with no resource logic underneath it, the plan is too thin.

Implementation tip: Add one explicit scope boundary for each of these areas. User group, data sources, systems integrated, automation authority, and geography. These are the most common scope creep paths.

Stage 4: Use the Project Plan as a Management Tool, Not a Static Document

A project plan should help the team make decisions, not just satisfy governance.

The responsible parties are the project manager, product owner, sponsor, PMO, and workstream leads. Governance should use the plan to track control readiness, not only delivery progress.

The critical artifacts are the live project plan, milestone status report, risk log, decision log, and change request tracker. These should be reviewed regularly and updated when assumptions change.

What to implement: Identify risks early and define mitigation actions before they become blockers. Keep the plan adaptable so it can reflect new insights, technical findings, or business changes. Review and update the plan regularly. Use it as a communication tool to keep stakeholders informed, aligned, and involved throughout the project lifecycle.

This matters because AI projects almost always generate new information after the first tests. Data quality may be weaker than expected. A model may perform differently on real scenarios. User adoption may be slower than hoped. A static plan cannot absorb that well.

Implementation tip: Review the plan against the objective, not just the calendar. A milestone met on time is less meaningful if it moved the project away from its business purpose.

Stage 5: Map Objectives to Practical AI Use Cases

Clear objectives become useful when they connect to actual implementation patterns. The examples below show how common business objectives translate into AI project choices.

Objective to enhance decision-making

This objective fits use cases where the business needs better forecasting, stronger risk insight, or more informed planning. Examples include transaction acceptance, market trend forecasting, scenario planning, and risk assessment.

What to implement: Deploy predictive analytics for strategic planning or operational decisions where better prediction improves timing, prioritization, or resource allocation. Define how decisions will be influenced, reviewed, and measured. If AI provides risk scores or forecasts, set rules for when humans must challenge or override them.

Implementation tip: Tie decision-support projects to a specific decision moment. If the output does not change a real decision, the value case is weak.

Objective to increase productivity

This is one of the most common AI objectives and one of the easiest to oversimplify. Productivity gains usually come from reducing repetitive work, improving retrieval, assisting with drafting, or supporting employees in complex tasks.

What to implement: Identify repetitive tasks suitable for automation through AI agents, copilots, AI-assisted process automation, or quality and compliance support. Use analytics to improve resource allocation. Apply text generation where internal or external materials can be drafted more efficiently. Plan staff training so people can use the tools effectively and know when to verify outputs.

Implementation tip: Measure net productivity, not only task automation. If AI saves time in one step but creates rework later, the gain may be overstated.

Objective to increase revenue

Revenue-focused AI projects need especially careful objective setting because commercial impact is often influenced by many variables at once.

What to implement: Use AI to identify market opportunities, improve segmentation, personalize recommendations, support targeted campaigns, or optimize pricing where appropriate. Make sure the project distinguishes between direct revenue outcomes and supporting signals such as conversion quality, lead prioritization, or offer relevance.

Implementation tip: Use supporting commercial indicators early and reserve direct revenue claims for later when enough evidence exists.

Objective to improve customer experience

This objective often includes personalization, 24/7 support, faster response times, sentiment analysis, or loyalty support. It is a powerful objective and a risky one if teams focus on efficiency more than quality.

What to implement: Deploy AI-powered personalization, virtual support agents, feedback analysis, and proactive support features. Define what better customer experience means in measurable terms such as reduced waiting time, improved resolution quality, higher satisfaction, or smoother journeys.

Implementation tip: Pair customer experience metrics with complaint and escalation metrics. Faster service is not better if trust declines.

Objective to develop competitive advantages

This objective usually fits research and development, predictive maintenance, inventory planning, competitor analysis, benchmarking, or product development support. It can be valuable, but it must still connect to concrete operational outcomes.

What to implement: Use AI in targeted research, planning, design, or optimization efforts where it creates a meaningful edge. Define how the project supports differentiation, cost structure, speed to insight, or product quality. Avoid vague claims about “innovation leadership” unless the business can explain what that means operationally.

Implementation tip: Competitive advantage is strongest when tied to a distinctive asset such as proprietary data, workflow knowledge, or customer context. Say which one matters.

Stage 6: Revisit Objectives as the Project Learns

Strong AI goals are stable in purpose but flexible in detail. As the project moves through testing and adoption, teams will learn things that should refine the objective, expected outcomes, or rollout path.

The responsible parties are the sponsor, product owner, PMO, analytics team, AI lead, and governance. The business owner should approve objective changes when they materially affect the value case or scope.

The critical artifacts are the updated objective log, lessons learned register, revised KPI set, and steering decisions. These keep the project aligned without pretending nothing has changed.

What to implement: Revisit objectives as new insights emerge. Refine expected outcomes based on actual model behavior, workflow fit, user adoption, and business conditions. Keep the strategic direction stable where possible, but update the path to reflect reality.

This is where projects either mature or start drifting. If you revise objectives too casually, accountability weakens. If you never revise them, the project becomes disconnected from what the team has learned.

Implementation tip: Separate objective refinement from objective rewriting. Adjusting a target or narrowing a scope is different from changing the fundamental reason the project exists.

Tips for AI Goals and Objectives

These tips apply throughout the lifecycle.

Tip 1: Start narrower than feels comfortable

Teams often assume broader goals create more strategic value. They usually create more confusion.

Implementation tip: Define the smallest meaningful business outcome the first version can achieve. That produces cleaner delivery and stronger evidence.

Tip 2: Use examples to make objectives real

Abstract objectives lead to abstract decisions.

Implementation tip: For each objective, include one specific example of how a user, customer, or business process will behave differently if the project succeeds.

Tip 3: Keep metrics balanced

A project can improve one dimension while damaging another.

Implementation tip: Use business, operational, and quality metrics together. That gives a fuller view of whether the objective is being met responsibly.

Tip 4: Keep the plan alive

A project plan should evolve with the project, not sit in a folder after kickoff.

Implementation tip: Review objectives, scope, milestones, and metrics together at regular checkpoints. Seeing them side by side reveals drift early.

Setting AI Goals and Objectives

If you want a stronger front-end structure for AI project planning, ground the work in recognized management and AI governance sources.

Here are the references I would use.

ISO/IEC 42001, AI management systems
ISO/IEC 42005, information to include in an AI impact assessment
ISO/IEC 23894, AI risk management
NIST AI Risk Management Framework 1.0
Internal PMO standards for business cases, stage gates, and delivery plans
Product management frameworks for outcome-driven planning
Change management and operational readiness frameworks
Privacy, security, continuity, and sector-specific compliance requirements relevant to the project

If your organization already uses portfolio planning, OKRs, business case reviews, or architecture stage gates, connect AI project objectives into those processes. That creates consistency and reduces AI-specific confusion.

Why AI Goal Setting Fails When Treated as a Kickoff Exercise

When teams treat goals and objectives as something to finish at kickoff, they produce broad ambition, weak scope, and generic metrics. The project starts moving, but nobody has a shared understanding of what success means, what the first version is actually meant to deliver, or how to judge progress honestly. That confusion usually shows up later as scope creep, stakeholder frustration, and pressure to overstate results.

When teams treat goals and objectives as the backbone of delivery, they create clarity. The objective is tied to a real business result. The scope is bounded. The milestones mean something. The metrics reflect actual progress. The team can learn without losing direction.

A strong AI project succeeds because its goals were specific enough to guide action and realistic enough to survive contact with reality.

If you reviewed your current AI portfolio today, which weakness would show up first: vague objectives, weak metrics, scope creep, unrealistic stakeholder expectations, or milestones disconnected from business value?

The 45 AI Threat Vectors That Your Security Team Probably Isn't Tracking

Thu, 12 Mar 2026 00:00:00 +0000

A Practitioner’s Field Guide

Most AI threat models are incomplete. Not slightly incomplete. Fundamentally incomplete.

Last year I reviewed the threat model for a financial services company deploying a credit decisioning AI. Their security team had identified seven threat vectors. Seven. They covered the obvious ones: data breaches, unauthorized access, denial of service. They missed 38 others, including 15 that were specific to AI systems and had no equivalent in their traditional IT threat catalog.

Three months after deployment, the model started producing subtly biased outputs. Not because of an external attack. Because a data scientist on the team had inadvertently introduced a feature engineering flaw that created a proxy variable for a protected characteristic. It was a negligence threat, an internal one, and it was not on anyone’s radar because the threat model only considered adversarial external actors.

This pattern repeats across almost every organization I work with. Security teams build threat models based on their experience with traditional systems. They focus on external attackers, malicious intent, and system-level exploits. They miss the internal negligence threats that cause the majority of real-world AI failures. They miss model-specific attack vectors that have no parallel in conventional cybersecurity. They miss the human and organizational threats that create the conditions for technical failures.

This post catalogs 45 distinct AI threat vectors organized across a two-dimensional taxonomy: intent (adversarial versus negligent) and target category (data, model, system, and human). Each vector includes a clear explanation, practical context, and implementation guidance. Use this as a working reference to audit the completeness of your own AI threat models.

Why Traditional Threat Models Fail for AI

Traditional threat modeling frameworks like STRIDE, PASTA, and even MITRE ATT&CK were designed for conventional information systems. They handle network attacks, authentication bypasses, privilege escalation, and data exfiltration well. They were not designed for systems where the “logic” is learned from data rather than written in code, where the attack surface includes the training pipeline itself, and where some of the most damaging threats come from well-intentioned internal teams making honest mistakes.

AI systems introduce three categories of threat that traditional frameworks handle poorly.

First, the model itself is an attack surface. Traditional systems have deterministic logic. If you protect the infrastructure and the data, the system behaves as designed. AI models are different. An attacker can manipulate the model’s behavior by carefully crafting inputs, without ever breaching the perimeter or accessing the infrastructure. They can extract sensitive training data by querying the model’s API. They can clone the model’s functionality through systematic probing. None of these attacks require the kind of infrastructure compromise that traditional threat models focus on.

Second, the training pipeline is a persistent vulnerability. Traditional systems are vulnerable during operation. AI systems are vulnerable during development. Poisoned training data, biased labels, flawed feature engineering, and compromised pre-trained models all introduce vulnerabilities before the system ever reaches production. By the time the model is deployed, the damage is already embedded in its weights.

Third, negligence threats cause more cumulative damage than adversarial threats. In traditional cybersecurity, the adversary is the primary concern. In AI, the internal team building and operating the system creates more risk through oversight, insufficient testing, poor documentation, and inadequate monitoring than external attackers do through deliberate exploitation. A threat model that only considers malicious actors misses the majority of the threat landscape.

Original implementation tip: When I first expanded an organization’s AI threat model beyond traditional categories, the security team pushed back hard. “We already cover insider threats,” they said. They did, but their insider threat model focused on malicious insiders who steal data or sabotage systems. It did not cover the data scientist who chooses features without considering proxy discrimination, the ML engineer who skips robustness testing under deadline pressure, or the architect who fails to build monitoring into the deployment pipeline. These are not insider “threats” in the traditional security sense. They are negligence risks that require fundamentally different controls. Treat them as separate categories in your threat model, not as subcategories of “insider threat.”

The Taxonomy Structure

This threat taxonomy organizes 45 vectors across two dimensions.

The first dimension is intent. Adversarial threats involve deliberate, intentional actions designed to compromise the AI system. Negligent threats involve unintentional actions or omissions that create vulnerabilities or cause harm. This distinction matters because adversarial and negligent threats require different controls. You defend against adversaries with detection, deterrence, and response. You defend against negligence with process, training, governance, and automation.

The second dimension is agent. External agents operate outside the organization’s boundary, including hackers, competitors, nation-state actors, and third-party vendors. Internal agents operate within the organization, including developers, data scientists, architects, operators, and end users.

Within each combination of intent and agent, threats target one of four categories: data (the information the system processes and learns from), model (the learned representations, algorithms, and parameters), system (the infrastructure, APIs, and operational environment), and human (the people who build, operate, and interact with the system).

The result is a comprehensive matrix that surfaces threats most organizations overlook because they fall outside the traditional “external attacker targeting our infrastructure” frame.

Adversarial External Threats: The Ones You Expect

This quadrant contains the threats most security teams already think about, plus several AI-specific vectors they likely do not.

Data Threats from External Adversaries

Two vectors target the data layer from outside the organization.

Data exfiltration occurs when attackers extract sensitive data from the AI system, either during training or inference. This differs from traditional data theft because the AI model itself can leak data. An attacker who gains access to model outputs, gradients, or confidence scores can sometimes reconstruct training data without ever accessing the database directly. The model becomes an unintentional data disclosure channel.

Data poisoning occurs when attackers introduce malicious or corrupted data into the training dataset. This is uniquely dangerous because the corruption happens before the model is deployed. If an attacker can influence any data source that feeds into the training pipeline, whether through compromised public datasets, manipulated web scraping sources, or tampered third-party data feeds, they can embed biases or backdoors that persist through every subsequent version of the model until the poisoned data is identified and removed.

Data poisoning is the adversarial data threat I worry about most because it is the hardest to detect and the longest-lasting in impact. Traditional data validation checks look for obvious anomalies: missing values, out-of-range entries, format errors. Poisoned data is designed to look normal. The individual data points are plausible. The corruption is statistical, not syntactic. Defending against it requires comparing training data distributions across time windows to detect subtle shifts, validating training data provenance to verify that sources have not been compromised, and testing model behavior on held-out validation sets from trusted sources. If your training data comes from any source you do not fully control, you have exposure to this vector.

Model Threats from External Adversaries

Six vectors target the model itself, and these are the threats most traditional security teams have the least experience with.

Deception involves crafting inputs designed to make the AI model produce incorrect predictions or classifications. Think of carefully modified images that cause a computer vision system to misclassify objects, or subtly altered text inputs that cause a natural language model to produce wrong outputs. The modifications are often imperceptible to humans but effective against the model.

Evasion is deception’s cousin, focused specifically on bypassing detection or classification mechanisms. An attacker modifies malicious inputs to evade an AI-based security system, such as altering malware signatures to bypass AI-powered threat detection or modifying fraudulent transactions to avoid AI-based fraud scoring.

Exploitation targets weaknesses in the AI system’s implementation rather than the model’s learned behavior. Insecure APIs, poor input validation, missing authentication, and misconfigured endpoints are the entry points. This vector bridges traditional cybersecurity and AI-specific risk because the vulnerabilities are conventional but the assets being targeted (models, training pipelines, inference endpoints) are AI-specific.

Inversion attacks reverse-engineer the model to infer sensitive information about training data. By carefully analyzing the model’s outputs across many queries, an attacker can deduce characteristics of the data the model was trained on. For models trained on medical records, financial data, or personal information, this vector directly threatens privacy even if the underlying database is perfectly secured.

Membership inference is related but distinct. Instead of reconstructing training data, the attacker determines whether a specific known data point was part of the training set. “Was this person’s medical record used to train your diagnostic model?” is a question that, if answerable through the model’s API, creates privacy and compliance exposure regardless of whether the attacker can reconstruct the full record.

Oracle attacks (also called model extraction or model stealing) involve an attacker querying the model extensively to understand its decision boundaries, then building a replica model that reproduces the original’s behavior without access to the training data. The attacker essentially steals the intellectual property embedded in the model through its public-facing API.

Transfer learning attacks exploit vulnerabilities in pre-trained models or the transfer learning process itself. Many organizations build their AI systems on top of pre-trained foundation models. If the foundation model contains embedded vulnerabilities, biases, or backdoors, every downstream model inherits them. This vector is growing in importance as more organizations build on third-party foundation models they did not train and cannot fully audit.

Of these six model-level threats, oracle attacks and transfer learning attacks are the two most underestimated. Oracle attacks are underestimated because organizations assume their model’s logic is protected by keeping the code proprietary. It is not. If the model is accessible through an API, its behavior can be replicated through systematic querying. Rate limiting helps but does not eliminate the risk. Transfer learning attacks are underestimated because organizations treat pre-trained foundation models as trusted components without verifying what is in them. I worked with a team that fine-tuned a publicly available language model for customer service automation. Nobody audited the base model for embedded biases or vulnerabilities. The assumption was “it’s from a reputable provider, so it’s safe.” That assumption is not supportable. If you use pre-trained models, document your trust assumptions about those models explicitly, test for bias and adversarial vulnerability in the fine-tuned model, and include the base model in your threat surface.

System Threats from External Adversaries

Eight vectors target the infrastructure and operational environment.

Advanced persistent threats involve state actors or organized crime groups using sophisticated tools to compromise the AI system over an extended period. These are the most resource-intensive attacks and typically target high-value AI systems in critical infrastructure, financial services, or defense applications.

API-based attacks exploit vulnerabilities in the APIs used to interact with the AI system. This includes injecting malicious data through API endpoints, exploiting authentication weaknesses, or abusing API rate limits to conduct model extraction.

Denial of service overwhelms the AI system with excessive traffic or resource demands. AI systems can be particularly vulnerable because inference workloads on complex models consume significant computational resources, making resource exhaustion attacks more efficient than against simpler services.

Model freezing attacks target the AI system’s update mechanism, preventing the model from receiving updates. A model that cannot be updated remains vulnerable to known threats and cannot adapt to data drift, effectively turning a dynamic system into a static one.

Model parameter poisoning introduces malicious updates or perturbations directly into the model’s parameters (weights, biases) rather than through training data. This vector is particularly relevant for federated learning systems where multiple parties contribute model updates.

Poorly designed APIs represent a threat vector that straddles adversarial and negligent categories. While the design flaw is internal, external attackers exploit it. Insecure API design that exposes model internals, lacks input validation, or provides excessive information in error messages creates the conditions for multiple other attacks.

Side-channel attacks exploit non-functional characteristics of the AI system, such as timing differences in inference responses, power consumption patterns during computation, or electromagnetic emissions, to extract information about the model or its data. These attacks do not target the model’s logic directly but extract information through observable physical or computational characteristics.

Supply chain compromise involves attackers tampering with hardware, software components, or third-party services in the AI system’s supply chain. Compromised ML libraries, poisoned pre-trained models distributed through public repositories, or tampered GPU firmware all fall into this category.

Supply chain compromise is the system-level external threat I spent the most time helping organizations address last year. The AI supply chain is broader and less controlled than most organizations realize. A typical ML pipeline might include open-source libraries (PyTorch, TensorFlow, scikit-learn), pre-trained models from public repositories (Hugging Face, GitHub), data from third-party providers, cloud services for training and inference, and container images from public registries. Each component is a potential entry point. The most practical defense is maintaining a software bill of materials (SBOM) for your AI systems that includes not just code dependencies but also model provenance, training data sources, and infrastructure components. When a vulnerability is discovered in any component, the SBOM tells you immediately which AI systems are affected. Without it, you are guessing.

Human Threats from External Adversaries

One vector targets the human element from outside.

Social engineering involves attackers manipulating developers, data scientists, or users into revealing sensitive information or interacting with the AI system in ways that compromise security. AI teams are often targeted because they have access to valuable IP (models, training data, feature engineering pipelines) and may not have received the same security awareness training as traditional IT staff. A data scientist who shares a model architecture diagram on a conference poster may not realize they have disclosed information useful for a model extraction attack.

Adversarial Internal Threats: The Ones You Underestimate

This quadrant contains only two vectors, but both are high-impact.

Human Threats from Internal Adversaries

Data sabotage occurs when insiders intentionally alter or destroy data used for training or inference. Unlike external data poisoning, an insider has legitimate access to data systems and can make changes that appear routine. A disgruntled data engineer who subtly modifies preprocessing scripts or alters label distributions can compromise model performance in ways that are extremely difficult to trace.

Subversion involves authorized developers or contractors intentionally sabotaging, exfiltrating, or manipulating the AI system. This goes beyond data sabotage to include embedding backdoors in model code, exfiltrating trained model weights for competitors, or introducing vulnerabilities that can later be exploited. The insider’s authorized access makes traditional perimeter defenses irrelevant.

Internal adversarial threats against AI systems are harder to detect than their equivalents in traditional IT for one specific reason: the normal behavior of an AI developer already includes activities that would be flagged as suspicious in other contexts. A data scientist routinely downloads large datasets, modifies data processing logic, changes model parameters, and deploys updated models. These are their job functions. Distinguishing between a legitimate model update and a sabotage event requires understanding what the model should be doing, not just what the developer is doing. The most effective control I have found is mandatory peer review for all changes to training data, feature engineering code, and model parameters before they reach production. Not automated testing, though that helps too. Human review by a second qualified person who can assess whether the change makes sense in context. This catches both intentional sabotage and unintentional errors.

Negligent External Threats: Your Vendors and Dependencies

This quadrant covers unintentional risks introduced by parties outside your organization.

Human Threats from External Negligence

Supply chain negligence occurs when third-party vendors introduce vulnerabilities through insecure libraries, dependencies, or tools used during AI system development, deployment, or maintenance. Unlike supply chain compromise (which is intentional), this vector reflects genuine negligence: a vendor fails to patch a library, releases an update with a security flaw, or provides tooling that does not meet security standards. The impact on your AI system is the same whether the vulnerability was introduced deliberately or carelessly.

Third-party data risk arises when organizations rely on external data sources that may be of poor quality, biased, or inadvertently altered. The third party is not acting maliciously. They simply do not maintain the data quality standards your model requires. Training on degraded external data produces degraded model performance, and the organization consuming the data may not detect the quality decline until outputs start failing.

System Threats from External Negligence

Outdated dependencies represent the use of unsupported software libraries in AI systems that introduce known, exploitable vulnerabilities. This is technically a negligence issue, an external provider stops maintaining a library, but the security impact is the same as an adversarial exploit because attackers actively scan for systems using deprecated dependencies.

Third-party data risk is the negligent external threat I encounter most frequently in practice. Organizations build models on external data feeds and assume the data quality will remain stable. It does not. I worked with a company whose fraud detection model degraded over four months because a third-party transaction data provider changed their data formatting without notification. The change was minor, a modification to how categorical fields were encoded, but it silently corrupted the feature engineering pipeline. The model’s accuracy dropped from 91% to 78% before anyone noticed. The fix was straightforward but the damage was done. For every external data dependency, establish a data quality SLA with the provider that specifies format, completeness, timeliness, and quality metrics. Monitor incoming data against those SLAs automatically. When a deviation occurs, alert before the data enters your training pipeline, not after your model degrades.

Negligent Internal Threats: Where Most AI Failures Actually Originate

This is the largest quadrant in the taxonomy, containing 24 of the 45 vectors. That distribution is not an accident. It reflects reality. The majority of AI failures in production stem from internal negligence, not external attacks.

Data Threats from Internal Negligence

Two vectors target the data layer through internal oversight.

Inaccurate data labeling occurs when developers fail to properly label training data during preparation. Labels are the ground truth the model learns from. If a medical imaging dataset contains mislabeled scans, the model learns to associate the wrong visual patterns with the wrong diagnoses. Unlike external data poisoning, this is not malicious. It is the predictable result of insufficient quality assurance in the annotation process, often driven by time pressure, undertrained annotators, or ambiguous labeling guidelines.

Bias in data occurs when data scientists or developers incorporate biased data during training, producing discriminatory or unfair outputs. This can result from historical bias embedded in the data itself (past lending decisions that reflected discriminatory practices), selection bias in how data was collected (underrepresenting certain populations), or measurement bias in how variables were recorded. The developers are not trying to create discriminatory outcomes. They are training on data that reflects existing inequities.

Bias in data is the negligent data threat with the highest regulatory and reputational impact. It is also the one where I see the most dangerous misconception. Teams believe that removing protected characteristics like race or gender from the training data eliminates bias. It does not. Other variables in the dataset frequently serve as proxies for protected characteristics. Zip code correlates with race. Job title correlates with gender. Part-time employment status correlates with caregiving responsibilities. Removing the protected characteristic while leaving the proxy variables in the feature set gives the appearance of fairness while producing the same discriminatory outcomes. The effective control is to test model outputs for disparate impact across protected groups, regardless of whether protected characteristics appear in the input features. Test the outputs, not the inputs.

Model Threats from Internal Negligence

Five vectors target the model through internal oversight.

Data and model drift occurs when developers fail to account for changes in data distribution or underlying concepts over time. A model trained on data from 2022 may not perform well on data from 2025 if customer behavior, market conditions, or the relationships between variables have shifted. This is not a one-time risk. It is an ongoing degradation that accelerates the longer a model operates without retraining or recalibration.

Feature engineering flaws result from unintentionally introducing vulnerabilities through poor feature selection. Selecting features that are easily manipulated by adversaries, including features that leak future information (data leakage), or failing to consider how feature distributions might shift in production all fall into this category.

Overfitting occurs when developers create models that fit too closely to the training data, learning noise and idiosyncrasies rather than genuine patterns. An overfit model shows excellent performance in testing and poor performance in production. From a security perspective, an overfit model is also more predictable to an adversary who understands its training data, making it easier to craft adversarial inputs.

Overfitting to noise is a specific variant where the model learns from irrelevant data rather than meaningful signal. The model performs well on training metrics but produces unreliable results in real-world deployment because it has memorized artifacts in the training data rather than learning the underlying relationship.

Unexplainability results when developers create AI systems too complex and opaque to understand or explain. This is a threat vector because opacity prevents detection of errors, biases, and vulnerabilities. A model whose decisions cannot be explained cannot be audited, cannot be debugged when it fails, and cannot satisfy regulatory requirements for explainability. Opacity does not cause harm directly, but it creates the conditions under which every other threat vector becomes harder to detect and address.

Data and model drift is the negligent model threat that causes the most cumulative financial damage because it is slow, silent, and continuous. I have never worked with an organization that detected drift proactively on their first AI deployment. They always detected it reactively, after business outcomes degraded enough for someone to notice. The detection lag ranged from three weeks to nine months depending on how closely business stakeholders monitored the model’s downstream effects. The fix is statistical monitoring of input feature distributions and output prediction distributions, compared against baseline distributions from the training period. When statistical tests detect a significant shift, trigger an alert. Do not wait for business outcome metrics to degrade, because by then the model has been making suboptimal decisions for weeks or months. Population Stability Index (PSI) is a good starting metric. Monitor it weekly at minimum for production models.

Human Threats from Internal Negligence

This is the richest subcategory in the entire taxonomy, containing 12 vectors. Each represents a different way that well-intentioned people create AI risk through oversight, insufficient skill, or organizational failure.

Inadequate documentation occurs when teams provide insufficient documentation for AI models, data sources, and decision-making processes. Without documentation, models cannot be maintained by anyone other than their original developer, security reviews cannot assess the system’s design assumptions, and regulatory compliance cannot be demonstrated.

Inadequate monitoring results from failing to build proper monitoring mechanisms to detect anomalies, adversarial activity, or model degradation in real time. An unmonitored model is a model whose failures go undetected until they manifest as business losses, customer complaints, or regulatory actions.

Inadequate maintenance occurs when teams fail to regularly update and maintain AI models, leaving them vulnerable to known attacks, data drift, and exploits as the system ages. Models, like all software, require ongoing maintenance. Unlike traditional software, model maintenance includes retraining, recalibration, and feature re-evaluation in addition to patching.

Inadequate testing results from failing to sufficiently test and validate models before deployment. This includes insufficient unit testing of data pipelines, absence of adversarial robustness testing, incomplete validation against held-out datasets, and failure to test for bias and fairness.

Inadequate training affects end-users and operators who receive insufficient instruction on how to interact with or manage AI systems. A model that is technically sound can still produce harmful outcomes if the humans using it do not understand its limitations, do not know when to override its recommendations, or cannot recognize when its outputs are unreliable.

Insecure design results from architects or developers failing to build secure AI systems from the start. This includes objective functions susceptible to manipulation, model architectures that leak information through their outputs, and deployment configurations that expose internal model details.

Insider threat (unintentional) covers authorized team members who inadvertently cause harm. A developer who accidentally pushes a model trained on test data to production. A data engineer who modifies a preprocessing script that breaks feature normalization. An operations team member who changes a configuration parameter without understanding its downstream effects.

Insufficient access control results from poorly managed access permissions that allow unauthorized access to models, data, or systems. This is a process failure, not a technology failure. The tools to enforce access control exist. The organization simply has not implemented them with sufficient rigor for AI-specific assets.

Lack of governance occurs when organizations fail to establish proper governance frameworks for AI development and deployment. Without governance, teams operate independently, security practices are inconsistent, accountability is undefined, and risk accumulates without visibility.

Over-reliance on AI results from decision-makers depending on AI outputs without sufficient human oversight or fail-safes. When users treat AI predictions as infallible, they stop applying the human judgment that catches model errors. This vector is particularly dangerous in high-stakes domains like healthcare, criminal justice, and financial services.

Unclear AI accountability occurs when nobody is clearly defined as accountable for AI-related decisions, model performance, or security. When accountability is unclear, risks go unmanaged because everyone assumes someone else is responsible.

Of these 12 human negligence vectors, inadequate monitoring and unclear accountability are the two that create the most cascading damage. They amplify every other threat in the taxonomy. An adversarial attack against an inadequately monitored system succeeds for longer. Data drift in a system with no accountable owner goes unaddressed for months. Bias in a model that nobody monitors against fairness metrics persists indefinitely. If you can only address two human negligence vectors immediately, address these two. Assign a named individual, not a team, as accountable for each production AI system. Then build monitoring that runs at the same speed as the model’s decision-making. Everything else becomes more manageable once you have visibility and ownership in place.

System Threats from Internal Negligence

Five vectors target the system infrastructure through internal oversight.

Inadequate incident response results from failing to develop or implement effective response plans for AI-specific incidents. AI incidents differ from traditional IT incidents. A model producing biased outputs is not a “system down” event. It does not trigger the same alerts. It requires different diagnostic procedures and different remediation steps. If your incident response playbook does not include AI-specific scenarios, your response will be improvised when it matters most.

Inadequate logging results from insufficient logging and monitoring infrastructure that makes it difficult to detect, investigate, or respond to incidents. If you cannot see what the model received as input, what it produced as output, and how its behavior has changed over time, you cannot diagnose problems or provide evidence for regulatory investigations.

Insecure data storage results from failing to implement secure storage for AI-specific data assets. Training data, model weights, feature engineering code, and hyperparameter configurations all contain sensitive intellectual property and potentially personal data. Storing them with the same (or lesser) security controls as general-purpose data creates exposure.

Insufficient redundancy results from failing to build AI systems with adequate fail-safes. A single point of failure in the model serving infrastructure, the data pipeline, or the monitoring system can take down the entire AI capability. AI systems often have complex dependency chains that create hidden single points of failure.

Misconfiguration results from incorrectly configuring security settings, environments, or system components. Cloud environment misconfigurations, exposed model endpoints, overly permissive IAM roles, and unencrypted data storage are the most common variants.

Inadequate incident response for AI systems is the system-level negligence threat I have spent the most time remediating. Traditional incident response plans categorize incidents by severity and system type, but they almost never include AI-specific incident categories. What do you do when a model starts producing outputs that are statistically different from its validation period behavior? What do you do when a fairness audit reveals disparate impact? What do you do when you discover that training data was contaminated three months ago and every model version since then is potentially compromised? These are AI incidents that require AI-specific response procedures. Build an AI incident response appendix for your existing plan. Include at minimum: model rollback procedures, retraining triggers, bias investigation protocols, adversarial attack containment steps, and stakeholder notification procedures. Then tabletop exercise these scenarios. The first time you run through an AI incident scenario, you will discover gaps in your response capability that are fixable before a real incident occurs.

Cross-Cutting Implementation Tips

These apply across all four quadrants of the threat taxonomy.

Assess your taxonomy coverage quarterly. AI threat vectors evolve as attack research advances, new model architectures emerge, and regulatory requirements expand. A taxonomy that was comprehensive six months ago may have gaps today. Assign someone to monitor AI security research (MITRE ATLAS updates, conference proceedings from NeurIPS and USENIX Security, regulatory guidance from NIST and the EU AI Office) and flag new vectors that should be added.

Original implementation tip: When reviewing your taxonomy coverage, do not just ask “have any new threat vectors emerged?” Also ask “have any of our existing vectors changed in severity or likelihood?” The relative importance of threat vectors shifts as your AI systems mature. Early in deployment, development negligence threats (inadequate testing, insecure design) are most relevant because the system is new and untested. Six months into production, operational negligence threats (inadequate monitoring, data drift, inadequate maintenance) become dominant. Twelve months in, adversarial threats increase as your AI system becomes a known, valuable target. Reassess priority rankings at each quarterly review.

Balance adversarial and negligence controls in your budget. Security teams naturally gravitate toward adversarial controls because they are more dramatic and more familiar. Adversarial robustness testing, penetration testing, red-teaming: these feel like “real” security work. Process controls, governance frameworks, training programs, and documentation standards feel like bureaucracy. In practice, the negligence controls prevent more incidents.

Original implementation tip: I track a simple metric with every organization I advise: the ratio of AI incidents caused by adversarial action versus negligence. Across 14 organizations over three years, the ratio has consistently been approximately 15% adversarial and 85% negligence. Yet budget allocation for adversarial controls versus negligence controls is typically inverted: 60-70% on adversarial defenses and 30-40% on process and governance. Reallocate to match the actual threat distribution. This does not mean reducing adversarial defenses. It means increasing investment in monitoring, documentation, testing processes, training, and governance until the budget reflects where incidents actually originate.

Map each vector to specific assets and controls. A threat vector without a corresponding asset mapping tells you what could happen but not where it could happen to you. A threat vector without a corresponding control tells you what to worry about but not what to do. For each vector in this taxonomy that applies to your AI systems, document three things: which specific assets are exposed, which controls currently mitigate the vector, and what residual exposure remains.

The most effective way I have found to operationalize a threat taxonomy is to create a traceability matrix with four columns: threat vector, exposed assets, current controls, and residual risk rating. Populate this matrix for every production AI system. When a new asset is deployed, add rows. When a new threat vector is identified, add rows. When a control is implemented or modified, update the current controls column and reassess residual risk. This matrix becomes the working document for your AI security program. It tells you at any point what threats you have addressed and what gaps remain. Without it, the taxonomy is an intellectual exercise. With it, the taxonomy drives action.

Key References and Standards

This threat taxonomy draws from and aligns with the following authoritative frameworks.

MITRE ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems) provides the primary reference taxonomy for AI-specific adversarial threats.

NIST AI RMF (AI 100-1) provides the risk management framework for identifying and addressing AI threats across the lifecycle.

ISO/IEC 27005:2022 provides the information security risk management process for integrating AI threats into enterprise risk assessment.

ISO/IEC 23894:2023 provides AI-specific risk management guidance including threat identification.

ISO/IEC 42001:2023 provides AI management system requirements for governance of the threat landscape.

OWASP Machine Learning Security Top 10 provides a practitioner-focused list of common ML security threats.

EU AI Act (Regulation 2024/1689) establishes regulatory requirements that make several of these threat vectors compliance-relevant.

NIST SP 800-30 Rev. 1 provides guidance on threat identification and risk assessment methodology.

ENISA Threat Landscape for AI provides European regulatory perspective on AI-specific threats.

Using This Taxonomy to Find Your Gaps

Organizations that treat this taxonomy as a reference list will read it, nod, and return to their existing threat models unchanged. They will continue to overweight adversarial external threats because those threats are familiar and dramatic. They will continue to underweight internal negligence threats because those threats feel mundane and uncomfortable to discuss. When an AI failure occurs, and it will, they will discover that the vector was sitting in a taxonomy they read but never operationalized.

Organizations that treat this taxonomy as an audit tool will do something different. They will take each of their production AI systems and map every applicable threat vector to the specific assets, existing controls, and residual risk for that system. They will discover gaps, mostly in the negligent internal quadrant, and they will prioritize closing them. They will update their incident response plans to include AI-specific scenarios. They will build monitoring that catches negligence-driven failures before they reach customers. They will assign accountability for each production model to a named individual who cannot hide behind a team name.

The threat your AI system faces tomorrow is almost certainly already in this taxonomy. The question is whether you have mapped it to your assets, built controls for it, and assigned someone to watch for it.

Which quadrant of this taxonomy has the least coverage in your current AI threat model? For most organizations, the answer is negligent internal. That is where your biggest gap probably lives.

About the Author

The AI Risk Taxonomy Most Organizations Never Build

Thu, 12 Mar 2026 00:00:00 +0000

Top Risk Scenarios and Controls That Actually Protect Your AI Project

A risk register with 15 vaguely worded AI risks and a color-coded heat map is not a taxonomy. It is a liability.

I reviewed an organization’s AI risk assessment last year that listed “AI bias” as a single risk with a “medium-high” rating. That was it. No decomposition into the dozen distinct ways bias manifests. No distinction between bias in training data, bias from proxy variables, bias from temporal misalignment, or bias from feedback loops. No specific controls mapped to specific failure modes. When their credit model produced discriminatory outcomes six months later, nobody could trace the failure to a gap in their controls because their taxonomy was too shallow to reveal where the gaps were.

The difference between organizations that manage AI risk effectively and those that just talk about it comes down to granularity. You need a taxonomy that decomposes AI risk into specific, actionable scenarios, each linked to a named control with concrete activities. This post provides exactly that: a structured taxonomy of 100 AI risk scenarios across 14 domains, with recommended controls mapped to COBIT 2019 governance objectives. Every scenario follows a consistent structure: what can go wrong, why it matters, and what to do about it.

This is a long reference piece. Use it as a working document, not a single-sitting read.

Why Most AI Risk Taxonomies Fail

The typical AI risk taxonomy fails for three reasons.

First, it operates at the wrong altitude. “Model risk” is not a scenario. It is a category that contains dozens of scenarios, each with different causes, different impacts, and different controls. When you treat a category as a scenario, your controls become generic and your residual risk unmeasurable.

Second, it ignores organizational and process risks. Most AI taxonomies obsess over technical risks like adversarial attacks and data poisoning while overlooking the governance, people, and operational risks that cause the majority of real-world AI failures. A model that degrades because nobody owns monitoring in production is not a technical failure. It is a governance failure.

Third, it lacks traceability from risk to control. Identifying a risk without mapping it to a specific, implementable control activity is an academic exercise. The taxonomy must create a direct line from “what could go wrong” to “what are we doing about it” to “how do we verify it is working.”

The taxonomy presented here addresses all three failures. It spans 14 domains from strategy through business continuity, covers 100 distinct scenarios, and links each one to a named control with specific activities. I have organized it to follow the natural lifecycle of AI in an enterprise, from strategic planning through development, deployment, operations, and ongoing governance.

Original implementation tip: When I first built an AI risk taxonomy for a European bank, I started with the technical risks because that is where the AI team’s attention naturally went. We ended up with 40 technical scenarios and 5 organizational ones. After the first major incident, which was caused by unclear model ownership between data science and IT operations, we realized our taxonomy was inverted. The organizational and governance risks caused more actual damage than the technical ones. Start your taxonomy with strategy, governance, and people risks. Then layer in the technical domains. This sequencing forces the right conversations early.

Domain 1: Business Value Risks

Strategy risks sit at the top of the taxonomy because every other risk domain inherits from them. If your AI strategy is flawed, your technical controls cannot compensate.

Two scenarios define this domain.

The first is strategy deficiency. Wasted resources and reputational damage may occur when an organization lacks a clear enterprise-wide AI strategy, leading to inefficient investments and potential misuse of AI. This is a Priority 1 risk.

The recommended control is an enterprise AI strategy. Develop and put in place a comprehensive AI strategy aligned with overall business objectives. Create clear guidelines for AI adoption and integration across departments. Establish governance structures with defined roles, responsibilities, performance metrics, and risk management protocols. Document policies and maintain evidence of governance through reports and records. Review and update the strategy regularly to reflect changes in technology, business needs, and regulatory requirements. Communicate the strategy across the organization to achieve alignment and stakeholder buy-in.

The second scenario is misaligned strategy. Missed opportunities may occur when insufficient stakeholder engagement leads to AI systems that do not support business goals or expose the organization to unacceptable risks. Also Priority 1.

The recommended control is stakeholder alignment. Establish stakeholder engagement processes to ensure AI systems align with business goals. Develop communication protocols and document policies for continuous alignment. Collect and maintain meeting records, stakeholder feedback, and communication logs as evidence.

Original implementation tip: The strategy risk I see most often is not the absence of a strategy. It is the presence of multiple competing strategies. The data science team has a roadmap. The IT department has an automation strategy. The business units each have their own AI wishlists. These strategies contradict each other in ways nobody notices until budget conflicts or architectural incompatibilities surface months later. Before you write a strategy document, conduct a strategy reconciliation exercise. Collect every existing AI-related plan, roadmap, and initiative list across the organization. Map them on a single page. The conflicts will be immediately visible. Resolve those conflicts first. Then write the unified strategy.

Domain 2: Governance Risks

Governance is where principles become operational. Eight scenarios span this domain, and most organizations have gaps in at least half of them.

Misaligned ethics is a Priority 1 scenario. Reputational damage may occur when AI decisions conflict with organizational cultural and ethical values, leading to poor decisions, negative public perception, and legal repercussions. The control is responsible AI principles: develop AI ethics guidelines, establish an ethics review board, and integrate ethical considerations into the design, development, and deployment of AI systems.

Overconfidence in automation is equally critical. Wasted resources result from unrealistic expectations about AI capabilities, leading to disappointment, wasted investments, and erosion of trust. The control is capability and limitations communication: communicate the limitations and potential risks of AI technologies and avoid overstating capabilities to ensure realistic expectations and informed decision-making.

Governance erosion occurs when AI negatively impacts existing governance mechanisms, reducing control over data processing and increasing breach risk. The control is control integration: update existing governance frameworks to incorporate AI-specific considerations and ensure alignment with established policies and risk management protocols.

Compliance failure carries the most immediate financial consequences. Regulatory penalties and reputational damage result from non-compliance with internal or external AI requirements. The control is compliance audit: regularly test, audit, monitor, and assess AI system compliance with internal policies, external regulations, and ethical guidelines, and report findings to relevant stakeholders.

Vendor lock-in limits flexibility when exit strategies for AI systems are absent. The control is exit planning: include exit strategy considerations in the design and procurement of AI systems, ensuring the ability to migrate to alternative providers.

Three Priority 2 governance scenarios round out this domain. Trust deficit limits innovation when organizations lack trust in AI technologies. The control is an AI framework that documents and communicates limitations and capabilities, provides clear explanations of AI decisions, and establishes processes for independent verification. Communication breakdown results from a lack of common language for AI concepts. The control is AI glossary management: develop and maintain a glossary of AI terms and ensure consistent terminology across the organization. Ownership vacuum leads to unauthorized AI development and security breaches when ownership and operating models are undefined. The control is operating model definition: establish clear roles, responsibilities, and accountabilities for AI initiatives, including appropriate segregation of duties.

Original implementation tip: The governance risk that causes the most silent damage is the ownership vacuum. I worked with a technology company where three separate teams claimed partial ownership of a production ML model. Data science owned the algorithm. Platform engineering owned the infrastructure. The business unit owned the use case. Nobody owned the model in production. When performance degraded, each team assumed another team was monitoring it. The model served degraded predictions for 11 weeks before a customer complaint triggered investigation. Fix this by creating a RACI matrix for every production AI system that names one individual, not a team, as the accountable party for model performance in production. One name. Not a distribution list.

Domain 3: Decision-Making Risks

Two Priority 1 scenarios address how AI risk integrates into enterprise decision-making.

Risk integration gap occurs when organizations fail to integrate risk assessment and controls into the AI framework. Unidentified or unmitigated risks result, along with potential compliance violations and reputational damage. The control is mandatory risk and impact assessments: create and enforce a comprehensive policy mandating systematic identification, evaluation, and management of AI-related risks. Include quantitative model impact assessments with statistical analyses of threat prevalence and potential losses. Integrate these processes with existing framework protocols covering confidentiality, integrity, availability, compliance, contracts, and responsible AI principles.

AI model risk exposure results from outdated quantitative model risk management practices. The control is a quantitative risk model: develop and maintain up-to-date practices for AI model risk management, conduct impact assessments and statistical analyses to evaluate accuracy and reliability, address model metrics and acceptance criteria, and incorporate regular reviews of data, operational practices, and cybersecurity controls.

Original implementation tip: Most organizations attempt to integrate AI risk into their existing risk framework by adding a few AI scenarios to their enterprise risk register. This approach fails because the existing register was designed for risks that behave differently. AI risks are dynamic. A model that was within tolerance last quarter may be outside tolerance this quarter because the underlying data distribution shifted. Instead of simply adding AI rows to your existing register, create a parallel cadence of AI-specific risk reviews that feed into the enterprise register. Monthly AI risk reviews that update quarterly enterprise risk reports. This gives AI risks the attention frequency they require while maintaining integration with enterprise governance.

Domain 4: People Risks

Three scenarios cover the human element of AI risk.

Resource misalignment (Priority 1) occurs when unclear resourcing requirements in the AI strategy lead to staffing inefficiencies. The control is staff planning: define and document human resource requirements, including recruitment, role profiles, training, retention strategy, and third-party involvement, in alignment with the AI strategy and roadmap.

Talent flight (Priority 1) results when poor development and retention of human talent produces AI solutions misaligned with organizational values. The control is talent alignment: establish HR processes to recruit, develop, and retain talent aligned with the AI strategy, including continuous professional development and performance evaluations.

Knowledge deficit (Priority 1) creates ineffective AI operations and poor incident response when IT knowledge is not retained and developed. The control is knowledge continuity: assign and document specific individuals to fulfill business-as-usual roles and sustainment functions. Ensure ongoing knowledge retention through formal knowledge management practices, continuous training, and documentation of key processes and incidents.

Original implementation tip: The knowledge deficit risk is particularly dangerous with AI systems because the knowledge required is specialized and often held by a single individual. I have seen organizations where one data scientist understood the feature engineering pipeline, and when that person left, nobody could retrain the model. The entire production system became fragile overnight. For every critical AI system, maintain a “bus factor” register. For each key knowledge area, list how many people can perform the function. If the number is one, you have a Priority 1 risk that requires immediate cross-training or documentation. Yeah, this sounds obvious. But count how many of your production AI systems depend on a single person’s knowledge. The number will concern you.

Domain 5: Architecture Risks

Architecture risks span seven scenarios across three priority levels.

Unexplainability (Priority 1) is the inability to understand or explain AI decisions due to missing functionality. The control is explainability by design: integrate explainability as a functional requirement in design, build, and testing phases. Ensure explanations are clear and accessible, with documentation of explainability features and traceability of decision-making processes.

Incompatibilities (Priority 1) cause operational issues from integration, scalability, and compatibility problems. The control is compatibility testing: develop testing procedures ensuring the AI model is compatible with the production environment, scalable to meet business needs, and integrated with other systems. Perform thorough compatibility testing across software, hardware, and network environments. Conduct scalability assessments including stress testing. Develop standardized integration protocols covering data formats, API usage, and security requirements.

Misaligned architecture (Priority 2) prevents unified automation when AI architecture is undefined. The control is architecture alignment: define and document an enterprise AI architecture including preferred technologies, design concepts, logging protocols, security controls, and monitoring requirements.

Segregation deficiency (Priority 2) creates security and data integrity losses in cloud or multi-tenant environments. The control is architectural segregation: define IT architecture principles enforcing segregation of AI system components and data from other infrastructure.

Unavailability (Priority 2) disrupts business operations due to insufficient AI system resilience. The control is high availability: define and monitor availability metrics, establish redundancy plans, and test for system reliability.

Ineffective security (Priority 3) results from failing to embed security by design. The control is security by design: incorporate security principles into the development methodology, ensuring all components adhere to established security standards.

License noncompliance (Priority 3) creates legal and financial exposure. The control is license management: establish a license management system ensuring appropriate licenses and timely renewals for all AI system components.

Original implementation tip: Explainability by design is the architecture control most frequently treated as an afterthought. Teams build complex ensemble models or deploy large language models, and only when a regulator or auditor asks “how does this model make decisions” do they realize explainability was never a requirement. Retrofitting explainability onto a deployed model is expensive and sometimes impossible. Add explainability to your definition of done for model development. If the development team cannot demonstrate how the model produces its outputs before deployment, the model does not deploy. This one requirement, enforced consistently, prevents an entire category of compliance and trust risks.

Domain 6: Lifecycle Risks

This is the largest domain, spanning 10 scenarios, because the AI lifecycle from data to deployment contains the most failure points.

Poor hypothesis (Priority 1) produces unreliable outcomes from inadequate governance around hypothesis development. The control is hypothesis testing: establish governance controls requiring approval of hypotheses based on predefined criteria, with regular reviews for ongoing relevance.

Poor algorithms (Priority 1) leads to ineffective performance. The control is algorithmic controls: develop governance policies for algorithm development and maintenance, including validation and testing procedures with regular reviews.

Flawed logics (Priority 1) produces unreliable outputs from inaccurate model parameters. The control is logic validation: establish rigorous logic validation and testing procedures, with approval requirements for any changes to model logic.

Data accuracy failures (Priority 1) cause erroneous outputs. The control is data accuracy verification: develop and enforce data accuracy verification standards including validation, error detection, and correction before and during model use, with continuous monitoring and automated alerts.

Six Priority 2 scenarios complete this domain. Unallocated roles create regulatory and operational failures through unclear data governance responsibilities. The control is data ownership: establish roles including data owners and stewards with regular audits. Data corruption results from unintended interactions between AI and other systems. The control is data integrity monitoring: implement controls to monitor data interactions with automated alerts for corruption incidents. Integration failure occurs from corrupted data inputs or outputs between systems. The control is integration testing: develop strict testing procedures with continuous monitoring for data anomalies. Poor data results from inadequate governance over learning and production data. The control is data governance: enforce quality checks with documented standards and periodic audits. Incomplete inputs lead to incorrect outcomes. The control is data completeness control: implement validation processes with protocols for handling incomplete datasets.

At Priority 3, inaccurate results from models not reflecting underlying parameters are addressed by model validation: comprehensive validation protocols including sensitivity analysis and performance benchmarks.

Original implementation tip: The lifecycle risk that consistently surprises organizations is data accuracy failure during the transition from development to production. The training data has been cleaned, validated, and verified. The model performs beautifully in testing. Then in production, the live data feed introduces formats, edge cases, and quality issues that never appeared in the training set. I watched a fraud detection model go from 94% accuracy in testing to 71% in the first week of production because the live transaction data contained encoding inconsistencies that the training data had been cleaned of. Build a data reconciliation step between your training pipeline and your production pipeline. Compare distributions, formats, and quality metrics between the data the model was trained on and the data it receives in production. Do this before go-live and continuously afterward.

Domain 7: Development Risks

Sixteen scenarios cover the development phase, making it the second largest domain.

At Priority 1, four scenarios demand immediate attention. Design flaw occurs when poor methodology is not consistently applied. The control is development standards: establish and maintain AI development standards integrated with broader development standards. Inaccurate model results from undefined model universe definition. The control is model universe: define and document the AI model universe including data sources, quality, transformations, and assumptions, updated regularly. Variable misalignment causes incorrect results from mistaking correlation for causality. The control is relationship modeling: establish quality controls ensuring relationships between variables are defined correctly, including interdependencies. Overfitting causes loss of reliability when models perform well on training data but poorly on new data. The control is overfitting mitigation: design algorithms for flexibility with documented testing and validation.

Learning bias (Priority 1) deserves special attention. Loss of accuracy and reliability occurs from data bias producing discriminatory outcomes. The control is bias mitigation: implement controls considering sensitivities across ethical, political, ethnic, racial, gender, and cultural groups, with documented evaluation processes and evidence of bias checks.

At Priority 2, five scenarios address operational development risks. To-be inaccuracy results from poor knowledge of desired processes. The control is to-be analysis: maintain documentation of user stories and end-to-end process flows with program sponsor approval. Insufficient segregation occurs when testing environments do not match production. The control is environment segregation: maintain separate development, QA/test, and production environments. Temporal misalignment causes accuracy loss when data time scales conflict. The control is synchronization verification: establish controls ensuring data source alignment with the AI system’s time scale. Data duplication produces inflated insights from processing duplicate data. The control is duplication mitigation: implement file and data validation checks with documentation. AutoML issues create complexity and lack of transparency. The control is AutoML guides: develop guidelines for automated machine learning use with regular complexity assessments and explainability tool integration.

At Priority 3, six scenarios cover remaining development risks. As-is ignorance results from poor knowledge of current processes. The control is as-is analysis: document pre-automation process narratives during the design phase. Undefined controls create vulnerabilities. The control is a control matrix covering all key areas. Control gap occurs when controls are not implemented in the developed solution. The control is control testing to verify processes align with design. Weak traceability compromises logging effectiveness. The control is bot identification with unique identifiers. Improper testing results from insufficient go-live strategy. The control is testing execution with comprehensive documentation. User acceptance deficiency results from inadequate business input. The control is test approvals with documented feedback and sign-off.

Original implementation tip: Variable misalignment, the correlation-versus-causation problem, is the development risk I find most often in production AI systems. And it is rarely caught by automated testing because the model’s statistical metrics look fine. A model might achieve high accuracy by using a variable that correlates with the target in training data but has no causal relationship. When the correlation breaks, which it eventually does, the model fails silently. The most effective countermeasure I have found is a mandatory “causal review” step in the development process where a domain expert, not a data scientist, reviews the feature set and challenges each variable’s causal relationship to the outcome. Data scientists are trained to find patterns. Domain experts are trained to question whether those patterns make sense. You need both perspectives before deployment.

Domain 8: Project Risks

Five scenarios cover project-level risks.

Operational misalignment (Priority 2) results from lacking strategic alignment between AI initiatives and organizational strategy. The control is a business case: establish a strategic alignment framework with formal approval and periodic review by stakeholders.

Problem mismatch (Priority 2) occurs when model design does not match the business problem. The control is iterative development: adopt approaches like Agile for continuous testing and refinement, with prototyping to identify mismatches early.

Management gap (Priority 2) results from poor project management methodology. The control is program management: implement project timelines, resource allocation, stakeholder engagement plans, and continuous alignment with business requirements.

Poor benefits (Priority 3) occurs when benefits management fails to track ROI. The control is impact value: develop a benefits management framework with metrics for short, medium, and long-term tracking.

Assurance deficit (Priority 3) results from lacking independent assurance. The control is assurance: engage an independent function to evaluate AI program setup, with regular reports on quality, costs, benefits, compliance, and internal control.

Original implementation tip: Problem mismatch is the project risk that wastes the most money. I worked with a retail organization that spent eight months building a demand forecasting model to solve what turned out to be a supply chain visibility problem. The model was technically excellent but solved the wrong problem. Their forecast accuracy improved by 15%, but the real issue was that they could not see inventory positions across warehouses in real time. The fix required a dashboard, not a model. Before approving any AI project, require the project team to answer one question in writing: “Why does this problem require machine learning, and what would the non-ML alternative look like?” If they cannot articulate why ML is necessary, there is a good chance a simpler solution would be more effective.

Domain 9: Operations Risks

Nine scenarios cover the operational phase where most AI failures actually manifest.

Performance drift (Priority 2) is the operational risk with the highest real-world impact. Model accuracy degradation from data drift and concept drift occurs when stability checks are insufficient. The control is stability monitoring: implement model stability checks requiring ongoing validation, benchmarking, and performance evaluation to detect drift.

Resource laxity (Priority 2) results from inadequate control over IT resource usage given AI’s unpredictable demands. The control is project monitoring: implement controls to monitor IT resource demands more closely than other systems.

Error oversight (Priority 2) leads to unauthorized changes and incidents from undetected errors. The control is incident management: establish a consistent approach with clear procedures, timely resolution, and integration with regular incident management.

Undetected error (Priority 2) causes delayed resolution from lacking procedures. The control is error resolution: perform timely exception processing with issue and performance monitoring.

Unsupported jobs (Priority 2) results from insufficient job monitoring. The control is job monitoring: monitor system jobs and interfaces ensuring completeness and timeliness.

Capacity issues (Priority 2) arise when availability and capacity management cannot meet evolving demand. The control is capacity management: implement availability and capacity management with scalability embedded in design.

At Priority 3, shadow AI is the scenario most organizations underestimate. Inability to ensure AI aligns with strategy and risk appetite occurs when the organization lacks an inventory of all AI solutions. The control is AI inventory: maintain a complete, up-to-date inventory of all AI platforms, solutions, and use cases, including dependencies and ownership.

IP loss (Priority 3) occurs when AI system intellectual property held by third parties is at risk. The control is IP protection: establish a repository of relevant IP, accessible in-house, secured with regular backups.

AI component blindness (Priority 3) results from lacking understanding of IT components and relationships. The control is configuration management: establish a configuration management database fed through change management.

Original implementation tip: Shadow AI is growing faster than most governance teams realize. Every time an employee uses ChatGPT to draft a customer response, builds a quick predictive model in a Jupyter notebook, or connects a third-party AI tool to company data through a browser extension, they create shadow AI. I conducted a shadow AI audit at a financial services firm last year. The governance team believed they had 12 AI systems in production. We found 47 AI tools and models being used across the organization, most without any risk assessment, data governance, or access controls. The 35 unknown systems included four that processed customer PII. Start your shadow AI inventory not by asking teams to self-report, which underestimates the problem, but by auditing network traffic, SaaS subscriptions, cloud resource usage, and API calls for AI-related activity.

Domain 10: Monitoring Risks

Four scenarios address the monitoring function that keeps deployed AI systems safe.

Outcome blindness (Priority 1) occurs when AI system behavior is not monitored against business and ethical requirements. The control is outcome monitoring: implement regular review of AI system outcomes using data analytics to ensure performance aligns with requirements. Maintain audit trails and ensure controls operate at the same pace as monitored activities.

Monitoring ineffectiveness (Priority 1) reduces operational effectiveness from inadequate monitoring. The control is operational monitoring: develop a real-time monitoring and alerting framework to detect anomalies, establish KPIs and KRIs as the basis for effective monitoring, and trigger alerts followed by documented follow-ups.

Undetected issues (Priority 1) cause financial losses and compliance fines from lacking post-deployment monitoring. The control is post-deployment monitoring: develop a monitoring framework defining specific metrics, thresholds, and alerts. Implement automated tools for continuous real-time tracking. Define key performance indicators and review them regularly against business objectives.

Control override (Priority 2) leads to financial loss when automated stop/loss controls fail. The control is automated stop/loss: design controls to halt unintended AI behavior with an override process for exceptions, assessing exceptions against risk appetite and business impact.

Original implementation tip: The monitoring risk that catches organizations off guard is the gap between monitoring cadence and AI decision speed. I worked with an organization that monitored their AI system’s output quality weekly. The system made 50,000 decisions per day. By the time they detected a quality degradation in their weekly review, the system had already made 350,000 decisions at reduced quality. Match your monitoring frequency to your decision frequency. If your model makes real-time decisions, you need real-time monitoring. If your model runs daily batch predictions, daily monitoring may suffice. But weekly monitoring for a real-time system is a control that exists on paper but provides no actual protection.

Domain 11: Security Risks

Seven scenarios span security from Priority 1 through Priority 3.

Lack of auditability (Priority 1) prevents validation of AI outcomes. The control is auditability: securely store and ensure timely retrieval of data and algorithms, comply with data privacy regulations, prevent data context loss, and apply the vault principle.

Unauthorized access (Priority 2) leads to inappropriate changes to AI learning and processing data. The control is data access: securely configure AI input datasets to prevent unauthorized changes with completeness and accuracy checks.

At Priority 3, five scenarios address specific security concerns. Security breach results from inconsistent security management. The control is cyber security: apply a consistent approach integrated with regular security processes, aligned with ISO 27001. Malware attack affects AI environment integrity. The control is malware protection: implement protection systems and monitor patches, protecting self-learning components against malicious attacks. Data breach occurs from insecure handling of temporary files. The control is encryption: encrypt code, data storage, and network communications. Vulnerability blindness results from undetected security weaknesses. The control is vulnerability testing: conduct periodic penetration tests and red-team reviews.

Original implementation tip: The security risk unique to AI that most security teams miss is the attack surface created by the model itself. Traditional security teams protect the infrastructure around the model, the servers, networks, APIs, and databases. But the model is an attack surface too. An adversary who can query a production model thousands of times can extract information about the training data through model inversion attacks. They can find decision boundaries through systematic probing. They can manipulate outputs through carefully crafted inputs. Your security testing must include model-specific attack scenarios, not just infrastructure penetration testing. If your red team does not include someone who understands adversarial machine learning, your testing has a blind spot.

Domain 12: Access Control Risks

Twelve scenarios cover access management for both human users and automated bots. All are Priority 3, but their aggregate effect is significant.

These scenarios cover compromised bot accounts, compromised user accounts, excessive bot access, excessive user access, inadequate account provisioning, inadequate access revocation, undetected bot access, undetected user access, excessive privileged access, segregation of duties conflicts, weak authentication, and unauthorized third-party access.

The controls follow a consistent pattern: bot control and user control for accountability, bot access authorization and user access authorization for least-privilege enforcement, account provisioning for formal approval processes, access revocation for timely deprovisioning, bot access review and user access review for periodic validation, privileged access authorization for restricting powerful accounts, access segregation for preventing conflicts, authentication for strong credential management, and third-party control for extending security standards to external users.

Original implementation tip: The access control risk specific to AI that most organizations handle poorly is bot account management. When a bot, an automated process, accesses systems, it typically uses a service account. These service accounts often accumulate privileges over time as the bot’s functions expand, but nobody conducts the same periodic access reviews for bot accounts that they do for human accounts. I audited one organization where a bot account for a data preprocessing pipeline had accumulated database administrator privileges, access to the production model repository, and write access to the training data store. Nobody had reviewed the bot’s access in 18 months. Treat bot accounts with the same access governance rigor as human accounts. Include them in quarterly access reviews. Apply least-privilege principles. Document and approve every privilege.

Domain 13: Change Management Risks

Seven scenarios address how changes to AI systems introduce risk.

IT impact assessment (Priority 1) is the most critical. Disruptions to other IT services may occur from AI system changes with insufficient impact analysis. The control is IT impact assessment: mandate thorough impact analysis for all AI changes, focusing on effects on related IT services, requiring integration testing with documented results.

Inadequate ongoing testing (Priority 1) causes missed defects. The control is testing protocol: establish comprehensive testing protocols for ongoing AI validation with pre- and post-implementation tests executed by independent teams.

At Priority 2, undetected errors result from inadequate automated monitoring. The control is automated error monitoring: deploy tools that continuously validate the AI system after changes, detecting anomalies in real time.

Five Priority 3 scenarios cover unauthorized changes, untracked modifications, poor change control, and insufficient validations. Controls include formal change management processes, modification logging, change control procedures, and validation procedures requiring pre-deployment tests across functional, security, and performance criteria.

Original implementation tip: The change management risk specific to AI that organizations consistently underestimate is the cascading impact of retraining. When a model is retrained on new data, the outputs change. Sometimes subtly, sometimes dramatically. If downstream systems or business processes depend on the model’s output characteristics, for example expected score ranges, output distributions, or decision thresholds, retraining can break those dependencies without triggering any traditional change management alerts. Treat model retraining as a change that requires the same impact assessment, testing, and approval as a code deployment. Because functionally, it is one.

Domain 14: Third-Party and Business Continuity Risks

The final domain covers four third-party scenarios and five business continuity scenarios.

For third parties, black box solution (Priority 2) creates business disruption when the organization cannot understand the AI system’s logic. The control is contract review: define intellectual property ownership, include escrow agreements, ensure right to audit, and outline roles and responsibilities. Third-party default (Priority 3) exposes the organization to lower control maturity. The control is due diligence: subject third parties to at least the same level of control as internal operations. Third-party dependency (Priority 3) creates concentration risk. The control is third-party segmentation: identify and categorize suppliers by criticality with contingency plans. Shadow third-party (Priority 3) results from lacking an updated vendor inventory. The control is third-party management: develop a comprehensive inventory integrated into risk and continuity planning.

For business continuity, inability to recover (Priority 1) causes prolonged disruptions when rollback mechanisms are absent. The control is roll-back: establish mechanisms to identify and recover the last known good AI state, with processes, algorithms, and cleansed data available for rapid retraining. Ineffective backups (Priority 1) results from inability to restore AI services. The control is backup restoration: implement appropriate backup and snapshot procedures, including frequent snapshots of learning data, with ability to roll back completely.

Ineffective fallback (Priority 2) results from lacking alternative processing facilities. The control is fallback facility: establish alternative processing capabilities with regular risk assessments. Fragility (Priority 3) and ineffective response (Priority 3) address business continuity planning and testing, with controls for continuity planning aligned with ISO 22301 and continuity testing through regular BCP simulations.

Original implementation tip: The business continuity risk most specific to AI is the inability to recover the model’s learned state. Traditional systems can be restored from backups because their logic is deterministic. An AI model’s “logic” is its trained weights, which are the product of specific training data processed in a specific sequence with specific hyperparameters. If you lose the trained model and do not have the exact training data, preprocessing pipeline, and training configuration documented and backed up, you cannot recreate it. I have seen an organization lose a production model to a storage failure and spend six weeks recreating it because they had backed up the model artifacts but not the training pipeline configuration. Back up everything: the model, the training data, the preprocessing code, the feature engineering pipeline, the hyperparameter configuration, and the training environment specification. Test restoration by actually rebuilding the model from backups at least annually.

Implementation Tips

These four principles apply across all 14 domains and 100 scenarios.

First, prioritize by actual exposure, not by perceived sophistication. The scenarios rated Priority 1 in this taxonomy are not necessarily the most technically interesting. They are the ones that cause the most organizational damage when they materialize. Strategy deficiency, ownership vacuum, and compliance failure cause more real-world harm than adversarial machine learning attacks. Fund controls for Priority 1 scenarios before you invest in exotic defenses against lower-probability technical attacks.

Original implementation tip: When presenting this taxonomy to leadership, resist the temptation to lead with the technically impressive scenarios like adversarial attacks or model inversion. Lead with the governance and strategy scenarios that connect to business outcomes leadership already cares about. “We lack a defined owner for our production AI models” resonates more with a board than “we are vulnerable to model extraction attacks.” Start with the risks they can feel, then build toward the ones they need to understand.

Second, map controls to your existing control framework. This taxonomy aligns to COBIT 2019 objectives across four domains: Evaluate, Direct and Monitor (EDM), Align, Plan and Organize (APO), Build, Acquire and Implement (BAI), and Deliver, Service and Support (DSS), plus Monitor, Evaluate and Assess (MEA). If your organization uses a different framework, map these controls to your existing structure. The worst outcome is creating a parallel AI control framework that nobody integrates into operational governance.

Original implementation tip: When mapping these controls to your existing framework, do not create 100 new control activities. Many of these AI controls are extensions of controls you already have. Data access controls for AI systems should be managed through the same access management processes you use for other systems. Change management for AI should follow the same change management framework with AI-specific additions. Identify which controls are genuinely new (explainability by design, bias mitigation, stability monitoring) and which are extensions of existing controls. New controls need new processes. Extensions need updated procedures. The distinction matters for implementation cost and adoption speed.

Third, document decisions and rationale, not just outcomes. For every control in this taxonomy, maintain evidence that demonstrates not just that the control exists, but why specific decisions were made. When an auditor or regulator asks why you accepted a particular residual risk, “because we assessed it and decided it was within tolerance” is insufficient. They need to see the assessment, the alternatives considered, and the governance approval.

Original implementation tip: Create a standard decision record template with five fields: the decision, the alternatives considered, the rationale for selection, the assumptions that must remain valid, and the conditions that would trigger reassessment. Use this template for every Priority 1 and Priority 2 control decision. It takes five minutes per decision and saves hours of reconstruction during audits. I have seen organizations that adopted this practice clear regulatory examinations in half the time of those that relied on informal documentation.

Fourth, reassess at a cadence that matches your risk velocity. AI risks change faster than traditional IT risks. Models degrade, data drifts, new attack techniques emerge, and regulations evolve. A taxonomy that is reviewed annually is a taxonomy that is wrong for 11 months of the year. Review Priority 1 controls quarterly, Priority 2 semi-annually, and Priority 3 annually at minimum. Update the taxonomy itself whenever a new risk scenario materializes that is not covered.

References and Standards

This taxonomy draws from and aligns with the following authoritative frameworks.

ISO/IEC 27005:2022 for the information security risk management process structure.

ISO/IEC 23894:2023 for AI-specific risk management guidance.

ISO/IEC 42001:2023 for AI management system requirements covering governance, ethics, and accountability.

COBIT 2019 for IT governance and management objectives, providing the control mapping framework used throughout this taxonomy.

NIST AI RMF (AI 100-1) for the AI risk management lifecycle framework.

EU AI Act (Regulation 2024/1689) for risk-based regulatory requirements governing AI systems in EU markets.

ISO 22301 for business continuity management systems referenced in the continuity domain.

ISO/IEC 27001 for information security management systems referenced in the security domain.

MITRE ATLAS for the adversarial threat landscape specific to AI and machine learning systems.

FAIR (Factor Analysis of Information Risk) for quantitative risk analysis methodology when assessing the scenarios in this taxonomy.

Making This Taxonomy Work

Organizations that treat this taxonomy as a reference document to satisfy an audit requirement will miss its value entirely. They will have a comprehensive list of 100 scenarios that nobody operationalizes, controls that exist in policy but not in practice, and a false sense of security that evaporates at the first real incident. The taxonomy becomes shelfware, and the organization remains exposed to the same risks it cataloged so carefully.

Organizations that treat this taxonomy as a living operational tool will use it differently. They will map their existing AI systems against these 100 scenarios to identify gaps. They will prioritize control implementation based on the priority ratings and their own risk appetite. They will assign named owners to each applicable control. They will review and update the taxonomy as new AI capabilities are deployed, new threats emerge, and new regulations take effect. Their risk conversations will be specific, traceable, and grounded in concrete scenarios rather than abstract categories.

A taxonomy that names 100 things that can go wrong is only useful if it drives 100 decisions about what to do right.

Which of these 14 domains has the biggest gaps in your organization right now? If you are honest with yourself, I suspect the answer is not the technical domains. It is strategy, governance, or people. Start there.

Ai-Risks |

New Book AI Risk Quantification: A Practical Roadmap for Chief AI Officers

A practitioner framework for turning ambiguous AI exposure into decision-grade evidence.

What Every Chapter Actually Delivers

Part 1. Foundations: Risk Management as Decision Support

Chapter 1. The Expensive Risk Theater, page 1

Chapter 2. Assess the Plan, Not the Danger List, page 25

Chapter 3. From Risk Registers to Risk-Adjusted Plans, page 42

Part 2. Core Operating Framework: The Quantitative Engine for Decisions

Chapter 4. Model the Failure, Protect the Objective, page 65

Chapter 5. Measure What Seems Unmeasurable, page 99

Chapter 6. Prioritizing Against Capacity, Not Intuition, page 127

Chapter 7. Choosing the Risk Response That Pays, page 151

Chapter 8. Monitor What Matters, page 179

Chapter 9. Updating Risk Before It Updates You, page 198

Part 3. Domain Applications: One Framework, Sharp Edges for Each Risk Type

Chapter 10. AI Risks: Assess AI Before It Acts, page 222

Chapter 11. IT Risks: Quantify Cyber Risk Exposure, page 273

Chapter 12. Compliance Risks: Price Obligations Before Commitment, page 294

Chapter 13. Project Risks: Know the True Odds of Delivery, page 322

Chapter 14. Third-Party Risks: Assess Dependency Before It Fails, page 346

Chapter 15. Financial Risks: Measure What the Spreadsheet Hides, page 371

Chapter 16. Strategic Risks: The Bets That Shape Your Future, page 412

Chapter 17. Continuity Risks: The Survival of Critical Services, page 443

Chapter 18. Sustainability Risks: The Transition Penalty, page 487

Chapter 19. People Risks: Prevent Behavioral Failures, page 523

Part 4. Advanced Practice: Deeper Certainty for the Numbers That Matter Most

Chapter 20. Build the Probability Engine, page 565

Chapter 21. Aggregate Risk Correctly, page 615

Chapter 22. Simulate Your Risk Before It Hits, page 649

Chapter 23. The Emerging Risk Modelling Approach, page 708

Chapter 24. Predictive Risk Models: Machine Learning, page 727

Chapter 25. Build Agentic Risk Controls, page 761

Chapter 26. The Decision-Ready Blueprint, page 778

The prEN 18228 Problem: Why Your AI Risk Assessment Will Fail the First Real Test

The Standard Brings Structure. It Does Not Solve the Decision Problem.

The Risk Definition Works for Products, but AI Behaves Differently.

The Terminology You Need to Understand Before You Start

Terms Relating to the EU AI Act

Terms Related to the Risk Management System

Terms Relating to Testing

Terms Related to Users and Affected Persons

Terms Related to Risk

How the Standard Actually Runs Risk Management

A Continuous Process, Not a One-Time Assessment

Management Owns the Process, Not Just the Outcome

Risk Acceptability Is a Policy Decision, not a Technical Detail

Competence Is a Collective Requirement

Defining Risk Acceptability Criteria

How Criteria Are Established and Updated

Looking at the Overall Residual Risk

Planning the Work

The Risk Management File as the Backbone

Risk Management Process and AI System Life Cycle

General Requirements for AI Risk Analysis

Risk Identification from the Intended Purpose

Risk Identification: Reasonably Foreseeable Misuse

Risk Identification: AI System Characteristics Related to Risks

Risk Identification: Hazards, Risk Scenarios, and Hazardous Situations

Risk Estimation

Risk Evaluation Turns Analysis into Decisions

Testing Is Evidence, Not Validation Theater

Test Plan

Real-World Conditions Testing

Test Monitoring and Test Reporting

Risk Control Follows a Clear Hierarchy

Implementation and Verification of Risk Control Measures

Residual Risk and When to Stop

Completeness of Risk Control

Looking at the AI System and Residual Risk as a Whole

Reviewing the Process, Not Just the Outcome

Learning from the Pre-Market and Post-Market Activities

Information Collection

Information Review

Actions to Take

Understanding How AI Risks Unfold in Practice

Example 1: Skin Cancer Detection App

What the system does

What causes the hazard

What the hazard is