← No Hallucinations

Governance · October 19, 2026

Beyond High-Medium-Low: Scoring AI Risk Defensibly

By David Monahan · Data Defenders, LLC · a worked example, a five-step rubric, and the method behind both

Effective governance depends on risk tiering, but most organizations stop at "high/medium/low" and general "vibes" rather than defensible scoring. The previous post established that a Governed program carries risk tiers on every inventory entry and uses those tiers to gate deployment. This post walks through what makes a tier defensible: the dimensions that feed the score, a worked example applying them to a real AI use case, the five steps to build your own rubric, and when a dollar-based quantitative method earns a place alongside it.

Why AI risk doesn't fit the cyber risk playbook

Traditional cyber risk assumes a human attacker exploiting a vulnerability to reach a bounded system. AI risk breaks four of those assumptions at once. The attack surface is novel — prompt injection, model inversion, adversarial examples, and jailbreaking aren't in the ISO 27001 Annex A controls. The scale is unlike traditional IT: an algorithmic hiring decision runs across 50,000 candidates in minutes, so a bias defect ships at that scale. The model is opaque in ways no database table ever was. And the model is usually controlled by a third party, so its behavior can change in a vendor update with no change on your side. Impact and likelihood still end the scoring; what feeds them for AI is different.

The seven dimensions of AI risk

A defensible AI risk score is a composite of dimensions you can describe, score, and justify. The chart below shows each at Low, Medium, High, and Critical with observable criteria. Assign the dimension's score (1 through 4) based on how many of the band's criteria apply — this is the starting rubric for the five-step method and the worked example that follow.

Click the chart to open it full size.

Build your own rubric: five concrete steps

1. Start from a published dimension rubric — don't invent. Use the seven dimensions above or an alternative from NIST AI RMF or ISO/IEC 42001. Save the dimension names and the observable Low/Medium/High/Critical criteria as your organization's starting template. Inventing dimensions from scratch is how organizations end up with a scoring sheet nobody can defend to an auditor.

2. Assign dimension weights per deployment context. Not every dimension matters equally for every system. A chatbot weights model behavior higher; a hiring system weights decision influence and regulatory classification higher. Document the weighting rationale in writing — auditors treat undocumented weighting as a missing control.

3. Set tier thresholds. Define numeric bands that map a composite score to a tier. A common pattern on the 1–4 scale is Low (1.00–1.74), Medium (1.75–2.49), High (2.50–3.24), Critical (3.25+). Tune the bands to the organization's risk tolerance and get them approved by the executive champion from blog-post-4.

4. Set single-dimension hard stops alongside the composite. Any single dimension scored Critical (4) blocks deployment regardless of the composite average. This prevents a strong overall score from papering over one critical weak point — as the applicant screener example below demonstrates, where three dimensions each hit Critical on their own.

5. Map each tier to a deployment gate. Low: baseline monitoring, standard release process. Medium: named governance lead sign-off, tightened monitoring. High: human-in-the-loop controls, bias audit, legal review before deployment. Critical: blocked from production until the triggering dimension drops below Critical; executive sign-off required to deploy with compensating controls. The tier is only useful if it triggers an actual control.

Scoring one AI use case: an applicant screener

With the rubric in hand, take a mid-market company's HR team using an AI applicant screening tool. The tool uses a third-party LLM with custom prompts, integrates with the applicant tracking system, and processes about 50,000 applications a year across 100 open roles. The LLM ranks each applicant and surfaces the top-scoring candidates for human review. Here is what scoring that system looks like, dimension by dimension:

• Model behavior: 3 (High). Opaque, generative, documented drift — three High criteria apply.
• Data sensitivity & provenance: 4 (Critical). PII plus inferable protected-class attributes at scale; vendor provenance unverifiable at the model layer — Restricted plus regulated-data territory.
• Decision influence: 4 (Critical). Automates the first-pass filter — rejected applicants never reach a human.
• Scale of use: 3 (High). 50,000 decisions/year, cross-functional.
• Autonomy level: 3 (High). Agentic ranking + filtering within one scope, supervised review.
• Regulatory classification: 4 (Critical). High-risk under the EU AI Act (Annex III employment), EEOC enforcement priorities, the Mobley v. Workday precedent, the Colorado AI Act, and NYC Local Law 144 bias-audit requirements. Risk management, bias testing, and human oversight required in multiple jurisdictions.
• Reversibility: 3 (High). Rejected applicants rarely know or re-apply; career effects compound downstream.

Composite: average 3.4 → Critical tier. Three dimensions scored Critical (4) — data sensitivity, decision influence, regulatory classification — each trigger single-dimension hard stops.

Controls that triggers: mandatory human review of all rejections, quarterly bias audit against protected classes, vendor conformity assessment under EU AI Act Article 43, legal sign-off before any prompt or model update, and an EEOC-ready evidence package per hiring season. None of these would be required by a "high/medium/low" vibe score; they are required because the triggering dimensions were documented and scored.

Where the frameworks and regulations fit

The rubric above is framework-agnostic by design. NIST AI RMF (MAP function: Context, Categorization, Capabilities, Risks and Benefits, Impacts and Risk Tolerance) and ISO/IEC 42001 (AI management system and impact assessment) define what to document and how to structure risk acceptance. The jurisdictional AI regulations that apply (EU AI Act, Colorado AI Act, NYC Local Law 144, South Korea AI Basic Act, UK AI framework, and the growing list of others) define what tiers trigger what obligations in your market. OWASP Top 10 for LLM Applications (2026) adds the threat-side view, drawn from 6,639 recorded AI incidents. All three live on top of the rubric; they do not replace it.

Qualitative or quantitative — pick the one that matches the decision

The rubric method above is qualitative scoring. It is the right default for operational prioritization, deployment gating, and framework reporting. Most organizations need one more method on top of it for the conversations qualitative scores do not answer well.

Qualitative composite (the method above) — use for Monday-morning decisions: which systems get reviewed first, what controls get triggered, what gets blocked. Fast to apply, easy to re-score on a trigger (model update, use-case change, vendor change), easy for a cross-functional governance committee to agree on.

Quantitative — FAIR-AIR — use when the CFO, the board, or a cyber insurance underwriter needs dollar exposure. The FAIR Institute's FAIR-AIR extension translates AI risk into three loss categories (regulatory and legal, operational and productivity, intellectual property and competitive erosion) and tiers third-party AI into mission-critical, business-integrated, and productivity. Where qualitative tells you which AI to look at first, quantitative tells leadership how many dollars of exposure the organization is carrying — the language that gets AI governance funded.

The failure modes to avoid

Risk theater. The organization has a scoring spreadsheet, but the score doesn't gate anything. The inventory shows "High" on systems already in production without the controls a High tier requires. That is the posture a regulator finds first.

Self-scoring by the owner. The team that wants to deploy the AI is the team scoring it. Scores are predictably low. Fix: a second-party reviewer, with disagreements escalated to the governance committee.

One-and-done scoring. The score is set at launch and never revisited. A model update, a use-case expansion, or a vendor change can lift a Medium into Critical overnight. Set re-scoring triggers in writing.

The takeaway

High, medium, and low are tier labels, not a method. A defensible score is a composite of seven observable dimensions, each scored 1–4, weighted for context, gated by both tier thresholds and single-dimension hard stops, and mapped to the controls each tier triggers. Build it in five steps, apply it one AI system at a time, and layer FAIR-AIR on top when leadership needs dollars behind the tier.

Next in the series: what audit-ready evidence collection looks like in practice — the artifacts that turn a governance program into a defensible one.

Sources: NIST AI RMF 1.0 (AI 100-1), MAP function and sub-categories; ISO/IEC 42001:2023 AI Management System and AI impact assessment; EU AI Act (Regulation 2024/1689), four-tier risk classification, enforcement timeline through August 2, 2026; OWASP Top 10 for LLM Applications, 2026 edition (6,639 recorded incidents, 75% practitioner consensus); FAIR Institute, FAIR-AIR extension for third-party AI risk; Openlayer AI risk scoring methodology (2026); Colorado AI Act (SB24-205, SB26-189); NYC Local Law 144 bias-audit requirements for AI hiring tools; Mobley v. Workday, Inc., N.D. Cal. 4:23-cv-00770; EEOC algorithmic hiring enforcement priorities.

← Back to No Hallucinations

Talk to Us Directly

Free 25-minute consultation. Bring your questions.