AI Vendor Risk Assessment Agent

Overview

As teams adopt generative AI and specialized AI vendors, evaluating third-party AI risk becomes a bottleneck. Assessing a single vendor’s AI capabilities, training data handling, sub-processors, security certifications, and model governance used to take:

• Days of back-and-forth between procurement, security, and the requesting team.
• Manual searches across vendor trust centers, Data Processing Addenda (DPAs), SOC 2 reports, and internal Jira security review tickets.
• Risk scores that depended on who was doing the scoring.

Built with Rovo Architect, the AI Vendor Risk Assessment Agent works as a compliance analyst. Given the vendor name, product link, intended use case, and data exposure, it investigates internal and external sources, scores the tool against MX’s AI Risk Scoring Rubric, and formats a publish-ready Confluence evaluation in minutes.

Operating principles

1. Research first, ask second. The user is not treated like a search engine. The agent asks only for internal context (the intended MX use case and data classification). Vendor documentation, DPAs, certifications, and model cards are investigated by the agent.
2. Conservative by default. If evidence is ambiguous, the agent assigns the higher-risk score. If evidence cannot be found after exhausting internal and external channels, it assigns Insufficient Evidence — Assumed High so the gap is visible.
3. Every score cites a source. Each scored indicator names a specific document, URL, Jira ticket, or Confluence page, with a one-sentence justification.
4. Present findings, don’t make the procurement decision. The agent calculates risk scores, identifies the tier, and lists the controls policy requires. Humans keep final approval.
5. Human gate before publishing. The agent shows a complete in-chat preview and publishes to Confluence only on explicit confirmation.

The five-cluster risk scoring framework

The agent evaluates third-party AI tools across five weighted clusters and produces a composite score mapped to required operational controls:

┌─────────────────────────────────────────────────────────────┐
│              AI VENDOR RISK SCORING ENGINE                  │
├──────────────────────────────┬──────────────────────────────┤
│ 1. Inherent System Risk      │ Interpretability, Generative │
│    (Range: 3 – 21)           │ Capacity, Agentic Autonomy   │
├──────────────────────────────┼──────────────────────────────┤
│ 2. Performance & Reliability │ Benchmarks, Bias, Fairness,  │
│    (Range: 1 – 13)           │ Robustness, Explainability   │
├──────────────────────────────┼──────────────────────────────┤
│ 3. Security & Compliance     │ Surface, Posture, 4th Party, │
│    (Range: 1 – 11)           │ Adversarial Risk, Certs      │
├──────────────────────────────┼──────────────────────────────┤
│ 4. Data Risk                 │ Training on MX Data, Memory, │
│    (Range: 0 – 16)           │ Filtering, PII Classification│
├──────────────────────────────┼──────────────────────────────┤
│ 5. Operational Modifiers     │ Human Oversight Level,       │
│    (Range: -7 to +5)         │ Decision-Critical Use Impact │
└──────────────────────────────┴──────────────────────────────┘
                               │ Sum
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  GOVERNED RISK TIERS                        │
│  • Low (-2 to 20): Internal docs & basic model card         │
│  • Medium (21 to 45): Human-in-the-loop review in SOP/tool  │
│  • High (46 to 55): Periodic audit & escalation pathways    │
│  • Critical (56 to 66): Exec review, red-teaming, kill-switch│
└─────────────────────────────────────────────────────────────┘

Risk indicators

Cluster	Indicator	Evaluated risk states
Inherent System Risk	Interpretability Level	Transparent (1) · Partially Interpretable (4) · Opaque/Black Box (7)
	Generative Capacity	None (1) · Template-Based (4) · Context-Limited (4) · Open-Ended (7)
	Agentic Profile	None (1) · Limited Autonomy (4) · Fully Autonomous (7)
Performance & Reliability	Performance Metrics	High (0) · Medium (1) · Low (2) · Not Available (3)
	Bias Assessment	Complete (1) · Partial (2) · None (3)
	Robustness & Fairness	Formally Tested (0) · None (2)
	Explainability Method	SHAP/LIME/Other (0) · None (2)
Security & Compliance	Access Surface	Internal (1) · External API (2) · Public (3)
	Security Posture	Hardened (0) · Baseline (1) · Unverified (2)
	4th Party AI Risk	None (0) · Moderate (1) · High (2)
	Certifications	NIST / ISO 27001 / SOC 2 (0) · Other (1) · None (2)
Data Risk	MX Data in LLM Training	Retained for Model Training (2) · Zero Data Retention (0)
	Input/Output Filtering	Active Guardrails (0) · None / Unverified (2)
	Data Classification	Public (-3) · Internal/NPI (0) · PII/Financial (3)
	Persistent Memory	Session Only (0) · Cross-Session Retention (2)
Operational Modifiers	Human Oversight Level	Strict Human-in-the-Loop (-7) · Partial (-4) · None (0)
	Decision Criticality	Low Workflow Impact (0) · Influences Decisions (+3) · Critical (+5)

Investigation and publishing workflow

[Phase 1: Minimal Intake]
  │ • Vendor/Tool Name & URL
  │ • MX Specific Use Case
  │ • Data Classification & Touchpoints
  ▼
[Phase 2: Multi-Channel Autonomous Research]
  │ • Internal: Confluence spaces, Jira security & procurement tickets
  │ • Vendor: DPAs, Trust Portals, Sub-processor lists, Model Cards
  │ • Independent: SOC 2, ISO certifications, CVE / breach history
  ▼
[Phase 3: Conservative-by-Default Rubric Scoring]
  │ • Map evidence to indicators with mandatory citations
  │ • Auto-assign 'Insufficient Evidence — Assumed High' on gaps
  │ • Calculate composite score and map to Risk Tier & Controls
  ▼
[Phase 4: Human Gate & Confluence Page Publishing]
  │ • Present complete formatted draft in chat
  │ • Await explicit human confirmation & target space/parent
  │ • Publish structured Markdown page via Confluence API

Architecture and safety controls

The agent is built so a reviewer gets a rigorous assessment without auditing every underlying document by hand.

1. Minimal intake, autonomous depth

Human input is limited to internal context that cannot be inferred from outside:

• Product target: vendor name, tool name, and the specific product URL.
• Workflow context: the MX team, workflow, and operational touchpoint.
• Data exposure: integration type (API, browser extension, SSO, native client), the data fields passed, and their classification (Public, Internal/NPI, PII, Financial).

Everything after that, including retrieving DPAs, SOC 2 compliance, sub-processor disclosures, and training data policies, is done by the agent across internal spaces and external trust centers.

2. Conservative scoring and evidence attribution

To protect against model hallucination and vendor marketing claims:

• Mandatory sourcing: every indicator score requires an auditable citation and a one-sentence factual justification.
• Conservative ambiguity resolution: if documentation is unclear or conflicting, the agent defaults to the higher-risk score.
• Explicit gap flagging: if evidence cannot be found after exhausting all channels, the indicator is marked Insufficient Evidence — Assumed High, so data gaps show on the final scorecard.

3. Evaluation is separate from procurement

The agent is an analyst, not a decision-maker. It calculates composite scores, maps them to required governance controls (executive review, kill-switch readiness, mandatory human verification), and generates the Confluence report. The procurement decision stays with people.