AI Vendor Risk Assessment Agent

Overview

As enterprise teams adopt generative AI and specialized AI vendors, evaluating third-party AI risk becomes a severe operational bottleneck. Evaluating a single vendor’s AI capabilities, training data handling, sub-processors, security certifications, and model governance typically required:

• Days of back-and-forth between procurement, security, and the requesting team.
• Manual searches across vendor Trust Centers, Data Processing Addenda (DPAs), SOC 2 reports, and internal Jira security review tickets.
• Subjective risk scoring prone to inconsistent human interpretation.

Built with Rovo Architect, the AI Vendor Risk Assessment Agent acts as an autonomous compliance analyst. Given only the vendor name, product link, intended use case, and data exposure profile, the agent executes an exhaustive internal and external investigation, scores the tool against MX’s formal AI Risk Scoring Rubric, and formats a complete, publish-ready Confluence evaluation in minutes.

---

Core Operating Principles

The agent operates under strict governance rules to ensure enterprise-grade reliability:

1. Research First, Ask Second: The user is never treated like a search engine. The agent asks only for internal context (the intended MX use case and data classification); all vendor documentation, DPAs, certifications, and model cards are investigated autonomously.
2. Conservative by Default: If evidence is ambiguous, the agent assigns the higher-risk score. If evidence cannot be found after exhausting internal and external channels, it assigns Insufficient Evidence — Assumed High to highlight the gap.
3. 100% Evidence Attribution: Every scored indicator must cite a specific source (document name, URL, Jira ticket, or Confluence page) with a one-sentence justification.
4. Present Findings, Don’t Make Procurement Decisions: The agent calculates risk scores, identifies risk tiers, and specifies required controls per policy. Humans retain final approval authority.
5. Human Gate Before Publishing: The agent generates a complete in-chat preview for review and only publishes to Confluence upon explicit human confirmation.

---

The 5-Cluster Risk Scoring Framework

The agent evaluates third-party AI tools across five weighted clusters, producing an objective composite score mapped directly to required operational controls:

┌─────────────────────────────────────────────────────────────┐
│              AI VENDOR RISK SCORING ENGINE                  │
├──────────────────────────────┬──────────────────────────────┤
│ 1. Inherent System Risk      │ Interpretability, Generative │
│    (Range: 3 – 21)           │ Capacity, Agentic Autonomy   │
├──────────────────────────────┼──────────────────────────────┤
│ 2. Performance & Reliability │ Benchmarks, Bias, Fairness,  │
│    (Range: 1 – 13)           │ Robustness, Explainability   │
├──────────────────────────────┼──────────────────────────────┤
│ 3. Security & Compliance     │ Surface, Posture, 4th Party, │
│    (Range: 1 – 11)           │ Adversarial Risk, Certs      │
├──────────────────────────────┼──────────────────────────────┤
│ 4. Data Risk                 │ Training on MX Data, Memory, │
│    (Range: 0 – 16)           │ Filtering, PII Classification│
├──────────────────────────────┼──────────────────────────────┤
│ 5. Operational Modifiers     │ Human Oversight Level,       │
│    (Range: -7 to +5)         │ Decision-Critical Use Impact │
└──────────────────────────────┴──────────────────────────────┘
                               │ Sum
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  GOVERNED RISK TIERS                        │
│  • Low (-2 to 20): Internal docs & basic model card         │
│  • Medium (21 to 45): Human-in-the-loop review in SOP/tool  │
│  • High (46 to 55): Periodic audit & escalation pathways    │
│  • Critical (56 to 66): Exec review, red-teaming, kill-switch│
└─────────────────────────────────────────────────────────────┘

Risk Indicators Breakdown

Cluster	Indicator	Evaluated Risk States
Inherent System Risk	Interpretability Level	Transparent (1) · Partially Interpretable (4) · Opaque/Black Box (7)
	Generative Capacity	None (1) · Template-Based (4) · Context-Limited (4) · Open-Ended (7)
	Agentic Profile	None (1) · Limited Autonomy (4) · Fully Autonomous (7)
Performance & Reliability	Performance Metrics	High (0) · Medium (1) · Low (2) · Not Available (3)
	Bias Assessment	Complete (1) · Partial (2) · None (3)
	Robustness & Fairness	Formally Tested (0) · None (2)
	Explainability Method	SHAP/LIME/Other (0) · None (2)
Security & Compliance	Access Surface	Internal (1) · External API (2) · Public (3)
	Security Posture	Hardened (0) · Baseline (1) · Unverified (2)
	4th Party AI Risk	None (0) · Moderate (1) · High (2)
	Certifications	NIST / ISO 27001 / SOC 2 (0) · Other (1) · None (2)
Data Risk	MX Data in LLM Training	Retained for Model Training (2) · Zero Data Retention (0)
	Input/Output Filtering	Active Guardrails (0) · None / Unverified (2)
	Data Classification	Public (-3) · Internal/NPI (0) · PII/Financial (3)
	Persistent Memory	Session Only (0) · Cross-Session Retention (2)
Operational Modifiers	Human Oversight Level	Strict Human-in-the-Loop (-7) · Partial (-4) · None (0)
	Decision Criticality	Low Workflow Impact (0) · Influences Decisions (+3) · Critical (+5)

---

Investigation & Publishing Workflow

[Phase 1: Minimal Intake]
  │ • Vendor/Tool Name & URL
  │ • MX Specific Use Case
  │ • Data Classification & Touchpoints
  ▼
[Phase 2: Multi-Channel Autonomous Research]
  │ • Internal: Confluence spaces, Jira security & procurement tickets
  │ • Vendor: DPAs, Trust Portals, Sub-processor lists, Model Cards
  │ • Independent: SOC 2, ISO certifications, CVE / breach history
  ▼
[Phase 3: Conservative-by-Default Rubric Scoring]
  │ • Map evidence to indicators with mandatory citations
  │ • Auto-assign 'Insufficient Evidence — Assumed High' on gaps
  │ • Calculate composite score and map to Risk Tier & Controls
  ▼
[Phase 4: Human Gate & Confluence Page Publishing]
  │ • Present complete formatted draft in chat
  │ • Await explicit human confirmation & target space/parent
  │ • Publish structured Markdown page via Confluence API

---

Operational Architecture & Safety Controls

The agent’s operational architecture ensures high analytical rigor without requiring human operators to manually audit every underlying document:

1. Minimal Intake vs. Autonomous Depth

The agent limits human input to essential internal context that cannot be inferred externally:

• Product target: Vendor name, tool name, and specific product URL.
• Workflow context: The exact MX team, workflow, and operational touchpoint.
• Data exposure: Integration type (API, browser extension, SSO, native client), specific data fields passed, and data classification levels (Public, Internal/NPI, PII, Financial).

All subsequent investigation—including retrieving Data Processing Addenda (DPAs), SOC 2 compliance, sub-processor disclosures, and training data policies—is conducted autonomously across internal databases and external trust centers.

2. Conservative Scoring & Evidence Attribution

To protect against model hallucinations and unverified vendor marketing claims:

• Mandatory sourcing: Every indicator score requires an explicit, auditable citation and a one-sentence factual justification.
• Conservative ambiguity resolution: If documentation is unclear or conflicting, the agent defaults to the higher-risk score.
• Explicit gap flagging: If evidence cannot be found after exhausting all channels, the indicator is marked Insufficient Evidence — Assumed High. This makes data gaps immediately visible on the final scorecard.

3. Separation of Evaluation from Procurement

The agent is explicitly designed as an analyst, not a decision-maker. It calculates composite scores, maps scores to required governance controls (such as executive review, kill-switch readiness, or mandatory human verification), and generates a publish-ready Confluence report. The final procurement decision remains strictly in human hands.