AI governance process

Pylon AI Lifecycle Process

The gate process that moves a customer-facing AI behavior from build to production — alpha readiness, controlled Support beta, feedback scorecard, and a multi-condition exit — piloted on the MFA Loop Triager.

My role: Author & process owner
Tools: Pylon, Process design, CPMAI governance methodology

The problem

AI behaviors were shipping into a customer-facing support tool without a shared definition of "ready," and the person building a behavior was also the one approving it for production. The vendor had no answer either — Pylon confirmed it had no standard practice for promoting a skill from idea to production, only informal proximity: one person builds it, pulls in one other to sanity-check it.

Approach

Mapped the existing practice and found there wasn't one — confirmed directly with Pylon's own customer success team, who had no promotion standard of their own to borrow.
Designed a three-stage path: manual invocation on live tickets, then a keyword-triggered background agent that drafts a response a human accepts or rejects, then vetted promotion into the always-on support agent.
Authored AlphaGate, the evaluator skill that scores a candidate skill's fitness for support consumption and gates its promotion — structurally separating the person who builds a skill from the person who approves it.
Piloted the lifecycle on the Pylon MFA Loop Triager, validated against a 14-case set with 79% target-pattern coverage.
Ran a skill-inventory audit to resolve a stalled archive-vs-keep disagreement: usage telemetry alone was ambiguous (a never-used skill could mean it's bad, or that nobody knew it existed), so verdicts were taken per-skill with the support SME instead.
Designed the in-line feedback loop (/agent-feedback) inside Pylon itself, avoiding the token, latency, and workflow cost of routing feedback through an external system.

Outcomes

Pilot modeled at 100–200 support hours returned annually at 100 applicable cases/month.
Pylon, the vendor, had no promotion standard of its own and asked MX to pilot an evaluation approach and share results back — MX's lifecycle became reference work for the platform it runs on.
The pilot and process became the evidence base for a proposed AI Operations function inside TechOps.

What I learned

Zero usage is ambiguous evidence, not a verdict — a skill nobody used and a skill nobody knew about look identical in the data. Archiving instead of deleting preserved the difference until it could be checked by hand.
The hardest governance question isn't building the gate — it's defining what the gate is even measuring. Letting the evaluator emit its own scored output on every run made the audit trail self-documenting, instead of needing a separate pass-criteria doc to stay in sync with it.

The lifecycle exists because two things were true at once: MX had no shared definition of “ready” for an AI behavior touching customers, and its vendor didn’t either. Pylon told us directly that skill promotion runs on informal proximity — a builder, a second set of eyes, then it ships. That gap became the argument for a formal gate rather than a nice-to-have.

The design principle that mattered most was separating build from approval. Whoever writes a skill should not be the one deciding it’s safe for production — so AlphaGate, the evaluator, scores a candidate skill independently and emits that score on every run, which makes its own output history the audit trail instead of a document someone has to maintain in parallel.

The MFA Loop Triager was the proof point: a real background agent, graded by the support team that would actually use it, validated against a real case set before it went anywhere near “vetted.” That pilot — and the governance model around it — became the evidence base for the AI Operations function this work was ultimately arguing for.