Deployment gate · AI governance

Pylon AI Lifecycle Process

The gate that moves a customer-facing AI behavior from build to production: alpha readiness, a controlled Support beta, a feedback scorecard, and a multi-condition exit. Piloted on the MFA Loop Triager.

My role: Author and process owner
When: 2024 – present
Tools: Pylon, Process design, CPMAI governance methodology

The problem

AI behaviors were shipping into a customer-facing support tool without a shared definition of ready, and the person building a behavior was also the one approving it for production. The vendor had no answer either. Pylon confirmed it had no standard practice for promoting a skill from idea to production, only informal proximity: one person builds it and pulls in one other to sanity-check it.

Approach

Mapped the existing practice and found there was not one. Confirmed directly with Pylon's customer success team, who had no promotion standard of their own to borrow.
Designed a three-stage path: manual invocation on live tickets, then a keyword-triggered background agent that drafts a response a human accepts or rejects, then vetted promotion into the always-on support agent.
Authored AlphaGate, the evaluator skill that scores a candidate skill's fitness for support consumption and gates its promotion. It structurally separates the person who builds a skill from the person who approves it.
Piloted the lifecycle on the Pylon MFA Loop Triager, validated against a 14-case set with 79% target-pattern coverage.
Ran a skill-inventory audit to resolve a stalled archive-versus-keep disagreement. Usage telemetry alone was ambiguous (a never-used skill could be bad, or nobody knew it existed), so verdicts were taken per skill with the support SME instead.
Designed the in-line feedback loop (/agent-feedback) inside Pylon itself, avoiding the token, latency, and workflow cost of routing feedback through an external system.

Outcomes

Pilot modeled at 100 to 200 support hours returned annually, at roughly 100 to 200 applicable cases a month.
Pylon had no promotion standard of its own and asked MX to pilot an evaluation approach and share results back. The lifecycle became reference work for the platform it runs on.
The pilot and the process became the evidence base for a proposed AI Operations function inside TechOps.
Where it stands: the criteria are authored and the process is piloting. It is not yet validated at scale.

What I learned

Zero usage is ambiguous evidence, not a verdict. A skill nobody used and a skill nobody knew about look identical in the data. Archiving instead of deleting preserved the difference until it could be checked by hand.
The hardest governance question is not building the gate. It is defining what the gate measures. Letting the evaluator emit its own scored output on every run made the audit trail self-documenting, instead of needing a separate pass-criteria doc to keep in sync.

The lifecycle exists because two things were true at once: MX had no shared definition of ready for an AI behavior touching customers, and its vendor did not either. Pylon told us directly that skill promotion runs on informal proximity: a builder, a second set of eyes, then it ships. That gap became the argument for a formal gate rather than a nice-to-have.

The design principle that mattered most was separating build from approval. Whoever writes a skill should not be the one deciding it is safe for production. So AlphaGate, the evaluator, scores a candidate skill independently and emits that score on every run, which makes its own output history the audit trail instead of a document someone has to maintain in parallel.

The MFA Loop Triager was the proof point: a real background agent, graded by the support team that would use it, validated against a real case set before it went anywhere near vetted. That pilot, and the governance model around it, became the evidence base for the AI Operations function this work was arguing for.