Is Your AI Pilot Production-Ready? 12 Checks Before You Ship

The Gate That Separates Pilot From Production

Most AI pilots die in the gap between demo and production. The demo works on clean data in a controlled environment. Production introduces adversarial inputs, multi-tenant data isolation requirements, regulatory evidence obligations, and latency budgets that the demo never faced. In my former role as an AI Strategy Core Team Lead at a Fortune 100 company, I managed 53 enterprise applications through this gap. The pilots that survived shared one trait: they passed a structured readiness gate before touching production traffic.

This is not a generic checklist. Each check below maps to a specific failure mode I have seen in production AI systems, the architectural control that prevents it, and the evidence artifact required for audit. If your pilot cannot produce these artifacts, it is not ready for production.

Check 1: Data Provenance and Lineage Verification

Every input feature must trace back to a verified source with a documented transformation pipeline. In production, RAG poisoning occurs when untrusted content enters the knowledge base without lineage controls. The readiness check requires a data lineage map showing every source, transformation step, and access control point.

Failure mode: Untrusted data enters the pipeline through a vendor API or user-generated content, poisoning model outputs.

Architectural control: Implement a data provenance layer that tags every input with source, timestamp, and trust score. Use row-level security to isolate tenant-specific data. For RAG systems, enforce a content quarantine where new documents are validated against schema and content policies before entering the vector database.

Evidence artifact: Data lineage diagram showing source-to-feature mapping, with access control points marked. See secure enterprise RAG architecture for the reference pattern.

Check 2: Model Robustness Under Adversarial Load

Stress testing in staging does not predict production behavior. Production traffic includes prompt injection attempts, malformed inputs, and edge cases that staging never sees. The readiness check requires adversarial testing results showing model behavior under attack scenarios.

Failure mode: Model produces harmful outputs when subjected to prompt injection, jailbreak attempts, or distribution shift.

Architectural control: Run the model through an adversarial test suite before deployment. Include prompt injection patterns from the OWASP Top 10 for LLMs, data exfiltration attempts, and model extraction attacks. Measure the rejection rate for each attack class. Set a threshold: if the model fails more than 2% of adversarial tests, it does not pass this gate.

Evidence artifact: Adversarial test report with pass/fail rates per attack class, signed by the security team.

Check 3: Multi-Tenant Data Isolation

If your AI system serves multiple tenants, data isolation is not optional. Cross-tenant data leakage is a regulatory violation under GDPR, HIPAA, and most enterprise contracts. The readiness check requires proof that Tenant A cannot access Tenant B's data through any query path.

Failure mode: RAG retrieval returns documents from the wrong tenant. Vector search bypasses row-level security because the embedding model operates on shared indexes.

Architectural control: Implement tenant-scoped vector indexes or enforce row-level security at the retrieval layer. Test isolation by attempting cross-tenant queries with Tenant A credentials against Tenant B data. The test must fail 100% of the time. For shared infrastructure, use namespace isolation in the vector database and verify that the retrieval filter chain enforces tenant boundaries. See implementing RAG row-level security for multi-tenant AI for the implementation pattern.

Evidence artifact: Isolation test results showing cross-tenant query rejection rate (must be 100%).

Check 4: Runtime Monitoring and Drift Detection

A model that passes pre-deployment testing can drift in production. Input distributions shift, model behavior changes, and degradation is often invisible until a user complains. The readiness check requires a deployed monitoring stack with defined alert thresholds before the pilot goes live.

Failure mode: Model quality degrades silently over weeks. The team discovers the issue when a customer reports incorrect results, by which point the damage is done.

Architectural control: Deploy three monitoring layers before production traffic: (1) input distribution monitoring to detect data drift, (2) output quality monitoring to detect response degradation, and (3) latency monitoring to detect infrastructure issues. Set alert thresholds based on baseline measurements from staging. Use runtime verification tools to capture deterministic checks on probabilistic outputs.

Evidence artifact: Monitoring dashboard URL, alert configuration with thresholds, and escalation runbook.

Check 5: Latency Budget Compliance

Every AI system has a latency budget. For voice agents, the budget is 800ms round-trip before callers notice degradation. For RAG systems, the budget is typically 2-3 seconds before users abandon the query. The readiness check requires measured latency at p50, p95, and p99 under production-equivalent load.

Failure mode: System works in staging with 10 concurrent users. In production with 500 concurrent users, latency spikes to 12 seconds and users abandon the system.

Architectural control: Run load testing at 2x expected peak traffic. Measure p50, p95, and p99 latency for the full request path: input processing, model inference, retrieval, response generation. If p99 latency exceeds the budget, optimize before deployment. Common fixes: implement response caching, use a model router to send simple queries to faster models, or pre-compute embeddings for high-frequency queries.

Evidence artifact: Load test report showing p50/p95/p99 latency at 2x peak load, with pass/fail against the latency budget.

Check 6: Fallback and Escalation Architecture

Production AI systems fail. The question is not whether, but when, and what happens when they do. The readiness check requires a documented fallback chain: what happens when the model is unavailable, when latency exceeds thresholds, and when outputs fail quality checks.

Failure mode: Model API goes down. The AI system returns errors to users. No fallback exists. The pilot is labeled a failure and the project dies.

Architectural control: Implement a three-tier fallback chain: (1) cached response for common queries, (2) deterministic rule-based response for queries the model cannot handle, (3) human escalation for high-stakes decisions. For voice agents, this means IVR fallback, SMS escalation, and human handoff. See why AI voice agents fail in production for the reference architecture.

Evidence artifact: Fallback flowchart showing the decision tree for model unavailability, latency timeout, and quality failure.

Check 7: Compliance Evidence Pipeline

Under TRAIGA, the EU AI Act, and NYC Local Law 144, you must produce evidence that your AI system was tested, monitored, and governed. A compliance dashboard that shows current status is not evidence. Evidence is an immutable, timestamped record of what happened, when, and who approved it. The readiness check requires a deployed evidence pipeline before production traffic.

Failure mode: Regulator requests evidence of bias testing. The team has a dashboard showing current bias metrics, but no historical record of when tests were run, what the results were, or who signed off. The organization faces a regulatory penalty for insufficient documentation.

Architectural control: Implement an immutable audit trail that captures every model decision, input, output, and human override. Use append-only storage (not a mutable database). Capture: timestamp, user ID, model version, input hash, output, confidence score, and any human review action. See AI governance and evidence architecture for the reference implementation.

Evidence artifact: Evidence pipeline architecture diagram, sample audit trail export, and retention policy documentation.

Check 8: Security Threat Model and Penetration Test

AI systems have unique attack surfaces that traditional security testing misses. Prompt injection, model extraction, data exfiltration through crafted queries, and adversarial inputs that bypass content filters are all AI-specific threats. The readiness check requires a threat model that covers these attack vectors and a penetration test that validates the defenses.

Failure mode: Attacker uses prompt injection to extract system prompts, customer data, or model weights. The team did not test for this because traditional pen testing does not cover AI-specific attack vectors.

Architectural control: Build a threat model using the NIST AI RMF framework. Include: prompt injection, data exfiltration, model extraction, denial of service through expensive queries, and supply chain attacks via model weights or plugins. Run a penetration test that attempts each attack class. Document the defenses and their effectiveness.

Evidence artifact: Threat model document, penetration test report with attack-by-attack results, and remediation plan for any failed tests.

Check 9: Human-in-the-Loop Decision Points

Not every AI decision should be automated. High-stakes decisions (financial, legal, healthcare) require human review. The readiness check requires a documented decision framework that specifies which outputs are automated, which require human review, and which require human approval before action.

Failure mode: AI system automates a decision that should have had human review. The decision causes harm. The organization is liable because there was no human checkpoint.

Architectural control: Define a decision matrix mapping output risk levels to review requirements: (1) low risk: automated, (2) medium risk: human review after action, (3) high risk: human approval before action. Implement the matrix as a policy layer in the application, not as a documentation note. The policy layer must be testable and auditable.

Evidence artifact: Decision matrix document, policy implementation code, and test results showing that high-risk outputs are blocked until human approval.

Check 10: Cost and Resource Budget

AI systems have variable costs that can spike unpredictably. A prompt that costs $0.01 in staging can cost $5.00 in production if a user crafts a query that triggers a long chain of tool calls. The readiness check requires a cost model showing projected monthly spend at expected traffic, and a circuit breaker that cuts off spending before it exceeds the budget.

Failure mode: Pilot goes live. A user discovers a query pattern that triggers expensive model calls. The team gets a $50,000 API bill in the first week.

Architectural control: Implement a cost monitoring layer that tracks spend per user, per query, and per model. Set a circuit breaker that rate-limits or degrades service when spend exceeds 80% of the daily budget. Build a cost model that projects monthly spend at expected traffic, and validate it against the first 48 hours of production traffic.

Evidence artifact: Cost model spreadsheet, circuit breaker configuration, and 48-hour cost validation report.

Check 11: Rollback and Version Management

When a production AI system degrades, you need to roll back to the previous version quickly. This requires versioned models, versioned prompts, versioned retrieval indexes, and a rollback procedure that takes minutes, not hours. The readiness check requires a tested rollback procedure.

Failure mode: New model version degrades output quality. The team tries to roll back but discovers the previous model version was overwritten, the prompt template was changed without version control, and the vector index was rebuilt with different embeddings. Rollback takes 6 hours instead of 6 minutes.

Architectural control: Version every component: model weights, prompt templates, retrieval indexes, and configuration. Store previous versions with retention policies. Test the rollback procedure by deploying a new version, then rolling back, and measuring the time. The rollback must complete in under 15 minutes.

Evidence artifact: Version management policy, rollback test results with measured time, and rollback runbook.

Check 12: Documentation and Operational Runbook

The team that builds the pilot will not be the team that operates it in production. The readiness check requires operational documentation that a new on-call engineer can follow to diagnose and resolve issues without contacting the original builders.

Failure mode: Pilot goes live. The original builders move to the next project. Production issue occurs at 2 AM. The on-call engineer has no documentation, no runbook, and no idea how the system works. Mean time to resolution: 4 hours.

Architectural control: Produce three documents before production: (1) system architecture diagram showing all components and data flows, (2) operational runbook with step-by-step procedures for common incidents, (3) on-call guide with escalation contacts and decision trees. Have a new engineer read the documentation and attempt to diagnose a simulated incident. If they cannot, the documentation is insufficient.

Evidence artifact: Architecture diagram, operational runbook, on-call guide, and the results of the new-engineer documentation test.

The Gate Decision

These 12 checks are not suggestions. They are the gate between pilot and production. If any check fails, the pilot does not go live. If the team cannot produce the evidence artifact for any check, the pilot does not go live.

In my Fortune 100 experience, this gate prevented at least 4 deployments that would have caused production incidents. In each case, the team believed the system was ready because the demo worked. The gate revealed what the demo could not: adversarial vulnerabilities, missing isolation controls, absent evidence pipelines, and undocumented rollback procedures.

If you are deploying high-stakes enterprise AI and need an independent systems advisor to audit your readiness before regulators or production traffic do, schedule a strategic evaluation.

FAQ

What is the most common reason AI pilots fail the readiness gate? Missing evidence pipelines. Teams build the system but do not build the audit trail. When regulators or internal auditors request evidence, there is nothing to show.

How long should the readiness check process take? For a well-architected system, 1-2 weeks. For a system that has not considered these checks, 4-8 weeks. The checks are not the bottleneck. The architectural work to satisfy them is.

Can these checks be automated? Checks 1, 3, 5, 8, and 11 can be partially automated with test suites and monitoring tools. Checks 2, 4, 6, 7, 9, 10, and 12 require human judgment and architectural review. Do not attempt to fully automate the gate.

What happens if a check fails? The pilot does not go to production. The team addresses the failure, re-runs the check, and re-presents at the gate. There is no partial pass.