AI Voice Architecture

AI Voice Agent Architecture:
How Kestrel Voice Works

Custom Python orchestration, realtime voice, deterministic guardrails, RAG, business tools, fallback modes, and operational evidence — the complete architecture behind a production voice agent platform.

By Subodh KC · July 15, 2026 · 30 min read

Last reviewed: July 15, 2026

Scope: Production architecture for AI voice agent platforms — telephony, orchestration, RAG, tools, security, fallback modes, post-call intelligence, learning, and multi-channel coverage

Disclosure: Kestrel Voice and HAIEC are platforms developed by Subodh KC. This article refers to Kestrel as the product through which these architectural principles have been applied directly, and HAIEC as the companion AI security, governance, and compliance-readiness platform. Neither platform eliminates AI risk. Their purpose is to make that risk more visible, controlled, testable, and supportable with evidence.

Educational notice: This article provides general information about AI voice agent architecture. Legal and regulatory applicability depends on the use case, jurisdiction, data, call purpose, and caller population. Telephony-specific obligations such as TCPA, HIPAA, and state biometric laws require case-by-case analysis.

Table of Contents

The Direct Answer Architecture at a Glance What Happens When a Call Arrives Four Ways Kestrel Can Handle Calls Adaptive Orchestration Tenant-Specific Business Runtime Deterministic vs Generative Answers Governed RAG Tools and Verified Actions Failure and Degraded Modes Post-Call Operations Learning Pipeline Beyond Inbound: Outbound, Meetings, and Copilot Multi-Channel: SMS, Video, and WebRTC Security Architecture Technology Decisions What the Architecture Does Not Guarantee Frequently Asked Questions

Synopsis

A production AI voice agent is not a model connected to a phone number. It is a layered operational system coordinating telephony, tenant identity, deterministic rules, realtime AI, retrieval-augmented knowledge, tool execution, fallback modes, post-call intelligence, learning, and evidence. This article documents the architecture behind Kestrel Voice — the platform I built to handle real calls for service businesses — covering inbound and outbound call flows, adaptive orchestration, RAG, security, compliance, and the operational pipeline that makes the system trustworthy enough to answer a phone.

AI Voice AgentsVoice OrchestrationRAGSecurityMulti-ChannelKestrel Voice

Most AI voice agent architecture discussions start and end with the model. Which frontier model powers the conversation? How natural is the voice? How low is the latency?

These questions matter. But they are the smallest part of a production system.

When I built Kestrel Voice, the model was the first decision I made and the easiest one to change. The harder work was everything around it: how to resolve which business a call belongs to, how to answer differently for a plumber versus a veterinary clinic, how to handle emergencies before the model speaks, how to ground answers in business-specific knowledge, how to verify that a booking actually succeeded before telling the caller it did, how to degrade gracefully when the realtime API fails, how to prevent toll fraud, how to record calls with proper consent, how to learn from patterns without introducing unsafe behavior, and how to produce evidence that a human can review after the call ends.

The model handles the conversation. The architecture handles the business.

This article documents that architecture — not as an abstract framework, but as the actual system running in production today.

The Direct Answer

Kestrel Voice is a multi-tenant AI voice agent platform built on FastAPI, deployed on Modal, and backed by Supabase and Postgres. When a call arrives, the system resolves the called number to a tenant, loads that tenant business configuration, selects an AI mode based on plan and credits, and routes the call through a layered orchestration pipeline: deterministic hard interrupts first, fast-path answers second, and realtime AI reasoning third. RAG grounds the model in tenant-specific knowledge. Tools execute bookings and CRM operations with result verification. A four-level degradation system handles failures. Post-call intelligence extracts structured outcomes. A learning pipeline promotes recurring patterns into fast paths. And every step produces evidence — transcripts, tool calls, prompt hashes, and outcome records — that a human can review.

The architecture is not a demo. It is an operational system handling real calls for real businesses.

Architecture at a Glance

Figure 1 — Five-layer architecture: Presentation (Next.js), Voice Runtime (FastAPI on Modal), Telephony (Twilio), Demand Engine, and Data (Supabase/Postgres with pgvector).

The diagram shows the five layers of the system. Telephony (Twilio) handles the phone call and media stream. The Voice Runtime (FastAPI on Modal) processes the call through tenant resolution, mode selection, deterministic intercepts, AI conversation, and tool execution. The Data Layer (Supabase/Postgres with pgvector) stores tenant configurations, call logs, RAG chunks, and analytics. The Demand Engine handles CRM, onboarding, and reporting. The Presentation Layer (Next.js) provides the dashboard, RAG management, and configuration UI.

These are logically separated services with a consolidated deployment option — the voice runtime, demand engine, and scrapers share a single Modal container for cost efficiency, while the frontend runs on Vercel. The separation is logical, not physical, which simplifies operations without sacrificing isolation.

What Happens When a Call Arrives

A call arrives at a Twilio phone number. What happens next is a nine-step sequence that runs before the AI ever speaks.

Phone rings

Twilio receives the inbound call and sends a webhook to the FastAPI voice runtime. The webhook is authenticated via Twilio request signature validation.

Tenant resolution

The called number is mapped to a tenant configuration. This determines which business identity, greeting, hours, services, and emergency rules apply. A call to a plumbing company will never use a veterinary clinic configuration.

Mode selection

The system selects an AI mode based on the tenant plan, available credits, and time of day. Realtime uses WebSocket streaming. Adaptive uses a three-layer intent router. Gather is a turn-based fallback. IVR handles deterministic routing.

Session initialization

A call session is created with a unique ID, tenant context, and conversation state. For realtime and adaptive modes, a WebSocket connection is established between Twilio media stream and the voice runtime.

System prompt assembly

The tenant business configuration is assembled into a system prompt using a three-tier hierarchy: tenant-specific override, industry template default, then safe fallback. Today date is injected to prevent date hallucination. The prompt source and hash are logged for evidence.

Deterministic intercept

Before the AI processes any user speech, the system checks for hard interrupts: emergency phrases, transfer requests, compliance stop phrases. If matched, the model is cancelled and a pre-written response is injected.

AI conversation

The model handles the conversation within the guardrails of the system prompt. For knowledge questions, RAG retrieves tenant-specific chunks and injects them as context. For actions, the model calls tools that execute in application code.

Tool execution and verification

When the caller requests a booking, the model gathers required fields, the caller confirms, and the booking API executes. The model reports success or failure based on the authoritative tool result — not its own assumption.

Evidence capture

The call produces a transcript, recording, tool call logs, prompt evidence, outcome classification, and cost record. Post-call intelligence runs asynchronously to extract summary, key points, decisions, action items, and sentiment.

Steps 1 through 5 happen in under a second. Step 6 runs on every user utterance. Steps 7 and 8 happen in real time during the conversation. Step 9 runs after the call ends without blocking teardown.

Four Ways Kestrel Can Handle Calls

Gather — Turn-Based Fallback

Turn-based voice agent. The caller speaks, the system transcribes, the AI responds, and the cycle repeats. Lowest cost mode. Used as a fallback when realtime WebSocket connections fail and as the primary mode for tenants on basic plans. No streaming — each turn is a complete request-response cycle.

Realtime — WebSocket Streaming

Lowest latency mode. Audio streams bidirectionally over WebSocket between Twilio media stream and the voice runtime, which proxies to the realtime API. The model can interrupt itself, handle barge-in, and respond in near-real time. Highest cost per minute. Used for tenants on premium plans or when conversation quality is critical.

Adaptive — Three-Layer Intent Router

Realtime-first with deterministic intercepts. The model handles the conversation, but every user utterance passes through a three-layer router first: hard interrupts (emergency, transfer), fast paths (hours, pricing, booking), and AI reasoning. When a fast path matches, the model is cancelled and a deterministic response is injected — saving cost and improving consistency. Credit-based pricing.

IVR / Forwarding — Deterministic Routing

No AI model. The call is routed deterministically based on tenant configuration: sequential ring (ring the business owner phone first, fall back to AI), time-based routing (different destinations for business hours vs after hours), or direct transfer. Used for tenants who want human-first routing with AI as fallback.

The mode is selected per call based on the tenant plan, available credits, and configuration. A smart mode router can escalate or downgrade mid-call based on conversation signals — for example, switching from adaptive to gather if the realtime API degrades.

Adaptive Orchestration

Adaptive mode is the most architecturally interesting mode. It runs a realtime AI session but intercepts every user utterance through a three-layer router before the model processes it. The goal: answer deterministic questions deterministically, reserve the model for reasoning.

Layer 1: Hard Interrupts

Regex-based, zero-latency. Matches emergency phrases ("gas leak", "fire", "carbon monoxide"), transfer requests ("speak to a human"), compliance stop phrases, and repeat requests. When matched, the model is cancelled mid-response and a pre-written text-to-speech message is injected. This ensures emergency routing happens in milliseconds, not model-response time.

Layer 2: Fast Paths

Keyword and lookup-based, zero-latency. Matches common business questions: hours, location, service area, pricing, booking intent, availability, greeting. When matched, the model is cancelled and a deterministic response is generated from the tenant canonical configuration — not the model interpretation. This means hours are always correct, pricing always matches the configuration, and the model never hallucinates a business hour.

Layer 3: AI Reasoning

The model handles everything that does not match L1 or L2: ambiguous questions, multi-part requests, contextual explanations, clarification, and natural conversation. No intercept is needed — the model simply responds naturally within the system prompt guardrails.

Circuit Breaker

If the realtime WebSocket fails three times for a tenant within a five-minute window, a circuit breaker trips. Subsequent calls for that tenant bypass realtime mode and route directly to Gather (turn-based) mode. The breaker recovers after a cooldown period, and the system automatically retries realtime.

Gather Fallback

On any WebSocket failure during a realtime or adaptive call, the call falls back to Gather mode without dropping the caller. The conversation continues in turn-based mode until the call ends.

Tenant-Specific Business Runtime

Every tenant has a unique business configuration stored in Postgres: business name, industry type, greeting, communication style, services offered, service areas, business hours, transfer destination, emergency behavior, appointment settings, and FAQ entries. This configuration is assembled into a system prompt using a three-tier hierarchy.

1Tenant-specific override — If the tenant has a custom prompt, it takes precedence.

2Industry template default — If no custom prompt, the system loads an industry template (HVAC, restaurant, veterinary, dental, legal, etc.) with pre-configured prompts, L2 intent patterns, and FAQ templates.

3Safe fallback — If neither is available, a generic voice agent prompt is used.

Prompt Evidence Logged

Tenant ID — Which business the prompt belongs to

Source — Tenant-specific, template, or fallback

Template name — Industry template used (if any)

Business name — Identity injected into prompt

Prompt hash — SHA-256 hash for version tracking

The prompt includes temporal context — today date and time — injected automatically to prevent the model from hallucinating dates. "What is your earliest appointment?" gets an answer grounded in the actual calendar, not the model training data.

Deterministic vs Generative Answers

A common architecture mistake is sending every user utterance to the model. This increases cost, latency, and hallucination risk. Kestrel separates answers into three categories:

Question Type	Answer Source	AI Involved?	Example
Hours, location, service area	Canonical tenant configuration	No	"What time do you close?"
Availability, booking	Live calendar system	No (tool call)	"Can I book for Tuesday?"
Policies, warranties, services	RAG (tenant-specific knowledge)	Yes (grounded)	"What is your warranty on repairs?"
Ambiguous, multi-part, contextual	AI reasoning	Yes (model)	"I think my AC is making a weird noise"

The deterministic layer means "What are your hours?" never goes to the model. The answer comes from the tenant configured business hours, injected as a fast-path response. This costs zero AI tokens and returns in zero model-response time. The model is reserved for questions that actually require reasoning — and when it does answer, RAG grounds it in the tenant own knowledge base rather than the model training data.

Governed RAG

Kestrel Voice includes a retrieval-augmented generation system that grounds the AI in tenant-specific knowledge. The pipeline has four stages: ingestion, storage, retrieval, and refresh.

Figure 2 — RAG pipeline: ingestion from three source types, tenant-scoped storage with RLS, keyword-triggered retrieval with similarity threshold, and SHA-256-based refresh tracking.

Stage 1: Ingestion

Website scraping — The tenant provides a URL. The system scrapes the page, extracts text content, and validates the URL to prevent server-side request forgery (private IPs, localhost, and internal domains are blocked).FAQ ingestion — FAQ entries from the tenant configuration are chunked and embedded with metadata identifying them as FAQ source type.Google Drive folder — Tenants can designate a Drive folder as a knowledge source. The system indexes the content and tracks storage usage.

Stage 2: Storage

Chunks are stored in Postgres with the pgvector extension. Each chunk includes: content text, embedding vector (1536 dimensions), tenant ID, source URL, source type (website, FAQ, feedback), content hash for versioning, and metadata. An approximate nearest neighbor index enables fast similarity search. Row-level security ensures tenants can only access their own chunks.

Stage 3: Retrieval

During a call, the system detects knowledge-seeking keywords in the user speech ("warranty", "pricing", "policy", "hours") and triggers a retrieval. The user utterance is embedded, and a Postgres RPC function performs cosine similarity search against the tenant chunks — filtered by tenant ID and optional source type. A minimum similarity threshold excludes low-quality matches. The top results are formatted as context and injected into the model prompt. RAG works in adaptive and realtime modes, not in Gather or IVR.

Stage 4: Refresh

Content changes are detected via SHA-256 content hashing. Sources older than seven days are marked as stale. Biweekly reminders prompt tenants to refresh their content. A manual re-scrape endpoint allows on-demand updates. FAQ re-indexing is triggered when the tenant updates their FAQ configuration.

RAG is per-tenant and opt-out. Each tenant RAG enablement is checked and cached. The retrieval path tries the frontend API first, then falls back to a direct database RPC — ensuring retrieval works even if the frontend is temporarily unavailable.

Tools and Verified Actions

When a caller asks to book an appointment, the system follows a strict sequence to ensure the action is real, not assumed.

Caller requestsAI gathers fieldsCaller confirmsBooking API executesAuthoritative resultAI reports success/failure

Calendar Service

Book, reschedule, and cancel appointments with Google Calendar sync. The AI gathers required fields (name, phone, service, date), the caller confirms, and the booking API executes. The AI reports the result based on the tool return value — not its own assumption. The system prompt explicitly instructs the model to confirm bookings only after tool success.

CRM Integration

Create and update leads in the tenant CRM. Phone numbers are normalized to E.164 format. The system stores leads in Postgres and is extensible to Salesforce and HubSpot.

Extraction-to-Booking

After a call, extracted data (caller name, requested service, preferred time) can be automatically drafted into a booking with confidence scoring. High-confidence extractions can auto-book; low-confidence ones require human review.

Failure and Degraded Modes

Production systems fail. The architecture must handle failure without dropping the caller or producing unsafe behavior. Kestrel has a four-level degradation system.

Level 0: Full AI

Frontier model with premium voice synthesis. Best quality. Used when all services are healthy.

Level 1: Fast AI

Cost-efficient model with standard voice synthesis. Reduced latency and cost. Used when the frontier model API is degraded or rate-limited.

Level 2: Rule-Based

State machine only. No language model. Handles greetings, basic intent routing, and voicemail capture. Used when AI APIs are unavailable.

Level 3: Human Transfer

Transfer to a human or voicemail. Last resort. Used when all automated modes have failed.

Failure	Response
Realtime WebSocket fails	Fall back to Gather (turn-based)
3 WebSocket failures in 5 min	Circuit breaker trips → Gather mode
AI API degraded	Degrade to Fast AI or Rule-Based
All AI unavailable	Human transfer or voicemail
Tool fails	AI reports failure, offers alternative
Emergency detected	Immediate transfer, no AI deliberation

The system tracks success and failure metrics at each level and can automatically recover when service health returns. Manual override is available for operators who need to force a specific level.

Post-Call Operations

After a call ends, an asynchronous pipeline runs without blocking call teardown. Two stages: intelligence extraction and action dispatch.

Call endsPost-call intelligenceAction engineWebhook + email + dashboard

Intelligence Extracted

Summary — 2-3 sentence narrative

Key points — Bullets of what was discussed

Decisions — Commitments and agreements

Objections — Concerns raised by the caller

Action items — Follow-ups with priority and owner

Recommendations — Next best action, risk flags, missed opportunities

Sentiment trend — Positive, neutral, negative, or mixed

Meeting insights — Topic segments and sentiment (meeting calls only)

Action Engine

The action engine maps intelligence output to persistent call actions. High-priority items trigger email alerts. If the tenant has configured a webhook, the engine dispatches the intelligence payload with three retries and exponential backoff. All actions are deduplicated and stored for dashboard review.

The call also produces a transcript, optional recording, tool call logs, prompt evidence with hash, and outcome classification. Calls can be replayed with audio and transcript sync, tagged for categorization, and reviewed in the dashboard.

Learning Pipeline

Kestrel Voice learns from call patterns. The learning pipeline promotes recurring fast-path patterns into active fast actions — but with guardrails.

Record

Every L2 fast-path match is recorded with tenant ID, intent, user input, and timestamp. This is the raw signal.

Promote

Patterns that appear above a minimum occurrence threshold are promoted to the fast actions table. Guardrails filter the promotion: only whitelisted intents are eligible, blocked phrases are excluded, and pattern length is limited.

Review Window

Promoted patterns enter a time-delayed review window. Email reminders are sent to the tenant before auto-approval, giving them an opportunity to reject. This is not a human-in-the-loop gate — it is a notification window with opt-out.

Activate

After the review window passes, patterns are auto-approved and become active in future calls. The tenant can deactivate any pattern from the dashboard.

Learning Guardrails

Intent whitelist — Only approved intent types can be promoted
Blocked phrases — Patterns containing blocked phrases are excluded
Length limits — Pattern text must be within configured bounds
Minimum occurrences — Patterns must appear multiple times before promotion
Tenant-scoped — Patterns are never shared across tenants

Template FAQs

Separately from learned patterns, industry template FAQs provide pre-written answers for common questions in each business type. A restaurant template includes FAQs about party size, reservations, and menu questions. An HVAC template includes FAQs about service calls, estimates, and emergency dispatch. These template FAQs are injected into the system prompt and matched against user input — no AI call needed.

Pattern Versioning

A pattern versioning system allows database-stored patterns to merge with hardcoded template patterns, with an approval workflow (draft, pending, approved, rejected). This enables safe pattern evolution without code changes.

Beyond Inbound: Outbound, Meetings, and Copilot

Kestrel Voice handles more than inbound calls. The platform supports outbound dialing, meeting bridges, live AI copilot, and sequential ring — each with its own state machine and guardrails.

Outbound Dialer

A full call state machine: initiating → dialing → ringing → connected → active → ended. The AI can join in three modes: silent (transcribe only), assist (provide suggestions to the caller), or active (speak on behalf of the user). Credits are only consumed when the AI is actively engaged — not during dialing or ringing. Twilio signature validation protects outbound endpoints.

Meeting Bridge

Dial into external meetings (Zoom, Teams, Google Meet) via Twilio. DTMF tones handle meeting ID and passcode entry. The AI participates in three modes: silent (transcribe only), assist (real-time suggestions), or active (speaks during the meeting). Reuses the outbound session model with meeting-specific billing.

AI Copilot

During any live call, a copilot panel provides real-time AI assistance to the human operator: live notes, suggested responses, chat Q&A with context, and action item extraction. Uses a cost-efficient model for low-latency assistance. RAG integration allows the copilot to look up knowledge-base answers during the call. All copilot events are persisted for post-call review.

Sequential Ring

Ring the business owner phone before the AI answers. Customizable timeout, sequential or simultaneous ring modes, and time-based routing. If the human does not answer within the configured timeout, the call falls back to AI. Only activates if the tenant has enabled it — existing call flow is preserved for tenants who want AI-first.

Multi-Channel: SMS, Video, and WebRTC

Voice is the primary channel, but Kestrel Voice also supports SMS, video sessions, and browser-based calling.

SMS

Outbound SMS via Twilio with two modes: internal (system notifications, no credit deduction) and user (business messaging, credit-based). Inbound SMS handles TCPA-required STOP, HELP, and START keywords through a dedicated webhook. The inbound SMS router has been migrated from the backend to the frontend for simpler maintenance.

Video Sessions

Video consultation sessions via Daily.co API. Session management includes creation, participant tracking, recording, and status lifecycle. Tenants can create video sessions for remote consultations, and recordings are stored with consent tracking.

WebRTC

Browser-based calling via Twilio Client. A prebind session binds the tenant ID before the device connects, ensuring deterministic tenant resolution for browser-initiated calls. This enables click-to-call from the dashboard and web-based phone functionality.

Compliance

Call recording consent is handled per state law. Two-party consent states receive a consent prompt before recording begins. The system maps area codes to states and generates the appropriate TwiML consent prompt. SMS complies with TCPA opt-out handling.

Security Architecture

Security in a voice agent platform spans telephony, API, data, and compliance. Kestrel implements defense-in-depth across six domains.

Telephony Authentication

All Twilio webhook endpoints validate request signatures using Twilio request validator. The system is fail-secure in production — validation bypass is blocked outside development environments. Multiple URL candidates are checked to handle proxy and deployment URL variations.

API Key Management

Three-tier API key system: admin keys for administrative endpoints (validated with constant-time comparison to prevent timing attacks), developer keys for external integrations (SHA-256 hashed storage, never stored plaintext, tenant-scoped), and internal service-to-service keys for platform-internal communication.

Spam and Fraud Protection

Multi-layer spam enforcement with score-based routing: normal, limited (voicemail only), and blocked. Caller reputation memory tracks repeat callers. Tenant-configurable sensitivity levels (low, medium, high). Four whitelist types for false-positive protection: contacts, recent callers, IVR-passed, and manual.

Toll Fraud Prevention

Country-specific call rate limiting with daily, hourly, and maximum duration limits. High-risk country detection for IRSF (International Revenue Sharing Fraud) prevention. Call attempts are tracked and blocked when exceeding configured thresholds. Fraud protection endpoints are admin-API-key protected.

Recording Consent

TCPA-compliant call recording consent. Area code to state mapping automatically detects two-party consent states. Consent prompts are generated as TwiML before recording begins. SMS handles STOP, HELP, and START keywords as required by TCPA.

Tenant Isolation

Defense-in-depth tenant isolation across five layers: ASGI middleware extracts tenant ID from JWT on every request, all database queries pass tenant ID as a parameter, Postgres row-level security enforces tenant-scoped access at the database level, server-side auth extraction never trusts client-provided tenant IDs, and internal service-to-service calls use a separate authentication path.

Tenant Isolation — Five Concentric Layers

ASGI Middleware — Extracts tenant ID from JWT on every request

Parameter Scoping — All database queries pass tenant ID as a parameter

RLS Policies — Postgres row-level security enforces tenant-scoped access

Auth Extraction — Server-side, never trusts client-provided tenant IDs

Internal API — Separate authentication path for service-to-service calls

Additional measures include CORS policies restricting origins to the application domain, API documentation disabled in production, file upload security with MIME type whitelisting and virus scanning, and SSRF prevention in the RAG scraper (blocking private IPs, localhost, and internal domains).

Technology Decisions

The key technology decisions and why they were made. This is not an exhaustive dependency list — it covers the decisions that shaped the architecture.

FastAPI

Chosen for async-first design, native WebSocket support for realtime voice streaming, automatic OpenAPI documentation, and Pydantic v2 validation. The voice runtime needs to handle simultaneous WebSocket connections, HTTP webhooks, and background tasks — FastAPI async model handles this naturally.

OpenAI (API-based)

All inference is API-based — no local models. The realtime API powers voice streaming, the frontier model handles conversation, a cost-efficient model handles copilot and post-call intelligence, and the embedding model powers RAG. API-based inference provides lower latency than self-hosted models on serverless infrastructure.

Twilio

Pinned SDK version for stability. Twilio handles voice (programmable voice, TwiML, media streams), SMS, phone number provisioning, and request signature validation. The telephony layer is the entry point for every call.

Supabase + Postgres + pgvector

Postgres with pgvector for RAG vector search, Supabase for auth and row-level security, real-time subscriptions for dashboard updates. Using Postgres for both relational data and vector search avoids a separate vector database while providing RLS for tenant isolation.

Modal

Serverless deployment with auto-scaling. Minimum container count prevents cold-start errors on the first Twilio webhook. Consolidated deployment serves the voice runtime, analytics, and scrapers from one container for cost efficiency. Secret management via Modal secret store.

Redis (optional)

Optional session store for turn-based conversations. Provides multi-instance state sharing and persistent session storage across restarts. Graceful fallback to in-memory storage if Redis is unavailable — the system degrades but does not fail.

What the Architecture Does Not Guarantee

Honest architecture documentation includes what the system does not guarantee. The following limitations are inherent to AI voice agent platforms, including Kestrel Voice:

Zero hallucinations

The model may still invent facts, mishear information, or claim actions succeeded when they failed. Deterministic fast paths, RAG grounding, and tool-result verification reduce this risk but cannot eliminate it.

Perfect emergency detection

Regex-based emergency matching catches known phrases but cannot detect every possible emergency description. The system errs on the side of caution but cannot guarantee zero missed emergencies.

Legal compliance in every jurisdiction

TCPA, HIPAA, state biometric laws, and international regulations vary by use case and jurisdiction. The platform provides configurable controls but does not make any business compliant by default.

Permanent uptime

The system degrades gracefully but is not immune to provider outages, network failures, or infrastructure issues.

Complete immunity from attacks

Twilio signature validation, API key management, spam enforcement, and tenant isolation reduce attack surface but cannot prevent every possible attack vector.

Autonomous production improvement

The learning pipeline promotes patterns with guardrails and a review window, but it does not autonomously change business logic or override tenant configuration.

Frequently Asked Questions

What is an AI voice agent architecture?

An AI voice agent architecture is the complete system design that surrounds a voice AI model — including telephony integration, tenant configuration, deterministic rules, retrieval-augmented generation, tool execution, fallback modes, security, and post-call intelligence. The model handles conversation; the architecture handles the business process.

What is custom voice orchestration?

Custom voice orchestration is the practice of building application logic around the AI model to control business workflows, enforce safety rules, verify tool results, and produce evidence — rather than relying on the model alone to manage the call. Kestrel Voice uses custom Python orchestration via FastAPI.

What is adaptive AI voice?

Adaptive AI voice is a mode where the system runs a realtime AI session but intercepts every user utterance through a three-layer router: hard interrupts (emergencies, transfers), fast paths (hours, pricing, booking), and AI reasoning. Deterministic answers are injected without model involvement; the model handles only what requires reasoning.

Does Kestrel Voice use RAG?

Yes. Kestrel Voice includes a full RAG pipeline: website scraping, FAQ ingestion, and Google Drive integration as source types; chunking with sentence boundary detection; embedding generation; tenant-scoped storage in Postgres with pgvector; cosine similarity retrieval with a minimum threshold; and content refresh tracking with stale detection.

What happens when realtime AI fails?

The system falls back to Gather (turn-based) mode without dropping the caller. If three WebSocket failures occur within five minutes for a tenant, a circuit breaker trips and routes subsequent calls directly to Gather. The system also has a four-level degradation system: full AI, fast AI, rule-based, and human transfer.

Can Kestrel Voice make outbound calls?

Yes. The outbound dialer supports a full call state machine with three AI participation modes: silent (transcribe only), assist (suggestions), and active (speaks on behalf of the user). Credits are only consumed when the AI is actively engaged.

Does Kestrel Voice support video?

Yes. Video consultation sessions are available via Daily.co integration, with session management, recording, and participant tracking.

Can Kestrel Voice connect to my calendar or CRM?

Yes. Google Calendar sync is built in for booking, rescheduling, and cancellation. CRM integration supports lead creation and management in Postgres, with extensibility for Salesforce and HubSpot.

Does Kestrel Voice support multiple businesses?

Yes. The platform is multi-tenant with defense-in-depth isolation: ASGI middleware, parameter-scoped queries, Postgres row-level security, and server-side auth extraction. Each tenant has independent configuration, RAG, prompts, and call data.

How does Kestrel Voice handle spam and fraud?

Multi-layer protection: score-based spam enforcement with reputation memory, tenant-configurable sensitivity levels, whitelisting for false-positive protection, country-specific call rate limiting, and IRSF (International Revenue Share Fraud) prevention.

Is call recording compliant?

The system includes TCPA-compliant recording consent with state-aware prompting. Two-party consent states receive a consent prompt before recording begins. SMS handles STOP, HELP, and START keywords. Compliance depends on the use case and jurisdiction — the platform provides controls but does not make a business compliant by default.

Does Kestrel Voice learn from calls?

Yes. A learning pipeline records fast-path pattern matches, promotes recurring patterns above a threshold to active fast actions, and provides a time-delayed review window where tenants can reject promotions before they activate. Guardrails include intent whitelisting, blocked phrases, and length limits. Industry template FAQs provide a complementary mechanism for pre-written answers.

What is the AI copilot?

The copilot is a live call assistant that provides real-time notes, suggested responses, chat Q&A, and action item extraction to a human operator during an active call. It uses a cost-efficient model and integrates RAG for knowledge lookups. All copilot events are persisted for post-call review.

Can I use Kestrel Voice from my browser?

Yes. WebRTC support enables browser-based calling via Twilio Client, with a prebind session that ensures deterministic tenant resolution before the call connects.

What technology stack does Kestrel Voice use?

FastAPI (async voice runtime), OpenAI (realtime API, embeddings, copilot — all API-based, no local models), Twilio (telephony), Supabase and Postgres with pgvector (data, RAG, auth, RLS), Modal (serverless deployment), and Redis (optional session store with in-memory fallback). Next.js powers the frontend dashboard and RAG management.

Related Resources

AI Voice Agent Architecture Checklist

Get a practical checklist covering call flow design, deterministic vs generative answer mapping, RAG pipeline setup, tool verification, degradation planning, security controls, and evidence requirements for production AI voice agents.

For the failure modes this architecture prevents, read Why AI Voice Agents Fail in Production. For the RAG governance framework behind the retrieval pipeline, read Secure Enterprise RAG Architecture. For the full AI compliance stack — NIST, ISO, SOC 2 — read How to Secure and Govern AI.

Ready to Deploy a Production-Grade AI Voice Agent?

Kestrel Voice handles the call. HAIEC handles the assurance. Start with a self-service setup or schedule a managed deployment consultation.

AI Voice Agent Architecture:How Kestrel Voice Works