Low-Latency AI Voice Pipeline: Modal, FastAPI & Postgres
The hard part of a production voice agent is not connecting speech recognition to an LLM. It is deciding what must happen immediately, what can wait for reasoning, where session state belongs, and what the system should do when one dependency stops behaving.
Updated August 8, 2026
Most AI voice tutorials end at approximately the same place.
Audio comes in.
Speech becomes text.
The text goes to a language model.
The model returns an answer.
Text-to-speech turns the answer back into audio.
It looks something like this:
Caller
↓
Speech-to-Text
↓
LLM
↓
Text-to-Speech
↓
Caller
That is enough to prove the concept.
It is not enough to run a production voice system.
The problems begin when several things happen at once.
A caller interrupts the assistant while it is speaking.
Another customer calls while the first model request is still running.
One tenant has a calendar integration and another does not.
The model provider slows down.
The database is temporarily unreachable.
An appointment API returns successfully but the acknowledgement is lost.
A configuration changes while a call is already active.
A new container starts cold.
Someone asks a question that can be answered in 30 milliseconds, but the architecture sends it through a multi-second reasoning path anyway.
The naive pipeline has only one route:
everything
↓
LLM
A production architecture needs considerably more judgment.
The architecture I have come to prefer separates the real-time media path, the orchestration path, and the persistent control plane. It also treats graceful degradation as part of the normal design rather than something added after the first outage.
The combination of FastAPI, Modal and Postgres/Supabase can work well for this kind of system.
But they should not be treated as three interchangeable pieces of a stack.
FastAPI is the application and ASGI framework.
Modal is the execution and scaling environment.
Postgres is the durable system of record.
Putting each one in the right part of the architecture matters more than simply having all three.
The architecture I would not build
A tempting first implementation looks like this:
Incoming audio
↓
FastAPI WebSocket
↓
Write audio event to Postgres
↓
Load tenant configuration from Postgres
↓
Send transcript to LLM
↓
Write LLM response to Postgres
↓
Read latest state from Postgres
↓
Call booking tool
↓
Write result
↓
Generate speech
It is logically clean.
It is operationally expensive.
Every additional round trip sits inside the user's conversational turn.
That means database latency, model latency, tool latency and network latency all begin accumulating in series.
A better mental model is:
REAL-TIME MEDIA PLANE
Caller
↓
Audio ingress
↓
Turn / speech detection
↓
Streaming transcription
↓
In-memory session orchestrator
↓
Routing decision
↓
Response generation
↓
Speech synthesis
↓
Caller
CONTROL PLANE
Authentication
Tenant resolution
Agent configuration
Tool permissions
Fallback policy
Rate limits
Session policy
PERSISTENCE PLANE
Postgres / Supabase
Tenant configuration
Call/session record
Configuration versions
Tool execution ledger
Transcript/event history
Bookings / outcomes
Audit metadata
The key idea is simple:
Postgres should know what happened. It does not need to sit between every two milliseconds of what is happening.
That distinction alone can remove a surprising amount of unnecessary latency.
What FastAPI should do in this architecture
FastAPI is well suited to the orchestration boundary because it sits on ASGI and supports both normal HTTP endpoints and WebSockets.
FastAPI's own documentation supports WebSocket routes directly. (fastapi.tiangolo.com)
For a voice platform, I would typically use FastAPI for several different interface types:
HTTP webhooks
|
+-- call started
+-- call completed
+-- recording available
+-- provider status
+-- booking callback
WebSockets
|
+-- media stream
+-- browser voice session
+-- live transcript / state
Normal APIs
|
+-- configuration
+-- session history
+-- tools
+-- admin actions
The important word here is orchestration.
FastAPI should not become the place where every expensive computation is performed synchronously.
If your WebSocket event loop is also doing CPU-heavy processing, blocking model SDK calls and large database operations, the gateway that is supposed to coordinate the conversation can become the bottleneck itself.
That is the same architectural problem I discussed in the context of FastAPI WebSocket 1006 failures: slow upstream work and an event loop that has actually stopped making progress are different failure classes.
Where Modal fits
Modal changes the deployment model.
It can serve a FastAPI application directly as an ASGI app:
@modal.asgi_app()
def fastapi_app():
return web_app
Modal's current documentation specifically supports FastAPI and other ASGI applications, and its ASGI-hosted functions support WebSockets. (modal.com)
That matters for AI voice because the same environment can also give you access to:
- autoscaled compute;
- CPU and GPU functions;
- secrets;
- container lifecycle hooks;
- concurrency controls;
- warm containers;
- region selection;
- independently scaled functions.
But "serverless" does not mean "latency disappears."
There are several Modal behaviors I would account for explicitly before putting a voice path into production.
Cold starts are part of the latency budget
When no ready container exists for a Modal Function, Modal starts one.
Modal calls this a cold start and identifies two main sources of additional latency:
- waiting for a container to become available;
- initialization work inside the newly started container.
For a batch job, that can be irrelevant.
For voice, the user is already waiting.
Current Modal exposes three particularly useful controls:
min_containers
buffer_containers
scaledown_window
min_containers maintains a warm baseline.
buffer_containers gives the autoscaler spare capacity while traffic is active.
scaledown_window controls how long idle containers remain alive before scaling down; the current default maximum idle period is 60 seconds if you do not change it. (modal.com)
There is a direct tradeoff:
more warm capacity
↓
lower cold-start probability
↓
higher idle cost
There is no correct universal number.
For a production voice system, I would derive it from call-arrival patterns and P95/P99 startup latency rather than setting min_containers=10 because somebody's tutorial did.
Modal WebSockets have one non-obvious consequence
Modal documents a WebSocket connection as a single Function call for the lifetime of that connection. It also supports concurrent inputs so multiple WebSockets can be handled by the same container. (modal.com)
That has an operational consequence.
Modal Functions currently have a default execution timeout of 300 seconds, although the timeout can be configured from one second up to 24 hours. (modal.com)
So if your WebSocket represents a phone call that may reasonably last longer than five minutes, I would not leave the Function timeout implicit.
I would configure it deliberately.
The exact value depends on your call/session policy.
A 30-minute maximum conversation and a 24-hour browser connection are very different products.
A production-shaped Modal + FastAPI entry point
A simplified version could look like this:
import modal
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
app = modal.App('voice-orchestrator')
image = (
modal.Image.debian_slim()
.pip_install('fastapi[standard]')
)
web_app = FastAPI()
@web_app.websocket('/v1/voice')
async def voice_session(websocket: WebSocket):
await websocket.accept()
try:
while True:
event = await websocket.receive()
# Decode and route the event.
# Keep expensive/blocking work off the ASGI loop.
except WebSocketDisconnect:
pass
@app.function(
image=image,
# Illustrative values only.
# Benchmark these for your workload.
min_containers=1,
buffer_containers=1,
scaledown_window=300,
# Explicitly choose a session lifetime appropriate
# for your product rather than depending on defaults.
timeout=3600,
)
@modal.concurrent(
max_inputs=20,
target_inputs=10,
)
@modal.asgi_app()
def api():
return web_app
The numbers above are examples, not production recommendations.
20 simultaneous WebSocket sessions may be trivial for one workload and disastrous for another.
Modal's concurrency documentation makes the underlying behavior clear: individual containers can handle multiple inputs concurrently, while demand beyond the configured level causes Modal to scale additional containers. target_inputs can also influence when that scale-out begins so traffic is less likely to wait for cold containers. (modal.com)
Benchmark the work that actually happens inside your connection.
Geography matters more than many voice-agent diagrams admit
This is easy to overlook.
Suppose:
Caller Dallas
Modal runtime Oregon
Postgres Virginia
LLM another region
TTS another region
No individual component may be "slow."
The architecture has simply created several cross-country round trips inside each conversational turn.
Modal currently routes Function inputs through us-east by default, although it supports compute-region selection and now has regional routing in beta for selected routing regions. Modal specifically calls out latency-sensitive applications and proximity to external databases as reasons to control region placement. (modal.com)
Supabase also deploys a project into a primary region and recommends choosing a region close to users for performance. (supabase.com)
For voice, I would draw the physical architecture, not only the logical one.
USER REGION
|
v
MEDIA INGRESS
|
v
ORCHESTRATOR REGION
|
+------ POSTGRES REGION
|
+------ STT REGION
|
+------ LLM REGION
|
+------ TTS REGION
Then measure the actual network legs.
Model benchmarks do not include the latency you created by placing the system three regions apart.
Postgres should be the memory of the system, not its nervous system
Postgres is excellent at things a voice application needs:
- tenant configuration;
- durable call records;
- tool execution history;
- bookings;
- agent versions;
- permissions;
- transcripts;
- audit events;
- analytics.
It is much less attractive as a per-audio-frame message bus.
I do not want this:
audio frame
↓
INSERT
↓
audio frame
↓
INSERT
↓
token
↓
INSERT
inside the live turn.
Instead:
Real-time state
=
in-memory session state
Durable state
=
Postgres
Then persist at useful boundaries:
session started
turn completed
tool requested
tool succeeded / failed
booking confirmed
fallback activated
session ended
You can still stream operational state elsewhere when required.
Just do not make the database a mandatory hop for every audio or token event unless you have a specific reason.
Supabase changes the Postgres operational model, but not the principle
Supabase gives each project a full Postgres database and layers products such as Auth, Realtime and Storage around it. (supabase.com)
For backend connections, Supabase currently recommends different connection modes depending on workload:
- direct connections for long-lived backend clients where connectivity permits;
- session-mode pooling for persistent backends on IPv4-only networks;
- transaction-mode pooling for temporary/serverless-style connections;
- dedicated transaction pooling as another option on applicable plans.
That distinction matters with an autoscaling Modal application.
A small number of warm, long-lived containers behaves differently from hundreds of containers appearing and disappearing during a spike.
Do not let every new container create an uncontrolled collection of Postgres connections.
Capacity-plan the database as part of the autoscaling design.
The multi-tenant problem starts before the first query
In a SaaS voice system, a call does not simply belong to "the application."
It belongs to a tenant.
So the very beginning of the session should resolve:
connection
↓
trusted channel / number / token
↓
organization
↓
voice agent
↓
approved configuration version
↓
tool permissions
I do not want arbitrary downstream code repeatedly accepting:
organization_id = request.organization_id
and trusting it because the caller supplied one.
The tenant boundary should be established from authenticated or otherwise trusted context.
Then it should follow the session.
Put tenant identity on the durable records
A minimal relational model might include:
organizations
organization_members
voice_agents
organization_id
voice_agent_versions
organization_id
phone_channels
organization_id
call_sessions
organization_id
voice_agent_version_id
conversation_turns
organization_id
call_session_id
tool_executions
organization_id
call_session_id
bookings
organization_id
For user-facing access, Supabase's Postgres Row Level Security can enforce authorization directly at the database row boundary.
Supabase recommends enabling RLS on exposed schemas and describes policies as effectively adding authorization predicates to every query. (supabase.com)
A simplified membership policy could look like:
create policy 'members can read organization calls'
on call_sessions
for select
to authenticated
using (
exists (
select 1
from organization_members m
where m.organization_id = call_sessions.organization_id
and m.user_id = auth.uid()
)
);
The exact policy needs to match your schema and indexes.
The important point is that the tenant boundary is not merely a React filter.
For a deeper treatment of multi-tenant data isolation patterns, see implementing RAG row-level security for multi-tenant AI.
One Supabase detail can undo an otherwise good tenancy design
Server-side Supabase credentials deserve special attention.
Supabase explicitly documents that a client using the service-role authorization bypasses Row Level Security. Its newer secret-key model serves the same category of trusted backend use and must not be exposed to customers. (supabase.com)
So:
RLS enabled
does not automatically mean:
every backend query is tenant-isolated
If your trusted voice backend operates with a credential that bypasses RLS, your application code now owns tenant authorization for those queries.
That is a major architectural boundary.
I prefer either:
- scoped database access where RLS remains meaningful; or
- an explicit trusted-backend authorization layer that validates the resolved tenant before every privileged operation.
What I would not do is assume the word "RLS" somewhere in the architecture means the multi-tenant problem is solved.
Snapshot the voice-agent configuration when the call begins
This is a small design choice that becomes extremely important later.
consider a call begins at 10:02.
At 10:04 an administrator changes:
system prompt
business hours
transfer policy
model
temperature
booking rules
Should the call that started at 10:02 immediately inherit all of those changes?
Sometimes yes.
Usually I would rather have a consistent session.
At admission:
call begins
↓
resolve tenant
↓
resolve active agent version
↓
snapshot configuration
↓
session uses version 184
Persist:
call_session.voice_agent_version_id = 184
Now when someone asks:
Why did the assistant say this?
you can answer with the configuration that actually governed that call.
Not whatever happens to be current today.
This is as useful for debugging as it is for governance.
The three-layer routing pipeline
This is the part of the architecture that made the largest conceptual difference for me.
A voice turn should not automatically enter the most expensive and least predictable reasoning path.
I think of the router in three layers.
USER EVENT
|
v
+---------------------+
| 1. SESSION CONTROL |
+----------+----------+
|
not handled
v
+---------------------+
| 2. BOUNDED FAST PATH|
+----------+----------+
|
not handled
v
+---------------------+
| 3. LLM REASONING |
+---------------------+
This is an architecture pattern, not a feature supplied automatically by FastAPI, Modal or Postgres.
Each layer exists for a different reason.
Layer 1: deterministic session controls
Some events should never wait for an LLM.
Examples include:
WebSocket disconnected
call ended
DTMF event
hard timeout reached
authentication failed
session cancelled
explicit tool acknowledgement
provider status event
rate limit exceeded
These are state-machine events.
They belong in deterministic application logic.
For example:
if event.type == 'disconnect':
await close_session()
return
if event.type == 'dtmf' and event.digit == '0':
await request_human_transfer()
return
if session.deadline_expired():
await initiate_timeout_flow()
return
One qualification matters here.
A caller saying:
"Stop."
is not automatically deterministic if you are relying on speech recognition to detect the word.
STT itself may be probabilistic.
The deterministic piece starts after the event has been classified into a trusted session command.
That sounds pedantic until a voice agent hangs up because it hallucinated the word "goodbye."
Layer 2: the bounded fast path
A large fraction of customer interactions do not require general reasoning.
Examples might include:
What time do you close?
What is the address?
Do you service this ZIP code?
Do you have my appointment?
Can I leave a callback number?
What is the status of my booking?
Some of those can be answered from authoritative structured data.
A good fast path might use:
known command
↓
validated lookup
↓
templated response
or:
retrieval
↓
high-confidence approved content
↓
short response
instead of:
construct massive context
↓
send to strongest reasoning model
↓
wait
The goal is not to avoid LLMs.
It is to avoid paying reasoning latency for a problem that does not require reasoning.
Fast path does not mean "hard-code every conversation"
There is an easy mistake to make here.
A developer hears "deterministic fast path" and creates:
if 'hours' in transcript:
return BUSINESS_HOURS
That becomes fragile almost immediately.
Natural language still needs interpretation.
The architecture should distinguish:
bounded answer source
from:
perfectly deterministic intent recognition
The source of truth can be deterministic while the route into it uses a classifier, a small model or semantic matching.
For higher-impact actions, require stronger evidence.
For example:
"What time do you close?"
↓
low-risk FAQ fast path
is very different from:
"Cancel every appointment I have."
↓
identity + explicit confirmation
↓
authorized tool
Latency optimization must not erase authorization.
Layer 3: real-time LLM reasoning
The general reasoning path is where the LLM earns its cost.
This is the route for turns requiring:
- synthesis across several facts;
- ambiguous conversational context;
- multi-step planning;
- tool selection;
- qualification;
- complex natural-language explanation.
The LLM should receive enough context to work.
Not every object the company has ever stored.
A good request envelope might contain:
tenant policy
agent configuration version
conversation state
bounded relevant history
retrieved knowledge
allowed tools
authorization context
deadline
fallback policy
Notice what is in that list:
deadline
The reasoning engine does not own unlimited conversational time simply because the WebSocket is still open.
A routing function should return more than an answer
Conceptually:
from dataclasses import dataclass
from enum import Enum
class Route(Enum):
CONTROL = 'control'
FAST = 'fast'
REASONING = 'reasoning'
@dataclass
class RouteDecision:
route: Route
reason: str
confidence: float | None = None
async def choose_route(event, session) -> RouteDecision:
control = match_session_control(event, session)
if control:
return RouteDecision(
route=Route.CONTROL,
reason=control,
)
fast_match = await find_bounded_answer(
event=event,
session=session,
)
if fast_match.is_acceptable:
return RouteDecision(
route=Route.FAST,
reason=fast_match.source,
confidence=fast_match.confidence,
)
return RouteDecision(
route=Route.REASONING,
reason='requires_general_reasoning',
)
The important thing is not this exact Python.
It is making the route observable.
Later you can measure:
What percentage of turns use the fast path?
How often does the fast path fall back?
Which route produces poor outcomes?
How much latency does each route add?
Without an explicit route decision, "the AI was slow" is almost impossible to decompose.
Do not put Postgres in the routing hot loop unless the query is necessary
Suppose the fast path requires:
business opening hours
Do you need to query Postgres every time the caller asks?
Probably not.
You can load the tenant configuration when the session begins:
session started
↓
load immutable config snapshot
↓
keep in memory
↓
serve repeated bounded reads
For data that genuinely changes during the call, query the authoritative source.
The rule I use is:
cache configuration; verify transactions.
Business hours can be cached.
Whether a specific appointment slot is still available should probably come from the scheduling authority.
Those are different consistency requirements.
The most dangerous latency optimization is caching the wrong truth
consider this:
10:00:00
availability = 2 PM
10:00:05
another customer books 2 PM
10:00:08
voice agent uses cached availability
10:00:10
"You're booked for 2 PM."
Fast.
Wrong.
For transactional actions, the authoritative tool must win.
The LLM can suggest:
2 PM looks available
but your system should only say:
You're booked for 2 PM
after the authoritative booking operation succeeds.
That distinction is not about latency.
It is about truth.
Side effects need idempotency
Voice systems are full of retries.
Networks fail.
Providers retry webhooks.
Users repeat themselves.
A model retries a tool.
The booking succeeds, but the acknowledgement times out.
Now the orchestrator thinks it failed.
If you blindly retry:
book 2 PM
you may create two appointments.
For side-effecting tools, I want:
session_id
turn_id
tool_name
idempotency_key
request_hash
status
external_reference
persisted in a durable execution ledger.
Conceptually:
create table tool_executions (
id uuid primary key,
organization_id uuid not null,
call_session_id uuid not null,
idempotency_key text not null,
tool_name text not null,
status text not null,
external_reference text,
created_at timestamptz not null default now(),
unique (organization_id, idempotency_key)
);
Now a retry can ask:
Did this logical operation already execute?
before reproducing the side effect.
Graceful degradation without idempotency can create a more serious incident than the one you were trying to survive.
Four levels of graceful degradation
A production voice system should have an answer to:
What happens when the LLM is unavailable?
"Return HTTP 500" is not a conversational strategy.
I prefer explicit degraded modes.
Normal operation might look like:
NORMAL
Primary reasoning
RAG
Tools
Full conversation policy
If that path fails, move through controlled levels.
Degradation Level 1: fail over the reasoning provider
The primary model is unavailable or exceeds its deadline.
If your risk, privacy and commercial requirements permit it:
Primary model
X
↓
Secondary approved model/provider
The important word is approved.
A fallback provider should not suddenly receive data your organization deliberately prohibited from leaving another boundary.
The secondary path needs its own:
model configuration
tool policy
data policy
timeout
evaluation history
Provider redundancy is useful only if the backup path is genuinely operational.
Degradation Level 2: remove free-form reasoning
If general model generation is not trustworthy or available:
Reasoning unavailable
↓
Bounded knowledge + authoritative tools
The assistant may still be able to:
- report business hours;
- provide a verified address;
- check an appointment;
- create a callback request;
- route a call;
- perform an explicitly authorized deterministic workflow.
It should stop pretending it can handle open-ended questions.
A user message can be honest:
"I'm having trouble with some of my answering capabilities right now, but I can still check your appointment or take a callback request."
That is better than generating confident nonsense through a damaged reasoning path.
Degradation Level 3: capture and route
If even the bounded answer path is unavailable:
Capture:
name
number
reason
urgency
preferred callback time
Then:
transfer
OR
queue for callback
OR
create service request
This is not a glamorous AI experience.
It can still preserve the business outcome.
For many service businesses:
accurately capture lead
is preferable to:
keep talking until something breaks
Degradation Level 4: fail safely
Sometimes there is no trustworthy action remaining.
Database unavailable.
Tool authorization cannot be verified.
Telephony state is inconsistent.
Session integrity is uncertain.
At that point:
do not guess
do not retry destructive tools
do not fabricate confirmation
Escalate, offer a safe alternate contact path, or terminate the automated interaction cleanly.
A production system needs the ability to say:
"I'm unable to complete that action right now."
Failure disclosure is part of reliability.
The degradation ladder should be a state machine
I would not scatter fallback behavior across dozens of exception handlers.
Represent it:
NORMAL
|
primary path unavailable
v
DEGRADED_1
secondary model
|
unavailable
v
DEGRADED_2
bounded answers / tools
|
unavailable
v
DEGRADED_3
capture + route
|
unsafe/unavailable
v
DEGRADED_4
fail safely
Persist the transitions:
session_id
from_mode
to_mode
reason
dependency
timestamp
Now a reliability report can answer:
How often do callers experience degraded mode?
instead of simply:
Uptime was 99.98%.
Those are different measures.
Not every failure should trigger model failover
This is another place where broad except Exception logic becomes dangerous.
If the primary LLM returns:
401 Unauthorized
do not automatically send the user's content to another provider.
If the request was blocked because:
tenant is suspended
a model failover is wrong.
If the tool failed because:
authorization denied
a retry against another tool is wrong.
Fallback logic should understand failure classes.
For example:
TIMEOUT
-> eligible for provider failover
PROVIDER_5XX
-> perhaps eligible
RATE_LIMIT
-> eligible if fallback approved
AUTH_FAILURE
-> do not reroute
POLICY_BLOCK
-> do not reroute
INVALID_TOOL_REQUEST
-> repair or reject
TENANT_NOT_FOUND
-> terminate
DATA_BOUNDARY_VIOLATION
-> fail closed
Graceful degradation should preserve policy.
Not bypass it.
The latency budget should be measured per stage
One metric matters more than "LLM latency":
How long from the user's end of speech until useful response audio begins?
That is the conversational latency the user experiences.
Break it down:
User stops speaking
|
v
Turn detection
|
v
STT finalization
|
v
Routing decision
|
v
Knowledge/tool/model
|
v
First response text
|
v
TTS first audio
|
v
User hears response
Instrument each boundary.
I would record at least:
speech_end_at
transcript_ready_at
route_selected_at
llm_requested_at
llm_first_event_at
tool_started_at
tool_completed_at
tts_requested_at
tts_first_audio_at
response_playback_at
Now:
"The model felt slow."
can turn into:
STT 180 ms
routing 22 ms
DB 310 ms
LLM 890 ms
tool 2300 ms
TTS 190 ms
The slowest thing may not be the model.
Measure P95 and P99, not just the demo
A prototype tells you:
this worked once
Production needs to tell you:
this usually works within the expected latency
I would monitor:
- end-of-speech to first response audio;
- model time to first event/token;
- TTS time to first audio;
- tool latency;
- database query latency;
- event-loop lag;
- cold-start frequency and duration;
- routing distribution;
- fallback activation rate;
- abandoned generation rate;
- WebSocket abnormal-close rate;
- transfer success;
- booking success;
- duplicate side-effect attempts.
For multi-tenant systems I would add one metric with a very different target:
cross-tenant data exposure
Target:
zero
A fast voice assistant with weak tenant isolation is not production-ready.
Cold-start latency needs to be visible as its own metric
Averages hide it.
consider:
95 calls:
700 ms response startup
5 calls:
4.5 sec response startup
Your average may still look respectable.
Those five callers experience a different product.
Log:
container_warm: true / false
or the closest runtime evidence you can derive, along with Modal task and region metadata.
Modal exposes runtime information such as MODAL_REGION, MODAL_TASK_ID, image ID and cloud provider in the container environment. (modal.com)
That gives you enough context to investigate whether a latency anomaly clusters around:
cold container
region
image revision
deployment
instead of guessing.
Keep model initialization out of the first user turn
Modal supports @modal.enter lifecycle handlers for one-time initialization when a container starts. It specifically calls out model loading and expensive package initialization as uses. (modal.com)
For an inference class:
@app.cls(...)
class ModelRuntime:
@modal.enter()
def load(self):
self.model = load_model()
@modal.method()
async def generate(self, request):
...
That still contributes to cold-start time when the container first appears.
But it avoids accidentally reloading the same model for every user turn.
For latency-critical workloads, combine:
container initialization
+
warm capacity
+
startup profiling
rather than optimizing only the inference call.
Separate the gateway from GPU inference when the workloads scale differently
The FastAPI gateway and the model do not necessarily belong in the same Modal Function.
They have different scaling characteristics.
The gateway may need:
many lightweight concurrent connections
while a local model may need:
few expensive GPU containers
That suggests:
FASTAPI / ASGI
lightweight orchestration
|
async remote call
|
v
GPU FUNCTION
inference / reasoning
Modal Functions scale independently, and its asynchronous API allows a coroutine to invoke remote functions without blocking the async caller. (modal.com)
This separation is especially useful if:
- you self-host a model;
- GPU capacity is expensive;
- media connections outnumber simultaneous generations;
- the inference layer needs a different concurrency policy.
Do not put a GPU behind every open socket if only a fraction of those sockets are actively generating.
The database and the gateway may need a different topology too
A Modal container can be ephemeral.
A Postgres connection is stateful.
Those facts can collide.
If each scale-up creates:
50 containers
x
20 DB connections
=
1,000 connections
the autoscaling tier has transferred its problem into Postgres.
Use a bounded application pool and choose Supabase's direct/pooler mode deliberately.
For bursty serverless-style connections, Supabase specifically recommends transaction pooling. For long-lived backend processes, direct or session-mode connections may make more sense depending on network constraints and connection budgets. (supabase.com)
Autoscaling compute and fixed-capacity databases have to be designed together.
Supabase Realtime can help dashboards, but I would not put it in the audio path
Suppose an operator dashboard needs to show:
call connected
AI speaking
appointment booked
transfer requested
call ended
Supabase Realtime can be useful there.
Supabase currently recommends Broadcast as the more scalable and secure approach for many realtime database-change scenarios, while Postgres Changes provides a simpler model with different scaling characteristics. (supabase.com)
That is a good control-plane use case.
I would not turn this into:
audio
↓
Postgres
↓
Realtime
↓
voice runtime
The operator's dashboard and the caller's audio stream have very different latency requirements.
Do not force them through the same path simply because both are "real-time."
Voice session state should have an owner
A call may have:
tenant
agent version
caller
conversation history
current speaker
active generation
current tool
pending confirmation
degradation level
transfer state
deadline
Where does that state live?
If the answer is:
some of it in Redis
some in Postgres
some in a Modal global
some in the WebSocket handler
some in the LLM messages
you will eventually get state disagreements.
For each state field, define:
authoritative owner
durability
lifetime
recovery behavior
For example:
| State | Owner | Durability |
|---|---|---|
| Active audio buffer | session runtime | ephemeral |
| Current speaking state | session runtime | ephemeral |
| Agent configuration version | Postgres to snapshot | durable |
| Booking | scheduling system | authoritative external |
| Tool attempt | Postgres ledger | durable |
| Transcript | session + durable sink | eventually durable |
| Degradation mode | runtime + event log | recoverable |
| Call outcome | Postgres | durable |
This is not glamorous architecture work.
It prevents extraordinarily confusing failures.
Barge-in is a state transition, not simply another prompt
A live voice user will interrupt.
If the system is speaking and new speech begins, the orchestration path may need to:
detect new user turn
cancel/stop TTS playback
possibly cancel generation
invalidate queued audio
preserve relevant context
begin listening
resume reasoning
If you treat the interrupt as:
append new text to chat history
you can end up playing stale audio after the user has already changed the conversation.
That is why I think of real-time voice more as a distributed state machine than a chatbot with a microphone attached.
Cancellation needs to cross every layer
Suppose the user interrupts while an expensive reasoning request is running.
You stop TTS.
Good.
But what happens to:
LLM generation?
tool execution?
database operation?
queued audio?
background task?
If they all continue, the user experiences one conversation while the backend continues another.
The correct cancellation boundary depends on whether the action is reversible.
For generation:
cancel if supported
For a tool that may have already committed:
do not assume cancellation means rollback
That is exactly why tool execution needs idempotency and durable state.
The voice stack should have two clocks
I like separating:
TURN DEADLINE
from:
SESSION LIFETIME
A call may last 20 minutes.
One reasoning turn should not automatically be permitted to consume those 20 minutes.
The turn has:
STT budget
routing budget
reasoning budget
tool budget
TTS budget
The session has:
overall call lifetime
idle policy
authentication lifetime
resource limits
If those clocks are not distinct, a stuck turn can effectively own the entire connection.
Avoid one global timeout
This:
TIMEOUT = 30
is attractive.
It is usually ambiguous.
Thirty seconds for what?
database query?
LLM generation?
tool call?
session idle?
WebSocket heartbeat?
HTTP request?
Modal Function?
Each timeout should answer a specific operational question.
For example:
MODEL_FIRST_EVENT_DEADLINE
TOOL_EXECUTION_DEADLINE
DATABASE_QUERY_DEADLINE
SESSION_IDLE_DEADLINE
CALL_MAX_DURATION
Once a timeout has a name, its fallback behavior becomes easier to design.
A production routing envelope
I would want every turn to carry something close to:
@dataclass
class TurnContext:
organization_id: str
session_id: str
agent_version_id: str
allowed_tools: set[str]
degradation_mode: str
turn_deadline: float
request_id: str
trace_id: str
Then:
reasoning
retrieval
tool calls
persistence
all receive the same context.
That avoids rediscovering tenant identity from untrusted parameters deep inside the call path.
It also makes logs traceable.
One trace ID should cross the whole call
A voice incident becomes painful when every service uses its own identifier.
I would propagate:
organization_id
call_session_id
turn_id
trace_id
across:
telephony/media provider
FastAPI
Modal Function
STT
LLM
tool execution
TTS
Postgres
A useful event:
{
"event": "reasoning_fallback",
"organization_id": "org_...",
"session_id": "call_...",
"turn_id": "turn_...",
"trace_id": "trace_...",
"from": "primary_llm",
"to": "bounded_fast_path",
"reason": "first_event_deadline_exceeded"
}
Now an incident can be reconstructed.
Without correlation, you are reading five log systems and hoping the timestamps line up.
Do not log everything simply because voice is hard to debug
A voice system handles unusually sensitive data.
Potentially:
phone numbers
names
addresses
appointment details
health information
financial details
raw audio
transcripts
credentials spoken accidentally
Observability is important.
Unlimited logging is not.
Separate:
operational metadata
from:
conversation content
and define retention independently.
Do not put full transcripts into every exception log simply because it is convenient.
Modal Secrets belong at the runtime boundary
Modal provides Secret objects specifically for injecting credentials into Functions as environment variables rather than hard-coding them into images or source. (modal.com)
For example:
@app.function(
secrets=[
modal.Secret.from_name('voice-production'),
],
)
def runtime():
...
I would still keep secrets scoped.
One enormous secret bundle containing:
database admin
telephony
LLM provider
calendar
CRM
payments
means every function receiving it has all of those capabilities.
Prefer the smallest credentials required by each execution surface.
Dev and production should not share one mutable runtime
Modal Environments isolate applications and resources such as Secrets between environments, making them suitable for dev/prod separation. (modal.com)
That matters for voice.
A developer testing:
transfer()
should not accidentally trigger the production transfer target because both deployments shared the same secrets and state.
Environment boundaries should include:
Modal app
Secrets
database project/schema
telephony configuration
model/provider keys
tool endpoints
where practical.
"Environment" should describe an actual isolation boundary.
Not a string in a log message.
The architecture should fail locally before it fails conversationally
Here is the principle I keep returning to.
A production voice agent has enough uncertainty already.
The control plane should remove uncertainty where it can.
Before a call begins, validate:
tenant exists
agent version exists
channel belongs to tenant
configuration is valid
required credentials exist
tool policy is loadable
fallback policy exists
Then admit the session.
Do not wait until minute four of a customer conversation to discover:
there is no calendar credential
if the system could have detected that before answering the call.
The four things I would benchmark before calling the stack production-ready
Not "Does the demo work?"
These four.
Normal latency
Under expected traffic:
How long does a normal conversational turn take?
Break it down by stage.
Burst behavior
When ten calls become fifty:
Does latency degrade gradually,
or does the system fall off a cliff?
Watch Modal queueing, cold starts, external APIs and database connections.
Dependency failure
Deliberately remove:
primary LLM
database
booking API
TTS
one at a time.
Does the system enter the intended degraded state?
Tenant isolation
Attempt:
tenant A session
↓
tenant B identifier
across every privileged route.
The expected result is not:
usually blocked
It is:
blocked
every time.
A CTO should care about degradation rate as much as uptime
consider two systems.
System A:
99.99% process uptime
but:
8% of calls spend >10 seconds waiting for response
4% silently enter broken booking path
System B:
99.9% process uptime
but:
dependency failures trigger a clear fallback
calls continue in capture-and-route mode
no false booking confirmations
Which system is more reliable?
Infrastructure uptime alone cannot answer.
For real-time AI I would put these on the executive dashboard:
successful conversation rate
P95 turn latency
P99 turn latency
fallback activation rate
fallback recovery rate
tool success rate
false confirmation incidents
transfer success
abandoned calls during AI wait
cold-start exposure
tenant-isolation failures
That is closer to the actual product.
Why the standard "LLM wrapper" architecture fails
A wrapper tutorial usually assumes:
input
↓
model
↓
output
A production voice system actually behaves more like:
SESSION STATE
|
v
AUDIO -> STT -> ROUTER -> POLICY -> REASONING
| | |
| | +-> LLM
| |
| +-> TOOLS
|
+-> FAST PATH
|
+-> CONTROL EVENTS
|
v
TTS
|
v
AUDIO
^ ^
TENANT CONTROL DURABLE EVIDENCE
Postgres Postgres
The LLM is one component.
That is the architectural shift.
Where Kestrel Voice changed my view of this
Working on Kestrel Voice made this distinction much more concrete for me.
The public product spans AI phone answering, appointment booking, RAG-powered answers, transcripts, call intelligence and integrations with external business systems. (kestrelvoice.com)
Once a voice system crosses that many boundaries, "the model answered correctly" is only one success condition.
The interaction can still fail because:
the booking tool did not confirm
the media connection disappeared
the wrong tenant configuration loaded
the fallback repeated a side effect
the caller interrupted stale TTS
the database could not persist the outcome
the secondary provider violated a policy boundary
That changed the way I think about voice architecture.
The most important component is not always the model.
It is often the orchestration layer deciding what is allowed to happen next.
For a deeper look at the product that shaped this thinking, see the KestrelVoice architecture detailed look.
The real three-tier performance strategy
If I had to reduce the entire architecture to one diagram, it would be this:
INCOMING TURN
|
v
+---------------------+
| CAN APP LOGIC |
| HANDLE IT SAFELY? |
+---------+----------+
|
yes | no
v | v
CONTROL ACTION| BOUNDED FAST PATH?
| |
| yes | no
| v | v
| FAST | GENERAL
| ANSWER | REASONING
| |
+----+---+
|
v
VERIFIED
ACTION
|
v
TTS
The performance insight is not:
use a smaller model.
It is:
do not invoke general reasoning when the problem has a safer, faster bounded path.
Frequently Asked Questions
Can Modal host a FastAPI voice agent?
Yes. Modal can serve ASGI applications through @modal.asgi_app(), including FastAPI, and Modal's ASGI-hosted Web Functions support WebSockets. Modal also supports input concurrency so a container can handle multiple simultaneous WebSocket connections. (modal.com)
Does Modal support WebSockets?
Yes. Modal currently supports the WebSocket protocol for @modal.asgi_app, @modal.wsgi_app and @modal.web_server functions. Modal documents each WebSocket as one Function call per connection. It currently limits individual WebSocket messages to 2 MiB and does not support RFC 8441 WebSockets over HTTP/2 or the permessage-deflate extension. (modal.com)
Should I use modal.fastapi_endpoint for a voice WebSocket?
For an application with actual WebSocket routes, use a FastAPI ASGI application exposed through @modal.asgi_app() rather than treating the connection like an ordinary fastapi_endpoint. Modal explicitly documents WebSocket support for its ASGI/web-server hosting modes. (modal.com)
How do I reduce Modal cold-start latency for a voice agent?
Measure first, then consider min_containers, buffer_containers, a longer scaledown_window, smaller initialization work and lifecycle initialization through @modal.enter. Modal documents these as controls for reducing queueing and container-initialization latency. They trade cost against latency. (modal.com)
What is Modal's default Function timeout?
Current Modal Functions have a default execution timeout of 300 seconds. Modal allows configured timeouts from one second to 24 hours. Since Modal documents each WebSocket connection as one Function call, long-lived voice or browser sessions should use an explicitly chosen Function timeout appropriate to the session policy. (modal.com)
Should Postgres store live voice audio frames?
Usually I would keep raw real-time media out of the transactional Postgres hot path. Postgres is better suited to durable session, configuration, transcript, tool and outcome records. Raw recordings or media are generally better handled through media/object storage designed for that payload, while Postgres stores references and metadata.
Why use Supabase/Postgres in an AI voice platform?
Postgres is useful as the durable control plane: organizations, users, voice-agent versions, sessions, tools, bookings, transcripts and audit metadata. Supabase adds Auth, Realtime and managed Postgres capabilities on top of that foundation. (supabase.com)
How should I handle multiple tenants in a voice-agent database?
Resolve tenant context from trusted authentication or channel information, include the organization identifier on tenant-owned data and enforce authorization at every access layer. With Supabase, RLS can provide row-level controls for user-facing access. Be aware that trusted service-role credentials can bypass RLS, so privileged backend paths require their own explicit authorization controls. (supabase.com)
Should every voice-agent query go through Postgres RLS?
That depends on the connection model. RLS is highly useful for user-scoped Supabase access. A trusted backend using credentials that bypass RLS must enforce tenant isolation itself or use a more constrained database role/access pattern. Do not assume RLS protects a query that is deliberately running with bypass privileges.
Should my AI voice agent send every question to the LLM?
No. Session-control events should be handled deterministically, and bounded questions can often use authoritative data or a constrained fast path. General LLM reasoning is most useful for turns that genuinely require synthesis, ambiguity resolution or planning.
What is a fast path in an AI voice architecture?
A fast path is a bounded response route that avoids full general reasoning when the answer can be obtained from an approved source or deterministic workflow. Examples include business hours, location information, appointment lookup or structured routing. The route may still use probabilistic language classification, so the confidence and risk of the action should determine when it is safe to use.
What should happen if the LLM provider goes down?
Design this before production. A useful degradation sequence is: approved secondary model/provider, bounded deterministic or retrieval/tool capabilities, capture-and-route mode, and finally safe escalation or termination when no trustworthy operation remains.
Should I automatically retry failed booking tools?
Not blindly. A booking may have succeeded even if the acknowledgement failed. Side-effecting operations should use idempotency keys and a durable execution ledger so retries can determine whether the logical operation already occurred.
How should I measure AI voice latency?
Measure the user-facing turn from end of speech to first useful response audio, then decompose it into speech recognition, routing, retrieval/database, LLM time to first event, tool execution and TTS time to first audio. Track P95 and P99 as well as averages.
Does Supabase Realtime belong in the live audio path?
I would generally use it for control-plane experiences such as dashboards, call-state notifications or database-driven updates rather than as the audio transport itself. Supabase currently recommends Broadcast for scalable database-change notification use cases, while Postgres Changes is simpler but has different scaling characteristics. (supabase.com)
Should Modal and Supabase run in the same region?
For latency-sensitive applications, keeping compute near users and data can remove avoidable network delay. Modal supports region selection and beta regional request routing, and Supabase offers multiple database deployment regions. Benchmark the actual network path rather than assuming geography is negligible. (modal.com) (supabase.com)
Production Checklist: Modal + FastAPI + Postgres AI Voice
Before calling the architecture production-ready:
[ ] Separate media, orchestration and persistence paths
[ ] Host WebSocket routes through an ASGI application
[ ] Keep blocking work off the FastAPI event loop
[ ] Measure Modal cold-start latency
[ ] Configure warm capacity from traffic data
[ ] Configure WebSocket Function timeout deliberately
[ ] Benchmark per-container WebSocket concurrency
[ ] Align compute, database and model geography
[ ] Bound Postgres connection pools
[ ] Resolve tenant before loading session configuration
[ ] Test tenant isolation across every privileged endpoint
[ ] Understand which backend credentials bypass RLS
[ ] Snapshot the agent configuration version at session start
[ ] Keep high-frequency audio/token state out of Postgres hot loops
[ ] Handle session controls before general reasoning
[ ] Create a bounded fast path for authoritative answers
[ ] Reserve general LLM reasoning for turns that need it
[ ] Verify transactional actions before confirming them to callers
[ ] Make side-effecting tools idempotent
[ ] Give each turn an explicit deadline
[ ] Give the session a separate lifetime
[ ] Implement an approved provider failover path
[ ] Implement bounded degraded operation
[ ] Implement capture-and-route fallback
[ ] Implement a fail-safe state
[ ] Persist degradation transitions and reasons
[ ] Propagate one trace ID across the full interaction
[ ] Measure end-of-speech to first response audio
[ ] Measure P95/P99, not just averages
[ ] Load-test concurrent calls
[ ] Chaos-test LLM, DB, tool and TTS failures
[ ] Verify cancellation and barge-in behavior
A low-latency voice system is not created by finding the fastest model and putting a WebSocket in front of it.
It comes from removing unnecessary work from the critical path.
Keep session state close to the live connection.
Keep durable truth in Postgres.
Keep tenant boundaries explicit.
Handle deterministic controls before reasoning.
Use the fast path when the answer is bounded.
Use the LLM when reasoning is genuinely required.
Verify side effects before telling a caller they happened.
And when the intelligence layer fails, degrade into something smaller and trustworthy rather than something equally conversational but less reliable.
That is the part most voice-agent demos do not show.
It is also the part that decides whether the architecture still works after the demo traffic becomes real traffic.
Continue Reading
Get new articles in your inbox
Occasional emails when I publish something worth reading. Unsubscribe anytime.
Subodh KC
AI Advisor & AI Systems Architect. Former Sr. Program Manager, HP Inc. Founder of HAIEC - Holistic AI Ethics & Compliance. Builds production AI systems from startups to global enterprise.