Overview
The three call modes, and how to choose between them.
A call is a room with something in it. What varies is how much of the pipeline you are asking for.
The three modes#
mode is a parameter on start_call. It decides what the
worker publishes, not what the caller may do.
| Mode | The agent publishes | Needs | Use it when |
|---|---|---|---|
voice | audio only | nothing extra | the default, and the right answer most of the time |
avatar | audio and a rendered video track | avatar_id | the person should see a face |
phone | audio only, by nature | a SIP trunk, which this kit does not yet include | — |
Two honest caveats:
phoneis accepted and not implemented here. The tool schema lists it, and there is no telephony integration in this repository. A call placed today is a web or app call. Treat the enum value as reserved.avatarwithout anavatar_idis refused at the call, not at the render. The error nameslist_avatarsand suggestsvoice— failing early is deliberate, because the alternative is a call that connects and then has no face for reasons nobody can see.
The pipeline underneath#
Independent of mode, the agent's own loop is one of two shapes:
- cascaded — speech to text, a language model, text to speech. Slower, and it keeps a text seam, which is where tool calls and retrieval hook in.
- realtime — a speech-to-speech model. Lower latency, and there is no seam to hook.
Choose by whether the agent needs tools mid-conversation. If it does, the seam is not a cost, it is the feature.
What latency you should expect#
About 1.6 seconds is the realistic floor when the worker is far from the model provider, measured rather than estimated. Sub-second is not a tuning problem — it comes from co-locating compute, which means putting the worker near the model, not adjusting parameters.
Everything the pipeline exposes, with its current defaults, is returned live by
describe_call_config. Prefer asking it over trusting a
table in a document, including this one.
Where to go next#
- Starting a call — the one call that does everything
- Access tokens and grants — what a participant may do
- Live state and history — two questions, opposite failure modes