Phone assistants that take calls: how they work

How is a phone assistant different from an IVR?

A classic IVR plays recorded prompts and follows a fixed menu: press 1 for billing, press 2 for support. The caller adapts to the machine. A phone assistant works the other way round: the caller speaks naturally, the assistant understands the request, asks follow-up questions and acts on it.

What matters to a company is not the voice itself. It is the actions. A useful phone assistant can look up an incident, check an order or open a ticket while the caller is on the line. That needs the same identity checks, policies and audit trail as any other path into your systems.

How does a phone assistant turn speech into action?

Every phone assistant runs the same loop, several times per call. Each step adds delay, and each step can fail in its own way, so it helps to name them.

• Capture: audio arrives from the browser microphone or from the phone network.
• Speech-to-text and end-of-turn detection: the audio becomes a transcript, and the system decides when the caller has finished speaking.
• Reasoning: the assistant reads the transcript with the conversation so far, and decides whether to answer, ask, or call a tool.
• Tools and policy: any tool call is checked against the caller’s identity and your organisation’s policies, and may pause for human approval.
• Text shaping: the reply is cleaned for speech (no Markdown, no code, no bullet symbols) and cut into sentences as it streams.
• Text-to-speech and playback: each sentence is synthesised and played while the next one is still being generated.

Browser voice vs phone calls: what changes?

Talking in a web app and calling a phone number use the same brain, but the edges differ. In the browser the person is usually signed in, so you know who is speaking. On a phone call you only have a caller ID, which is not proof of identity.

Audio quality differs too. The phone network carries narrowband audio, typically sampled at 8 kHz, so speech recognition has less to work with than it does with a browser microphone. That is one more reason to confirm names, numbers and amounts out loud.

Why do barge-in and latency make or break a call?

People are fast at turn-taking. A study of ten languages found that speakers everywhere avoid talking over each other and keep silence between turns short, with language averages differing only within about 250 ms. An assistant that pauses for two seconds before every answer feels broken, even when the answer is right.

The practical fix is streaming at every step: start speaking the first complete sentence while the rest is still being written. Filler phrases help when a tool is slow. A short “One moment, I’m checking that” is better than silence, but it should be capped per turn so the assistant does not ramble.

Barge-in is the caller interrupting. Two things must happen at once: the audio already queued for playback is cleared (with a text relay such as Telnyx Conversation Relay, the provider stops its own playback and tells the application about the interruption), and the current turn is abandoned. The subtle part is timing. Cutting a turn while a tool is half-way through a write can leave a ticket or an email in an unknown state, so the safe pattern is to mark the turn as stale and abort it as soon as the tool returns.

How do you connect an assistant to a real phone number?

The phone side runs through a telephony provider. For an inbound call, the provider notifies your webhook and your server tells it how to connect the call: either a bidirectional audio stream over a WebSocket, or a text relay where the provider runs speech recognition and text-to-speech and exchanges only text with your assistant. For an outbound call, your server asks the provider to dial and connects the call the same way once it is answered.

Botify does this with Telnyx. Telnyx carries the call and handles speech recognition and text-to-speech, and Botify exchanges text with it over Telnyx Conversation Relay. Calls can be inbound or outbound on a number Botify provisions, and outbound calls can also show your own number once it is verified as a caller ID. Each turn runs on the same path as chat: the same tools, the same organisation policies, the same approvals and the same audit trail.

• Verify every provider webhook signature and reject anything unsigned.
• Bind the real-time connection (audio stream or text relay) to the call with a single-use token, so a stray or replayed connection cannot attach to someone else’s session.
• Detect answering machines on outbound calls and decide up front: hang up, or leave a short message.
• Keep voicemail messages generic. A voicemail box is not an authenticated channel, so say who called, not why.
• Set a maximum call duration and a concurrency cap per assistant and per organisation.

What guardrails does a phone assistant need?

Outbound calling is where phone assistants get expensive and dangerous. Telecom fraud guides, such as TransNexus’s introduction to telecom fraud, describe revenue-sharing schemes: traffic pushed to high-cost destinations that share the revenue with the fraudster, billed to the account that placed the calls. An assistant that can be talked into dialling any number is an open door to that fraud.

The rule of thumb is deny by default, then allow narrowly. In Botify, outbound calling is off until an organisation lists the countries it may call, and known premium-rate and special-service prefixes are blocked even inside allowed countries. Hourly limits and per-number cooldowns cap the damage of a loop or an abuse attempt. Placing a call (phone.call) always waits for a human approval, transferring a call follows organisation policy, and hanging up runs directly.

• Country allowlist per organisation; an empty list means no outbound calls at all.
• Premium-rate and special-service prefixes blocked regardless of the allowlist.
• Hourly call limits, per-number cooldowns and concurrent-call caps.
• Human approval before the assistant places any call.
• Only call numbers that belong to a known contact, and honour do-not-call flags.
• Recording disclosure spoken at the start of recorded calls.
• Step-up verification (for example a keypad PIN) before sensitive actions on inbound calls.

What changes for Arabic phone assistants?

Arabic callers rarely speak textbook Modern Standard Arabic on the phone. Expect Gulf, Levantine or Egyptian dialect, and frequent switching into English for technical terms, product names and numbers. Test with real speakers from the region you serve, over a real phone line, not with clean studio recordings.

The text side matters as much as the audio. Sentence splitting must understand Arabic punctuation such as ؟ and ، so replies are spoken in natural chunks. Language can change mid-call, so detect it per turn and switch fixed phrases (greetings, “one moment”, approval notices) with it. Dates, amounts and phone numbers should be read back and confirmed, because they are where recognition errors hurt most.

How do you measure a phone assistant before going live?

Measure the experience, not only the transcript. The useful numbers are call setup time, time from the end of the caller’s speech to the first audio, how often callers interrupt, how many tools each turn calls, and how often fillers were needed. Rising interruptions usually mean answers are too long or too slow.

Then review outcomes: was the task completed, was the call transferred, did any action wait for approval, and was the transcript accurate on names and numbers. Run these checks separately for each language you support.