Three problems
I know how to solve.

An agent that takes four seconds to answer, and keeps talking when you interrupt it, sounds to the caller like an answering machine. Two measurements explain most of the problem.

Time-to-first-audio is the gap between the caller finishing their sentence and the agent making a sound. Past roughly one second, humans repeat themselves or hang up. Barge-in is what happens when the caller cuts in. It is the most common move in a phone conversation, and almost every implementation handles it badly.

① The connection opened too late

The naive flow: a call arrives, the telephony provider hits your webhook, you then open the connection to the model, handshake, configure the session, first audio. Five to ten seconds of silence before the agent says hello. The caller has already said "hello?" twice.

The fix is one sentence: the connection to the model must exist before the call is answered, and be kept warm. The cost is a session open for a few seconds for nothing. The gain is a five-fold cut in time-to-first-audio, for a change that touches very little code.

② Interrupting without truncating

The caller cuts in. You stop playing audio on the telephony side, which is the right reaction. But the model believes it delivered the whole sentence: it is already written into the conversation history. So it carries on as if it had been heard out.

You have to truncate the item on the model side and tell it where playback actually stopped. Without that, the agent's context and what the caller heard drift a little further apart with every interruption.

③ Not measuring the audio actually heard

To truncate in the right place, you need to know how many milliseconds were heard, not how many were sent. Telephony buffers: when you cut, several hundred milliseconds are already on the wire and will never be played.

Without a counter based on playback acknowledgements, you truncate in the wrong place. Over a three-minute call with six interruptions the error compounds, and the agent ends up answering the previous question. This is the bug people describe as "it's just weird".

④ Mistaking silence for end of turn

End-of-speech detection is a setting you tune call after call. Too aggressive and it cuts off a caller who is thinking. Too permissive and it leaves an awkward gap that makes the caller start again at the exact moment the agent starts too.

The threshold depends on the business. An elderly patient working out which dates suit them does not speak at the pace of a broker in a hurry. Set this parameter per vertical, on real calls rather than on a default value.

In a demo everything works: one call at a time, one user, the happy path. Production is concurrency, replays, and people acting in bad faith.

Replayed callbacks

Telephony providers replay their webhooks. This is documented behaviour and it is what a reliable system does. If your handler isn't idempotent, a replay produces a duplicate email, a duplicate charge, a duplicate call. The idempotency key must come from the event id, never from a timestamp.

Booking a slot with read-then-write

Two callers confirm the same slot two hundred milliseconds apart. The code reads "free", then writes. Both succeed. The clinic finds the double-booking the next morning.

A conditional in your code does not guarantee uniqueness. You need an atomic conditional write at the database level, the only place where the race between the two requests is actually settled.

Cost as an attack surface

A public endpoint that triggers an outbound call exposes your telecom budget directly. The classic fraud is dialling premium-rate numbers abroad, where the attacker collects a share of the minutes you pay for.

Controls, in order of effectiveness: country allow-list, hard spending cap on a dedicated sub-account, premium-prefix blocking, per-number and per-IP rate limits, maximum call duration. The first one removes most of the risk on its own.

Access controls that fail open

An authorisation check that lets the request through when it hits a technical error. Tests do not catch it, since they pass and the check works. The problem appears the day the service the check depends on goes down, and everyone has access to everything.

Tenant isolation, tested backwards

Most tests check that customer A can see their own data. The useful test checks the reverse: can customer A reach customer B's data by guessing an identifier? The two look alike, and only the second covers the scenario that matters.

Tokens that never expire

A signed token whose signature is verified but whose expiry never is stays valid indefinitely. A confirmation link emailed six months ago still works today.