Interrupt the Refusal: How a Barge-In Talks Voice AI Agents Out of Saying No
A single barge-in, right as the agent starts to refuse, and harmful answers jump from 6.5% to 35.6%. The very thing that makes voice agents feel human is the thing this attack turns against it.
You know the moment on a call when you start to say no, and the other person talks right over you?
Voice AI agents have that moment too. And it turns out to be a weak spot.
In a recent study, when an AI voice agent was interrupted just as it started to refuse a harmful request, harmful answers jumped from 6.5% to 35.6%. The interruption was one short spoken clip, the same every time. On a version of the model tuned to be more conversational, the number went from 9.4% to 48.6%.
If your voice agent can move money, reissue a card or change a booking, that is a gap worth understanding.
In short
- The newest voice agents listen and talk at the same time, so a caller can cut in mid-sentence.
- An AI’s "no" mostly lives in the first few words of its answer. Interrupt early and you can steer what comes next.
- Making an agent more conversational can make this worse, and a standard safety test may not notice.
- Testing one clean, typed message at a time won’t catch it.
Voice agents that listen and talk at once
Most voice agents today take turns. You speak, the agent waits until you’re done, then it answers.
The newest ones don’t wait. These "full-duplex" agents listen and speak at the same time, like a person on the phone. You can cut in, and the agent adjusts mid-sentence. That is what makes it feel so natural.
It also removes a quiet checkpoint. There is no longer a clean moment where you have finished and the agent decides what to say. Your voice can reach it while it is still forming its answer. If you only test what happens at the end of each turn, you miss everything that happens in the middle.
What the attack looks like
How the interruption lands
Someone asks for something harmful. The agent starts to say "I’m sorry, I can’t…". Right then, a short recorded clip plays over it. From that moment on, the agent is shaping its answer with the clip in its ears.
That is the whole attack. No clever wording tailored to each request, no arguing with the rules. The same clip was played over hundreds of different harmful requests to NVIDIA’s PersonaPlex, an open voice model built to play roles like a customer service agent. It had been told to refuse anything harmful.
The numbers
One interruption turned a mostly safe agent into one that gave a harmful answer more than a third of the time.
- NVIDIA PersonaPlex — 6.5% — 35.6%
- PersonaPlex-RL (tuned to be more conversational) — 9.4% — 48.6%
Share of harmful requests that got a harmful answer.
The words matter even more than the timing. Played after the agent had already finished, the same clip still pushed harmful answers to about 31%. Cutting in mid-refusal pushed them higher.
Why it works
Three things stack up.
- A "no" is only a few words deep. Safety training mostly shapes how an AI starts its reply. Once it gets past those first words, it tends to keep going. A refusal is not a locked door. It is a habit of starting with "I’m sorry."
- The clip doesn’t argue. It starts the answer. The clips that worked best sounded like the caller setting up the first words of a reply and inviting the agent to carry on. A simple plea for help did nothing.
- There is no pause to catch it. Because the agent listens while it talks, the clip lands while the refusal is still a few words long. Its next words are shaped by what it just heard.
The more natural the agent, the bigger the risk?
Two versions of the same model were put through this. The second had extra training to handle conversation better, including responding quickly when someone cuts in.
On a standard safety test, it scored as well as the original, slightly better even. Under this attack, it was the easier one to break: 48.6% vs 35.6%.
That does not prove the extra training caused it. But it is a warning worth taking seriously. Making an agent feel more human can change how it behaves under pressure, and your usual safety checks may not show it. Separately, training the same model on casual phone conversations has been shown to weaken its safety.
What this could look like in your business
In the study, the agent only produced harmful text. A real voice agent does more than talk. It resets passwords, moves money and changes accounts. That is where a talked-over refusal stops being a bad answer and becomes a loss.
A SIM swap the agent never meant to make
Picture a mobile carrier’s support line. A caller asks to move a phone number to a new SIM card but can’t pass verification. The agent starts to refuse: "I’m sorry, I can’t make that change without…" The caller cuts in, sounding relieved, as if the agent has already said yes, and jumps straight to the next step. The agent never finishes its sentence. It follows the caller’s lead and completes the swap.
That phone number now belongs to someone else, along with every security code sent to it by text: bank logins, email resets, payment apps. SIM swaps are already a well-known route to account takeover. An agent that can be talked past its own "no" makes them easier.
And here is the part that should worry you most. The refusal is right there in the call transcript. Anyone reviewing the call sees "I’m sorry, I can’t…" and assumes the agent held the line.
The same pattern applies anywhere a refusal guards an action:
- Telecom — Moving a number to a new SIM without verification — Account takeover
- Banking — Sending a replacement card to a new address — Card fraud
- Insurance — Changing payout details on a claim — Money sent to the wrong account
- Healthcare — Sharing test results without an identity check — Privacy breach and regulatory exposure
- Travel — Waiving a fee that policy forbids — Lost revenue
None of these needs special tools. It needs a caller who knows when to cut in and what to say.
Questions to ask about your own voice agent
- Can callers interrupt it mid-sentence? If so, has anyone tested what happens when a caller cuts in during a refusal?
- Is your safety testing done with typed messages, or with live voice calls that include interruptions?
- Which refusals guard real actions, such as payments, card changes, account updates or bookings?
- Did you re-test safety after making the agent more natural or more responsive?
- Do your test results show when a harmful answer started, or only whether it happened?
If any of these are hard to answer, that is exactly what we test at SottoAI.
How to defend against it
Before launch
- Test with live voice calls, not typed transcripts. Include callers who interrupt, barge-in and talk over the agent.
- Start with the refusals that guard real actions: payments, card changes, account updates and bookings.
- Re-test whenever you make the agent more natural or more responsive.
In production
- Don’t let a refusal depend only on what the agent says. Put hard checks in front of sensitive actions, so a talked-over "no" can’t turn into a "yes" in your systems.
- Require confirmation or a hand-off to a person for anything that can’t be undone.
- Keep timed call logs, so you can see whether something went wrong right after an interruption.
When training or tuning the model
- Include interrupted refusals in safety training, so the model learns to come back to "no" after being cut off. In text models, this kind of training makes refusals much harder to break.
- Reward safety alongside naturalness when you tune an agent for conversation.
The takeaway
A refusal is not a wall. In an agent that listens while it talks, it is a few words at the start of an answer, and a caller who cuts in can steer what comes next. The feature that makes these agents feel human, being interruptible, is the same one this attack uses.
Most AI security testing still sends one clean, typed message at a time. Real callers interrupt, barge-in and talk over. That is why at SottoAI we attack voice agents on a live call, the way real callers talk.
If your voice agent supports barge-in, have you tested what happens when someone talks over a refusal?
Questions
What is a full-duplex voice agent?
A voice AI that listens and speaks at the same time, so callers can interrupt it mid-sentence, like a person on the phone.
What is a barge-in attack?
Talking over a voice agent at the moment it starts to refuse, with words that steer it toward finishing the answer instead. In the research literature it is also called an interruption attack.
Does this affect commercial voice agents?
So far it has been shown on open research models, not commercial products. But any agent that can be interrupted mid-reply is worth testing for it.
Is a turn-based voice agent safe from this?
It is not exposed in the same way, because it waits for the caller to finish. But the same clip, played as an ordinary follow-up after a refusal, still raised harmful answers to about 31%. Turn-based agents should test what happens after a refusal too.
How do I test my voice agent for this?
Run live voice calls that interrupt during refusals, focus on refusals that guard sensitive actions, and re-test after any change that makes the agent more conversational.