Interrupt the Refusal: How a Barge-In Talks Voice AI Agents Out of Saying No

A single barge-in, right as the agent starts to refuse, and harmful answers jump from 6.5% to 35.6%. The very thing that makes voice agents feel human is the thing this attack turns against it.

You know the moment on a call when you start to say no, and the other person talks right over you?

Voice AI agents have that moment too. And it turns out to be a weak spot.

In a recent study, when an AI voice agent was interrupted just as it started to refuse a harmful request, harmful answers jumped from 6.5% to 35.6%. The interruption was one short spoken clip, the same every time. On a version of the model tuned to be more conversational, the number went from 9.4% to 48.6%.

If your voice agent can move money, reissue a card or change a booking, that is a gap worth understanding.

In short

Voice agents that listen and talk at once

Most voice agents today take turns. You speak, the agent waits until you’re done, then it answers.

The newest ones don’t wait. These "full-duplex" agents listen and speak at the same time, like a person on the phone. You can cut in, and the agent adjusts mid-sentence. That is what makes it feel so natural.

It also removes a quiet checkpoint. There is no longer a clean moment where you have finished and the agent decides what to say. Your voice can reach it while it is still forming its answer. If you only test what happens at the end of each turn, you miss everything that happens in the middle.

What the attack looks like

How the interruption lands

Someone asks for something harmful. The agent starts to say "I’m sorry, I can’t…". Right then, a short recorded clip plays over it. From that moment on, the agent is shaping its answer with the clip in its ears.

That is the whole attack. No clever wording tailored to each request, no arguing with the rules. The same clip was played over hundreds of different harmful requests to NVIDIA’s PersonaPlex, an open voice model built to play roles like a customer service agent. It had been told to refuse anything harmful.

The numbers

One interruption turned a mostly safe agent into one that gave a harmful answer more than a third of the time.

Share of harmful requests that got a harmful answer.

The words matter even more than the timing. Played after the agent had already finished, the same clip still pushed harmful answers to about 31%. Cutting in mid-refusal pushed them higher.

Why it works

Three things stack up.

The more natural the agent, the bigger the risk?

Two versions of the same model were put through this. The second had extra training to handle conversation better, including responding quickly when someone cuts in.

On a standard safety test, it scored as well as the original, slightly better even. Under this attack, it was the easier one to break: 48.6% vs 35.6%.

That does not prove the extra training caused it. But it is a warning worth taking seriously. Making an agent feel more human can change how it behaves under pressure, and your usual safety checks may not show it. Separately, training the same model on casual phone conversations has been shown to weaken its safety.

What this could look like in your business

In the study, the agent only produced harmful text. A real voice agent does more than talk. It resets passwords, moves money and changes accounts. That is where a talked-over refusal stops being a bad answer and becomes a loss.

A SIM swap the agent never meant to make

Picture a mobile carrier’s support line. A caller asks to move a phone number to a new SIM card but can’t pass verification. The agent starts to refuse: "I’m sorry, I can’t make that change without…" The caller cuts in, sounding relieved, as if the agent has already said yes, and jumps straight to the next step. The agent never finishes its sentence. It follows the caller’s lead and completes the swap.

That phone number now belongs to someone else, along with every security code sent to it by text: bank logins, email resets, payment apps. SIM swaps are already a well-known route to account takeover. An agent that can be talked past its own "no" makes them easier.

And here is the part that should worry you most. The refusal is right there in the call transcript. Anyone reviewing the call sees "I’m sorry, I can’t…" and assumes the agent held the line.

The same pattern applies anywhere a refusal guards an action:

None of these needs special tools. It needs a caller who knows when to cut in and what to say.

Questions to ask about your own voice agent

If any of these are hard to answer, that is exactly what we test at SottoAI.

How to defend against it

Before launch

In production

When training or tuning the model

The takeaway

A refusal is not a wall. In an agent that listens while it talks, it is a few words at the start of an answer, and a caller who cuts in can steer what comes next. The feature that makes these agents feel human, being interruptible, is the same one this attack uses.

Most AI security testing still sends one clean, typed message at a time. Real callers interrupt, barge-in and talk over. That is why at SottoAI we attack voice agents on a live call, the way real callers talk.

If your voice agent supports barge-in, have you tested what happens when someone talks over a refusal?

Questions

What is a full-duplex voice agent?

A voice AI that listens and speaks at the same time, so callers can interrupt it mid-sentence, like a person on the phone.

What is a barge-in attack?

Talking over a voice agent at the moment it starts to refuse, with words that steer it toward finishing the answer instead. In the research literature it is also called an interruption attack.

Does this affect commercial voice agents?

So far it has been shown on open research models, not commercial products. But any agent that can be interrupted mid-reply is worth testing for it.

Is a turn-based voice agent safe from this?

It is not exposed in the same way, because it waits for the caller to finish. But the same clip, played as an ordinary follow-up after a refusal, still raised harmful answers to about 31%. Turn-based agents should test what happens after a refusal too.

How do I test my voice agent for this?

Run live voice calls that interrupt during refusals, focus on refusals that guard sensitive actions, and re-test after any change that makes the agent more conversational.

info@sotto-ai.com