Latency Is the Real UX Problem in AI Avatars, Not the Voice

<h1> Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture (and Why Function Calling Is Harder Than It Looks) </h1> <h2> Why this matters </h2>

<p>Most developers underestimate how hard voice AI actually is.</p> <p>To build a production-ready calling agent, you need to integrate:</p> <p>– SIP signalling</p> <p>– Real-time

<p><strong>Something went wrong. The agent said something bizarre, or paused for three seconds, or completely misunderstood the user. The real question is: which layer broke? The

<p>Most AI tutors today feel like glorified chatbots with a text-to-speech layer slapped on top. But what if students could actually <em>talk</em> to their tutor, pause

<p>Most voice AI teams optimize for one thing: speed. Fewer are asking what happens when the network itself is the problem, not the model.</p> <p>We wrote up an approach to this:

<p><em>By <a href="https://parvejshah.com" rel="noopener noreferrer">Parvej Shah</a> — Full-Stack Web Developer &amp; AI Systems Engineer based in Dhaka, Bangladesh.</em></p>

<p>When I was building <a href="https://intervu.dev" rel="noopener noreferrer">intervu.dev</a>, an AI mock coding interviewer that conducts full voice interviews in the browser,

<p>Adding voice messages to an AI companion is one of those features that sounds simple until you try to ship it. "Just use a TTS API and send the audio" -sure, in theory. In