Microsoft’s new speech models tackle the wait for a reply

Microsoft has introduced a speech-recognition model that begins producing text while a person is still talking, alongside two new models for turning a reply back into speech. Released October 1, the products address the delays at both ends of a voice assistant’s conversation: hearing a request and saying something useful in return.[1]

The listening model, MAI-Transcribe-2-Streaming, sends provisional text that can change as more audio arrives, followed by confirmed segments. Microsoft lists live captions, dictation, call centers and voice assistants among its intended uses. Its documentation also describes the service as a public preview, without a service-level agreement, and says it is not recommended for production workloads.[2]

For a reader watching captions, quicker words can make an explanation easier to follow. For someone talking to an automated service, the opportunity is more demanding. The system has to hear the correction at the end of a sentence, decide whether a pause means the person is finished, check the information needed for an answer and avoid talking over the next thought.

One limitation in Microsoft’s technical instructions makes that distinction unusually clear. Its raw Realtime transcription endpoint has no server-side speech detection or automatic signal to finish a turn. The application must request completion itself. Faster recognition supplies words sooner; the surrounding software still has to decide when it has enough to answer.[3]

The words arrive before the thought is finished

Consider a hypothetical caller arranging a delivery: “Send it to the office on Tuesday—actually, Wednesday, after two.” A useful assistant could begin looking up the order while that sentence unfolds. Scheduling the delivery for Tuesday would be premature. The valuable head start is the work that remains useful after the speaker corrects the request.

Speech systems already distinguish several kinds of completion. Deepgram’s documentation separates a finalized piece of transcript from a detected pause at the end of an utterance. A long spoken request can contain several finished transcript segments before the speaker stops. Its examples require applications to collect those segments together, rather than treating the last returned fragment as the whole request.[4]

Microsoft’s interface likewise distinguishes stable text from a provisional ending. New provisional text replaces that ending; confirmed text accumulates. A request to finish the buffered audio eventually returns the complete transcript for that interval. These are instructions for handling changing evidence, not a guarantee that the recognized words are correct.[3]

The delivery example shows why the distinction matters beyond a moving caption. Looking up an order is reversible preparation. Changing its delivery day creates a consequence. A well-designed service can start the first task early while waiting to confirm the second. The user should hear the requested day and time accurately enough to correct them before the change takes effect.

Human conversation sets a demanding pace. In a 2009 study of informal conversations across ten languages, Tanya Stivers and colleagues found that the most common response timings in every language fell between zero and 200 milliseconds after the previous turn. The overall median was 100 milliseconds, with differences between languages and kinds of response. The study examined human exchanges, including visible gestures; it did not establish a universal speed requirement for software.[7]

The researchers also found that confirming answers generally came faster than disconfirming ones. That is a useful qualification to the idea that every extra pause is a defect. In conversation, timing carries information. An assistant checking an uncertain delivery date can reasonably take longer than one acknowledging that it heard a name. What matters is whether its response fits the work being done.[7]

What the speed measurement actually measures

Independent testing provides evidence of progress in the recognition step. The public Artificial Analysis results retrieved October 3 put MAI-Transcribe-2-Streaming at about 2.5 percent final word error rate and a 0.13-second wait for a final transcript after the endpoint signal. Grok Voice Transcribe 2.0’s corresponding results were about 2.7 percent and 0.49 seconds. These are results for the benchmark’s audio and configuration, rather than measurements of complete customer-service conversations. Streaming results. The Daybreak has not performed hands-on tests of these models.[5]

The clock has a specific starting point. On its standard testing path, Artificial Analysis streams audio in real time, uses a separate detector to identify the end of speech, then asks the recognition service to finalize. It measures the wait from that signal until the final transcript arrives, including network delay. Services without forced finalization use a different fallback procedure.[6]

The accuracy test combines roughly eight hours of audio. Half its overall score comes from the evaluator’s held-out voice-agent dataset; the other half is divided between English parliamentary speech and earnings calls. Word error rate counts substitutions, omissions and additions after text normalization. It does not grade whether an assistant makes the correct reservation, identifies the right person or gives a useful answer.[6]

The breakdown also shows why an overall winner need not win every job. Microsoft’s result was better on the voice-agent dataset, while Grok’s was better on the parliamentary and earnings-call subsets. The weighted headline score rewards Microsoft’s particular balance of results. A company transcribing financial calls might reasonably care more about the subset resembling its own recordings.[5]

None of those numbers establishes how a voice assistant will behave beside a television, with two people speaking at once or when a caller slowly spells an unfamiliar name. Those conditions can be represented in a practical trial, with the errors inspected individually. Losing one digit from an order number and dropping an inconsequential word may each contribute to an error score while producing very different outcomes for the caller.

That distinction is visible in competitors’ own testing. SpaceXAI’s September 18 announcement for Grok Voice Transcribe 2.0 described separate internal evaluations for telephone calls, conversations, spoken credentials and short multilingual commands. Those company-run tests are not independent evidence of superiority, but their categories identify real jobs hidden inside the broad label of transcription. Its announced streaming price was $0.20 per audio hour.[9]

Taking turns in a noisy room

Microsoft offers more of the surrounding conversation machinery through Azure Voice Live. Its documentation describes noise suppression, removal of the system’s own acoustic echo, and turn detection that can use meaning as well as sound. It also provides controls for interruptions, including whether the assistant stops its response when the user starts talking. Those features belong to the wider service and should not be assumed to be present in the raw transcription endpoint.[13]

The details expose tradeoffs a polished demonstration can hide. Voice Live can be configured to ignore selected short filler responses during an ongoing answer, reducing accidental interruptions. Its multilingual semantic detector supports a specified language set. Recognition, deciding who has the floor and generating the spoken answer each have their own capabilities; one advertised language total does not describe the entire experience.[13]

Imagine the delivery assistant is reading available times when the caller says “mm-hm,” then adds “wait, use my home address.” The first sound may mean keep going; the second sentence changes the task. A system that stops at every sound can feel brittle. One that continues through a correction can become unusable, even if the words eventually appear correctly in its transcript.

A simpler approach detects silence. Deepgram lets developers choose how long a pause should trigger an endpoint, recommending longer settings for conversations with pauses inside a thought. That is an explicit compromise: more waiting can preserve the full request, while less waiting can make the system respond sooner. Changing the silence threshold affects behavior independently of the transcription model’s accuracy.[4]

Speaker identity is another separate question. In a shared room, a faithful transcript can contain a request from one person and a correction from another. An application handling an account needs a way to determine whose instructions count. Faster text alone cannot settle that question, and a label separating voices should not be presented to users as proof of identity.

Microsoft’s broader Speech documentation makes that boundary explicit for speaker separation. The feature distinguishes voices within a recording and labels their contributions, then discards the signals used for that separation. Microsoft says it does not provide identity enrollment or tracking of an individual across audio files. This describes the documented Speech feature, rather than a claim that every MAI streaming integration includes it.[11]

A voice that can answer in more languages

On the speaking side, MAI-Voice-2.1 and its Flash variant support 23 languages. Microsoft positions the standard version for expressive, longer material and Flash for interactions where responsiveness matters. The documentation describes prebuilt voices, controls over speaking style and gated voice cloning from a short reference recording. Access to cloning requires approval and uploaded audio consent.[8]

Microsoft says the new listening model supports 60 languages with continuous automatic language detection. Its introductory transcription price is $0.54 per audio hour through year-end. The speech-generation prices are $22 per million characters for MAI-Voice-2.1 and $15 for Flash. These are charges for components, so they do not represent the total cost of an assistant that also reasons or calls other services.[1]

For a tutoring application, switching the language of a spoken explanation could remove friction without changing the lesson’s interface. But the application would still need to establish that the explanation is accurate and that a learner’s question was understood. Clear pronunciation can make an answer sound authoritative; it cannot supply missing subject knowledge or resolve an ambiguous question.

The voice documentation carries the same public-preview warning as the transcription service. That matters for a school, service desk or publisher considering a dependable everyday product. A demonstration can show what a voice sounds like, while a deployment requires evidence about the complete interaction under the conditions in which people will actually use it.[8]

Where the conversation goes

The processing path also determines what happens to spoken information. Microsoft says its real-time Speech-to-text service processes incoming audio in server memory without storing it at rest. Its batch service has different arrangements: customers choose storage for recordings and transcripts and control retention. A live interaction and an uploaded recording should therefore not be treated as the same data-handling choice.[11]

Voice Live introduces another layer. Microsoft says that service itself does not retain customer data, but connected features can store it as required for their operation. Connections to third-party services can also move processing or storage outside Azure’s compliance boundary. The relevant question for an assistant is the path of the whole conversation, including text sent to a reasoning model and information returned by outside tools.[12]

Microsoft also documents optional logging for Voice Live support investigations. When a customer enables it for a support ticket, speech data is stored in Microsoft-managed Azure storage in the resource’s region, with controlled engineer access and automatic deletion after 30 days. That is a concrete reason to examine configuration alongside the service description. A choice made to diagnose a malfunction can change what is retained from the conversation.[12]

Developers also have an alternative to sending every transcription request to a hosted service. Mistral released the weights for Voxtral Realtime under Apache 2.0 in February. Its four-billion-parameter model supports 13 languages and is designed for streaming, with a configurable delay that trades waiting time against transcription quality. Mistral describes deployment on local devices as an intended use. That establishes an available route; it does not demonstrate performance on every laptop or phone.[10]

Mistral’s own evaluation illustrates that tradeoff: it reports matching its batch model at a 2.4-second delay, with some accuracy lost when reducing the delay to 480 milliseconds. Those figures come from its evaluation and cannot be substituted into the Artificial Analysis comparison. They describe a choice the application can make. Subtitles may tolerate more delay to obtain steadier text; a back-and-forth exchange makes a long wait more noticeable.[10]

For an organization, keeping recognition on its own equipment could give it greater control over audio handling, but the rest of the application still matters. A locally produced transcript subsequently sent to a hosted assistant has left that equipment. Evaluating privacy requires following both forms of the conversation: the sound and the words extracted from it.

The strongest near-term test for Microsoft’s new models is a complete, ordinary exchange. Can the assistant catch Wednesday after hearing Tuesday, wait through a thoughtful pause, stop when corrected and confirm the final delivery details? Quicker recognition gives the application more time to do those things. The improvement a caller will notice is receiving the right answer without having to fight for a turn.