How AI Speech-to-Text Works: From Sound Waves to Sentences

A microphone is a pressure gauge with very fast reflexes. It converts moving air into a changing voltage, and software converts that voltage into a long list of numbers. Everything a speech recogniser does happens to those numbers. It never hears a word the way you do.

The machinery behind that trick was rebuilt over the past fifteen years. Hand-written dictionaries of speech sounds gave way to neural networks trained on hundreds of thousands of hours of audio. The journey itself has not changed. Sound becomes signal, signal becomes features, features become probabilities, and probabilities become words.

The pipeline in one pass

Speech recognition looks like a single clever trick from outside. Inside, it is an assembly line. Each station solves one narrow problem and hands its output to the next. Knowing the stations is what lets you predict where a transcript will go wrong.

Step one: turning air into numbers

Speech starts as a disturbance in the air. Your vocal folds vibrate, the shape of your mouth changes, and air pressure fluctuates rapidly. The microphone diaphragm moves with that pressure and produces a voltage. That signal is still analogue, and computers cannot store a continuous curve.

So the first station samples it. Sampling measures the height of the wave many thousands of times a second and records each measurement as a number. Each number is usually stored with 16 bits of resolution, which is plenty for speech. One minute of 16 kHz audio is already close to a million numbers.

Why 16,000 samples a second?

The figure comes from the Nyquist–Shannon sampling theorem. To preserve a frequency, you must sample at least twice as fast as that frequency. The information that separates one speech sound from another sits mostly below 8,000 hertz. Sampling at 16 kHz therefore captures what matters and discards what does not.

Music production uses higher rates because instruments carry real energy above the human voice. Recording speech at 48 kHz does no harm, since the system simply resamples it down. It also brings almost no benefit, which is why microphone placement beats specification chasing every time.

What a spectrogram actually shows

Raw samples are too detailed to model directly, so the audio is condensed. The signal is cut into frames of roughly 25 milliseconds, stepped forward 10 milliseconds at a time. Each frame passes through a fast Fourier transform, which reports how much energy sits at each frequency.

Stack those reports side by side and you get a spectrogram. Time runs left to right, frequency runs bottom to top, and intensity appears as brightness. Most systems then group the frequencies using the mel scale. That scale copies the ear, which hears fine detail low down and coarser detail higher up. Around 80 mel bands per frame is a common choice.

The dark bands you can see in a vowel are formants, the resonances of the vocal tract. They are how a spectrogram distinguishes ee from ah. What reaches the neural network, then, is not sound. It is a sequence of compact numerical snapshots, each covering a fraction of a second.

Step two: how models learn speech sounds

Spoken language is built from a small inventory of basic sounds. Linguists call them phonemes, and most varieties of English use around forty-four. The difference between bat and pat is one phoneme doing its work. The study of these sounds, phonetics, gave early recognition systems both their vocabulary and their strategy.

The phoneme era: lexicons and hidden Markov models

For roughly three decades, a recogniser was a chain of hand-assembled parts. An acoustic model, usually hidden Markov models paired with Gaussian mixtures, guessed which phoneme each slice of audio resembled. A pronunciation lexicon listed how each word breaks into phonemes. An n-gram language model then ranked candidate word sequences by plausibility.

It worked well enough to power the dictation software and phone systems of the 1990s and 2000s. It was also brittle. Every language needed a hand-built lexicon, and every unfamiliar accent exposed an assumption somebody had baked in.

The end-to-end shift

Around 2012, deep neural networks replaced the statistical heart of the acoustic model and error rates fell sharply. Researchers soon found that networks could learn the whole mapping from audio to text, with no pronunciation dictionary at all. Three ideas made that practical.

• Connectionist temporal classification: a training method that lets the network emit a blank symbol when no new character is due. Alignment is learned rather than supplied.
• Attention and the transformer: introduced in the 2017 paper Attention Is All You Need. Attention lets the decoder look back across the whole recording and pick the stretch of sound that matters now.
• Subword tokens: models write fragments such as tran and scribe rather than whole words, so a fixed vocabulary can spell rare and invented words.

Scale supplied the rest. Modern systems train on hundreds of thousands of hours of loosely labelled audio. That approach is set out in Robust Speech Recognition via Large-Scale Weak Supervision. Open corpora such as Mozilla's Common Voice add donated sentences across dozens of languages and accents. A model only knows what its data taught it, which matters shortly.

A speech recogniser never hears words. It hears numbers, and it has learned which words those numbers most probably encode.

Step three: choosing words with context

Listen to the acoustics alone and their, there and they're are identical. So are write and right, or ate and eight. No amount of audio quality separates them, because the difference is not in the sound. It lives in the sentence.

Beam search in plain terms

Recognisers cope by refusing to commit early. The decoder keeps a shortlist of candidate transcripts alive at once, a method called beam search. Each candidate carries two scores: how well it fits the audio, and how likely it is as language. The phrase "I ate breakfast at eight" wins because both homophones land where the language expects them.

That sense of plausibility once came from a separate language model bolted onto the pipeline. In end-to-end systems, much of it is absorbed into the network itself. The same bet underpins the chatbots people use daily, as we describe in our guide to how large language models power AI chat assistants.

Punctuation, numbers and who spoke

Recognising the words is not the end of the job. Raw output arrives with no capital letters, no commas and no full stops, so a further model restores them. Spoken numbers need converting too, a step called inverse text normalisation. "Nineteen forty-five" may be a year, a time or a sum of money.

• Diarisation: clusters voice characteristics to answer who spoke when, turning a wall of text into a conversation.
• Timestamps: word-level timings anchor the text to the audio. They also feed the caption formats covered in our guide to how SRT and WebVTT store those timings.
• Formatting rules: currencies, dates, acronyms and units all get rewritten into the shapes readers expect.

Where accuracy breaks down, and what to do

Once you can see the assembly line, its failures stop looking random. Nearly every error traces back to one station, and most have a practical fix.

• Distance and reverberation: a speaker two metres from a laptop in a hard-walled room produces smeared features. Move the microphone closer, since no software fully repairs this.
• Overlapping speech: when two voices share frames, their frequencies add together. Separate channels or a chaired agenda help. Our advice on why meetings need turn-taking discipline comes before any software setting.
• Accent coverage: a model mirrors its training data. Thinly represented accents score measurably worse, not because they are harder, but because the network heard less of them.
• Rare words and names: a decoder trained to prefer the probable will swap an unfamiliar surname for a common word that sounds similar. The sentence still reads fluently, which is what makes the error easy to miss.
• Code-switching: speakers who move between languages mid-sentence defeat systems that lock onto one language at the start of a file.

All of this is measurable. The standard yardstick is word error rate, a deceptively simple number we unpack in our explainer on how word error rate is calculated. Where the stakes are high, the sensible design pairs machine speed with human judgement. Everything we publish on the underlying technology is gathered on our speech recognition topic page.

What changes next for listening machines

The frontier is moving in three directions at once, and each one changes how the pipeline is used rather than what it is.

• Streaming: models that emit words within a fraction of a second now make live captioning practical, at a small cost in accuracy.
• On-device recognition: running the model on a phone or laptop means audio need never leave the machine. That changes the privacy calculation around recorded speech.
• Merged understanding: the same family of networks that writes the words now summarises, translates and answers questions about them.

That last shift deserves care. Transcription reports what was said, while a summary asserts what it meant, and an assertive model can invent a detail that was never spoken. The failure mode is familiar from why chatbots make things up. Keep the verbatim transcript, and treat the summary as a draft.

The shape of the problem stays constant. Air moves, numbers are made, probabilities are weighed, and sentences appear. That engineering is what a service such as TRANSCRIPT.YOU stands on, and the rest of our Voice & Transcription hub builds on it. The machinery is elaborate, but the promise is plain: anything said aloud can become something you search, quote and keep.