How to test auto-caption accuracy in 10 minutes
Every caption tool claims it gets almost everything right. The only number that matters is what it gets right on your audio, with your slang and your accent. You can measure that in ten minutes with one minute of video and a bit of division.
How to test auto-caption accuracy in 10 minutes
Short answer: take 60 seconds of your own video, transcribe it by hand, run the same minute through the caption tool, and count how many words it swapped, dropped or invented. Divide by the number of words actually spoken. That is word error rate (WER), the standard speech recognition metric, and it lets you compare any tool on the only audio that matters: yours.
I am writing this as someone who sells a caption tool. Cut.Pro says it gets more than 99% of words right. That is our claim, and the point of this post is that you do not have to take it on faith, or anyone else's. The test below works on us too.
Why the number on the website says little about your video
An accuracy percentage without the audio behind it is close to decoration. It depends on three things vendors rarely disclose.
What audio was used. A studio read and a six-hour stream with game audio and a screaming chat are different problems. A system can shine on the first and struggle on the second.
How the text was normalized. Before comparing, text gets cleaned: lowercase, punctuation, "ten" versus "10". The OpenAI Whisper paper reports that on some test sets, WER dropped by up to 50% just from handling formatting quirks, such as a dataset splitting contractions from words with whitespace (Radford et al., 2022). Half the "error" was style, not recognition.
Which errors were counted. The metric treats every word equally. Swapping "a" for "the" costs the same as misspelling your guest's name. Even the metric's own definition flags this: it takes no account of how different error types affect the outcome (Wikipedia, Word error rate).
So the right test is small, on your material, with a second tally for the errors that actually hurt.
What WER is, in one line
WER = (S + D + I) / N
- S (substitution): a spoken word that became another word.
- D (deletion): a spoken word that vanished.
- I (insertion): a word nobody said that showed up.
- N: total words in the correct transcript.
The count uses the minimum number of edits to turn one text into the other, the same logic as Levenshtein distance applied to words. A detail that surprises people: WER can exceed 100%, when a tool invents more words than it gets right (Wikipedia).
Marketing "accuracy" is usually 1 minus WER. A 5% WER becomes "95% accurate".
The 10-minute test, step by step
1. Pick your worst minute (1 minute of work)
Do not pick the cleanest stretch. Pick the one that represents your real problem:
- slang and swearing the way they come out on stream;
- a strong accent, or two different accents;
- two people talking over each other;
- music or game audio underneath;
- names of people, games, brands.
If you clip podcasts, grab the moment three people laugh and talk at once. If you clip gaming streams, grab the tense stretch with the game loud. That is where captions break in real life.
2. Transcribe it by hand (4 to 5 minutes)
Use headphones and write exactly what was said. Not what should have been said.
Three rules so you do not sabotage the test:
- Write speech as it is: "gonna", "kinda", "y'know". If you clean up grammar in the reference, you punish the tool for being faithful.
- Decide up front how to handle numbers ("ten" or "10") and apply the same rule to both texts.
- If you cannot make out a word yourself, mark it and drop it from the test. If you cannot hear it, the tool cannot be judged there.
3. Generate captions in the tool (2 minutes)
Upload the same minute, trimmed identically, and export plain text (TXT or SRT). Comparing two tools? Use the exact same file in both.
4. Normalize both texts
Everything lowercase, no punctuation, no line breaks. This is what avoids the Whisper trap: you want to measure recognition, not formatting.
5. Count (2 minutes)
Put the texts side by side and mark every swap, every missing word and every extra word. A short example, just to show the math:
| Text | |
|---|---|
| Spoken | bro he literally jumped out of his chair i have never seen that |
| Caption | bro he literally jumped out of the chair i never seen that |
That is 13 spoken words. The caption swapped "his" for "the" (1 substitution) and dropped "have" (1 deletion). WER = 2 / 13 = 15.4%, or "84.6% accurate". In a slangy sentence, two small slips are enough to tank the number. That is why one sentence proves nothing and a full minute proves a lot.
If you would rather not count by eye, the open-source Python library jiwer does it in two lines: pip install jiwer, then wer(reference, hypothesis). For the industry reference tool, NIST maintains SCTK, which includes the sclite scorer.
The second tally: errors that actually hurt
WER gives you the average. A clip caption is not judged by the average. It is judged by the worst error on screen. So, on the same minute, keep a separate list with four columns:
- Wrong proper name. The guest, the streamer, the game. Gets you correction comments and looks careless.
- Wrong number. A price, a score, a year. Changes the meaning.
- Wrong key word. The word that carries the joke or the hook. If it comes out wrong, the clip loses its reason to exist.
- Sync. Pick three points in the minute and check whether the highlighted word matches the sound. Correct captions half a second late wear out anyone watching on mute.
In my experience, two tools with similar WER can have very different profiles on this second list. One misses articles and prepositions that nobody notices. The other misses names and slang that everyone notices. The second list is what decides.
Turning the result into a decision
The test is not about crowning an absolute winner. It answers one practical question: how much time will I spend fixing captions per clip?
A way to think about it that I use: count how many caption screens a 60-second clip has, and how many needed a fix in the test. If you are fixing one in three, that cost lands on your workflow every day, on every clip. If you are fixing one in twenty, the tool is earning its price.
Run it again when your material changes. A calm podcast channel and a gaming stream channel with a loud chat are two different tests, even on the same tool.
What this changes when choosing a tool
Three conclusions I would draw before looking at price:
Always compare on the same clip. Website number against website number tells you nothing. The same minute in two tools tells you a lot.
Weigh name and slang errors more than the rest. Those are the ones that show up in the comments.
Look at the correction editor. Every tool misses something. The difference between a ten-second fix and a two-minute fix is being able to click a word, change it, and have the caption re-fit on its own.
If you want to run the test on Cut.Pro, the free plan includes 15 credits a month, and one credit is one minute of analyzed video, so the test minute fits easily. Captions come out in 40+ languages, and you set the style in the editor. Compare it with whatever you use today, on the same clip, and keep the one that misses less of what matters for your channel.
Once accuracy is sorted, see how caption style changes clip retention. If you are choosing between the two caption tools mobile editors use most, the Submagic vs CapCut comparison applies this same reasoning. And if the next step is publishing in other languages, the post on translated captions in three languages starts where this one ends: good translation depends on good transcription.
The short version
- Advertised accuracy does not apply to your audio. Measure your worst minute.
- WER = (swaps + drops + insertions) / spoken words. It can exceed 100%.
- Normalize before counting. Formatting alone moved WER by up to 50% in published tests.
- Keep a second list for names, numbers, key words and sync.
- Decide on correction time per clip, not the average.
Ten minutes of testing saves months of fixing captions by hand. And from then on, you read any "99%" with the right question: measured on what?
Sources: Wikipedia, Word error rate · Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022 · jiwer, speech recognition evaluation library · NIST SCTK
