# WER — corpus EvalSTT officiels

- corpus: AgentPublic/eval-stt-officiels @ ae64e631ebd0f5cc376f0d733c0494152af0f7ab
- généré: 2026-06-23T13-57-49Z (local, tous modèles : remote albert/scaleway + local whisperx/kyutai/cohere)
- normalisation: standard_numbers
- ⚠️ régimes de référence mixtes : officiels lissés vs QAG verbatim strict (ne pas comparer en absolu). Albert/Whisper-l3 sur les QAG = plancher biaisé (a servi de seed).

## Aggregate

| WER | S | D | I | N |
| --- | ---: | ---: | ---: | ---: |
| 9.78% | 3562 | 3573 | 1211 | 85296 |

## By provider/model

| Provider | Model | WER | S | D | I | N | Samples | Transcripts |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| albert | openai_whisper-large-v3 | 3.22% | 144 | 170 | 95 | 12705 | 8 | 8 |
| kyutai-chunked-mlx | stt-1b-en_fr | 6.39% | 184 | 145 | 135 | 7257 | 5 | 5 |
| kyutai-native | stt-1b-en_fr | 50.42% | 2016 | 1367 | 276 | 7257 | 5 | 5 |
| omlx | cohere-transcribe-03-2026 | 6.66% | 280 | 393 | 173 | 12705 | 8 | 8 |
| omlx | kyutai_stt-1b-en_fr | 4.89% | 318 | 150 | 153 | 12705 | 8 | 8 |
| scaleway | voxtral-small-24b-2507 | 11.71% | 218 | 1167 | 103 | 12705 | 8 | 8 |
| whisperx | large-v2 | 4.50% | 272 | 119 | 181 | 12705 | 8 | 8 |
| whisperx | large-v3 | 3.95% | 130 | 62 | 95 | 7257 | 5 | 5 |

## Transcripts

| Sample | Provider | Model | WER | CER | S | D | I | N |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| assurance_chomage_pm_assemblee | albert | openai_whisper-large-v3 | 3.67% | 3.56% | 9 | 35 | 0 | 1200 |
| assurance_chomage_pm_assemblee | kyutai-chunked-mlx | stt-1b-en_fr | 4.08% | 2.29% | 20 | 4 | 25 | 1200 |
| assurance_chomage_pm_assemblee | kyutai-native | stt-1b-en_fr | 30.33% | 22.21% | 337 | 2 | 25 | 1200 |
| assurance_chomage_pm_assemblee | omlx | cohere-transcribe-03-2026 | 4.33% | 3.80% | 8 | 17 | 27 | 1200 |
| assurance_chomage_pm_assemblee | omlx | kyutai_stt-1b-en_fr | 3.25% | 2.02% | 15 | 5 | 19 | 1200 |
| assurance_chomage_pm_assemblee | scaleway | voxtral-small-24b-2507 | 1.33% | 0.95% | 8 | 1 | 7 | 1200 |
| assurance_chomage_pm_assemblee | whisperx | large-v2 | 3.67% | 2.90% | 25 | 18 | 1 | 1200 |
| assurance_chomage_pm_assemblee | whisperx | large-v3 | 1.50% | 0.95% | 12 | 6 | 0 | 1200 |
| forum_europeen_calcul_quantique | albert | openai_whisper-large-v3 | 8.88% | 5.69% | 40 | 40 | 63 | 1610 |
| forum_europeen_calcul_quantique | kyutai-chunked-mlx | stt-1b-en_fr | 11.18% | 6.72% | 63 | 44 | 73 | 1610 |
| forum_europeen_calcul_quantique | kyutai-native | stt-1b-en_fr | 54.47% | 35.39% | 727 | 38 | 112 | 1610 |
| forum_europeen_calcul_quantique | omlx | cohere-transcribe-03-2026 | 10.00% | 6.06% | 57 | 45 | 59 | 1610 |
| forum_europeen_calcul_quantique | omlx | kyutai_stt-1b-en_fr | 10.75% | 6.35% | 59 | 44 | 70 | 1610 |
| forum_europeen_calcul_quantique | scaleway | voxtral-small-24b-2507 | 8.82% | 5.84% | 35 | 40 | 67 | 1610 |
| forum_europeen_calcul_quantique | whisperx | large-v2 | 9.63% | 5.66% | 69 | 26 | 60 | 1610 |
| forum_europeen_calcul_quantique | whisperx | large-v3 | 9.38% | 5.83% | 45 | 42 | 64 | 1610 |
| impots_systeme_fiscal_francais | albert | openai_whisper-large-v3 | 2.96% | 1.04% | 34 | 5 | 2 | 1385 |
| impots_systeme_fiscal_francais | kyutai-chunked-mlx | stt-1b-en_fr | 2.96% | 1.09% | 28 | 11 | 2 | 1385 |
| impots_systeme_fiscal_francais | kyutai-native | stt-1b-en_fr | 24.04% | 15.18% | 266 | 14 | 53 | 1385 |
| impots_systeme_fiscal_francais | omlx | cohere-transcribe-03-2026 | 3.97% | 2.40% | 25 | 12 | 18 | 1385 |
| impots_systeme_fiscal_francais | omlx | kyutai_stt-1b-en_fr | 3.90% | 1.59% | 36 | 12 | 6 | 1385 |
| impots_systeme_fiscal_francais | scaleway | voxtral-small-24b-2507 | 4.33% | 2.98% | 25 | 34 | 1 | 1385 |
| impots_systeme_fiscal_francais | whisperx | large-v2 | 3.68% | 2.17% | 28 | 6 | 17 | 1385 |
| impots_systeme_fiscal_francais | whisperx | large-v3 | 3.10% | 1.14% | 34 | 7 | 2 | 1385 |
| nouvelle_caledonie_pm_senat | albert | openai_whisper-large-v3 | 1.78% | 0.79% | 20 | 3 | 5 | 1574 |
| nouvelle_caledonie_pm_senat | kyutai-chunked-mlx | stt-1b-en_fr | 3.43% | 1.32% | 41 | 8 | 5 | 1574 |
| nouvelle_caledonie_pm_senat | kyutai-native | stt-1b-en_fr | 52.86% | 42.55% | 683 | 69 | 80 | 1574 |
| nouvelle_caledonie_pm_senat | omlx | cohere-transcribe-03-2026 | 2.80% | 1.08% | 35 | 9 | 0 | 1574 |
| nouvelle_caledonie_pm_senat | omlx | kyutai_stt-1b-en_fr | 3.62% | 1.38% | 45 | 8 | 4 | 1574 |
| nouvelle_caledonie_pm_senat | scaleway | voxtral-small-24b-2507 | 1.33% | 0.59% | 17 | 4 | 0 | 1574 |
| nouvelle_caledonie_pm_senat | whisperx | large-v2 | 2.22% | 1.41% | 19 | 0 | 16 | 1574 |
| nouvelle_caledonie_pm_senat | whisperx | large-v3 | 1.72% | 0.85% | 18 | 3 | 6 | 1574 |
| qag_assemblee_2026_05_27 | albert | openai_whisper-large-v3 | 0.74% | 0.57% | 5 | 9 | 0 | 1884 |
| qag_assemblee_2026_05_27 | omlx | cohere-transcribe-03-2026 | 8.55% | 6.52% | 46 | 105 | 10 | 1884 |
| qag_assemblee_2026_05_27 | omlx | kyutai_stt-1b-en_fr | 3.50% | 1.74% | 42 | 16 | 8 | 1884 |
| qag_assemblee_2026_05_27 | scaleway | voxtral-small-24b-2507 | 25.27% | 22.85% | 44 | 432 | 0 | 1884 |
| qag_assemblee_2026_05_27 | whisperx | large-v2 | 2.87% | 2.18% | 26 | 11 | 17 | 1884 |
| qag_assemblee_2026_06_03 | albert | openai_whisper-large-v3 | 2.19% | 1.97% | 3 | 35 | 1 | 1781 |
| qag_assemblee_2026_06_03 | omlx | cohere-transcribe-03-2026 | 11.51% | 9.35% | 40 | 155 | 10 | 1781 |
| qag_assemblee_2026_06_03 | omlx | kyutai_stt-1b-en_fr | 3.76% | 1.66% | 44 | 18 | 5 | 1781 |
| qag_assemblee_2026_06_03 | scaleway | voxtral-small-24b-2507 | 20.83% | 19.77% | 44 | 323 | 4 | 1781 |
| qag_assemblee_2026_06_03 | whisperx | large-v2 | 3.59% | 2.48% | 30 | 16 | 18 | 1781 |
| qag_assemblee_2026_06_17 | albert | openai_whisper-large-v3 | 2.86% | 1.90% | 12 | 39 | 0 | 1783 |
| qag_assemblee_2026_06_17 | omlx | cohere-transcribe-03-2026 | 5.78% | 3.32% | 51 | 46 | 6 | 1783 |
| qag_assemblee_2026_06_17 | omlx | kyutai_stt-1b-en_fr | 4.88% | 2.43% | 44 | 36 | 7 | 1783 |
| qag_assemblee_2026_06_17 | scaleway | voxtral-small-24b-2507 | 20.42% | 20.87% | 30 | 332 | 2 | 1783 |
| qag_assemblee_2026_06_17 | whisperx | large-v2 | 5.55% | 4.01% | 46 | 37 | 16 | 1783 |
| reception_maires_elysee | albert | openai_whisper-large-v3 | 3.29% | 1.86% | 21 | 4 | 24 | 1488 |
| reception_maires_elysee | kyutai-chunked-mlx | stt-1b-en_fr | 9.41% | 7.04% | 32 | 78 | 30 | 1488 |
| reception_maires_elysee | kyutai-native | stt-1b-en_fr | 84.21% | 83.28% | 3 | 1244 | 6 | 1488 |
| reception_maires_elysee | omlx | cohere-transcribe-03-2026 | 4.37% | 2.46% | 18 | 4 | 43 | 1488 |
| reception_maires_elysee | omlx | kyutai_stt-1b-en_fr | 5.24% | 2.96% | 33 | 11 | 34 | 1488 |
| reception_maires_elysee | scaleway | voxtral-small-24b-2507 | 2.55% | 1.64% | 15 | 1 | 22 | 1488 |
| reception_maires_elysee | whisperx | large-v2 | 4.70% | 3.01% | 29 | 5 | 36 | 1488 |
| reception_maires_elysee | whisperx | large-v3 | 3.23% | 1.89% | 21 | 4 | 23 | 1488 |

## Top errors

```text
top errors
substitutions:
  monsieur → m  53
  madame → mme  37
  m → monsieur  30
  il → ils  29
  de → encore  22
  leur → leurs  19
  que → qu  18
  le → encore  17
  a → à  16
  d → de  15
insertions:
  et  302
  donc  88
  depuis  46
  alors  44
  encore  36
  en  31
  3  30
  on  24
  effet  24
  à  21
deletions:
  de  146
  et  122
  le  105
  la  104
  les  89
  à  74
  l  62
  vous  57
  qui  52
  des  45
```