The Open ASR Leaderboard Adds Its First Global South Language

The landscape of Automatic Speech Recognition (ASR) development is dictated by the benchmarks we choose to measure. When a model performs exceptionally on the Open ASR Leaderboard, it gains industry adoption and iterative momentum. Conversely, capabilities that remain unmeasured by these benchmarks often stagnate. While recent efforts have focused on making evaluation metrics more robust—through held-out private splits, benchmark-fitting analysis, and improved normalizers—the industry has largely remained tethered to a single, aggregate metric: the Word Error Rate (WER).

However, a singular number can be misleading. Extensive research has consistently demonstrated that ASR error rates are not distributed equitably across the population. Commercial systems have been shown to perform significantly worse for Black speakers compared to white speakers, with further disparities emerging across gender, age, and accent. These failures are often invisible on traditional leaderboards, not because they are intentionally hidden, but because the test sets themselves record only the audio and the transcript, ignoring the identity of the speaker.

To bridge this critical gap, Voice Arena and Hugging Face have partnered to introduce two new evaluation sets to the Open ASR Leaderboard: Monsoon en-IN (Indian English) and Monsoon hi-IN (Hindi). Hindi, a language spoken by over half a billion people, marks the first time an Indic language has been integrated into the leaderboard’s multilingual tab, which has historically been dominated by European languages.

---

Redesigning the Benchmark: The Monsoon Philosophy

A test set can only reveal failure modes that it is designed to capture. Most existing benchmarks are constructed from whatever audio is readily available. In contrast, the Monsoon dataset was engineered to vary across nine specific axes: geography, age, gender, vocabulary, device hardware, acoustic environment, speech type, speech rate, and the presence of multiple valid transcripts.

By recording these attributes, the researchers aim to move beyond the "aggregate WER" trap, where a model might appear accurate on average while failing catastrophically for specific demographic groups.

Key Dataset Composition

The Monsoon project consists of four distinct splits—two for Indian English and two for Hindi—all derived from a unified collection pipeline.

| Set | Language | Duration | Speakers | Districts | Devices | | :--- | :--- | :--- | :--- | :--- | :--- | | Monsoon en-IN (Public) | Indian English | 5.62 h | 1,444 | 428 | 556 | | Monsoon en-IN (Private) | Indian English | 5.58 h | 1,405 | 420 | 560 | | Monsoon hi-IN (Public) | Hindi | 1.33 h | 468 | 202 | 315 | | Monsoon hi-IN (Private) | Hindi | 4.47 h | 1,571 | 295 | 582 |

Each set is released as a public split for self-scoring, alongside a private, withheld split designed to prevent developers from "gaming" the benchmark through over-optimization.

---

Why Speaker Diversity Matters

The Monsoon dataset is intentionally designed to be "small" in terms of total hours but "large" in terms of speaker count. This design choice is a deliberate claim about variance rather than volume.

• No Single Voice Dominates: The top ten contributors account for less than 7% of the total duration. More than half of all speakers appear exactly once, ensuring that the benchmark reflects an average across hundreds of unique voices rather than a few individuals recorded at length.
• Geographic and Technical Breadth: The Indian English set draws from 428 native districts across 30 states and union territories. By utilizing contributors' own handsets—ranging from high-end smartphones to low-end devices on unstable connections—the dataset avoids the "lab-quality" bias that often plagues standardized corpora.
• Accent Representation: Indian English is not a monolith. The Monsoon dataset captures the linguistic diversity of all six Indian zones, with metadata recording the accent variations rather than assuming a standard "Indian English" accent.

"A result on Monsoon is an average over hundreds of distinct voices, not a small number of talkers recorded at length. Corpora collected on standardised hardware overfit to one microphone response; this one cannot."

---

Rigorous Quality Control and Transcription

Collecting spontaneous speech at scale introduces significant risks, including inattentive annotation, played-back audio, and gaming of the task. To mitigate these, the team implemented a multi-layered quality control process:

1. Recruitment: Contributors were sourced via the Voice Arena community, reaching rural and semi-urban areas often ignored by traditional datasets. 2. Elicitation: To avoid sparse, short responses, conversations were seeded with open-ended narrative cues covering topics like healthcare, agriculture, and digital services. 3. Gating Checks: Every recording underwent language identification verification, gender-label corroboration, and a "liveness" check to distinguish genuine conversation from pre-recorded audio. 4. Transcription Protocol: References were created using a five-level protocol. A first draft was generated by internal ASR models, followed by multiple rounds of human correction and independent verification. No linguist was permitted to audit their own work, ensuring the highest level of orthographic consistency.

---

Addressing Orthographic Complexity in Hindi

A major challenge in Hindi ASR is the lack of a standardized orthography. Everyday speech is heavily code-mixed, and there is no settled way to write English-origin words in Devanagari. Consequently, a single phrase can have ten valid written forms.

Traditional WER metrics penalize a model if it chooses a spelling that differs from the annotator's, even if the recognition is phonetically correct. To solve this, the Monsoon project introduces the Orthographically-Informed Word Error Rate (OIWER).

Instead of a single reference string, the Hindi sets ship with a lattice—a list of all accepted spellings for each span of the transcript. This ensures that models are rewarded for correct recognition rather than being penalized for orthographic choices. The team has also open-sourced the voi-oiwer implementation, allowing researchers to reproduce these results directly.

---

The Future of Global South Benchmarking

The introduction of Monsoon to the Open ASR Leaderboard is more than just an update to a leaderboard; it is a fundamental shift in how we evaluate AI fairness. By providing 12 metadata fields per segment—including occupation, education, and handset model—the project enables researchers to perform granular analysis that was previously impossible.

For instance, when testing eight leading models, researchers found that while they might appear identical on a standard leaderboard, their performance diverged significantly when broken down by region. One model might excel in the North while struggling in the South, while another shows the inverse. These differences are not inherent to the audio, but to the models themselves.

How to Get Evaluated

Developers can now submit their models to the Open ASR Leaderboard via the standard GitHub pull request process.

• Indian English: Integrated into the default column set, contributing to the headline Average WER.
• Hindi: Available on the Multilingual tab, providing a like-for-like comparison for models supporting the language.

As the AI industry continues to expand, the Monsoon dataset serves as a vital reminder: if we do not measure the diversity of our speakers, we cannot claim to serve them. By making these disparities visible, Voice Arena and Hugging Face are setting a new standard for transparency and inclusivity in the global AI ecosystem.