Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

Should LLMs, $\textit{like}$, Generate How Users Talk? Building Dialect-Accurate Dialog[ue]s Beyond the American Default with MDial

About

More than 80% of the 1.6 billion English speakers do not use Standard American English (SAE) and experience higher failure rates and stereotyped responses when interacting with LLMs as a result. Yet multi-dialectal performance remains underexplored. We introduce $\textbf{MDial}$, the first large-scale framework for generating multi-dialectal conversational data encompassing the three pillars of written dialect -- lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features -- for nine English dialects. Partnering with native linguists, we design an annotated and scalable rule-based LLM transformation to ensure precision. Our approach challenges the assumption that models should mirror users' morphosyntactic features, showing that up to 90% of the grammatical features of a dialect should not be reproduced by models. Independent evaluations confirm data quality, with annotators preferring MDial outputs over prior methods in 98% of pairwise comparisons for dialect naturalness. Using this pipeline, we construct the dialect-parallel $\textbf{MDialBench}$mark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve under 70% accuracy, fail to reach 50% for Canadian English, and systematically misclassify non-SAE dialects as American or British. As dialect identification underpins natural language understanding, these errors risk cascading failures into downstream tasks.

Jio Oh, Paul Vicinanza, Thomas Butler, Steven Euijong Whang, Dezhi Hong, Amani Namboori• 2026

Related benchmarks

TaskDatasetResultRank
Dialect IdentificationMDialBench RBT_model 1.0 (Turn 1)
Accuracy0.967
5
Dialect IdentificationMDialBench RBT_model Turn 2 1.0
Accuracy99.1
5
Dialect IdentificationMDialBench RBT_model 1.0 (Turn 4)
Accuracy99.7
5
Dialect IdentificationMDialBench RBT_model 1.0 (Turn 8)
Accuracy99.9
5
Dialogue GenerationMDial
AU100
2
Showing 5 of 5 rows

Other info

Follow for update