MBZUAI study finds AI models struggle to generate Arabic dialects

Artificial intelligence models may understand Arab culture, but getting them to speak like people across the region remains a challenge.

Researchers at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) have developed the first benchmark designed to assess how AI models understand and engage with Arab culture across Modern Standard Arabic (MSA) and 13 national dialects.

The research found a significant gap between AI models’ ability to recognize culturally appropriate responses and their ability to generate the dialects used in everyday conversations, according to Emirates News Agency (WAM).

The study was led by Fajri Koto, Assistant Professor in MBZUAI’s Department of Natural Language Processing, and researcher Muhammad Dehan. The researchers said addressing this gap could help advance Arabic-language AI systems that understand not only the language but also how it is used across different Arab societies.

Arabic is spoken by more than 400 million people worldwide, but Dehan said one challenge is that AI models are largely trained on Modern Standard Arabic. While this can make models appear fluent in conventional tests, their ability to navigate natural conversations and regional cultural nuances remains less developed.

To examine the issue, the researchers developed ArabCulture-Dialogue, which they describe as the first benchmark for testing Arabic cultural reasoning in multi-turn conversations across MSA and 13 national dialects.

The team recruited 26 native Arabic speakers from 13 countries to develop conversations covering 12 areas of everyday life, including weddings, food, parenting, agriculture, arts, and games.

Researchers then gave AI models three tasks: selecting a culturally appropriate response, translating between MSA and a particular dialect, and continuing a conversation in a specified dialect.

“We gave the models three tasks: pick the culturally appropriate reply from a set of options; translate between Modern Standard Arabic and a specific dialect; and continue a conversation in a named dialect on request,” Koto said.

The strongest models scored in the mid-90s when asked to identify culturally appropriate responses, even when conversations moved between MSA and dialects. But performance fell sharply when models were required to generate dialect themselves.

The study found that AI performed better when navigating customs shared broadly across the Arab world, while country-specific cultural practices were more challenging. North African dialects were among the most difficult for the models, alongside Emirati Arabic.

Models generated the dialect corresponding to the intended country correctly in only around half of the cases tested.

“The paradox is that the cultural knowledge is already present within the models, but they sometimes require only a small amount of guidance,” Dehan said. Researchers found that providing models with information about the country and region associated with a conversation improved accuracy.

The findings come as the UAE continues to invest heavily in artificial intelligence through initiatives including the UAE National Strategy for Artificial Intelligence 2031 and the development of homegrown AI models such as Jais.

For developers, the research also highlights the limitations of describing an AI system as supporting Arabic without considering the linguistic diversity within the language.

Koto said an AI system that “supports Arabic” does not necessarily understand the range of dialects, cultures, and identities embedded within how Arabic is spoken across the region.

The research paper, Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues, was presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2026).

The researchers say the benchmark could provide a foundation for developing AI systems that better reflect the Arab world’s linguistic and cultural diversity, while helping preserve local dialects and cultural identities as generative AI becomes more widely used.