Irish-language AI leaderboard - November update

19 November, 2026

min read

5 months ago, Údarás na Gaeltachta launched a first-of-its-kind public benchmarking leaderboard evaluating the performance of large language models (LLMs) in the Irish language. Developed in partnership with University College Cork (UCC), University of Limerick and independent researchers, the leaderboard brings increasing transparency and rigour to the question: how well does modern AI serve Irish speakers?

In this 2nd update, 7 additional models were evaluated, including 3 open-weighted models — Mistral-Medium 3.5, Mistral-Medium 3.5 (no-thinking), GLM 5.2 (no-thinking), Gemini Flash 3.5, GPT 5.2, GPT 5.5, and Claude Fable 5 — bringing the total number of models evaluated to 20.

While in the previous leaderboard, Google’s Gemini models represented the state of the art for LLM performance in Irish, Anthropic’s Claude Fable model outperformed them all, and now sits on top of the leaderboard. A newer Gemini Flash model, 3.5, made it into the top 5 but interestingly, still did not perform as well on the leaderboard’s benchmarks as earlier versions 3.0 and 3.1. OpenAI’s GPT 5.5 also broke through into the top 5.

Our two primary takeaways from this update:

  • As we have found before, iterative gains that models make in English by scaling up do not necessarily translate into stronger performance in low-resource languages such as Irish. In OpenAI’s GPT 5 family for example, while 5.5 does outperform 5.0, 5.2 did not. Meanwhile Google’s Gemini Flash 3.5 scores lower than Gemini 3.0 (especially in the Irish-BLiMP benchmark). What is being optimized by the Frontier AI researchers may be orthogonal to what is required to improve performance in Irish.
  • There continues to be a loose correlation between performance enhancements achieved by new generations of models and performance in Irish. In the case of this update, Claude’s Fable significantly outperformed Claude Opus on all benchmarks included in this leaderboard.

In this update, a benchmark that had become saturated was omitted and we plan next to evaluate newer frontier models and to facilitate the creation and inclusion of new benchmarks that dig deeper into LLMs performance as Gaeilge, beyond surface-level fluency.

The leaderboard is freely accessible on this site and is updated regularly from its underlying data source. Údarás na Gaeltachta welcomes continued collaboration with AI developers, academic institutions, and language bodies who share the goal of ensuring that Irish is not merely processed by AI systems, but genuinely used by them with the fluency, accuracy, and cultural understanding the language deserves.