
5 months ago, Údarás na Gaeltachta launched a first-of-its-kind public benchmarking leaderboard evaluating the performance of large language models (LLMs) in the Irish language. Developed in partnership with University College Cork (UCC), University of Limerick and independent researchers, the leaderboard brings increasing transparency and rigour to the question: how well does modern AI serve Irish speakers?
In this 2nd update, 7 additional models were evaluated, including 3 open-weighted models — Mistral-Medium 3.5, Mistral-Medium 3.5 (no-thinking), GLM 5.2 (no-thinking), Gemini Flash 3.5, GPT 5.2, GPT 5.5, and Claude Fable 5 — bringing the total number of models evaluated to 20.
While in the previous leaderboard, Google’s Gemini models represented the state of the art for LLM performance in Irish, Anthropic’s Claude Fable model outperformed them all, and now sits on top of the leaderboard. A newer Gemini Flash model, 3.5, made it into the top 5 but interestingly, still did not perform as well on the leaderboard’s benchmarks as earlier versions 3.0 and 3.1. OpenAI’s GPT 5.5 also broke through into the top 5.
Our two primary takeaways from this update:
In this update, a benchmark that had become saturated was omitted and we plan next to evaluate newer frontier models and to facilitate the creation and inclusion of new benchmarks that dig deeper into LLMs performance as Gaeilge, beyond surface-level fluency.
The leaderboard is freely accessible on this site and is updated regularly from its underlying data source. Údarás na Gaeltachta welcomes continued collaboration with AI developers, academic institutions, and language bodies who share the goal of ensuring that Irish is not merely processed by AI systems, but genuinely used by them with the fluency, accuracy, and cultural understanding the language deserves.