Údarás na Gaeltachta launches Ireland's first AI language benchmark leaderboard for the Irish Language

New public leaderboard assesses not just whether AI models can translate Irish, but whether they can genuinely use it, reasoning, generating, and engaging with the language across grammar, culture, and context.

Údarás na Gaeltachta today announced the launch of a first-of-its-kind public benchmarking leaderboard evaluating the performance of large language models (LLMs) in the Irish language. The leaderboard was developed in partnership with University College Cork (UCC), University of Limerick and independent researchers in Dublin. It is available at Leaderboard and brings transparency and rigour to a question with profound implications for Ireland’s linguistic heritage: how well does modern AI serve Irish speakers?

Irish is a low-resource language, one with comparatively limited digital text, training data, and linguistic tooling relative to dominant languages such as English or Spanish. As generative AI reshapes how people access information, create content, and communicate, low-resource languages face a disadvantage: the less data available, the weaker the model performance, and the less likely communities are to adopt tools that do not serve them well. This leaderboard aims to measure and track that challenge, providing researchers, developers, policymakers, and the public with comparable, reproducible data on LLM capability across several Irish-language dimensions.

Tomás Ó Síochán, CEO, Údarás na Gaeltachta:

“As the body charged with the social and economic development of Gaeltacht communities, we cannot stand by while the tools that will define the next generation of digital services fail to provide for the Irish language”

Gemini Flash is currently the top model for the Irish Language:

This leaderboard will be updated regularly as new models are released but in this inaugural edition, some patterns have already emerged.

In what is perhaps the most surprising of the first results of the benchmarks in the leaderboard, a “small” model, Gemini Flash, is currently outperforming larger models tested, including in some tests, other Gemini models. It appears that what adds performance to a model in English when enlarging it may be at the cost of performance in low resource languages like Irish.

There has been significant progress over two years in this sector, but this progress does not always favour low resource languages. Comparing the three LLaMA model scores on the IrishQA benchmark, for example, indicates a rapidly moving field but progress is not linear or guaranteed for low resource languages. Iterative model optimisations that boost English benchmark scores do not necessarily carry over to Irish.

Developers and users should not assume that a newer model version automatically represents an improvement for Irish-language tasks.  In one example,  Claude 3.7 appeared to perform at a lower level than 3.5 on many Irish tasks.

AI models are now performing at levels comparable to humans on Irish-language tasks, even in their evolution, and the implications are potentially profound. If these performance trends continue, AI has the potential to become a powerful tool for language preservation, supporting learners, as well as enabling Irish-language content creation at scale.

This new leaderboard exists to increase awareness amongst developers and the public while also showing leading LLM companies the effectiveness of their offerings to the Irish language speaking community.

Colm Ó Riain, AI Advisor

“The leaderboard we are launching today is a foundation, not a ceiling. Our next priority is to expand benchmark coverage, so that alongside raw linguistic performance, we are evaluating real-world utility, cultural alignment and safety alongside raw linguistic performance. Beyond that, we plan to add cost and energy metrics to inform the rankings, because sustainable AI matters as much as capable AI, perhaps more so. Ultimately, this leaderboard should become the reference point that drives model developers to invest seriously in Irish.  Low resource languages should be central in any technical advances in the generative AI era, not an afterthought.”

The leaderboard is freely accessible at  Leaderboard and is updated live from its underlying data source. Údarás na Gaeltachta welcomes collaboration from AI developers, academic institutions, and language bodies who share the goal of ensuring that Irish is not merely processed by AI systems, but genuinely used by them with the fluency, accuracy, and cultural understanding the language deserves.