Irish-language AI leaderboard adds four new models, including Claude Opus 4.8

One month on from launch, fresh benchmarking results show Gemini’s smaller model continuing to outperform its larger counterparts on Irish, while a new open model makes a strong entrance into the top five.

A month ago, Údarás na Gaeltachta launched a first-of-its-kind public benchmarking leaderboard evaluating the performance of large language models (LLMs) in the Irish language. Developed in partnership with University College Cork (UCC), University of Limerick and independent researchers, the leaderboard brings increasing transparency and rigour to the question: how well does modern AI serve Irish speakers?

In June, four additional models were added to the leaderboard, two further Gemini models, Anthropic’s Claude Opus 4.8, and Chinese company Z.ai’s just-released GLM 5.2. All four new entrants debut in the top five, and the latest results point to some notable developments.

Gemini 3 Flash retains its leading position on the leaderboard, but has now been joined in the top three by two other Gemini models, the larger Gemini 3 Pro and the older Gemini 2.5 Flash. The standout finding from the original launch was that a smaller Gemini model outperformed every larger model tested. That pattern has now deepened: Gemini 3 Flash even outperforms its own larger family member, Gemini 3 Pro, reinforcing the leaderboard’s central observation that the gains a model makes in English by scaling up do not necessarily translate into stronger performance in low-resource languages such as Irish. Together, the two Gemini models currently represent the state of the art for LLM performance in Irish.

Z.ai’s newly released GLM 5.2 has entered the top five and is now the best-performing open model on the leaderboard. Its improvement over other open models is striking, and it now ranks ahead of several widely used proprietary models on Irish-language tasks as well.

Anthropic’s most advanced currently available model, Claude Opus 4.8, shows overall stronger performance on Irish than any previously tested Claude model.

Meanwhile, GPT-5 from OpenAI has moved down the rankings as a result of these new entries, from 2nd to 6th position. More recent GPT versions are expected to be tested in July.

For Gaeltacht communities, learners, and Irish-language content creators, these shifts matter in practice as well as in theory: the everyday tools people choose, from chatbots to writing assistants, increasingly run on these same underlying models, so which one a person picks can shape how well their experience of using Irish actually is.

Colm Ó Riain, AI Advisor:

“A month in, these results already confirm two of our founding observations: that scale alone does not guarantee progress for low-resource languages like Irish, and that the field is moving quickly enough that leaderboard positions can shift within weeks. We’re looking forward to future evaluations of where the technology can support Pobal na Gaeilge and where it still falls short.”

The leaderboard is freely accessible at https://leaderboard.insight-centre.org/ and is updated live from its underlying data source. Údarás na Gaeltachta welcomes continued collaboration from AI developers, academic institutions, and language bodies who share the goal of ensuring that Irish is not merely processed by AI systems, but genuinely used by them with the fluency, accuracy, and cultural understanding the language deserves.