Irish language LLM Leaderboard

Evaluating Large Language Models on Irish language tasks.

Currently included Benchmarks:

  • IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation. arxiv.org, May 2025
  • Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting. arxiv.org, Oct 2025
  • Silicon in Irish. substack, Jan 2026

While these benchmarks evaluate LLM performance as Gaeilge on a number of important tasks, there is still plenty of scope for further work to evaluate how useful these LLMs serve Irish speakers in their work and daily lives.

Latest update (November)