Evaluating Large Language Models on Irish language tasks.
Currently included Benchmarks:
While these benchmarks evaluate LLM performance as Gaeilge on a number of important tasks, there is still plenty of scope for further work to evaluate how useful these LLMs serve Irish speakers in their work and daily lives.