LingOly-TOO
LingOly-TOO (L2) is a challenging linguistics reasoning benchmark designed to counteracts answering without reasoning (e.g. by guessing or memorizing answers). We accomplish this by permuting Linguistics Olympiad problems with Templates and Orthographic Obfuscations. By rewriting (obfuscating) parts of questions and answers, the chance of benchmark leakage in training data is minimized.
"How do top LLMs reason on unseen linguistic questions?"
Paper Code Blog Dataset Add Your Model
16
Total Models
8 Reasoning Models
5 Open Source Models
82
Linguistics Problems
6 Permutations per problem
Problems from Low-resource Languages
1.2k
Total Questions
6,995 sub-question and answer pairs including Match-Up, Multiple Choice and Completion tasks

Citation

@inproceedings{ khouja2026lingolytoo, title={{LingOly}-{TOO}: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation}, author={Jude Khouja and Lingyi Yang and Simeon Hellsten and Karolina Korgul and Vlad A. Neacșu and Harry Mayne and Ryan Othniel Kearns and Andrew M. Bean and Adam Mahdi}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://arxiv.org/abs/2503.02972} }