MEXA: Multilingual Evaluation of Open English-Centric LLMs via Cross-Lingual Alignment

About

We introduce MEXA, a method for assessing the multilingual capabilities of English-centric large language models (LLMs). MEXA builds on the observation that English-centric LLMs semantically use English as a kind of pivot language in their intermediate layers. Mexa computes the alignment between non-English languages and English using parallel sentences, estimating the transfer of language understanding capabilities from English to other languages through this alignment. This metric can be useful in estimating task performance, provided we know the English performance in the task and the alignment score between languages derived from a parallel dataset.

Code

https://github.com/cisnlp/MEXA

Details

We use parallel datasets from FLORES and the Bible. In the ARC style, we use mean pooling over layers, and the English score achieved by each LLM in the ARC benchmark is used to adjust the multilingual scores. In the Belebele style, we use max pooling over layers, and the English score achieved by each LLM in Belebele is used to adjust the multilingual scores.

Parallel Dataset

Choose parallel dataset!

FLORES Bible


llama3.1-70B	Swahili (individual language)	eng_Latn	0.8235	0.7014	0.9456	llama3.1-70B Swahili (individual language)


llama3.1-70B	English	eng_Latn	0.8235	0.7014	0.9456	llama3.1-70B English
gemma2-9B	English	eng_Latn	0.7976	0.6775	0.9178	gemma2-9B English
llama3.1-8B	English	eng_Latn	0.7281	0.5794	0.8767	llama3.1-8B English
llama3-8B	English	eng_Latn	0.7263	0.5836	0.8689	llama3-8B English
gemma-7B	English	eng_Latn	0.7168	0.587	0.8467	gemma-7B English
mistral-7B	English	eng_Latn	0.7126	0.5862	0.8389	mistral-7B English
olmo-7B	English	eng_Latn	0.6291	0.4872	0.7711	olmo-7B English
llama2-7B	English	eng_Latn	0.4975	0.5128	0.4822	llama2-7B English
llama1-7B	English	eng_Latn	0.4578	0.5	0.4156	llama1-7B English

Citation

@inproceedings{kargaran-etal-2025-mexa,
    title = "{MEXA}: Multilingual Evaluation of {E}nglish-Centric {LLM}s via Cross-Lingual Alignment",
    author = "Kargaran, Amir Hossein  and
      Modarressi, Ali  and
      Nikeghbal, Nafiseh  and
      Diesner, Jana  and
      Yvon, Fran{\c{c}}ois  and
      Schuetze, Hinrich",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.1385/",
    doi = "10.18653/v1/2025.findings-acl.1385",
    pages = "27001--27023",
    ISBN = "979-8-89176-256-5",
}