LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal

Abstract

Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document ($q$-$d$) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this internal signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL test collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings when compared with human judgments. Our results suggest that $q$-$d$ relevance is encoded as a depth-dependent internal signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments further suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.

Publication
Proceedings of the 35th International Conference on Information and Knowledge Management (CIKM 2026)
Date
Links