TREC 2025: Million LLMs Track

Overview

In the era of Large Language Models (LLMs), we envision a transformation in how information is served to users. Imagine a world where:

Content providers like Wikipedia, NY Times, Reddit, and Elsevier offer information through specialized LLMs
Some LLMs provide cited responses with original source materials, while others offer direct answers
Different levels of cooperation exist among LLMs, from those sharing embeddings and internal weights to those providing only text outputs
Various access models coexist, from free-to-query to cost-based services
Millions of LLMs operate with distinct, sometimes overlapping expertise domains

Research Context

The challenge bears similarities to established fields such as:

Distributed Information Retrieval (DIR)
Federated Search (FS)
Meta-search

A key distinction lies in the nature of collecting statistics: while traditional search engines return thousands of results enabling reliable statistics, LLM responses are typically brief, making expertise assessment more challenging.

Key Challenges

1. Evaluation Framework Development

While we propose a specific simulation and evaluation framework, our assumptions may not fully reflect real-world conditions. Bridging this gap and developing more accurate evaluation methodologies remains a crucial challenge.

2. LLM Expertise Quantification

Unlike traditional DIR systems where search engines provide extensive results, LLMs typically respond with concise answers, sometimes with limited references. This brevity complicates the assessment of LLM expertise.

3. LLM Ranking Optimization

Even with robust expertise assessment, determining optimal LLM ranking strategies for meta-LLM agents remains complex. While this year's track focuses primarily on LLM ranking, future iterations may explore this challenge further.

4. Answer Synthesis

The challenge of effectively combining multiple LLM responses into coherent, comprehensive answers represents an open research question with limited existing work.