When Choosing an Embedding Model, Don't Look Only at MTEB Scores

Many people's first instinct when choosing an embedding model is to look at MTEB scores. A high score seems to equal strong capability. Karthik shared his RAG learning notes on X, and the conclusion is somewhat counterintuitive: a high MTEB score does not mean retrieval works well.

MTEB stands for Massive Text Embedding Benchmark and covers more than a dozen tasks such as classification, clustering, and semantic similarity. A high score indicates strong overall model capability, but RAG cares more about whether, given a query, the model can rank relevant documents near the top. There is a gap between these two objectives.

In his post, he mentioned RTEB (Retrieval Embedding Benchmark), a benchmark specifically designed to test retrieval and covering vertical domains such as code, healthcare, finance, and law. Compared with MTEB, it is closer to real RAG usage scenarios.

Another noteworthy point: open-weight models are now capable of competing with proprietary models. Open-source models such as Qwen3 and Nemotron are not inferior on retrieval tasks.

Beyond scores, context length, vector dimensions, hosting costs, and license terms also affect deployment. If the context is short, large document chunks cannot be fed in; if vector dimensions are high, storage and query costs rise; if licenses have many restrictions, commercial use can be blocked outright.

Karthik's final advice is very plain: do not choose a model based only on benchmark scores; test it with your own data and queries. Each model has its strengths, and what works for others may not suit you. The best model is the one that can correctly retrieve the content you need.

Someone in the comments added a practical method: build a small test set, collect real queries and their corresponding correct document chunks, and measure with recall@k. This is more reliable than looking at MTEB scores.