How do you choose an embedding model for RAG?

The right embedding model is the one that scores highest on your own documents and queries — not the one at the top of a public leaderboard. Start with a shortlist from a benchmark like MTEB, then test two or three candidates on a sample of your real data, and decide on retrieval quality first, using dimensions, context length, cost, and hosting as tie-breakers.

What should you actually test for?

1. Retrieval quality on your data — build a small eval set of real questions with known-correct chunks, and measure whether the right chunk shows up in the top results. This beats any leaderboard score.
2. Domain fit — a general model can stumble on legal, medical, or code text; check whether a domain-tuned or strongly multilingual model does better on yours.
3. Dimensions — higher dimensions can lift accuracy but cost more storage and slower search; 768 to 1024 dimensions is a common practical sweet spot.
4. Context length — the model's max input decides how large a chunk it can embed without truncating, so match it to your chunk size.
5. Cost and latency — hosted APIs charge per token and add a network round-trip; self-hosted open models are free to run but need a GPU.
6. Operational fit — data residency, privacy, and compliance often matter more than a couple of benchmark points.

Open-source or hosted API?

The performance gap has largely closed. Strong open families such as BGE-M3 or the Qwen3 embedding models now match or beat well-known hosted options from providers like OpenAI, Google, Cohere, and Voyage on general retrieval. So the choice is mostly operational: hosted means zero infrastructure and easy scaling but recurring cost and sending data out; open means you run it yourself, keep data in-house, and pay in GPU time and setup instead.

For regulated data, that compliance angle usually settles it.

The one rule that saves pain later

Whatever you pick, you're somewhat locked in — switching embedding models later means re-embedding your entire corpus, because vectors from two different models aren't comparable. So test properly before you index millions of documents, and always keep your raw source text so you can rebuild the index cheaply if you ever change models.