Traditional retrieval (BM25, TF-IDF) matches query keywords against document keywords, weighted by frequency and importance. It's fast, interpretable, and excellent for exact matches. Semantic retrieval encodes queries and documents as embeddings and finds nearest neighbors in vector space. It handles paraphrase and conceptual similarity but can miss exact keyword matches. Hybrid retrieval combines both, typically using reciprocal rank fusion to merge results.
For RAG, documents must be split into chunks before embedding. Chunk size is a critical design decision: too small and you lose context, too large and you dilute relevant information with noise. Common strategies include fixed-size chunks with overlap, sentence-level splitting, paragraph-level splitting, and recursive splitting that respects document structure (headers, sections). The optimal approach depends on your documents and queries.
A common pattern: retrieve a broad set of candidates (say 50) using fast retrieval, then rerank them using a more accurate (but slower) model. Cross-encoder rerankers (like Cohere Rerank or BGE-Reranker) process query-document pairs together, producing more accurate relevance scores than embedding similarity alone. This two-stage pipeline balances speed (fast initial retrieval) with accuracy (precise reranking of the top candidates).