International Journal of Scientific Research and Engineering Development

Retrieval used to mean one thing: match the words in a query against the words in a document, count how often they overlap, and rank accordingly. That approach carried search engines and enterprise document systems for the better part of four decades, and it still runs underneath a surprising number of production systems today. What has changed is what sits on top of it. Once large language models became capable enough to synthesize answers rather than just list documents, the retrieval problem quietly turned into a different problem: not "which documents contain these words" but "which passages, out of millions, actually let a model answer this question correctly." That shift in the question being asked is what has driven the last decade of retrieval research, and it is the throughline this paper follows. We trace the path from Boolean and TF-IDF retrieval through BM25, into dense embedding-based semantic search, approximate nearest neighbor indexing, hybrid lexical-semantic fusion, cross-encoder reranking, and finally into the multi-stage, permission-aware retrieval pipelines that sit inside modern RetrievalAugmented Generation (RAG) systems. Rather than treating each of these as an isolated technique, we look at why each one appeared when it did, which failure of its predecessor it was responding to, and what new problem it introduced in the process — because none of these transitions were free. We also present a generalized architecture, distilled from patterns observed across enterprise-grade retrieval deployments, that separates ingestion, indexing, retrieval, reranking, and generation into layers that can be reasoned about independently. Along the way we discuss chunking strategy, embedding dimensionality, index structures such as HNSW and IVF, similarity functions, metadata filtering, and the evaluation metrics — Recall@K, MRR, nDCG — that let teams actually measure whether a retrieval change helped or hurt. The paper closes by looking at where the field appears to be heading: adaptive, query-dependent retrieval strategies and agentic systems that decide for themselves how much retrieval a question actually needs.

Varun Reddy Maram, Nandan Reddy Maram, "The Evolution of Retrieval-Augmented Generation: From Traditional Information Retrieval to Intelligent Multi-Stage Retrieval Systems" International Journal of Scientific Research and Engineering Development, V9(4): Page(374-385) May-June 2026. ISSN: 2581-7175. www.ijsred.com. Published by Scientific and Academic Research Publishing.