Web Information Retrieval: A Literature Review of Search Engines, Semantic Search, Neural Information Retrieval, and Generative Artificial Intelligence

The growth in digital information available via the World Wide Web has made it more important than ever to develop efficient and intelligent information retrieval solutions. Despite the effectiveness of keyword-based search in matching terms accurately, traditional methods fall short in capturing intent and semantics. As a result, web information retrieval has gone through significant progress due to semantic search, machine learning, deep learning, knowledge graphs, transformers, and generative AI technologies. This literature review focuses on the developments in web information retrieval from 2016 to 2026. It discusses advancements in retrieval models, semantic search techniques, neural information retrieval, recommender systems, large language models, and generative AI-enabled retrieval systems. Major findings include a shift from keyword-based retrieval to context-based, intent-based, and knowledge-based retrieval approaches. Although there have been numerous developments in web information retrieval to ensure enhanced accuracy and user experience, the difficulties associated with scalability, misinformation, bias, interpretability, privacy, and computational efficiency continue to be considerable.