MindSearch: A Multi-Agent AI Framework Processing 300+ Web Pages in Under 3 Minutes to Enhance Information Retrieval and Integration

Information seeking and integration are critical processes that underpin analysis and decision-making across various fields. These processes demand significant time and effort, especially when dealing with complex queries that require thorough and precise information retrieval. Traditional search engines have reshaped how to seek information but often fall short when aligning with complex human intentions. The inefficiencies in retrieving and integrating information from the web have long posed challenges for users who need detailed and accurate data quickly.

One of the main issues with current information-seeking methods is their inability to handle complex queries effectively. Traditional search engines frequently provide fragmented and noisy search results, making it difficult to find the necessary information. This problem is exacerbated when dealing with complex queries that require detailed and precise responses. Furthermore, the overwhelming volume of irrelevant information and the limitations imposed by the maximum context length of large language models (LLMs) add to the complexity of information retrieval and integration.

LLMs and search engines are often used together to address these challenges. Despite the progress made by LLMs in reasoning, language understanding, and information integration, these methods still fail to perform satisfactorily for complex information-seeking tasks. Existing solutions often treat the information-seeking and integration task as a straightforward retrieve-augmented generation (RAG) task, which leads to sub-optimal performance. They need help to decompose complex queries effectively, manage the overwhelming volume of search results, and integrate information efficiently within the context length limits of LLMs.

Researchers from the University of Science and Technology of China and the Shanghai AI Laboratory have introduced MindSearch, a novel framework designed to mimic human cognitive processes in web information-seeking and integration. MindSearch is a multi-agent framework consisting of a WebPlanner and multiple WebSearchers. This innovative system leverages the strengths of both LLMs and search engines, providing a more effective solution for complex information-seeking tasks.

MindSearch operates by decomposing complex user queries into smaller, manageable sub-questions. The WebPlanner orchestrates this process by modeling the query as a dynamic graph. This graph construction process involves breaking down the user query into atomic sub-questions, represented as nodes in the graph. The WebSearcher then performs hierarchical information retrieval, addressing each sub-question and collecting valuable data for the WebPlanner. This multi-agent design enables MindSearch to seek and integrate information from a larger scale of web pages—more than 300—in just three minutes, a task that would take human experts approximately three hours to complete.

The WebPlanner in MindSearch functions as a high-level planner, orchestrating reasoning steps and coordinating the WebSearchers. It decomposes complex queries into multiple atomic sub-questions that can be solved in parallel. By leveraging the superior performance of current LLMs in code generation, the WebPlanner interacts with the dynamic graph through code writing. This process involves adding nodes and edges to the graph, progressively decomposing the query, and efficiently managing the information retrieval process. The WebSearcher, tasked with each sub-question, employs a hierarchical retrieval process to extract valuable data from the internet, significantly improving the efficiency of information aggregation.

MindSearch has demonstrated significant improvements in response quality. Experimental evaluations on closed-set and open-set question-answering tasks using GPT-4o and InternLM2.5-7B-Chat models have shown substantial enhancements in the depth and breadth of responses. In comparative analyses, human evaluators preferred responses from MindSearch over those from existing applications like ChatGPT-Web and Perplexity.ai. MindSearch’s ability to process over 300 web pages in under three minutes showcases its efficiency and effectiveness in handling complex queries.

MindSearch offers a simple multi-agent solution to complex information-seeking and integration tasks. Its explicit role distribution among specialized agents improves long-context management, facilitating more robust handling of complex and lengthy contexts. This design reduces the cognitive load on each agent and ensures that the information retrieval and integration processes are performed more efficiently. The framework’s ability to dynamically construct reasoning paths and manage context across multiple agents leads to better performance in solving complex problems.

In conclusion, MindSearch addresses the fundamental issues of traditional information-seeking methods by introducing a robust, multi-agent framework that combines the cognitive abilities of LLMs with the extensive data access of search engines. This innovative approach significantly improves the precision and recall of retrieved web information, making it a highly competitive solution for AI-driven search engines. MindSearch’s ability to efficiently decompose complex queries and manage the information retrieval process sets it apart from existing solutions.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter..

Don’t Forget to join our 47k+ ML SubReddit

Find Upcoming AI Webinars here

Nikhil is an intern consultant at Marktechpost. He is pursuing an integrated dual degree in Materials at the Indian Institute of Technology, Kharagpur. Nikhil is an AI/ML enthusiast who is always researching applications in fields like biomaterials and biomedical science. With a strong background in Material Science, he is exploring new advancements and creating opportunities to contribute.