Agent web index

Standard search engines were built around a simple assumption: a person types a query, scans a page of results, and clicks a link. AI agents need something different. They can read full documents, benefit from much higher recall, often need hundreds of source texts at once, and have no use for click-rank signals derived from human behavior. An agent web index is infrastructure designed for that reality: a large, fresh, structured crawl of the open web, exposed via an API that agents call the same way they call any other tool.

The distinction matters because the economics and design of a human search index diverge from what agents need. Relevance ranking for humans favors brevity, page authority, and click-through rate. Agents favor completeness, freshness, and document fidelity. An agent writing a research report needs the actual content of twenty primary sources, not a ranked list of URLs to visit. Startups like Keenable are building this category directly; established players like Perplexity and Brave are adapting their indexes toward agent consumption as well.

Builders integrating retrieval into agentic workflows should understand whether the index they are calling was built for agents or adapted from human search. The difference shows up in recall quality, latency at high document volumes, and how well the API maps to the context window patterns agents actually use. As agents become the dominant consumers of web information, agent-native indexes are likely to become their own infrastructure category, separate from the search products humans interact with directly.