Parag Agrawal’s newest project is trying to solve one of the messiest questions in AI: how to compensate content creators in a world where AI agents, instead of humans, are surfing the web.
The former Twitter CEO’s company, Parallel Web Systems, is launching Index, a platform that gives publishers, data providers, and independent creators visibility into how AI agents are using their content and a new way to be compensated for that use. Launch partners include publishers and distributors such as The Atlantic, Fortune, and PR Newswire; business and data intelligence providers including PitchBook, Enigma, RocketReach, and ZoomInfo; and independent creators including Alex Heath’s Sources, Packy McCormick’s Not Boring, and Mario Gabriele’s The Generalist
“The core thesis of the company was that agents will use the web a lot more than humans, and as a result of that, everything about the web will change, both the technology and the business models,” Agrawal told Fortune.
Rather than clicking around like human readers, AI agents can pull from dozens or hundreds of sources simultaneously to complete various tasks for users. Parallel already sells web access infrastructure to AI companies and developers, including Harvey, Notion, and Opendoor.
Index is aimed at trying to help content owners understand how AI agents use their work. The platform is built around a concept called Shapley value, a game theory idea for estimating how much each participant contributes to a collective outcome. In Parallel’s version, rather than paying only for content access or citations, Index tries to estimate how much a particular source contributed to an AI agent’s completed task and the value of the AI agent’s overall work. A source that is more unique or used in more valuable work should theoretically receive more compensation.
Parallel intends to pay participating publishers using that value-estimation model. The system will first apply to AI agents using Parallel’s own tools, though the company says it wants Index to eventually work with agents built outside Parallel too. Agrawal says that AI companies have an incentive to participate because their agents need access to high-quality content, especially as more publishers put up barriers to keep AI crawlers and agents out.
“AI agents are becoming the next major interface for accessing information, but the economics of the web have not caught up with that reality,” Nicholas Thompson, CEO of The Atlantic said in a prepared statement. “Parallel is tackling this by creating a dynamic and scalable model for recognizing and compensating publishers.” (Fortune is a member of the Parallel’s Index platform. As part of the program, the startup shares ad revenue with publications when their content is used by AI agents.)
It’s a different approach from the fixed-fee licensing deals that have dominated the relationship between AI companies and publishers so far. OpenAI, for example, has signed licensing deals with publishers including the Associated Press, Axel Springer, and News Corp.
Agrawal says these fixed-price deals won’t work for the AI agent age, and risk leaving smaller publishers and AI startups out of the market.
“If only a few large companies have access to the premium content and no one else does, how will anyone compete?” he said.
The agent economy
The launch comes amid several years of tension between AI companies and content owners.
The New York Times sued OpenAI and Microsoft in 2023, alleging that millions of Times articles were used to train AI systems without permission. Dow Jones and the New York Post have also sued AI startup Perplexity, alleging the company illegally used their copyrighted material. Authors have also challenged AI companies over books used in large models’ training data. Last year, Anthropic agreed to a major settlement in a copyright case brought by authors.
Parallel is not the only company trying to create a new economic model for AI access to the web. Last year, Cloudflare launched Pay Per Crawl in private beta, a product that lets publishers charge AI crawlers a flat per-request price across their domain.
Both Index and Cloudflare’s offerings are trying to move beyond the binary choice of blocking AI crawlers or allowing free access. The main difference is that Cloudflare’s model is based on crawl access, while Parallel’s Index is trying to tie compensation to the value of the agent’s work and the source’s contribution to it.
Index’s model depends on the difficult task of estimating which sources mattered most to an AI output. Publishers, already wary of AI’s effect on traditional revenue streams like traffic, may also be concerned about becoming too dependent on yet another intermediary.
Agrawal says that uncertainty is part of the reason to start with transparency. Index will let site owners enter their domain and see how Parallel’s agents are using their content. Over time, the company hopes more partners will join and provide feedback on the model.
Parallel, founded about two and a half years ago, has raised a $100 million Series B at a $2 billion valuation in April, five months after a $100 million Series A at a $740 million valuation.