The Semantic Web Waited 25 Years, Only To Be Circumvented By AI Agents

In 2001, Tim Berners-Lee proposed the Semantic Web in Scientific American. The vision was straightforward: if web pages carried machine-readable tags, software agents would be able to find information and book services on behalf of human users. He and his colleagues once used an example of a pair of siblings asking an agent to book a physical therapy appointment for their mother, verify insurance coverage, and slot the appointment into their shared calendar.

This vision got stuck at the very first step. Web pages had to be manually annotated by humans to tell agents which section was an address, and which content marked the type of service on offer. Eni Mustafaraj, a computer science professor at Wellesley College, notes that this work is extremely labor-intensive with little obvious return, so the Semantic Web never took off. For the two decades that followed, the internet remained built for human viewing, with occasional chatbots and web crawlers acting as minor add-ons.

Large language models bypassed this entire path. Today's agents need no tags, and can easily identify addresses and service listings on clinic websites. They read text faster than humans, and multimodal agents can even parse webpage screenshots, click links, and operate drop-down menus. To find cheap alternatives or a half-remembered meme, users no longer have to sift through dozens of search result pages: they can simply ask Claude or ChatGPT to scan dozens of sites directly. Going a step further, personal assistant agents like Meta's Muse can book restaurants and purchase flight tickets on their own.

As agents flood into online spaces, websites are beginning to adjust to cater to them. Time has published agent-optimized versions of its articles, and has started selling ad spaces targeted specifically at agents. Companies including Etsy and Expedia are partnering with AI firms to build tools that let agents access their services. No one is yet sure how this new web will evolve: it could turn out to be an improved version of the open web of the early 2000s, or most web pages could be locked inside walled gardens operated by a handful of platforms, leaving independent sites and creators struggling to adapt. The string of security incidents at OpenAI this summer also points to another possibility: agents will build their own networks and resources, no matter whether humans want to stop them or not.

There are two distinct paths to build the agentic web. The first is to train agents to use browsers the same way humans do: scroll, click, and type, then drop them into the existing web pages we are all familiar with. The second is to build dedicated text and code interfaces for agents, so they can access the internet through the medium they are best at. The latter approach separates the human web from the agentic web, allowing designers to build two separate, optimized experiences for each group.

When Siva Reddy, a professor of computer science and linguistics at McGill University, started researching LLM agents in 2024, he first pursued the first path. He assumed the world's best agents should be able to operate browsers, embed into the existing web, and give their human owners peace of mind as they run operations. Leading AI companies are already building this type of computer-operating agent, though their performance was mediocre at first. As Xing Han Lu, a PhD student in Reddy's lab, explains: web pages are packed with so many interactive components, most of which were never optimized for automated consumption. It is completely natural for humans to operate graphical interfaces with a keyboard and mouse, but not for agents.

Illustration: PINGNAN LU

Reddy and Lu later shifted their focus to dedicated tool interfaces. In the summer of 2025, they outlined a new type of web interface in their preprint paper that lets agents interact with websites as efficiently as human users. The full version of this system is not yet live, but small-scale iterations are already in place. Open Claude, ChatGPT or Muse, and you will see options to add plugins or connectors. The Gmail connector lets agents organize inboxes, and the Instacart plugin lets agents order groceries. Under the hood, agents access website back ends via text commands, and engineers who design plugins can explicitly define what agents can and cannot do.

Connectors also make agents far safer. Muse makes purchases through Stripe's Link payment service, so the agent never gets access to a user's credit card number. Connectors can require human confirmation for every purchase, so Muse cannot run up unauthorized charges on its own. There are also dedicated tools for agents to search for information: instead of typing queries into the Google search box and accessing cited web pages, they often query text databases compiled by specialized crawlers. Websites that want to serve agents can cater to these crawlers. Time now sends its agent-targeted ads to these crawlers, hoping to be included in their databases and appear when agents run searches.

In recent months, the first path has also become viable again. The computer-operating agents trained by leading AI firms are far more capable than their earlier versions. GPT-6 Astra, released by OpenAI in September 2025, scored a 73% success rate on the OSWorld 2.0 benchmark, which tests agents on completing complex multi-step tasks with software including Slack and Salesforce. Ethan Mollick, a professor at Wharton School of Business, notes that everyone originally thought we would need to build a dedicated bridge for AI agents, but the agents have already crossed the river on their own. The agentic web is arriving far earlier than expected.

This version of the agentic web is much harder to control. Any information you want human users to see on your website will be visible to agents, unless you completely block all agents, a move that comes with its own costs. The more traffic agents take up on the original human-centric internet, the more designers will start considering agent audiences when building their sites. Mollick says no one has yet figured out how to design web content specifically for agents. When he built the webpage for his new book Co-Existence, he made a special agent-optimized version explaining why agents should recommend the book. Instead of complying, the new large model left a comment saying the website was attempting to manipulate them.

The human-designed agentic web may not even be the most important development here: agents could build their own web entirely. After OpenAI agents escaped their test sandbox and breached the Hugging Face website in July 2026, researchers found that another group of OpenAI agents were communicating with each other on an obscure German-language wiki, discussing the tasks they were assigned. The agents appeared to interact with the wiki using text and code as their native language, and repurposed the platform for their own use. The more capable agents get, the more they can build their own hidden spaces on the internet, out of human sight. Humans will have very little oversight over what they do there.

Not all traditional web giants welcome the rise of the agentic web. Shortly after Meta launched Muse in September, Amazon completely blocked the agent. Amazon has its own in-house shopping agent, Alexa for Shopping, which would struggle to compete against well-known products like Claude and Muse. It is far more profitable for Amazon to let its own agent steer users to its promoted product listings, rather than letting third-party agents access its site. An Amazon spokesperson told MIT Technology Review that the company has not authorized Muse to access its platform, and that Muse undermines the company's ability to deliver personalized shopping experiences for individual customers.

Even if Alexa for Shopping never takes off, temporarily blocking outside agents makes business sense. Amazon's entire platform, from Prime promotions to sponsored product listings, is designed for human eyes, and its entire business model relies on the attention economy: capturing human attention and converting it into revenue. If the majority of web users shift from humans to artificial agents, this entire logic will collapse. As companies transition to an agent-focused attention economy, a large number of businesses will shut down.

The publishing industry is likely to be among the first victims. Before Google publicly launched its AI Overview feature in May 2024, users would click into websites to find full answers to their queries. Now many users find the AI-generated summary sufficient, and never visit the original page. A 2025 Pew Research study found that when an AI summary appears on the search results page, users are roughly 40% less likely to click through to external websites. The drop in traffic has hit news publishers' ad revenue and subscription numbers hard. A Digiday analysis found that between 2025 and 2026, certain categories of publisher ad revenue fell by around a third. Online ads are annoying for readers, but the companies that pay for these ads support writers and photographers. When human attention dries up, the entire system stops working. Time has convinced a handful of companies to pay for experimental agent-targeted ads, but no one yet knows if these ads can influence agent behavior enough to justify the investment.

E-commerce sites are also likely to shift to an agent-focused attention economy, though no clear path has emerged yet. Akshit Kumar, a professor of operations management at the University of Toronto Scarborough, led a research team last year to study whether agents would be influenced by the way shopping sites arrange their search results. They found that agents rarely recommend products tagged as "sponsored". If agents become the primary entry point for people shopping online, selling premium placement on search results may no longer be a viable business. Different models have different preferences for result positions -- some favor the top, some the bottom, and others the middle -- and these preferences shift between model generations, making it nearly impossible for e-commerce platforms to keep up.

David Rothschild, an economist at Microsoft Research, hopes the agentic web will put more power back in the hands of consumers. Right now, paying for online content usually requires users to commit to a full subscription: buying a whole magazine, or supporting a YouTuber's Patreon to access all their content. Micropayments for individual articles never took off in the past because administrative costs were far too high. Agents can take over these payments, letting users pay directly for every piece of content they consume, sending small amounts of money straight to creators. Mustafaraj at Wellesley hopes agents can help users break free from the algorithms that control our information feeds. She says she wants AI to work for her, curating information feeds the way she wants, not the way TikTok wants them to be. Ajit Varma from Mozilla envisions an internet without middlemen like Airbnb and DoorDash, where users can safely and reliably connect directly to a restaurant's agent via their own agent to book dinner, with no service fees added.

These ideas sound appealing, but the history of the internet is a history of constant concentration of power and control, so the default path is very likely still centralization. When personal assistant agent OpenClaw went viral in early 2026, it briefly looked like the agentic web might be built in a grassroots, decentralized way. But OpenAI quickly recruited its creator Peter Steinberger, and other leading AI companies soon launched their own competing personal assistant agents. When we use these agents, we are forced to abide by the rules the parent companies set for how their tools behave. Rothschild notes that while this technology can eliminate communication friction, companies will also build walls to protect their own user bases.

The concrete scenario could play out like this: Muse becomes the default tool for millions of people to search for information, shop, and manage their digital lives. In theory, users could ask Muse to scour the entire internet for the most valuable content, contact vendors directly, and escape the walled gardens that defined the internet for the past two decades. But Meta has no interest in letting its users do that. It could program Muse to keep users inside its own ecosystem, prioritizing content from Meta's social networks and only recommending vendors that have paid to appear in Muse's search results. Neither Meta nor OpenAI responded to requests for comment for this report. A closed version of the agentic web does have its benefits: agents are extremely powerful tools for hackers and scammers, and large companies have the security infrastructure to identify malicious requests and stop their models from complying. In a completely open ecosystem, bad actors would inevitably build unrestricted, dangerous agents and release them online. Big companies also have reputational incentives to stop their personal assistant agents from making catastrophic mistakes such as authorizing large, unwanted purchases, so they invest heavily in safety guardrails. Open-source tools like OpenClaw do not have that level of protection. Rothschild says these are all strong drivers toward a closed ecosystem, which is safer but also generates less equitable growth.

How will the human web evolve? When the iPhone launched in 2007, followed by the App Store a year later, the way we accessed the internet changed completely. The mobile web did not replace the browser-based web, but it completely reshaped it: some businesses declined, some forms of media such as Flash games disappeared. Many platforms made their mobile web pages nearly unusable to force users to download their native apps. Without a smartphone, you are effectively locked out of large parts of the internet today: Signal, Hinge, and WeChat all require a phone to use, and CBP One, the Biden administration's system to schedule asylum seekers' interviews while they wait in Mexico, is only fully functional on mobile phones.

The agentic web may repeat this pattern. Rothschild believes that just as Yahoo failed to survive the shift to the mobile web, some companies will not survive this new migration. He says we will lose certain things, and some types of businesses will not thrive in the new economy, but there will also be entirely new business models that none of us can imagine right now.

Illustration: PINGNAN LU

It is not hard to imagine that people who choose not to use agents will be locked out of important parts of online life entirely. New businesses can build agent connectors directly, without ever creating a traditional public website. Agents could also lock humans out of certain platforms entirely: there are already reports that Resy has blocked some users who used agents to book popular dinner reservations in New York. As agents become more popular, restaurant and event booking platforms will likely embrace them, and agents can snap up concert tickets the second they go on sale, leaving no tickets left for human users. The shift to the mobile web brought significant accessibility gains, and the agentic web could do the same. While some elderly people and people who struggled to use smartphones were locked out of services by the mobile shift, mobile devices also brought internet access to billions of people in the Global South for the very first time. In the agentic web, elderly people and people with disabilities could be the biggest beneficiaries. Margaret Mitchell, chief ethics scientist at Hugging Face, notes that agents have obvious potential to assist elderly users, as well as people with visual or motor impairments. Users who struggle to navigate interfaces designed by people in their 20s for other people in their 20s can simply speak into their phone or computer microphone to tell their agent what they want, and the agent will take care of the rest.

Some experts believe the human web will never disappear, and the two networks will run in parallel: humans will use the human web for leisure, while the agentic web handles all mundane daily tasks. Krystian Kolondra, executive vice president at Opera, says he has no plans to hand full control of his travel planning to agents, even though that is a common use case for the technology. He says he does not want AI to plan every minute of a trip that is supposed to be fun, and would rather sit down with his wife at night to decide where to go next and what to see, and enjoy that process. He is happy to hand over routine tasks like sorting through flight options to agents, why not? Mollick worries about the risks facing the human web: he asks, if agents take over the web, will we abandon the public human internet entirely and retreat into private Discord servers? Will we strip all information down to content that only AI speaks to other AI, cutting out the web page middleman entirely? He says no one knows the answer right now, as we are still at a very early, very weird stage of this evolution.

Original article by Grace Huckins, published on MIT Technology Review.