• | 8:00 am

Headless browsing is on the rise. Here’s why that could get scary for businesses

Once a tool for web developers and other programmers, headless browsers are how AI surfs the web, and that’s a problem.

Headless browsing is on the rise. Here’s why that could get scary for businesses
[Source photo: Rawpixel]

The rise of artificial intelligence in recent years, along with the surge in AI-generated online content, has given more credibility to a decades-old conspiracy theory known as the Dead Internet Theory. It holds that most of the content we encounter online isn’t actually produced by living humans but by lifeless bots.

AI is increasingly turning the once-fringe theory into a reality, but even today, at least one of the participants—the living, breathing observer browsing the web on the other side of the screen—is still usually a real, sentient being.

Yet this may not be true for much longer. Thanks to AI systems’ increasing reliance on a technology known as headless browsing, artificial intelligence is becoming a primary consumer of the internet. And if that happens at scale, the internet will truly be a land of the unliving.

Here’s what you need to know about headless browsing, a term you’ll likely hear increasingly often in the years ahead.

Headless browsing is nothing new

Nearly every web browser you’ve ever used—whether it’s Google Chrome, Apple’s Safari, or even Microsoft’s old Internet Explorer—is a traditional “visual” browser. It features a graphical user interface (GUI), which includes buttons, tabs, scrollbars, and, of course, a large window that displays content (i.e., a website) that you can see. You navigate a visual browser mainly by clicking with your mouse cursor on hyperlinks or other buttons on a web page. If the web page requires you to enter text, such as in a form, you click in the text entry field and use the keyboard to type your characters.

But for decades, another kind of browser has existed: the headless browser. A headless browser has no graphical user interface (GUI). It has no window that displays a webpage and does not support pointing and clicking with a mouse cursor.

Instead, a headless browser processes a website’s content by reading its code directly. It interacts with the site, such as “clicking” on a link to go to the next page or entering text into a form, all through direct interaction with its code.

Since humans are visual creatures, it’s clear why GUI browsers are the primary way most of the world accesses the internet. So then, what are headless browsers used for?

Historically, they have been tools for web developers, as enterprise proxy provider Oxylabs explains. Because every graphical user interface element on a webpage has corresponding code, an automated program designed to help devs find errors on a website running through a headless browser can interact with that website just like a person would—but much faster since no visual interface needs to be displayed. The traditional benefit of headless browsers is that websites become more stable and reliable because headless browsing allows errors to be found relatively quickly.

But human developers aren’t the only ones using headless browsers anymore.

Headless browsing in the age of AI

Once a tool for web developers and other programmers, headless browsers are now being employed by new users—who don’t have heads at all. Increasingly, artificial intelligence systems are the primary “users” of headless browsers.

AI browsers, such as Perplexity’s Comet, use headless browsing to scan websites to carry out your prompts quickly. For example, when you prompt an AI browser for a list of the capitals of the 50 United States, the browser’s AI will “read” the content of numerous websites via headless browsing to quickly retrieve the answer.

But headless browsing goes beyond letting AI scan a website to retrieve information. As artificial intelligence systems evolve from being simple answer bots to becoming personal assistants—known as AI agents—headless browsing is also being utilized by these agents to interact with websites on your behalf, performing tasks like clicking links, checking boxes, or even adding items to your shopping cart.

A large part of why an AI agent can perform tasks you prompt it to do so quickly is due to headless browsing. For example, say you prompt an AI browser to order the ingredients you need to make Thanksgiving dinner from multiple grocery websites. The browser’s AI agent isn’t actually perusing grocers’ websites through any visual interface and then clicking on “Buy Now” buttons to find and add items to your shopping cart. It’s using headless browsing to read and interact with the website’s code directly.

But while headless browsing makes AI more efficient and versatile, there’s a negative side to AI’s use of it, particularly if you’re a website or one of its advertisers.

An internet where AI is the main user, not humans

As more people turn to agentic AI and AI browsers, these AI systems will utilize headless browsing to visit websites and carry out tasks assigned by humans. This means that AI has the potential to be the primary type of “user” that is visiting a website. And there are already signs of this happening.

report from the AI monetization platform TollBit last month showed that, for the most recent quarter, human traffic to the websites that TollBit monitored declined by 9.4%, while AI traffic continued to rise. And it’s rising a lot. In the first quarter, TollBit found that 1 out of every 200 visitors to the sites it monitors was AI. By the second quarter, AI visitors accounted for 1 out of every 50 visitors. That’s a fourfold increase in less than a year.

TollBit’s report goes on to note that when AI agents visit a website, the website often has no way to tell that it’s an AI and not a human being. That’s terrible news for companies, which rely on web advertising to pay the bills. Advertisers sell things to human beings, and if advertisers can no longer trust whether a website knows precisely how many actual people are visiting it, they likely aren’t going to spend their limited ad dollars on that site.

For what it’s worth, an executive at an unnamed “large digital publisher” told Digiday that they believed headless browsing does not currently pose a major issue for publishers. However, they noted that if big players in the AI space, such as OpenAI or Google, adopt the technology for their AI agents, headless browsing could become a significant concern.

And if headless browsing does become the norm, it also means that the Dead Internet Theory could take on an expanded meaning. No longer would the phrase be used to signify only an internet where human beings do not make the majority of the content—but where the majority of the browsing is no longer done by humans either.

  Be in the Know. Subscribe to our Newsletters.

ABOUT THE AUTHOR

Michael Grothaus is a novelist, journalist, and former screenwriter. His debut novel EPIPHANY JONES is out now from Orenda Books. You can read more about him at MichaelGrothaus.com More

More Top Stories:

FROM OUR PARTNERS