Overview
This paper examines the effect of Google’s introduction of AI Overviews and AI Mode on user behavior and site visits for publishers on the platform. The authors conducted a preregistered field experiment with 1,100 U.S. participants who used Google Search in their everyday browsing for ten days. A browser extension randomly assigned each participant to one of three conditions—AI Overviews hidden, ordinary Google Search, or all searches redirected to AI Mode—and captured both real browsing behavior and post-experiment perceptions. Key findings include:
- AI search garners less trust. Participants using AI mode reported significantly less trust in information on Google, along with lower levels of perceived usefulness, satisfaction and agency. Those using AI mode were also more likely to search for information through a competitor like Bing.
- AI Mode substantially reduced engagement with the broader web. Participants assigned to AI Mode clicked through to external sites 18.8 percentage points less often and ran approximately one fewer search session per day. Participants using AI search also clicked through to external sites less often, including news sites (by 12.5 percentage points), Reddit (21.2), and Wikipedia (9.9). Notably, these users spent more time inside the AI chat window.
- AI search did not improve user experience. Relative to ordinary Google Search, AI Mode reduced trust in information found on Google and significantly lowered perceived usefulness, satisfaction, agency, and personalization. It also pushed more users toward competing search engines such as Bing. In open-ended responses, negative reactions outnumbered positive ones, with participants describing a loss of control regarding AI use, difficulty reaching specific websites they wanted, and fewer and less diverse sources.
Why Is This Important?
This paper provides evidence to show how Google’s introduction of AI Overviews reduces traffic for third-party publishers and that the conversational AI mode worsens user experience. As Google and other search engines experiment with AI responses, this paper shows that users are not broadly accepting of these changes, with some reporting less control and trust in AI mediated information on Google.