Google's AI Could Trick You Into Calling a Scammer

The Growing Concern of AI-Generated Search Results

Artificial intelligence has become an integral part of modern life, influencing everything from entertainment to communication. However, despite its widespread adoption, AI technology is not without flaws. While it may appear as if generative AI possesses vast knowledge, it can often make errors or even fabricate information. This concern is particularly relevant for tech reporters and users who rely on AI-driven tools in their daily routines.

One such tool that has seen significant changes due to AI integration is search engines. Since the late 1990s, people have relied heavily on search results to find the information they need. Most individuals simply open a search engine, type in a query, and accept the top results without much scrutiny. With the introduction of Google's AI Overviews, many users are now viewing AI-generated summaries at the top of search pages, often taking them at face value without further verification.

A Critical Issue with AI Overviews

A major issue with this new approach lies in the potential for misinformation. According to Digital Trends, when users search for a company's phone number, Google's AI Overviews and even AI Mode might recommend a scammer’s phone number instead. Scammers are effectively "hacking" Google's AI by manipulating the data it uses to generate results.

Digital Trends highlights several instances where this has occurred. One example involves Alex Rivlin, who tried to contact Royal Caribbean's customer service but couldn't find the support number on their website. He searched for "royal caribbean customer service phone number 24 hours usa" and called the number that appeared in the AI Overview. The interaction initially seemed legitimate, but Rivlin became suspicious when the representative asked for his date of birth, which he already had on file with the company. After hanging up, he discovered unauthorized charges on his credit card and reported the incident.

How Scammers Exploit AI Systems

The problem arises because scammers are listing fake numbers on fraudulent websites, tricking Google into sourcing that data. Google's AI then sees "Royal Caribbean" next to this phone number on a .gov site, which appears legitimate, and surfaces it in an AI-generated result. Even after the site is taken down, Google continues to pull the outdated information, leading users to fall victim to scams.

Another example comes from a Redditor who searched for "how to fix a misspelled name on Southwest." The AI Overview provided a scam number, but the user recognized the discrepancy and avoided falling for the trap. In another case, a man lost over $3,400 after calling a number that appeared for Swiggy, a food delivery service.

The Flaws in AI Technology

Google's AI Overviews is not inherently malicious; rather, it is flawed. The underlying technology struggles to differentiate between accurate and false information. It lacks the awareness to understand that a site impersonating a government page can host a number and claim it belongs to a specific business. To the AI, that means the number is legitimate, and it includes it in the results.

This flaw was evident during the initial rollout of AI Overviews, where the model pulled jokes from Reddit as legitimate sources. For instance, it once suggested that glue could make cheese stick to pizza, highlighting the risks of relying solely on AI-generated content.

Recommendations for Users

While AI can still provide accurate results from high-quality sources, there are inherent risks in trusting AI answers without verification. If you're unsure about the reliability of an AI-generated response, click through the sources to check where the information is coming from. If the source seems questionable, assume the answer may also be unreliable.

When dealing with company contact information, it's best to go directly to the official website. If a phone number isn't listed there, it may not exist, and you should seek alternative contact methods. Scammers are becoming increasingly sophisticated, making it crucial to remain vigilant and avoid relying solely on search results.