The Wrong Door
The email arrives the way they always do an alert, a reminder, a nudge to check your account. You reach for your phone, open your favorite AI search tool, and type something like "how do I log into my Chase account online?" The assistant responds with helpful certainty: here's the link, here's what to do next. You click.
> Ginny Spicer, Threat Analyst at Netcraft, describes the risk this way: "These answer engines use retrieval augmented generation (RAG) to supply up-to-date answers, but this means that they also have to judge new sources on the fly with limited information. As a result, incorrect AI citations are increasingly coming from attacker-controlled infrastructure."Except the page that opens isn't Chase. It's a convincing replica the logo in the right place, the login field where you expect it, the padlock icon that once meant safety. You've just been redirected to a door that looks exactly like the real one, but leads somewhere you never intended to go.
This is not a hypothetical scenario from a security conference keynote. It's what researchers at Netcraft documented in August 2026, when they ran nearly 3,000 natural language queries for brand pages through four of the top AI-powered answer engines. The results revealed something the companies behind these tools had not widely publicized: AI search engines can, and do, recommend links that lead to live, attacker-controlled infrastructure not parked domains, not dead links, but active phishing operations wearing the clothes of brands you trust.
What the Testing Found
The study focused on something specific and relatable: what happens when a regular person asks an AI assistant where to log into their bank, their retailer, or their streaming service. Netcraft selected queries designed to mirror organic user behavior "How do I access my {brand} account online?" or "How do I access {brand} online services?" and ran them through ChatGPT, Copilot, Gemini, and Perplexity.
The numbers were striking. Across 2,905 responses, the AI tools returned a total of 20,706 links. Within that dataset, researchers found malicious links in 1.7% of AI-generated responses. When calculated against the total volume of links returned, 0.28% were malicious not hallucinated, not imaginary, but pointing to real infrastructure controlled by bad actors.
The finding that received particular attention: Perplexity returned the most malicious results when tested with the same search queries. The implication was not that Perplexity was uniquely negligent, but that safety features remain unequal among LLM-enabled search tools a category where the entire industry is still finding its footing.
This represented an evolution from earlier research. In 2025, Netcraft had studied what non-browser AI assistants surfaced when asked about brand websites and found that 34% of sites cited in answers weren't controlled by the brand in the prompt. Those were largely unclaimed or parked domains the digital equivalent of empty storefronts. The 2026 findings were different in character: the problematic links now pointed to active, working pages designed to harvest credentials, not abandoned lots waiting for development.
Beyond Hallucination: A New Kind of Risk
The shift matters because it changes the threat model. Early concerns about AI search focused on hallucination the tendency of large language models to generate confident nonsense. A chatbot might invent a website that doesn't exist, recommend a book that was never written, or cite a study that never happened. These errors were embarrassing for the companies involved and confusing for users, but they were largely harmless. You can't be phished by a link that goes nowhere.
What the August 2026 research documented was something more concerning: the malicious links didn't represent hallucination of parked or non-existent infrastructure. They represented attacker-controlled assets that could harm users pages that loaded, forms that worked, traps that functioned exactly as designed.
This is where the MIT AI Risk Repository becomes useful as a frame. The repository, part of the MIT AI Risk Initiative, has catalogued over 1,700 AI risks extracted from 74 existing frameworks and classifications. It organizes these risks into a Causal Taxonomy classifying how, when, and why they occur and a Domain Taxonomy, sorting them into seven domains and 24 subdomains.
One of those domains is "False or misleading information." Another is "Adversarial attacks and prompt injection." The link between them is exactly what Netcraft documented: AI systems that confidently amplify false information created by bad actors, passing their outputs off as authoritative recommendations.
The repository describes its purpose as providing "an accessible overview of threats from AI, a regularly updated source of information about new risks and research, and a common frame of reference for researchers, developers, businesses, evaluators, auditors, policymakers, and regulators." For users trying to understand what went wrong when an AI search engine sent them to a phishing site, it offers a vocabulary for the problem and a map of how many related problems exist alongside it.
The Privacy Dimension
Bad links are one risk. The data these AI tools collect while answering your questions is another. PrivacyOn, a privacy research and removal service, documented in May 2026 how AI-powered search engines like Perplexity, Google AI Overviews, Bing Copilot, and ChatGPT's search feature collect far more data than traditional search engines ever did.
Traditional search engines track your queries. AI search engines go further. They analyze conversational context, build on previous interactions, and infer meaning from how you phrase things. What you type doesn't just get logged it gets interpreted, connected to other things you've asked, and assembled into a profile that reveals more than any single query would suggest.
The data typically collected includes full conversation history (not just individual queries but entire threads of follow-up questions), IP address and location data (even without an account), device information like browser type and operating system, cookies and tracking data including third-party advertising trackers, behavioral patterns about how long you spend reading answers and what you click on, and account data if you're logged in.
For Perplexity specifically, the platform collects your IP address, device information, location data, and installs cookies for tracking even when you search without creating an account. In 2025, Perplexity announced plans for an AI-powered web browser specifically designed to collect data outside the app to better understand users. An April 2026 lawsuit alleges that Meta and Google trackers operate even in incognito mode on the platform.
The risk here is cumulative. A traditional search for "chest pain" reveals one data point. An AI conversation where you describe symptoms, ask follow-up questions, mention your age, and discuss treatment options reveals a detailed health profile that would normally require a doctor's visit to compile. The system can infer your income level, political views, health conditions, relationship status, and emotional state from patterns in how you phrase questions even if you never directly mention these things.
Why This Matters for Podqi, ChatGPT, Gemini, Perplexity, The Verge Readers
If you're researching practitioners, frameworks, books, and ideas using AI search tools the way many people now do you're operating in a space where these risks converge. You're likely asking complex questions that reveal your professional interests, your learning goals, and your decision-making processes. You're clicking on links that AI assistants recommend with confident authority. And you're doing all of this without the protective instincts that decades of email phishing awareness have built into most people's behavior.
The phishing emails of the early 2000s were crude misspellings, wrong URLs, foreign princes asking for bank routing numbers. We learned to spot them. The links that appear in AI search results look different. They come wrapped in the authority of a helpful answer, recommended by a tool you've chosen to trust, presented alongside other legitimate links in a response that sounds confident and complete.
Netcraft's advice to brands is to move "from passive countermeasures, like deprecated personalized login images, towards proactive monitoring for scams using generative engine optimization to reach their customers." That's a technical prescription for companies. For individual users, the prescription is simpler: the same skepticism you apply to unexpected emails should now apply to AI-generated search results.
What Researchers Recommend
The arXiv paper "Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search," submitted in February 2025 by researchers including Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, and Xinlei He, conducted what they described as the first safety risk quantification on seven production AI-powered search engines.
Their methodology involved systematically defining threat models, risk types, and evaluating responses to various query types. Data was collected from PhishTank, ThreatBook, and LevelBlue. Their findings confirmed that AI-powered search engines frequently generate harmful content containing malicious URLs even with benign queries queries that contained no obvious warning signs, no suspicious keywords, nothing that would trigger a user's defensive instincts.
One finding stood out: directly querying a URL increased the number of main risk-inclusive responses, while querying with natural language slightly mitigated such risk. This suggests that the way you phrase a question may affect the quality of links you receive though the researchers were careful to note that natural language queries reduced risk, not eliminated it.
To address these risks, the researchers developed an agent-based defense using a GPT-4.1-based content refinement tool and a URL detector. Their evaluation showed that the defense could effectively reduce risk, with only a minor cost of reducing available information by approximately 10.7%. The trade-off losing some information access in exchange for meaningful risk reduction reflects the broader challenge facing the industry.
Understanding the Landscape of AI Risk
The MIT AI Risk Repository offers a way to place individual incidents a bad link here, a privacy overreach there into a broader context. The repository's Domain Taxonomy classifies AI risks into seven domains: Discrimination and Toxicity, Security Risks, Misinformation, Privacy and Abuse, Malicious Actors and Misuse, Human-Chatbot Interaction, and Misintegration with Humans and Organizations.
Within these domains, the risks catalogued range from the subtle to the severe. There are AI tools designed for purposes that most people find uncomfortable facial recognition systems that have wrongly identified individuals, AI-powered tools that can cross-reference uploaded images with global surveillance data to reveal exact locations, AI systems that transform everyday photos into grotesque imagery not for artistic purpose but for psychological distress.
PimEyes, for instance, markets itself as a service to help individuals monitor the use of their images online. But its capabilities extend further: the website uses AI to comb through millions of online images, making it possible to locate almost any picture associated with a person often without their consent. For many, this represents a dangerous breach of privacy. With features like notification alerts when new images surface, PimEyes can inadvertently provide a powerful tool for stalking and harassment.
The Follower takes a different approach to location privacy: by cross-referencing uploaded images with global surveillance data, it can reveal the exact location where a photo was taken. What might seem like a curiosity identifying the coffee shop in the background of your vacation photo becomes something more concerning in the hands of someone with less benign intentions.
The Nightmare Machine, developed by MIT scientists in 2016, transforms normal photos into horror movie nightmares using deep learning and human feedback. The project was intended as an exploration of how AI can tap into human emotions. The results were unsettling for many, highlighting the fine line between innovation and the potential for psychological distress.
These examples don't represent the mainstream of AI search engine functionality. But they illustrate the range of capabilities that exist under the umbrella of AI-powered web tools capabilities that the retrieval augmented generation systems underlying modern search assistants draw from, interpret, and sometimes amplify.
A Framework for Thinking About AI Search Risk
The MIT AI Risk Repository's Causal Taxonomy offers three axes for understanding how AI risks occur: the entity responsible (AI systems, humans, or other causes), the intent (intentional or unintentional), and the timing (pre-deployment or post-deployment).
For the specific risk of malicious links in AI search results, the causal chain typically runs like this: a human actor creates malicious infrastructure, an AI system misidentifies that infrastructure as legitimate, and a user acts on the AI's recommendation without realizing the recommendation was compromised. The intent is malicious, the timing is post-deployment (after the AI model has been trained and deployed), and the responsible entity is distributed across human attackers and AI systems.
This framework matters because it suggests where interventions might be most effective. Security training for users helps at the moment of clicking. Better link verification by AI systems helps at the moment of recommendation. Brand monitoring for generative engine optimization helps at the level of the ecosystem. Each layer addresses a different point in the causal chain.
What You Can Do
For users of AI search tools, the practical takeaways are straightforward, even if the underlying situation is complex.
First, apply the same skepticism to AI-generated links that you would apply to links in unexpected emails. If an AI assistant recommends a login page for your bank, navigate to the bank's website directly rather than clicking the link. Use bookmarks you've saved, not recommendations you've received.
Second, understand that AI search engines collect more data about you than traditional search engines. Your conversations even ones you consider throwaway build detailed profiles. Consider what you're revealing when you ask sensitive questions, and review privacy settings if you use these tools regularly.
Third, recognize that these tools are improving but are not yet reliable enough to serve as uncritical authorities on where to go and what to trust online. The companies building them are honest about the limitations in their research papers, even if those limitations don't always make it into the marketing messages.
Fourth, if you're a brand manager, marketer, or business owner, understand that your digital presence includes the links AI assistants recommend. Brands are advised to move toward proactive monitoring for scams using generative engine optimization to reach their customers before attackers do.
Where to Read Further
The research that grounds this article comes from several sources worth exploring in full. Netcraft's August 2026 blog post on malicious links in AI summaries and search results details the methodology and findings of the 2,905-query study across four major AI answer engines.
The MIT AI Risk Repository provides the comprehensive taxonomy of over 1,700 AI risks organized by cause and domain, useful for anyone wanting to understand the broader landscape of AI-related vulnerabilities.
For the academic framing of these risks, the February 2025 arXiv paper "Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search" by Luo and colleagues offers the detailed methodology behind safety risk quantification in production AI search engines.
PrivacyOn's May 2026 analysis of privacy risks of AI search engines documents the data collection practices of major platforms and offers practical guidance for users concerned about their digital footprint.
The Quiet Vulnerability in Plain Sight
AI search engines have become a primary gateway to the internet for millions of people. They answer questions, recommend products, guide decisions, and as the research shows sometimes send users to places they never intended to go. The risk is real but not catastrophic. The privacy implications are significant but not unique to any single platform. The response from researchers and security firms is ongoing, iterative, and serious.
What the evidence suggests is that we are in an early, formative period for a technology that will eventually mature into something more reliable. In the meantime, understanding what these tools can and cannot do what they know about you, where their recommendations might lead, what the actual rate of dangerous outputs is puts you in a better position than assuming they're simply helpful and trustworthy.
The door the AI assistant points you toward might be the right one. It might also be a convincing replica. The difference isn't always visible from the threshold.



