I rigorously tested the most popular AI search engines to discover their strengths and weaknesses.
I ran this test for two reasons:
- As a human being, I want to use the best search engine around. I spend my life on the internet. I want to find information in the fastest, most effortless way possible.
- As a marketer, I want to know where the tide is headed. Will AI search engines replace “traditional” search engines? Are AI chatbots really good enough to dethrone Google? Should I start investing more effort in LLM optimization?
There are basically a thousand products claiming to be AI search engines, but I focused on the biggest and most popular.
I tested out the AI features on offer from two existing search giants—Google’s AI Overviews and Bing AI—and two trendy AI upstarts—ChatGPT Search and Perplexity.
The best AI search engines are—in my opinion—ChatGPT Search for the best chatbot experience, and Google’s AI Overviews for the best all-around search engine.
Read on for pros and cons, screenshots, and short videos of the search experience.
| AI search engine | Best for… | Pricing |
|---|---|---|
| ChatGPT Search | Best AI chatbot search experience | Free; $20/month for Pro |
| Google AI Overviews | Best combination of traditional search and AI features | Free |
| Perplexity | Best for research and news | Free; $20/month for Pro |
| Bing AI | Best for simple searches | Free |
In order to properly judge the quality of each search engine’s results, I focused on a topic where I’m a world-class expert: digital marketing the role-playing video game Skyrim.
Because I’m an SEO, I also wanted to test as many different types of search queries as possible. I brainstormed tons of Skyrim searches grouped by different categories, including:
- Short-tail queries (e.g. skyrim)
- Long-tail queries (e.g. how to solve the mzulft oculory puzzle)
- Transactional queries (e.g. best paid skyrim mods)
- Informational queries (e.g. how to level up smithing quickly in skyrim)
- Navigational queries (e.g. uesp skyrim map)
- Local queries (e.g. buy skyrim near me)
- Image queries (e.g. skyrim landscapes)
- Trending queries (e.g. skyrim nolvus)
How to find tons of keywords on any topic
The Intents filter in Keywords Explorer was a big help for brainstorming keywords. I ran a search for “skyrim”, navigated to the Matching terms report, and filtered by different intents to find ideas.
I ran these searches through every AI search engine and recorded my experience. I focused particularly on:
- Relevance: were the results appropriate for my query?
- Accuracy: did the search engines return valid, up-to-date information?
- Usefulness: did I meet the intent of my search? Did I leave happy?
- Speed: how quickly did I get my answer?
- Ease-of-use: how much effort did it require?
This is a completely unscientific but very interesting analysis of the best AI search engines. Let’s get started.
My opinion: A research assistant that excels with news and summarization, but doesn’t really “feel” like a search engine.
Price: Free; $20/month for Pro
Link: https://www.perplexity.ai/
Pros
- Excellent at summarising lots of sources.
- Great for news-related topics.
- The “Related topics” feature is useful.
Cons
- Plenty of incorrect answers and hallucinations.
- Bad at image, commercial, and local searches.
- Didn’t really feel like a search engine.
Perplexity feels more like a Wiki or a knowledge base than it does a search engine. Many of its responses are terse bulleted lists, and it excelled in summarising technical queries (like DA09Dawnbreaker) or news-related topics (like elder scrolls vi news).
I also noticed lots of errors and hallucinations in its output. One example: for Should I become a vampire or a werewolf in skyrim?, Perplexity suggested that werewolves were immune to silver weapons—they actually take bonus damage from them. (C’mon Perplexity, honestly.)
Information often seemed outdated too. Perplexity said that Skyrim SE was on sale for $7.99 on Steam (it wasn’t), and the sale extended until May 30… 2024. It feels like Perplexity’s index isn’t anywhere near as fresh as other AI search engines.
Perplexity also regularly failed to “get” the intent behind my searches, and defaulted to word salad paragraphs of text. When I searched for “Steam skyrim”, I got a long, rambling response with no actual link to Steam. For “Skyrim creations”, it listed no actual creations, and just shared a Wiki-style overview of the Creation concept.
Perplexity also struggled with image searches, commercial queries were very unsatisfying, and local searches were decidedly not local, with responses that recommended websites and shops that were nowhere near me.
My opinion: A great all-rounder search engine that incorporates AI in limited—but useful—ways.
Price: Free
Link: https://www.google.com/
Pros
- AI Overviews were generally helpful (when they showed up).
- Does a great job at surfacing different media formats, like images and videos.
- Excels at commercial and transactional searches.
Cons
- Very cluttered search results, often with duplication between AI Overviews and Fatured Snippets.
- AI Overviews feel like a dead-end—you can’t continue to interact with them.
- Felt a little weird having Reddit in the majority of SERPs.
It was fun evaluating Google Search through fresh eyes. Unsurprisingly, Google was very good for commercial and local queries: interactive product carousels made it easy to find relevant products in no time, and map packs were fantastic for finding local shops.
I was also reminded of just how powerful Google’s integration with YouTube is: any queries that were best resolved with video content almost always included YouTube videos—often timestamped—in the SERP.
AI and machine learning underpin all of Google Search, but as far as specific AI features, AI Overviews only triggered for about 15% of my queries. These were all informational queries, and in most cases, they were very specific long-tail queries.
The AI Overviews were actually pretty good, relevant and accurate—but I definitely missed the ability to keep interacting with the AI.
AI Overviews generally felt like amped-up Feature snippets, providing direct, certain answers to long-tail queries in SERPs (and in fact, many of the AI Overviews competed directly with duplicate Featured snippets directly below them in the SERP).
It was also a little weird having Reddit present in so many SERPs. Some queries really are best-suited by user generated content—like those with no objective “best” answer—but Reddit muscled its way in everywhere.
My opinion: The best AI chatbot for specific, text-heavy searches (but still lacking some “classic” search features).
Price: Free; $20/month for Pro
Link: https://chatgpt.com/
Pros
- Excellent for very specific long-tail queries.
- The ability to ask follow-on questions is extremely good.
- “Memory” feature is useful.
Cons
- Outright failed on several occasions.
- Pretty bad for image searches and commercial searches.
- Verbose, likes to write long answers even for simple questions.
ChatGPT Search is fantastic for very specific long-tail queries. Answers to queries like Should I become a vampire or a werewolf in skyrim? were like reading a short blog post that was written just to summarise a bunch of sources specifically for your query—because, well, it was.
Couple this with the ability to ask follow-up questions, and ChatGPT’s ability to remember information from previous conversations (its “memory”), and asking long, complicated questions is a ton of fun.
But this strength is also a weakness. ChatGPT excels with text-heavy queries, and kinda sucks for many others. Short-tail queries felt way more limited than Google. Navigational queries (like skyrim creation kit download) were unsatisfying: ChatGPT would explain precisely how to download the Creation Kit (a modding engine for Skyrim), but it couldn’t actually let you do it.
It was also pretty weak with commercial queries, and very bad at image queries, with a default response limited to just four images (nowhere near enough to sate my appetite for hi-res skyrim landscapes). Although one surprise: ChatGPT Search incorporated playable YouTube videos directly into its responses.
ChatGPT Search outright failed for a few queries, requiring multiple refreshes to get results for very basic queries. All-in-all, it was strongest for queries that benefited from long, detailed text responses, and worse on almost every level for all the other queries.
My opinion: A solid second-choice search engine with experimental AI features that don’t really deliver.
Price: Free
Link: https://www.bing.com/
Pros
- Great at navigational and commercial searches.
- Interesting search features for short-tail queries.
Cons
- AI features were quite often broken and unhelpful.
- Slightly disappointing local and news searches.
Bing has two outwardly obvious “AI” features. One is a text snippet, very similar to Google’s Featured Snippets and AI Overviews. These worked okay, but not as well as Google’s AI Overviews.
Often, the AI answers didn’t quite understand my search (the response for Should I become a vampire or a werewolf in skyrim? didn’t actually include an answer—just steps to become either a vampire or werewolf).
The other was a dynamically generated “table of contents” module, and despite looking nice, it generally fell somewhere between not helpful and totally broken, with the different modules appearing in jumbled orders or missing steps.
Like Google Search, Bing performed perfectly at navigational queries and showed tons of products and inspiration for commercial searches. It returned good results for newer, trending topics (as did all the search engines, surprisingly).
Most of all, I loved the visual sidebar that appeared for the short-tail searches, offering tons of facts and trivia for skyrim. When the search intent is so so vague, it seems incredibly helpful to treat the SERP as a mood board of inspiration and possible wormholes to pursue.
Surprisingly, I was disappointed with local searches, and the news search (Elder scrolls vi news) was pretty awful.
Final thoughts
“Traditional” search engines have decades of experience in making searchers—and all the different types of searches they make—happy. This experience is obvious across commercial, local, and image searches where Google and Bing dominated the newer tools.
But “pure” AI search engines also unlock new types of search behavior. The query Skyrim bosmer stealth archer build backstory was sensational with ChatGPT, allowing me to actively brainstorm and refine a story in a way I never could with Google Search. Perplexity excelled at summarising news and complex information.
Are the strengths of AI search engines enough to make up for their many weaknesses? Will searchers abandon a company that controls 88.5% of the current search engine market to experiment with relatively unproven new tools?
More likely (in my estimation) is that we’ll see the existing search engines build, integrate, and partner with AI companies to bring the best features into the existing search experience (like Bing’s partnership with OpenAI.)
In the meantime, the best AI search engines are—in my opinion—ChatGPT Search for the best chatbot experience, and Google’s AI Overviews for the best all-around search engine.
