The State of Search Results: A Comparative Analysis
Is the quality of search results declining? We put Google, Bing, and alternative search engines to the test, comparing results on various queries to see if the concerns of tech enthusiasts are valid. Find out what we discovered.
The quality of search results has been a topic of debate among tech enthusiasts. While some argue that search results are still great, others claim that they have deteriorated significantly over the years. In this post, we'll delve into the world of search engines, comparing the results of Google, Bing, Marginalia, Kagi, Mwmbl, and ChatGPT.
To start, let's consider a simple experiment. Take an open-source project like yt-dlp and try to find it using a generic term like "YouTube downloader." You'll likely struggle to find it due to the plethora of content farms that dominate the top results. This is a common issue, and many tech folks I've spoken to agree that Google search results have worsened over the past decade.
To test this hypothesis, I ran a series of queries across the various search engines. The queries fell into three categories: representative queries used by a friend to set up her new computer, queries I had in high school that I couldn't find answers to at the time, and a local query for information I needed while writing this post.
The results were subjective, as it's challenging to define what constitutes a "good" result. However, I considered factors like relevance, accuracy, and the presence of scams or misleading information. I also turned off my ad blocker to simulate the experience of a non-expert user.
The queries I ran included:
- Download YouTube videos
- Ad blocker
- Download Firefox
- Why do wider tires have better grip?
- Why do they keep making CPU transistors smaller?
- Vancouver snow forecast winter 2023
The results were rated on a scale from Terrible to Great, with worse results being more red and better results being more blue. Here's a summary of the results:
- Marginalia performed relatively well, providing decent but not great answers and fewer scams than other search engines.
- ChatGPT showed promise, but its results were inconsistent, with some queries yielding great answers and others resulting in "hallucinations" or nonsense.
- Mwmbl allowed users to directly edit search result rankings, but this feature didn't significantly improve the results.
- Google and Bing struggled with scams, SEO garbage, and misleading information, often prioritizing ads over relevant results.
- Kagi had a mix of good and bad results, with some queries yielding relevant answers and others returning scams or irrelevant information.
One interesting observation was Google's propensity to return recent YouTube videos, which often seemed useless and were likely attempts to game the algorithm for ad revenue. I also found that Google's results were heavily influenced by localization, with many results tailored to my location in Vancouver, Canada.
ChatGPT's performance was inconsistent, with some queries yielding great results and others resulting in nonsense or "hallucinations." While ChatGPT can be impressive, its limitations are still evident, and it's not a reliable replacement for traditional search engines.
The debate surrounding search result quality is ongoing, with some arguing that the issues are overblown and others claiming that search engines have become less effective. Based on my analysis, it appears that search result quality has indeed declined, with many search engines prioritizing ads and SEO-optimized content over relevant, accurate information.
In conclusion, the state of search results is a complex issue, and there's no clear winner among the search engines I tested. However, by understanding the strengths and weaknesses of each search engine, users can develop strategies to find the information they need. Whether it's using alternative search engines, employing specific query techniques, or relying on expert knowledge, there are ways to navigate the ever-changing landscape of search results.
Appendix: Other Search Engines
- DuckDuckGo: Similar results to Bing, with some variation.
- wiby.me: Unique results, but often irrelevant or low-quality.
- searchmysite.net: Some relevant results, but many scams and low-quality links.
- indieweb-search.jamesg.blog: Currently offline due to server errors.
- Teclis: Closed due to bot abuse, with results available through Kagi's search engine.
Appendix: Queries that Return Good Results
- For the tire query, I found a decent answer by searching YouTube and following a comment thread that led to a relevant book.
- For other queries, I'll cover the strategies for finding good results in a future post.
Appendix: Summary of Query Results
- Download YouTube videos: No search engine returned a good result, with many scams and low-quality links.
- Ad blocker: ChatGPT provided the best result, with ublock origin as the top suggestion.
- Download Firefox: Bing and Mwmbl returned the most relevant results, with few scams or low-quality links.
- Why do wider tires have better grip?: No search engine provided a correct answer, with many incorrect or misleading explanations.
- Why do they keep making CPU transistors smaller?: No search engine provided a complete or accurate answer, with many non-answers or misleading explanations.
- Vancouver snow forecast winter 2023: No search engine provided a correct answer, with many scams, low-quality links, or irrelevant information.
Appendix: Detailed Query Results
- Download YouTube videos: Google and Bing returned many scams and low-quality links, while Marginalia and Mwmbl had fewer scams but still low-quality results.
- Ad blocker: Google and Bing returned many ads and low-quality links, while ChatGPT provided the best result.
- Download Firefox: Bing and Mwmbl returned the most relevant results, with few scams or low-quality links.
- Why do wider tires have better grip?: Google and Bing returned many incorrect or misleading explanations, while Marginalia and Kagi had fewer but still low-quality results.
- Why do they keep making CPU transistors smaller?: Google and Bing returned many non-answers or misleading explanations, while Kagi had some relevant but incomplete information.
- Vancouver snow forecast winter 2023: Google and Bing returned many scams, low-quality links, or irrelevant information, while Marginalia and Mwmbl had fewer but still low-quality results.
Appendix: Google "Knowledge Card" Results
- Many knowledge card results were incorrect or misleading, with some providing false information or irrelevant answers.
Appendix: FAQ
- This post is not a peer-reviewed study, but rather a subjective analysis of search results.
- The results are based on a limited set of queries and may not be representative of all search engine performance.
- Users may have different experiences with search engines based on their individual query techniques and preferences.