The Web is an incredibly huge, complicated, and ever-expanding entity. Is it possible for one tool – one search engine – to index, curate, and retrieve content from the Web in a meaningful way?
In a word, no. Search engines have millions of pages in their databases (Google alone indexes close to eight billion at the time of this writing), but none of them even come close to indexing the whole Web, let alone the Internet.
What are some pages that they exclude, and why?
- The invisible web, a vast repository of information that search engines don’t have access to, such as databases
- Private networks, called intranets, that are not actually hooked up to the Web
- Forms, like ColdFusion or CGI
- Password-protected sites, like a university library
- Sites that intentionally, for various reasons, keep their information from being indexed by search engine spiders
Will the Web ever be searchable from just one platform? Judging from the exponential growth of the Web day after day, week after week, and year after year, the odds are against it. This is one reason why expert searchers tend to not rely on just one search engine for their Web search needs; one search engine can’t deliver the fullest Web search experience that many people don’t even realize that they are missing out on. It’s smart to diversify your Web search streams; here are a few resources that can help you address that:
- The Ultimate Search Engine List: A list of search engines from general to incredibly niche that you can use as a jumping off point for your Web-based exploration.
- How to Pick the Right Search Engine for Your Needs: What if you’re not exactly sure what it is you need to find what you’re looking for? This article provides a walk-through of the steps you can take to determine that.