There’s been a lot of TV and movie coverage about something called the “Dark Web.” I’ve received several questions here at Web Search Secrets and within our Facebook community about it, so I thought it could be useful to write something up.
The Dark Web: what is it, really?
Before I get into that, it’s probably best to start by explaining that there are actually several “layers”, so to speak, of the Web; the Surface Web, the Invisible Web, and the Dark Web. The web that we’re all used to – the one that offers up our favorite sports websites, gossip news, online magazines, etc. – that is commonly known as the Surface Web. The Surface Web includes any content that is easily crawled, or indexed, by search engines.
However, there’s a limit to what search engines include in their indexes. That’s where the term “invisible web” comes into play. The term “invisible web” mainly refers to the vast repository of information that search engines and directories don’t have direct access to and are not including in their index, like databases and libraries and court records. Unlike pages on the visible Web (that is, the Web that you can access from search engines and directories), information in databases is generally inaccessible to the software spiders and crawlers that create search engine indexes. There’s generally nothing nefarious going on here; there are several different factors as to why a site would not be included in a search engine index, but basically they boil down to technical barriers and/or deliberate decisions on the part of the site owner(s) to exclude their pages from search engine spiders. For instance, university library sites that require passwords to access their information will not be included in search engine results, as well as script-based pages that are not easily read by search engine spiders. There’s also really large databases in there, both public and private; anything from the U.S. National Oceanic and Atmospheric Administration, NASA, the Patent and Trademark Office and the Securities and Exchange Commission’s EDGAR search system to databases like LexisNexis, which require a fee to search.
How do you access the Invisible Web?
It used to be that these pages were hard to get to, but search engines have gotten pretty sophisticated. Search engines’ crawlers and indexing programs have overcome many of the technical barriers that made it impossible for them to find “invisible” web pages. But there are still some pages that are not making it into search engines; you can still find them directly if you know how – basically, you can piggyback on search engines to drill down into databases to find these pages; for example, if you did a search for “weather” and “database”, you’d come up with some pretty fascinating information. From there you can drill down into the database to find what you’re looking for.
What is the Dark Web?
Now we can finally get to what the Dark Web – also known as DarkNet – really is. If the Surface Web is basically everything that a search engine offers up in its index, and the Invisible Web – which, incidentally, is estimated to be at least 500x times larger than the Surface Web – is basically information that a search engine does not or cannot include in its index, then the Dark Web is a relatively small portion of the Invisible or Deep Web, one that has a lot of different stuff going on, anything from drug trafficking to murder for hire to people who are looking to share information safely in an unsafe environment or culture, with complete freedom from censorship; in other words, it’s not all bad stuff going on there.
How do you get there?
The Dark Web is made up of sites that require a specialized browser and protocols in order to access. You can’t just type a Dark Web URL into your average Web browser and reach it, in other words. Access to these sites is not via the regular process of a .com site; and they’re not indexed by search engines, so navigation here is tricky.
Basically, it’s hidden and takes some level of computer sophistication to reach.
Most people aren’t going to just casually drop by the Dark Web. It’s not just a matter of following a link or using a search engine, which is what the majority of us are used to online.
You have to download specialized browser clients in order to access the Dark Web – and these tools are going to do two things for you: they’re going to connect you to the subset of networks that make up the Dark Web, and they are going to completely anonymize your every step by encrypting where you are, where’ you’re coming from, what you’re doing, everything. You are anonymous, which is the main draw of the Dark Web – however, this does NOT mean that you’re completely untraceable, as, if you listen to the news, you’ll be able to ascertain as we hear about people being caught doing some pretty illegal stuff via the Dark Web on a regular basis. Using these tools makes you much more difficult to track, but not impossible. And it’s also important to recognize that while downloading these encryption tools and clients is definitely not illegal, you can become a “person of interest” so to speak by using them; it seems to be a pattern with people who are breaking the law here that they start on the Dark Web and then end up somewhere else, so it’s just part of tracing that process.
Who uses it and why?
The Dark Web has somewhat of an unsavory reputation; if you’re a House of Cards fan you probably remember the whole scene with the reporter looking to dig up dirt on the Vice President and contacting someone on the Dark Web to do it. There’s been a lot of Dark Web stuff in the media as well.
The Dark Web’s offer of anonymity is definitely a huge draw for people who are looking to procure drugs, weapons, and other illicit items, but it’s also gained notoriety as a safe haven of sorts for journalists and people who need to share information but can’t share it safely.
For example, many people visited a storefront called the Silk Road on the Dark Web. The Silk Road was a large marketplace within the Dark Web, mostly infamous for the buying and selling of illegal narcotics, but also offering a wide variety of other goods for sale. Users could only purchase goods here using Bitcoins; virtual currency that is hidden inside the anonymous networks that make up the Dark Web. This marketplace was shut down in 2013 and is currently under investigation; according to several sources, there were over one billion worth of goods sold here before it was taken offline.
So while that definitely includes illegal activities – for example, buying stuff on the Silk Road, or digging up illegal images and sharing them – there are also people using the Dark Web who are legitimately in need of anonymity because their life is in danger or the information they are in possession of is too volatile to share publicly. Journalists have been known to use the Dark Web to contact sources anonymously or store sensitive documents.
Bottom line is that if you’re on the Dark Web, you’re there because you don’t want anyone to know what you’re doing or where you are, and you’ve taken very specific steps to make that a reality.
What can you find on the Dark Web?
So the Dark Web is organized – and I use that term very loosely – very differently than what we’re used to seeing, with web pages and links. There’s very little structure here; you can find different directories that have been organized by category to make browsing here a little easier, and these categories range from news to forums to hacking information. There’s a lot of stuff that is illegal or toeing the line here, but there’s also homegrown alternatives to various commercial services; like an alternative to Pandora, or Twitter, or Reedit. Scammers and people who are looking for easy marks abound on the Dark Web, too, unfortunately; there’s also the risk of really being scammed by someone trolling the Dark Web looking for people who are not doing a good job of covering their tracks; I’ve heard from a couple people that this is becoming a big issue.
To be honest though, the Dark Web is not the super exciting undercover haven that a lot of movies and TV shows are making it out to be. Many sites are non-functional. A lot of the black market stuff is still there, and there are always new sites popping up – you can definitely still find some pretty active forums that are mostly conspiracy-related. There’s a lot of crazy and terrible stuff on there, but there’s also some really interesting content. Is it worth visiting? That’s up to every individual to make up their minds about; the Dark Web has become a haven for all sorts of different activities; not all of them strictly aboveboard. It’s an important part of the Web that bears careful monitoring as privacy concerns grow in importance to society at large.
What are the future implications for the Internet? E.g. Avoid NSA scrutiny? Use in politically volatile places (e.g. Syria)?
Well, I can’t speak for the entire Internet, but yes, the Dark Web could have uses for people who want to stay anonymous, for whatever reason. Journalists use it now – for example, the New York Times has a secure lockbox on the Dark Web that people can send files anonymously to.
However, I do need to clarify that the Dark Web and the tools you use to access it – to stay anonymous – are two completely different things. Many people use anonymizers, the most well-known of which is Tor, to make sure their activities online are private, especially as we hear more about the NSA keeping track of everyday Americans. This is a major issue for many people and I don’t see that going away. The Dark Web is merely a destination.
And as for countries where Internet use is restricted; yes, anonymizing tools and proxies could help, definitely. But not just to get on the Dark Web; just to access the Surface Web, the Web that most of us use on a daily basis without any issues.
There are definitely legitimate applications to using TOR and other anonymizing tools; getting around censorship, having personal and private communication that is not monitored by government entities, etc.
I think the Dark Web will continue to grow and evolve; the appeal of an anonymous pipeline for various activities is just too appealing to resist. But I also believe that as more people are growing concerned about their completely legal online activities, communications, etc. are being monitored that tools that help us be more anonymous will also grow in popularity.