The internet contains a massive amount of information, but not all of it is accessible through a standard web browser. There is in fact a whole segment of the internet that lies beyond the surface, which can be divided into the deep web and the dark web. Though sometimes used interchangeably, these two terms refer to very different areas of the internet.
What is the Deep Web?
The deep web refers to online content that, while publicly available, cannot be accessed through a standard web search. This includes things like:
- PasswordProtected Sites: Subscription sites or sites that require a login like online banking, email, or social media.
- Unlinked Databases: Data that is not linked to by other sites, like library catalogs and government databases.
- Unindexed Content: Websites or pages that search engines cannot crawl, like dynamically generated content.
Essentially it refers to any content that does not show up on Google, Bing, or other conventional search engines.
How Big is the Deep Web?
The deep web is thought to make up over 90 percent of the internet. For comparison:
| Web Type | Size (in petabytes) | Percentage of internet |
|---|---|---|
| Surface web | 1 | <10% |
| Deep web | 95 | >90% |
Some deep web databases hold an incredible amount of valuable data. For example, the National Climatic Data Center contains over 275 PB of climate data alone. This information remains hidden from standard search engines but can be unlocked with the proper credentials or search tools.
Why Does the Deep Web Exist?
There are a few reasons why such a massive corner of the internet remains tucked out of search engines’ reach:
Privacy and Security
Pages like your email inbox contain sensitive personal data and are hidden behind passwords for privacy reasons. Keeping this content private and secure takes priority over findability.
Storage Practicality
Many databases add or update content far too rapidly for search engine crawlers to keep up. Keeping constantly changing data essentially unlinked simplifies data management.
Intentional Isolation
In some cases, such as government databases, content isolation serves an institutional purpose, allowing restriction of access as needed. Leaving documents unlinked essentially hides them in plain sight for those “in the know.”
Is the Deep Web Legal?
For the most part, yes. Much deep web content is mundane and legitimate, though harder find. While illicit or questionable content undoubtedly exists in some deep web back channels, the deep web itself is not synonymous with illegal activity. In fact much of the scientific, corporate, or personal data it shields serves legitimate confidentiality purposes. Simply being deep web content does not make something shady or unlawful.
What is the Dark Web?
The dark web refers specifically to websites that exist on encrypted networks like Tor or I2P that allow users to browse anonymously and hide their locations. These anonymous networks overlay the open internet, providing hidden services.
How Does the Dark Web Work?
Dark web sites are hosted on servers set up by individuals and specially configured to hide the identity and location of the servers’ owners. Accessing the dark web requires special routing software like Tor Browser or I2P which encrypt traffic and obscure users’ digital footprints.
This dark web network configuration enables a number of uses, both legal and illegal. Some examples include:
- Whistleblowing – Anonymously sharing information in oppressive political regimes
- Journalism – Communicating news without threat of retaliation
- Personal Privacy – Preserving privacy and defending against surveillance
- Unfortunately also:
- Illicit Marketplaces – Dealing weapons, drugs, exploitables
- Hacking/Phishing – Sharing tools and tips for illegal cyber activities
This ability to hide one’s identity and digital tracks means the dark web offers cover for unlawful schemes as well as legal aims like political speech or journalism.
Is the Dark Web Illegal?
The dark web itself is merely a routing configuration, which can be put to both legitimate and unlawful ends. But because this anonymous network shelters illicit dealings, the dark web is far more associated with cybercrime than the deep web.
Some estimate over 50% of dark web activity is illegal, though estimates vary. Unlawful offerings common on dark web markets include:
- Illegal drugs
- Firearms
- Hacking services
- Counterfeit documents
- Stolen credit cards
Accessing the Dark Web Safely
Browsing the dark web poses greater security risks than surface web browsing. But the anonymity of networks like Tor goes both ways hiding your tracks while seeing hidden services. Keeping safe on the dark web requires special precautions:
- Download Tor from the official website to avoid malware
- Never enter personal info on dark web sites
- Use end-to-end encryption tools for messages
- Verify PGP signatures on downloadable files
- Only transfer information over encrypted connections
Staying safe also means avoiding illegal offerings altogether. Curiosity can lead people to access illegal marketplaces, which puts them at serious risk of cyber attacks or legal penalties.
Surface Web vs Deep Web vs Dark Web
To summarize, here is how the surface web, deep web, and dark web differ:
| Publicly Indexed | Anonymizing Networks | Illegal Content | |
|---|---|---|---|
| Surface web | Yes | No | No |
| Deep web | No | No | No |
| Dark web | No | Yes | Sometimes |
So while the deep web hides in plain sight, the dark web actively anonymizes behind encryption, sometimes offering shadowy dealings as a result.
Conclusion
The deep web’s hidden caches of data and the dark web’s anonymous corridors differ greatly despite being lumped together. The deep web conceals immense stores of mundane information behind passwords and databases. Meanwhile the dark web lies beyond the reach of tracking, letting users anonymize both for better and worse.
Dark networks unavoidably shade some illicit dealings. But they also make crucial privacy, speech, and security tools possible for citizens, investigators, and journalists alike. The deep web similarly secures critical privacy around finance, medicine, intellectual property, and personal communication.
Far from full of sinister secrets, the untracked corners of the internet harbor the practical tools and datasets we use privately every day. While undoubtedly some unscrupulous activity slips between the web’s cracks, much of this hidden content is sensitive by necessity rather than nefarious by nature.
FAQs
Why can’t search engines index the deep web?
Much deep web content is locked behind logins or frequently updating databases, putting it beyond what search engine crawlers can access or keep up with indexing. This content is not impossible to get to, just impractical for search engines to continually crawl.
What is the dark web used for legally?
Whistleblowing, investigative journalism, political speech, law enforcement operations, and personal privacy measures are some examples of legal and ethical activities the high anonymity of dark web tools facilitates. The privacy afforded by dark networks enables many legal aims.
Can Google index dark web sites?
No. Google and other conventional search engines cannot index the content hosted on the encrypted anonymity networks that make up the dark web, like Tor or I2P. Accessing dark web services requires special routing software.
Is Tor the only way to access the dark web?
No, though it is the most widely used browser. Other software like I2P (Invisible Internet Project) offers similar anonymous network access to hidden dark web sites that typical connections cannot reach. Special routing configurations obscure traffic sources.
Is accessing the dark web dangerous?
Browsing the dark web does pose security risks, yes. Downloads or network tools may contain malware, trackers, or methods to deanonymize users and access devices. And legal penalties around viewing explicitly illegal offerings remain a risk. As such most advise against accessing the darkest corners out of curiosity.
- Claude 3 Context Windows Pricing (guide) 2024 - March 5, 2024
- How AI Enhances Customer Service in 2024 - March 1, 2024
- What is a Counterfactual Explanation in the Context of AI? - March 1, 2024