The term “Deep Web” was coined in 2001 by Computer scientist Mike Bergman in his white paper “The Deep Web: Surfacing Hidden Value”. He used the term to describe the parts of the web containing content which was not indexed by traditional search engines, claiming that it was “400 to 550 times larger than the commonly defined World Wide Web” (or “surface web”). In effect, this means that a Google search which produces millions of results only reveals the tip of an iceberg of data, the majority of which is hidden below the surface.

Why does the deep web exist? There is no simple answer to this question – and it’s not as obvious as the mass media like to portray. The Dark Web – which is sometimes confusingly used synonymously with the Deep Web – actually refers to a section of the Deep Web which is actively hidden and sometimes heavily encrypted, often because it’s being used for criminal purposes. The Dark Web uses a range of large networks, sometimes referred to as darknets, such as Freenet, I2P, and Tor or smaller peer to peer networks. One of the most famous examples of a darknet market was Silk Road, essentially an online black market, shut down by the FBI in 2013 after a controversial investigation, which catered for drug users, paedophiles, terrorists and other criminals – as well as many legitimate sellers and buyers.

However, the Dark Web only accounts for a small portion of the Deep Web. Much of the Deep Web consists of vast databases of information which cannot be indexed by Google because they do not exist as a set of static pages; instead the content of these datasets is dynamically generated upon direct searches of the database. One example is the Wayback Machine which archives internet pages and websites, many of which cease to exist entirely on the surface web and can only be found in the Deep Web” archive.

An interesting new development in the Deep Web arena is the announcement that NASA is joining forces with the Defense Advanced Research Projects Agency (DARPA) to help with its Memex program, an initiative designed to “access and catalog this mysterious online world.” The impetus for NASA in progressing Memex is that its Jet Propulsion Laboratory wants a better way of accessing information sent back to earth by its spacecraft, which is in a file format that Google finds hard to read, hence ending up on the Deep Web. It’s worth noting that DARPA, under its former guise of ARPA, developed ARPANET in the late 60s, which was essentially one of the earliest forms of the internet.

Relevance to lawyers

An understanding of the basics of the Deep Web is important for legal practitioners, especially those specialising in IT and IP law. Important evidence and information pertaining to a case may reside in the vast array of databases, and lawyers who know how to locate this needle in a haystack will be at an advantage.

The Dark Web, however, throws up more obvious legal challenges. Online intellectual property theft often involves Tor networks, and all manner of criminal activity takes place on darknets. Furthermore, as the general public becomes more aware of data protection and privacy concerns, increasing numbers may start to use the Dark Web – whether for criminal purposes or simply to remain anonymous in oppressive regimes.

The legal issues

Crime

According to a study carried out by University of Portsmouth computer science researcher Gareth Owen, the majority of Tor hidden services site visits were to online destinations with paedophilia materials (although child abuse sites represent only about 2 per cent of Tor hidden service websites). Around a quarter of these sites relate to drugs and online black markets, which provide a trading platform for weapons and terrorist materials and even facilitate crowdfunded assassination. Pirated films, music and software are readily available on Tor networks, so copyright infringement is another major criminal activity on this murky part of the Deep Web. Furthermore, trade secrets as well as sensitive personal information (eg stolen from government databases by malicious hackers) are up for grabs.

Data protection, whistleblowing and human rights

Although the Dark Web has a bad reputation – with Ross Ulbright who created Silk Road being portrayed by some media outlets as a modern day Al Capone – there is also a large chink of light to all the darkness. The anonymity afforded by darknets allow citizens living in oppressive regimes to exercise certain human rights enjoyed in the UK (at least before the Human Rights Act is scrapped), such as the freedom of expression and the right to education. Even those living in the “free world” who want to protect their data from advertisers and snooping governments (post – Snowden revelations) may choose to remain private for benign purposes by using services such as Tor.

Tor (an acronym of The Onion Router) uses a certain type of routing involving many layers (like those of an onion) to encrypt data and hide the identities of those transmitting the data. This technology can help whistleblowers and human rights workers to communicate with journalists and organisations such as WikiLeaks without the danger of being intercepted.

Cryptocurrencies

Although cryptocurrencies such as Bitcoin are still in their infancy, they are being used quite extensively on darknets. Cryptocurrencies are not intrinsically part of the Dark Web and can be used in the real world for innocuous purposes (such as buying a burger in east London), but they are particularly well suited to online black markets (and corrupt FBI agents investigating these black markets) due to their relatively anonymous nature. If cryptocurrencies take off, then this could significantly impact upon banking and finance law.

Further reading

Wikipedia: Deep Web.

Wikipedia: Dark Web.

The Guardian: Dark side of the internet.

The Guardian: US federal agent investigating Silk Road admits $800,000 bitcoin theft.

Fusion: NASA is indexing the Deep Web to show mankind what Google won’t.

The Journal of Electronic Publishing: White Paper: The Deep Web: Surfacing Hidden Value.

Wired: What Deep Web’s director learned investigating Silk Road.

Image: By Ryan Dickey on Flickr.