The Deep Web — etymological curiosity and urban legend
In May this year my writing was feature in my first academic-level publication, the ‘The SAGE Encyclopedia of the Internet’, providing a lengthy definition of term ‘deep web’. Alas only about a quarter of the content made the cut, thus I present the complete essay submission here. It is a compilation of materials previously written by me on this blog, elsewhere and RationalWiki.
- Preface
- Introduction
- History
- - Deep web search
- - The deep … and dark web?
- - Tor, Bitcoin and the Silk Road
- Language
- Dark web influences
- Legends
- Exploration
- Psychosocial influences
- Conclusions
Preface
As a cybercrime researcher, and amateur techno-sociologist, I’ve run a popular forum on the topic of the ‘deep web’ for almost two years now. I have experienced the confusion and ambiguity surrounding the subject on a daily basis, prompting my investigations into underground darknet markets, creatively horrific scams and seen the birth of new generation of conspiratorial technology-driven paranoia, bred by half-truths, misunderstandings and fabrications in the absence any kind of real authority on the matter.
This is my world and the world of the ‘deep web’.
Introduction
You’ve probably already heard of the ‘deep web’, a hidden part of the internet filled with criminality, mystery, advanced encryption technologies, hackers, government secrets and maybe even more? But how mysterious it is really? Is it the same as the ‘dark web’? Isn’t search indexing supposed to be involved at some point or it the ‘deep web’ something else altogether?
The term ‘deep web’ most commonly refers to hidden service .onion websites build on top of the Tor dark web. However, in its original sense, it referred to web content inaccessible via traditional search engines. Today the terms are incredibly nebulous with technology companies, darknet sociologists, technology journalists and Wikipedians all attempting to tease apart the overlapping and confusing definitions.
Lexicographer Jane Solomon reports in mid 2015 that ‘deep web’ now so often appears in place of or in combination with ‘dark web’ that it is taking on a life of its own.
One common misconception about the dark web and the deep web is that these two terms are interchangeable. This is simply not true.
However it turns out it’s more than just terminology that people are confused about.
“How can I access the deep web”
- internet forums 2011–2017
History
Deep web search
‘Deep web’ in a search context was coined in the year 2000 by technologist and internet search researcher Michael Bergman. Bergman was designing the next generation of internet search algorithm which would go beyond the scope of traditional web crawler technology. He divided the web into two parts, the parts traditional search engines were indexing he named the ‘surface web’, whereas his new ideas were about exploring the ‘deep web’. In ‘The Deep Web: Surfacing Hidden Value’ Bergman explains how the ‘deep web’ may be searched via methods such as passing queries into search forms to retrieve dynamic results, searching corporate intranets, accessing private content behind logins and pay walls and other types of sites such as classified ads, ecommerce, job boards and forums. Bergman would later go on to use the analogy of an iceberg to illustrate the how ‘deep’ this content was, as how the vast majority of information was ‘below the surface’.
In the intervening years, Google’s Page Rank and successor algorithms have been in a continuous state of improvement, with the specifics guarded as one of the tech giant’s most precious pieces of intellectual property in this commercially competitive space.
In 2015, the US Defense Advanced Research Projects Agency (DARPA) announced a partnership with NASA for a ‘Memex’ search engine to take things even further, indexing scientific materials, semantic meaning of multimedia content and even previous versions of websites potentially hosted by archive.org to create ever more powerful search technology. “NASA is indexing the ‘Deep Web’ to show mankind what Google won’t” exclaims the news headline, no doubt leading to much confusion amongst it’s readers.
This search term will be regularly referenced online in the context of ‘technically the deep web is just anything not indexed by Google’, with its users unaware of the conflation of meaning that has occurred in recent years. As of 2017, the majority of discussion about the ‘deep web’ will not be about search technology, rather something quite different altogether. But how did we get here?
The deep … and dark web?
In 2009 Guardian journalist Andy Beckett published an exploratory piece entitled “The dark side of the internet”, touching on the confluence of cybercrime, piracy, new networking technologies and ‘deep web’ search. Police and security specialists explained to him how criminal gangs were using new technologies to enable spamming, virus distribution and the distribution of child pornography. Beckett would explore Freenet, a peer to peer darknet and web-like software that allows censorship-free and cryptographically secure communications.
While a “darknet” is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of “the deep web”, spooky as it sounds, consists of unremarkable consumer and research data that is beyond the reach of search engines — Beckett 2009
Most importantly however, Michael Bergman was interviewed within the same article confirming how Freenet was indeed a part of the ‘deep web’ in the search context, not knowing the impact this article would have for years to come.
Tor, Bitcoin and the Silk Road
Onion routing was a type of sophisticated network encryption and obscuration technology first developed by the US Naval Research Laboratory in the mid 90’s. Tor, originally short for ‘The Onion Router’ was an obscure and unremarkable piece of experimental technology, passing over to control of the Electronic Freedom Foundation in 2004 and remaining quiet until the 2010’s. At a similar time in 2008 the person known pseudonymously as Satoshi Nakamoto would release his paper “Bitcoin: A Peer-to-Peer Electronic Cash System”.)
This ‘cryptocurrency’ digital cash and the latest version of the Tor browser would develop an significant following amongst tech-savy libertarians and anarchists online. In 2011, Gawker’s Adrian Chen shocked the internet with his article about Silk Road “The Underground Website Where You Can Buy Any Drug Imaginable”. He had stumbled upon a fully self-regulating underground ecosystem for drugs and stolen data which were utilising the near untracable currency of Bitcoin and privacy technology of Tor — both of which had seemingly developed overnight. It seemed an libertarian anarcho-capitalist enclave had been created, allowing reliable, violence-free exchange of state-prohibited goods and services, protected by these bleeding-edge cypherpunk technologies.
Except, that wasn’t the narrative that the media picked up on. The epitome of the reaction could be best captured by the UK’s Daily Mail headline in 2013, warning of “The disturbing world of the Deep Web, where contract killers and drug dealers ply their trade on the internet”.
Following a lengthy investigation and eventual shut down of Silk Road, journalists and bloggers alike from around the world lined up to report on the ‘dangers of the deep web’, a term now also adopted by the dark web community themselves, something Andy Beckett’s article and popular usage had assured everyone was the appropriate term. ‘Deep web’ fact and fiction would now start to blur at an accelerating rate.
“How can I stay safe on the deep web?”
- internet forums 2011–2017
Language
Today the term ‘deep web’ is poorly understood and communicated. Lexicographer Bryan Garner calls terms which are highly ambigious and confusing ‘skunked terms’ and recommends they be avoided in order to aid understanding. Similar contemporary examples include the recent shift in meanings of ‘literally’ and ‘decimate’.
To combat this confusion, tech news outlets such as Vice Motherboard and Andy Greenberg at Wired have employed style guides since late 2014, ensuring preference of ‘dark web’ over ‘deep web’ when reporting Tor and hidden service news. Despite this, the continued use of the term within the Tor community and beyond means only the small subsection of the media make this distinction routinely. Even creator of the deep web term Mike Bergman attempted to tackle the issue with “Clearing Up Confusion — Deep Web vs. Dark Web” in 2014. He was ignored.
To illustrate how these differentiation efforts have failed, one need look no further than 2015 documentary ‘Deep Web’ covering the story of Silk Road, which heavily featured Wired’s Andy Greenberg. Despite this, the film used the proscribed term and even featured the original irrelevant search meaning in its introduction.
Dark web influences
The real dark web on Tor contains a large amount of unsavoury and illegal material, and to the unfamiliar, actual cybercrime may be indistinguishable from fabrications. Combined with media and even political hysteria, reliable information on the area is hard to come by.
Drop sites such used by Wikileaks and later more mainstream news outlets were set up to allow the highly anonymised submission of sensitive documents by whistle-blowers. ‘Doxing’ sites feature malicious personal information dumps as well as some ‘revenge porn’ sites, often driven to Tor due to their increasingly illegal nature. The alleged hackers behind ‘The Fappening’ celebrity image leak was believed to have used secret communication infrastructure to trade their content. Finally, the misunderstanding of hackers, their abilities and computer security misconceptions is a vast topic unto itself. Concepts such as these feed into the idea one might find all kinds of hidden information via this vector.
The dark web has been associated with some of the most worst of the worst content the internet has to offer. Most notoriously the story of the video, ‘Daisy’s Destruction’ featured the rape and torture of an infant which was later sold via the dark web. The sexualised torture of animals known as ‘crush fetish’ porn makes actual zoophiles seem tame by comparison. Whilst the majority of people who hear about this ideas recoil, a significant minority are fascinated.
The meme of ‘the Underground Website Where You Can Buy Any Drug Imaginable’ has long since morphed into the idea that you can ‘find anything on the deep web’. If you limit ‘anything’ to drugs, fake IDs, stolen data and hacked accounts — you might be right.
“How can I access the deepest level of the deep web?”
- internet forums 2011–2017
Legends
For a few media cycles, reports of hitmen for hire and other other dubious claims were taken at face value, however as evidence of these failed to emerge the sensationalism began to die down in the main stream media. To fill the gap, an entirely new subculture emerged, one of conspiracy theories, half-truths and outright fabrications.
In late 2011 or early 2012 the info graphic below was created by an anonymous troll whose name is likely forever lost to history. It is a remarkable work of creativity, drawing upon Mike Bergman’s iceberg analogy that a large percentage of the content of the internet is ‘hidden’ on the ‘deep web’. The concept of ‘surface web’ and ‘deep web’ have evolved into a multitude a ‘levels’ in a fashion comparable to Dante’s Inferno and its circles of hell.
This info graphic, derivations and concepts within have been widely circulated on social media and new breeds of opportunistic YouTubers and bloggers have sprung up to peddle such fabrications as either fact or ‘possible’. The image is by 2017 standards ‘fake news’, but aimed at a younger mildly technical audience with insatiable curiosity about the unknown fringes of the internet.
In early 2015, I differentiated the terms ‘dark web’ and ‘deep web’ on Wikipedia. Finding limited mainstream analysis of the term, by mid 2015 I started researching the infographic’s sociological influence on the skeptic website RationalWiki.
Skeptic Alberto Brandolini’s eponymous law states:
The amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it.
And a considerable amount of energy has indeed been required. So many concepts featured on the info graphic have morphed into full blown urban legends in their own right. Themes include pornography, cybercrime, technological inventions and traditional conspiracy theories like the illuminati. Examples include
- A haunted computer game called ‘Sad Satan’ that will kill your PC,()
- A secret ‘CAIMEO’ government AI that’s trying to escape
- Tales of entirely fake internet assassins, payable in Bitcoin. Bitcoin scams remain prolific to this day.
- Rumours of easily available live-streamed murder in ‘Red Rooms’ that you might ‘stumble’ into. These are pretty much a retelling of the ‘snuff film’ legend for the YouTube generation.
- Fake top level domain names with other hidden internets to be accessed
- Government secrets documents and other hidden information
- The ‘Marianas web’ — named after the Mariana Trench, the deepest place on earth — is purported to be ‘deepest part of the deep web’
Slaves, rare animals, fixed sports matches, this is by no means an exhaustive list of scams associated with the ‘deep web’, tomorrow there will probably be another, and another after that. It is however worth noting that confused misunderstandings are nearly-exclusively associated with the term ‘deep web’, and seldom ‘dark web’ or ‘darknet’.
“What’s your deep web story?”
- Reddit, 2015
Exploration
Due to the short-lived nature of many Tor hidden services, poor informational resources and limited search capabilities, a phenomena of unstructured exploration has emerged particularly among teenagers and younger people. The most lurid and disturbing tales pass between peers in the form of ‘creepy pastas’ — short possibly-true horror stories — and create an unresolved psychological tension between the belief in these ideas and the ambiguity of their existence.
The market for ‘deep web stories’ has been fulfilled by a range of opportunist YouTubers, the most notorious of which is Dale Gill, aka Takedownman. YouTubers will provide video slide-show ‘tours’ of weird Tor sites they have been recommended and report them in a sensationalist fashion, deliberately withholding the site links and performing no background research. Unlike the ambiguity of ‘creepy pastas’, they provide a mix of esoteric site reviews and technical advice for a young YouTube audience fascinated with the concept of the ‘deep web’. The popularity of such content has made the term ‘deep web video’ synonymous with ‘creepy video’.
The theme of exploration is strongly associated the ‘deep web’ in general. Randomly clicking an index site such as one of the Hidden Wikis, finding a site one day which is gone the next, causes an ephemerality to pervade many hidden services and affects peoples’ attitudes towards their permanence. Children and young people today will ‘dare’ one another to go onto the ‘deep web’, in a similar fashion that previous generations would have dared one another to camp out in the haunted forest or knock on the door of the house where the witch lives. It may be true that the younger generation are ‘digital natives’ and have a more intuitive understanding of some technologies, but I believe this illustrates the low public understanding of advanced encryption technology and cybercrime in general.
Psychosocial influences
Media communications psychologists Mary Beth Oliver and Arthur Raney researched how individuals may seek out negative content on the internet for the reason of fulfilling cathartic and purging needs, making downward social comparisons, maintaining bad moods, and learning information that may ultimately help one deal with negative or unhappy circumstances. In “Captivated and Grossed Out: An Examination of Processing Core and Sociomoral Disgusts in Entertainment Media” Bridget Rubenkind and Annie Lang similarly explore the appeal of ‘disgusting’ content and why people seek it out for it’s engaging and emotionally visceral appeal, techniques frequently used in advertising.
Such explanations provide at least a partial rationalisation for the market for horrifying content, real and otherwise on the internet. Legends of the ‘deep web’ appear to have had an amplifying affect on this pre-existing phenomenon.
“I went too deep on the deep web”
- internet forums 2011–2017
Conclusions
‘Deep web’ continues to covers a complex melting-pot of ideas around our relationship with technology, arguably epitomising much of contemporary society’s techno-paranoia fueled by ignorance due to the taboo nature of the real content. This process can be seen as history repeating itself around terms such as ‘cyber’ and ‘hacker’ which have been greatly devalued through their continued misuse in the mainstream media.
The dark web is truly only understood by a minority of software developers, technically-minded cryptocurrency enthusiasts, researchers and privacy activists. The remainder is the domain of cybercriminals, geeky drug buyers and people you will likely never meet face to face. The resultant painful gap in public understanding is served by the term ‘deep web’, as the answer to any and all remaining questions.
It is the ongoing search for these answers which motivates the deep web explorers.
