The internet is a super massive exponentially expanding ice burg floating in a limitless ocean in volume and depth. Inside the ice burg is a small singularity much like a black hole that pulls information into it likely never to be see again, however the speed in which data is being pulled into the singularity is incredibly slow and as data is accessed it is pulled further away from the singularity. The observable internet which is the part of the ice burg above the water line, current estimates calculate represents at most 4% of the internet, the rest is known as the Darknet which is unknown to popular search engines. On average 99.995% of internet users will only directly interact with within the observable internet their entire life and are largely unaware the Darknet event exists. Much like when it was popular belief that the world was flat.
The majority of the users that do interact with the Darknet are comprised of intelligent agencies, large corporations and big data mining companies, both black and white hat hackers and criminal organizations. So much unorganized data exists in the Darknet that when most companies attempt a project in big data research that includes Darknet data, the often fail or retreat to working within the observable internet in favor of reduced complexity.
However there are a few creative big data architects and engineers that are able to see beyond the general disorder and chaos of the Darknet to discover insight. After many years in mining big data both in the observable and Darknet of the internet you come to accept the real challenge is not figuring out what to do with all that data but learning to develop methods to ask the right question of the data. The right question is infinitely more important than the answer, unless you don’t mind waiting forever for an answer.
To ask the write question you have to understand how data evolves within the internet:
- If something is put online it will exist online forever, at least beyond the human lifetime.
- If the internet is an ice burg, expansion occurs on the outer layer and internally within.
- Users interacting with the internet are like small lightning strikes on the visible surface of the internet, it starts like a small point of data then like the branches of a tree expands out across the observable internet then down into the dark net leaving pieces of data all along the way.
- Users leave data footprints all over the internet when they interact with services online and they may not be even be aware or accept it as the nature of online services.
There are many popular data mining designs and with the more widely available machine learning technologies we are discovering more and more data insights all the time. Once cloud computing and faster connectivity and storage becomes more widely available and adopted with the use of quantum computing answers to some of the most complex data questions will become easily accessible.