Deep Data Mining

September 30, 2017





Natures greatest gift to humanity - intelligence - especially our ability of pattern recognition, sharpened and honed over eons of evolution. Inevitably, this meant that we in our early days were very good at spotting prey and predator or discerning poisonous plants from the nourishing ones but more importantly at picking up subtle signs in our vicinity and making logical deductions. 


Soon enough, we looked to the heavens and saw the shining luminous points as all other creatures did, but we more than that recognised in them patterns and thus we read the messages written in the stars, taking notes about the weather conditions linked to each of the four seasons. And yet that was not all, behold we turned our gaze upwards once again and this time instead of using the skies as a calendar we deduced its usefulness as a map. Throughout history we looked and we learned but this was possible only because of our gift of pattern recognition.


While we may have come a long way from Stone Age to the Modern Age defined by our impact on the world, our mind works on the same basic principles as it did thousands of years ago. Today we seek to replicate our gift, we try to pass it on to other inorganic, artificial beings. And thus the beginning of ‘Deep Data Mining’. The name is quite misleading as it's not actually creation of new data, rather a search of patterns using computers in extremely large sets of data that are beyond the scope of human memory and retentive capacities. The overall goal of the data mining process is to extract information from a data set and transform it into an understandable structure for further use. This has many plausible uses in Security, Business and most of all in Science.


‘Big Data’ has long been collected by Internet-based retailers such as Amazon to predict consumer behaviour and recognise patterns in consumer demand. Oddly this is done under a less exquisite name i.e. ‘cookies’ (Packets of information that help compile the Big Data). 


However, it gets even better with emergence of social media networks. In 2009, a team from MIT won $40,000 cash prize from Defence Advanced Research Projects Agency (DARPA) for being the first to locate 10 red weather balloons like these spread across the United States by incentivising crowd-sourced reconnaissance on social media. 


The Network Challenge, as it was called, offered a hefty prize to the first team that could, on a specific day, identify the locations of the 10 red weather balloons scattered across the United States. The basic motive was that teams would use social media to help locate the balloons. The contest would assess the teams’ ability to leverage a network, working on how to motivate people to participate while weeding out possible fake sightings, and to do it quicker than other Teams. On Dec. 5, 2009, the day of the challenge, the biggest fear was that no team would identify all the balloons, undermining the point of the challenge. 


However, it took only nine hours for a team from MIT to win. They beat the competitors by using financial incentives that rewarded not just those who spotted balloons, but those who recruited others who successfully spotted balloons. It might sound all interesting but the science of deduction runs deeper, the DARPA object was of course not to provide an entertaining, ingenious competition to young detectives but to set up an experiment to assess the feasibility of prospects in Security and Intelligence. Social Networks such as Facebook and Twitter could be used to predict the location of terrorist attacks using same tactics!


With such clever breakthroughs in the field of science, it's hard to know what the future holds in the augury for us.








Share on Facebook
Share on Twitter
Please reload

Recent Posts

December 17, 2017

Please reload

Featured Posts

Inspiration from nature

October 6, 2018

Please reload

Follow Us

  • facebook_circle_color-256