This statement can be set as the abstract of a new research being carried out on data mining. According to this new research , by researchers from Indiana University at Bloomington and the Oak Ridge National Laboratory in Oak Ridge, TN,  tracking of organized criminal activities across the web by the cyber gangs will be much easier now.

This Research identifies dense clusters of ISPs that appear to be overly tolerant of malicious activity from anti-malware, anti-spam companies and phishing blacklists. They state that such patterns were particularly evident in Eastern Europe and the Middle East after comparing data from variety of services that Measure ISPs. Acording to them an ISP is classified as malicious if it harbored at least 2.5 percent of the malicious Internet addresses for a given data set, such as the list of phishing sites or malware-laced sites. They found 58 networks that each had more than 100,000 compromised hosts in their Internet address space ranges, while another 255 networks had between 10,000 and 100,000 systems blacklisted.

Measuring online threats largely depends on their geographic location and focus. The study includes information on phishing websites from Phishtank.com and the Anti-Phishing Working Group; botnet data from the Shadowserver Foundation; spam data from Indiana University, Spamhaus, SURBL, and Support Intelligence; malware hosting stats from organizations such as CleanMX, eSoft, and Malware Patrol.

Ukraine, Iran and Belarus were found to be in an alarming stage as they had more than 80 percent of their Internet address ranges blacklisted for a combination of spam, phishing, and hosting malicious software. Their ISP count were two, one and one respectively. On the other hand Turkey captured the limelight while analyzing (mining) the data on prevalence of servers that criminals use to control botnets. They covered almost 9.11% of the total internet  addresses  listed through a large broadband ISP.

Another strategy, that brought United States into notice, which identifies problem networks based on the number of blacklisted addresses for a given ISP. This method usually points to the world’s largest ISPs.

One more approach was quite successful in identifying zombie systems. It was to identify ISPs and hosting providers that had a disproportionate number of network peers that were malicious. With the help of this approach 22 networks were found to be purely malicious, while some 194 networks were found to be partially malicious.

This research will definately be of great help and development of  internet security and law enforcements in this field.

For more details : http://www.csiir.ornl.gov/shue/research/infocommini10.pdf

Today’s industry generates large volumes of data from all sectors such as  financial, retail,  factory, call centers, and customer products,  and so forth, SAS Analytics lets them realize the value within these growing volumes of data.

[Read more @ SAS]

Data mining and machine learning, even today, is being used in various fields, be it sciences, engineering or even entertainment. Data mining in customer relationship management applications can contribute significantly to the bottom line. Rather than randomly contacting a prospect or customer through a call centre or sending mail, a company can concentrate its efforts on prospects that are predicted to have a high likelihood of responding to an offer. Machine learning is enabling more real artificial intelligence in computer games. In Bioinformatics, it is being used to detect patterns present in the DNA sequence which is very important to the human genome project.

Photography: infocusmagazine.org

In the future data mining will help solve problems with far more bigger impacts on mankind, with scales which we have not seen till now. The complexity of these challenges will make usage of machines necessary, not just as a source of data management but also as a source of intelligence. Data mining and machine learning can provide very innovative, out of the box and practical solutions in the fields like climate change, energy, education, health etc.  Here we look at some of the challenges of the 21^st century and the way data mining and machine learning can be of some assistance.

Climate Change

Climate change is one of the biggest challenges we face and one which needs immediate attention. One of the major reasons for the current situation we are in is inappropriate usage of fossils fuel. To this end we are trying some alternative sources of energy. Data mining can play a crucial role in this by monitoring not only our usage but also figuring out patterns in global patterns which will help determine the best source of alternate energy for a region. We have been able to develop sources of energy using renewable sources like solar energy, wind based turbines as well as turbines harnessing massive source of energy present in the ocean currents. But all these sources of energy might not be appropriate for every region and we will need a detailed region wise study to gather information which will help in judging the best sources to be used during different seasons in a year for the region so as to provide a continuous flow of energy. The complexity and the huge amount of data involved in this study will make the use of computers indispensible. Data mining will enable in figuring out climate patterns and suggesting statistically appropriate results as to which is the best source of energies for a region.

Photo Courtesy: Brian Hayes, USA

Machine learning will also help us to counter some of the negative impacts of climate change like changes in the land usage patterns. Data mining, which is the partially automated search for hidden patterns in large databases, offers great potential benefits for applied Geographic Information Systems-based decision-making. Recently, the task of integrating these two technologies has become critical, especially as various public and private sector organisations possessing huge databases with thematic and geographically referenced data begin to realise the huge potential of the information hidden there. Environmental agencies are assessing the impact of changing climate conditions on land-use patterns data mining techniques on these vast sources of information. The days of Global Environmental Protection Agency(G-EPA) are here, and yes, these are the men in green!

Energy

Energy consumption leading to its depletions are on the rise. While looking out for renewable sources of energy is crucial, energy management in present time is equally important. We have large scale inefficiencies in the whole process, from energy production to energy distribution to energy consumption. Today these areas are mainly handles manually, using computers just for data management with minimal usage of computer based intelligence. But as with most of the problems, the complexity involved, will make computer based techniques like data mining crucial for an optimal solution to this problem [1].

In 2007, IBM formed a coalition of innovative utility companies to accelerate the use of smart grid technologies and move the industry forward through its most challenging transformation. The Global Intelligent Utility Network Coalition wants to change the way power is generated, distributed and used by adding digital intelligence to the current systems to reduce outages and faults, manage demand, and integrate renewable energy sources such as wind and power. Smart Grid is even being tested close to home at North Delhi Power Limited (NDPL) which is one of the biggest distributers of electrical power in Delhi, India.

Image Credit: http://my.reset.jp/~adachihayao/

Health

With the emergence of intelligent systems, in no time we will see a wide spread usage of machine intelligence in the fields of medicine and health. There is already some work going on in using data mining to identify the outburst of diseases. Public health services are searching for explanations of disease clusters by identifying common geographical, economical and social patterns existing there. This will greatly help in identifying a disease outbreak in advance, drugs management in a region and also in identifying preventive measures appropriate for a region.

Machine learning is increasingly being used in the process of generation of drugs which will me most effective on a person based on his/her DNA pattern. In the area of study on human genetics, the important goal is to understand the mapping relationship between the inter-individual variation in human DNA sequences and variability in disease susceptibility. In lay terms, it is to find out how the changes in an individual’s DNA sequence affect the risk of developing common diseases such as cancer. This is very important to help improve the diagnosis, prevention and treatment of the diseases. The data mining technique that is used to perform this task is known as multifactor dimensionality reduction [2 ].

Education

Today if a student wants to make decisions related to his/her career like which courses to take and which career path will suit his/her interests in the long run, on global scale these decisions are based most of the times on pure assumptions. We do have large sources of relevant information on the internet, but again this information is in the form of segregated and unstructured data. Such assumptions based decisions often results in a wrong choice which can impact ones career as well as his/her behavious in the long term. What we need is an intelligent systems which by using the past record of a student, educational or otherwise, to assist him/her in this decision making. This can again be done with the help of machine learning which will take records related to education as well as the psychology of the person as input to give a more logical and less assumption based decision [3 ].

Cyber Security

With the fast integration of our lives with internet, cyber security will be of utmost importance in the future. Though we have been able to take some measures with regards to this, but still there are some loop holes present which can be of grave threat to an individual.

Some very innovative work is being done in the field of data mining and machine learning to tackle the issue of cyber security. Army High Performance Computing Research Centre (AHPCRC) has developed an intrusion detection system called – Minnesota Intrusion  Detection System (MINDS) which used advanced data mining techniques to detect cyber threats. Its anomaly detection system Detect novel attacks/intrusions by identifying them as deviations from “normal”, i.e. anomalous behavior. It does so by defining what is normal and anything having different characteristics as a deviation and hence as a possible threat.

Space Exploration

Space Exploration is being given more and more importance as we try to figure out the laws which govern the universe, our origin and ultimately other pools of resources in our solar system. Till the last century the main problem with space exploration was the ways by which we can gather more information. As we make advance in space technology the processing of large amount of data we gather will become more and more complicated. The number of factors one has to study in the data is enormous. It can be related to physics, chemistry and may be biology too.

Photography: matricresearch.com

Data mining provide the necessary assistance in this learning process. Data mining techniques are being used to study the elemental differences between different planets of the solar system, which is being used to deduce the chronological order of the formation of the solar system as well as the core reasons for the presence of life on planet earth.

Security and counter-terrorism

One of the issues which will be faced by many countries of the world is global terrorism. One of the ways to tackle it is have strong knowledge about suspicious identities and there association. Two plausible data mining techniques in the context of combating terrorism include “pattern mining” and “subject-based data mining”.

In the context of pattern mining as a tool to identify terrorist activity, the National Research Council provides the following definition: “Pattern-based data mining looks for patterns (including anomalous data patterns) that might be associated with terrorist activity — these patterns might be regarded as small signals in a large ocean of noise.”

“Subject-based data mining” is a data mining technique involving the search for associations between individuals in data. In the context of combating terrorism, the National Research Council provides the following definition: “Subject-based data mining uses an initiating individual or other datum that is considered, based on other information, to be of high interest, and the goal is to determine what other persons or financial transactions or movements, etc., are related to that initiating datum.“

Photography: wired.com

We will continue to face bigger and more complex challenges and, we will continue to make machines more intelligent so as to be used as alternate brains or cyber brains.

References:

[1] http://www.ibm.com/smarterplanet/us/en/smart_grid/ideas/?&re=spf

[2] http://en.wikipedia.org/wiki/Data_mining

[3] http://www.educationaldatamining.org/EDM2009/uploads/proceedings/vialardi.pdf

- Kartik Rustagi, SDE Intern.

This is a clear indication of what we can expect in the near future.(Oh boy! Not another American Robot idol, The Amazing robot race, America’s next top robot model, or robot factor)

Read more about the event here and for sure visit the compiled photo gallery1 and gallery2 for more pictures.

The robot hand is capable of 24 movements and can be remote-operated with the CyberGlove Photograph: Kim Kyung-hoon/Reuters

Humanoid industrial robot 'Motoman-SDA5D', developed by the Yaskawa Electric Corporation, demonstrates its capabilities with Lego Photograph: Dai Kurokawa/EPA

A humanoid robot 'Manoi AT01', produced by Japan's toy robot maker Kyosho, performs a hip-hop dance Photograph: Yoshikazu Tsuno/AFP/Getty Images

A description:

Microsoft:

Amazon:

Yahoo!:

Google:

IBM:

Sources:

• Wikipedia,
• Yahoo! Media Relations,
• Microsoft PressPass,
• Google Press Center,
• Google Timeline
• PowerPivotPro.com
• Microsoft Research
• Amazon Media Room
• InformationWeek
• NYTimes

–  SAGAR JAUHARI, SDE Intern.

“Business Intelligence (BI) helps business people make more informed decisions by providing them timely, data-driven answers to their business questions. BI analyzes data stored in data warehouses, operational databases, and/or ERP systems (i.e. SAP®, Oracle, JD Edwards, Peoplesoft) and transforms it into attractive and easy to understand dashboards and reports. BI delivers the insight needed to make strategic planning decisions, improve operational efficiencies, and optimize business processes.” -Microstrategy.

What are BI tools?

Business Intelligence (BI) tools  are a set of software systems and practices that enable organizations to analyze data, and make better decisions based on the insight from that information. Companies can use this insight to take the following steps to improve overall corporate performance:

• Enhance cost-efficiency and productivity
• Build strong customer relationships
• Optimize revenue-generating strategies
• Increase revenue and maximize profitability
• Monitor trends and discover anomalies
• Forecast business opportunities
• Maintain compliance and perform risk management

Categories of BI Tools:

• Spreadsheets
• Reporting and querying software
• OLAP
• Digital Dashboards
• Data mining
• Process mining
• Business performance management
• Local information systems

State of art of BI in Industry today:

In one word, most companies do not use BI to their fullest potential according to market research.

Some stated reasons include:

• Lack of proper training, followed by limited staffing resources.
• Most custom reports remain very sophisticated
• IT staffs still create the majority of BI reports, followed by  business analysts.
• Dissatisfaction with BI technology is found among all user communities.
• The idea that BI applications can create more work

Now the latest of all is the move of BI from industry to everyday life.

A discussion room in an advertising company will now comprise of a few data miners as well. Yes, this is the latest trend. The question of “why data miners in an ad agency?” will be the first one which would occur to our  mind. The simplest answer that can satisfy our ask is, the company wants to market ads based on user needs. Based on the user data analysis, the marketing would bare more tastier fruit. But why not we have a data analysis done by a market researcher? The ad agencies are looking ahead and taking one step further. They are paying close attention to user needs. Does optimization of ads ring a bell?  Also, with the recession taking up the market, companies sought to take every advantage they can leverage, even combining marketing with digital technology for better ads and better ROI.

What has data mining got to do with the marketing?

The digital advent has highly increased the necessity for smarter ways of advertising through internet. Advertisers have started to thoroughly examine and debate data mining and other new sciences that will shape the interactive marketplace. It is sure to be broached in discussions about consumer trust, and maybe even in a Facebook session aptly titled “Knowing is Better.”

An instance of the same…..

An ironic prelude to the week-long fete of advertising’s digital future was the Sept. 18 settlement of a privacy lawsuit related to Facebook’s social ad experiment, Beacon. The short-lived, poorly executed program riled online consumers, whose purchase information with off-site retailers such as Zappos and Blockbuster was unexpectedly shared with their Facebook friends. Their only recourse was to “opt out” of the program after the damage was done. While there are mounting examples of online consumers trading their personal information and privacy for more targeted interactive results, Beacon assumed too much with its initial tacit user approval.

Going the giant’s way…

All major internet giants like  Google, Yahoo, Microsoft, etc. have already gone far and wide in dealing with user data. They have reports saying how their research with user data has proven to be fruitful. User insights, social connections, personal preferences and buying history — which Amazon and Google already masterfully manipulate — are building blocks for an interactive economy that relentlessly exploits links to generate revenues.

Forrester Research shows that even as digital grows from 12 percent of existing overall advertising spend to 21 percent (or $55 billion) by 2014, there is a pressing necessity for companies to master constructive interactive relationships with consumers and each other to generate many times that in digital sales and other transactions.

So why this sort of marketing now?

Though recession has considerably reduced the advertising budgets of the company and the consumer spending allocations, there are signs of few marketers who drift consumers and technology into the interactive future.

What exactly will advertisers and media do with those interactive connections, and the insights and information they yield? What do you think?

-Vidhya, Student Intern