» Events

RCOMM 2010: Rapid Miner Conference

Vikramaditya Jakkula — Sat, 17 Jul 2010 03:14:24 +0000

RapidMiner Community Meeting And Conference – RCOMM 2010

As RapidMiner has once again proved to be the most-used open source data mining tool among the community of data analysts world-wide in a recent poll, it is now the time to give a face to that community. Therefore, Rapid-I hosts the first RapidMiner Community Meeting And Conference (RCOMM 2010) and invites users and developers of RapidMiner to take part and share their RapidMiner experiences with other members of the community. The RCOMM 2010 intends to intensify the community life and strengthen the RapidMiner network by bringing together users and developers of RapidMiner from all backgrounds, may they be scientific or commercial, from the whole variety of applications and from all grades of knowledge. A vital exchange of ideas, application reports, and scientific results will help beginners to advance and will inspire the already advanced leading them to professionalism. Users will profit from in-depth knowledge of developers, who in turn will gain from picking up requirements and ideas for further development.

The RCOMM 2010 encompasses conference talks, in which invited lecturers will discuss aspects of state-of-the-art data mining with RapidMiner. A Call-for-Papers will be issued for those who would like to present their work in that scope. Workshops will be held to give participants a hands-on experience concerning several topics regarding RapidMiner usage. Additionally, attendees of the RCOMM 2010 will also have the option to participate in several courses given by professional RapidMiner consultants in the surrounding of the user meeting.

Dates & Deadlines:

Submission Deadline:August 6, 2010

Notification of Acceptance:August 13, 2010

Camera-ready Papers:August 20, 2010

Conference: September 13 – 16, 2010

Visit Conference Home: RCOMM
Registration Link: Registration
Schedule Link: Schedule

That’s one small step for robot, one giant leap for robotkind

Vikramaditya Jakkula — Sun, 03 Jan 2010 14:21:37 +0000

International Robot Exhibition 2009 has finished with a great success, this event was held during November 25, 2009 to November 28, 2009 at Tokyo International Exhibition Center in Tokyo,Japan. International Robot Exhibition 2009 show is designed to provide a place to exhibit robots and related equipments in order to enhance market awareness of new technology. At the same time, the show is to be a medium to promote new products and to develop new business through contributing the promotion of new technology. Some of the highlights have been illustrated below.

This is a clear indication of what we can expect in the near future.(Oh boy! Not another American Robot idol, The Amazing robot race, America’s next top robot model, or robot factor)

Read more about the event here and for sure visit the compiled photo gallery1 and gallery2 for more pictures.

The robot hand is capable of 24 movements and can be remote-operated with the CyberGlove Photograph: Kim Kyung-hoon/Reuters

Humanoid industrial robot 'Motoman-SDA5D', developed by the Yaskawa Electric Corporation, demonstrates its capabilities with Lego Photograph: Dai Kurokawa/EPA

A humanoid robot 'Manoi AT01', produced by Japan's toy robot maker Kyosho, performs a hip-hop dance Photograph: Yoshikazu Tsuno/AFP/Getty Images

The datamining journey so far ..

Thu, 31 Dec 2009 12:09:41 +0000

This new year, let us go through all the major developments that have taken place in the Data Mining industry over the years. Here is a quick glimpse:

A description:

1993	Development of WEKA begins: In 1993, the University of Waikato in New Zealand started development of the original version of Weka. Weka (Waikato Environment for Knowledge Analysis) is a popular suite of machine learning software written in Java, developed at the University of Waikato. WEKA is free software available under the GNU General Public License.
1996	CRISP-DM is conceived CRISP-DM stands for CRoss Industry Standard Process for Data Mining. It is a data mining process model that describes commonly used approaches that expert data miners use to tackle problems. Polls conducted later in 2002, 2004, and 2007 show that it is the leading methodology used by data miners
1998	KXEN established Founded in 1998, KXEN has corporate offices in San Francisco, California and Paris, France, with Fortune 1000 customers around the world.
1999	CRISP-DM 1.0 released After it was conceived in 1996, in 1997 CRISP-DM got underway as a European Union project under the ESPRIT funding initiative. The project was led by four companies: ISL, NCR Corporation,Daimler-Benz and OHRA. The first version of the methodology was released as CRISP-DM 1.0 in 1999.
2000	The ‘R’ Project considered stable for production R is an implementation of the S programming language with lexical scoping semantics inspired by Scheme. R was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand, and is now developed by the R Development Core Team.
2003	Appricon established In order to provide a better data mining solution, Analysis Studio® and the Analysis Studio® end-to-end logistic regression modeling solution were weaved into enterprise data mining projects in 2003. SAS 9.1 was released in 2003
2004	Rapidminer distributed with GNU license The initial version has been developed by the Artificial Intelligence Unit of University of Dortmund since 2001. It is distributed under a GNU license, and has been hosted by SourceForgesince 2004. SAS 9.1.2 was released in 2004.
2005	Amazon launches Mechanical Turk The service was launched publicly on November 2, 2005. In early- to mid-November 2005, there were tens of thousands of HITs, all of them uploaded to the system by Amazon itself for some of its internal tasks that required human intelligence. Most of these were related to music CD items. The number of Amazon’s Mechanical Turk HITs in the system soon decreased after its launch in november, and by December 20, there were less than 100 groups of HITs on the average page load Weka receives the SIGKDD Data Mining and Knowledge Discovery Service Award SAS 9.1.3 was released in 2005.
2006	Work on CRISP-DM 2.0 begins In July 2006 the consortium of CRISP-DM announced that it was going to start the process of working towards a second version of CRISP-DM. On 26 September 2006, the CRISP-DM SIG met to discuss potential enhancements for CRISP-DM 2.0 and the subsequent roadmap. Pentaho acquires exclusive ….. In 2006, Pentaho Corporation acquired an exclusive license to use Weka for business intelligence. It forms the data mining and predictive analytics component of the Pentaho business intelligence suite.
2008	COGNOS acquired by IBM Cognos (Cognos Incorporated) was an Ottawa, Ontario based company making business intelligence (BI) and performance management software. On January 31, 2008, Cognos was officially acquired by IBM. The Cognos name continues to be used, being applied to IBM’s line of business intelligence (BI) and performance management products. SAS 9.2 is the latest release (March 2008) and was demonstrated at SAS Global Forum (previously called SUGI) 2008.
2009	PASW/ SPSS PASW (formerly SPSS) is a computer program used for statistical analysis. Before 2009 it was called SPSS, but in 2009 it was re-branded as PASW (Predictive Analytics Software). The company announced July 28, 2009 that it was being acquired by IBM for US$1.2 billion.

Microsoft:

1996	Microsoft opens new team to build an OLAP product, codenamed Plato (permutation of letters from OLAP) Panorama Software delegation meets with Microsoft Microsoft announces acquisition of Panorama Software development team
1997	OLAP Services 7.0 (codename Sphinx) ships
2000	Analysis Services 2000 (codename Shiloh) ships Microsoft Analysis Services is part of Microsoft SQL Server, a database management system. Microsofthas included a number of services in SQL Server related to Business Intelligence and Data Warehousing. These services include Integration Services and Analysis Services. Analysis Services includes a group ofOLAP and Data Mining capabilities. Microsoft Corp. announces the beta release of the OLE DB for Data Mining specification, a protocol based on the SQL language, that provides software vendors and application developers with an open interface to more efficiently integrate data mining tools and capabilities into line-of-business and e-commerce applications.
2001	XML for Analysis SDK 1.0 ships
2004	ADOMD.NET and XML for Analysis SDK 1.1 ship
2005	Analysis Services 2005 (codename Yukon) ships
2008	Analysis Services 2008 (codename Katmai) ships
2009	Microsoft has decided to make the BI Conference into a biennial event, with the next conference in 2010. For 2009, we are excited to team with the Professional Association for SQL Server (PASS) to expand the BI tracks at PASS Summit 2009 and help deliver the content that BI architects, developers, and administrators need to get the most value from their Microsoft SQL Server and BI-based solutions. PowerPivot gives users the power to create compelling self-service BI solutions, facilitates sharing and collaboration on user-generated BI solutions in a Microsoft SharePoint Server 2010 environment, and enables IT organizations to increase operational efficiencies through Microsoft SQL Server 2008 R2-based management tools.

Amazon:

2003	“Search Inside the Book” is a feature which allows customers to search for keywords in the full text of many books in the catalog. The feature started with 120,000 titles (or 33 million pages of text) on October 23, 2003. There are currently about 250,000 books in the program. Amazon has cooperated with around 130 publishers to allow users to perform these searches.
2005	In November 2005, Amazon.com began testing Amazon Mechanical Turk, an application programming interface (API) allowing programs to dispatch tasks to human processors.
2006	Amazon launched an online storage service called Amazon Simple Storage Service (Amazon S3). An unlimited number of data objects, from 1 byte to 5 gigabytes in size, can be stored in S3 and distributed via HTTP or BitTorrent .In April 2006, Amazon introduced Amazon Simple Queue Service (Amazon SQS), a distributed queue messaging service.
2007	In January 2007 Amazon launched Amapedia, a collaborative wiki for user-generated content to replace ProductWiki In December 2007, Amazon introduced SimpleDB, a database system, allowing users of its other infrastructure to utilize a high reliability high performance database system.
2008	Amazon Web Services launched a public beta of Amazon Elastic Compute Cloud running Microsoft Windows Server and Microsoft SQL Server.

Yahoo!:

2002	Yahoo! HotJobs, previously known as hotjobs.com, is an online job search engine. It has been known as Yahoo! HotJobs since being acquired by Yahoo! in 2002. Yahoo! HotJobs provides tools and advice for job seekers, employers, and staffing firms.
2003	Yahoo! Introduces Smartsort Technology: Personalized Product Recommendation Tool The new Yahoo! Product Search powers the redesigned Yahoo! Shopping, providing consumers with the most comprehensive and relevant comparison-shopping site on the Web. The redesigned Yahoo! Shopping now boasts a variety of comparison-shopping features including: side-by-side product comparison, detailed buyer’s guides, tax and shipping calculator tool, consumer product and merchant ratings, unbiased expert product reviews etc. Yahoo! Shopping is the third largest multi-category commerce destination on the Web. (Nielsen//NetRatings, August 2003)
2004	Yahoo! Launches SmartView Technology: New Mapping Feature Creates Customized Visual Search Capability
2005	Yahoo! Search Launches Search Subscriptions Beta, Providing Select Deep Web Content to Users
2006	Yahoo! Opens Internet Time Capsule to Capture Life in 2006 SUNNYVALE, Calif., October 10, 2006 – Yahoo! Inc. (Nasdaq:YHOO) today announced the launch of what is expected to be the world’s largest time capsule in history. Starting today, Yahoo! is encouraging people from around the world to contribute personal photos, stories, thoughts, ideas, poems, home movies and art to this first-ever electronic…
2007	Yahoo! pipes: Yahoo! Pipes was released to the public in beta on 7 February 2007.Yahoo! Pipes is a web application from Yahoo! that provides a graphical user interface for building data mashups that aggregate web feeds, web pages, and other services, creating Web-based apps from various sources, and publishing those apps. The application works by enabling users to “pipe” information from different sources and then set up rules for how that content should be modified (for example, filtering).
2008	The software, called Hadoop, is part of Yahoo’s massive computing grid and is transforming the way Yahoo and corporate giants such as IBM extract meaning from enormous streams of data. Universities are also using the code – an open-source version of software Google relies on for daily operation – to train a new generation of computer scientists and engineers. On February 19, 2008, Yahoo! launched what it claimed was the world’s largest Hadoop production application. The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query. Yahoo joins OPEN SOCIAL: On Mar 25, 2008 Yahoo! also announced it has joined the initiative . OpenSocial is a set of common application programming interfaces (APIs) for web-based social network applications, developed by Google along with MySpace and a number of other social networks. It was released November 1, 2007. Applications implementing the OpenSocial APIs will be interoperable with any social network system that supports them, including features on sites such as Hi5.com, MySpace, orkut, Netlo], Sonico.com, Friendster, Ning and Yahoo!. Yahoo! Inc. announces the general availability of Fire Eagle (http://fireeagle.yahoo.net), an open platform that helps users take their location to the Web while giving them the ability to easily control how and where their location data Yahoo! Opens Up Search Technology Infrastructure for Innovative, New Search Experiences, Providing Third Parties with Unprecedented Access, Re-Ranking and Presentation Control of Web Search Results: BOSS: Build your own search service: The main goal and idea of BOSS is to give users, in this case developers, free access to the Yahoo! Search index. The results can be supplied into the developer’s website or program so that they can manipulate the resources according to their product’s requirements. BOSS allows the results to be returned back in XML, JSON, HTML, text and also allows the comprehensive search feature allowed in Yahoo like pulling the results by pages, searching inside PDF, etc. The ranking of the websites for a search term is same as the Yahoo! Searchranking since both of these are pulling from the same index and ranking.
2009	On June 10, 2009, Yahoo! released its own distribution of Hadoop.

Google:

1998	Google sets up workspace in Susan Wojcicki’s garage at 232 Santa Margarita, Menlo Park.
2000	The first 10 language versions of Google.com are released Google forges a partnership with Yahoo! to become their default search provider. Google search index reaches 1 billion pages Google toolbar is launched
2001	Image Search launches, offering access to 250 million images. Google is available in 26 languages Search index reaches 3 billion mark.
2002	The first Google hardware is released: it’s a yellow box called the Google Search Appliance that businesses can plug into their computer network to enable search capabilities for their own documents. Google releases a major overhaul for AdWords, including new cost-per-click pricing. Google releases a set of APIs, enabling developers to query more than 2 billion web documents and program in their favorite environment, including Java, Perl and Visual Studio. Users can search for stuff to buy with Froogle (later called Google Product Search). Partnership with AOL Google Labs is launched
2003	Google announces a new content-targeted advertising service, enabling publishers large and small to access Google’s vast network of advertisers. (Weeks later, on April 23, we acquired Applied Semantics, whose technology bolsters the service named AdSense.) Google acquires blogger.com
2004	Search index reaches 8 billion Orkut released Keyhole Acquired
2005	Urchin acquired Google Maps, code.google.com launched Google image search boasts of 1.1 billion images. iGoogle launched Google Earth, Google talk launched
2006	YouTube acquired Jotspot acquired Google docs and spreadsheets launched Google custom search launched
2007	Google hot trends launched Partnership with salesforce.com Postini acquired Joint supercomputing project with IBM
2008	DoubleClick acquired Google index: 1 trillion Google Chrome browser launched Google tracks flu trends

IBM:

1995	IBM acquires Lotus
1996	IBM launches its DB2 relational database. IBM acquires Tivoli.
1998	IBM launches the PowerPC 740/750 processors, the world’s first manufactured using IBM’s copper manufacturing technology. Two new AS/400s are introduced, as well as new products in the Aptiva, PC, and Thinkpad series.The IBM S/390 computing system for business is also launched.
1999	The S/390 G6 server, using IBM’s cop per technology, is introduced. IBM and Dell sign a $16 billion technoogy agreement, where Dell will purchase IBM components for use in Dell systems. IBM and Lotus found the Institute for Knowledge Management.
2000	IBM launches the NetVista line of PC devices. IBM launches the eServer line.
2002	Product offerings during 2002 include the eServer p650 eight-way UNIX server, the eServer i890, and the IBM eServer xSeries 440. IBM acquires Price Waterhouse Coopers’ business consulting and technology services unit for $3.5 billion in cash and stock.
2003	IBM and Cisco announce a set of open software technologies designed to advance the development of “self-healing” computer systems and networks. IBM and Siebel launch CRM OnDemand. IBM launches its WebSphere business integration software. Japan’s largest research organization orders an AMD Opteron based eServer 325 supercomputer, running Linux.
2005	IBM plans to expand its data-integration product line through a $1.1 billion acquisition of Ascential Software Corp.
2007	Google and I.B.M. Join hands in ‘Cloud Computing’ Research
2008	Researchers with IBM have developed a new set of software applications designed to improve the human memory. The software is designed to run on a smartphone or mobile handset and analyze collected pieces of data. The collected data is then used to help the user better remember faces and other information such as conversations.
2009	IBM boasts that its so-called Sequoia system will be capable of crunching numbers 20 times faster than IBM’s last record-breaker and 15 times faster than the current fastest machine.

Sources:

– SAGAR JAUHARI, SDE Intern.

Tim Berners-Lee on the next Web

Vikramaditya Jakkula — Thu, 24 Sep 2009 12:50:36 +0000

Lets hear what Tim Berner’s Lee talks about the next Web! Yes, its Linked Data! We need data patterns to understand users, linked data is the answer, and raw or unadulterated data is what we should ask for.

WEKA pens victory at the University of California San Diego student datamining competition

Thu, 24 Sep 2009 05:47:56 +0000

A machine learning algorithm processes a very large dataset to find the best fit in the observed data to learn, and also makes use of the prior knowledge of the system for learning or forecasting. Data mining forms an integral part of the search and understanding of the electronically stored data.

Graduate student Quan Sun’s win at the University of California San Diego student datamining competition surely highlights Weka. Sun claims he used open source software to claim his win in the elite “hard” category for graduate students. In fact, says Sun, at least half the competitors in the competition used the software, called Weka, which he describes as the “Microsoft Word of data mining”.

Waikato Environment for Knowledge Analysis (WEKA), developed at the University of Waikato, New Zealand, is a collection of machine learning algorithms. It has data preprocessing tools to provide inputs to these algoritms.This tool uses Java as its base and is compatible with both Windows and Linux. It is open source software available under the terms of GNU General Public License. Inputs to the machine learning algorithms implemented in WEKA are in the form of relational tables in Attribute Relational File Format (ARFF).

Some of the key features of WEKA

WEKA has inbuilt format converters to convert the dataset available in any format say a spreadsheet to ARFF type. In addition it also incorporates filter to delete specified attributes from the dataset.
WEKA trains and tests the learning algorithms that perform classification and regression and also allows the user to create his own classifier interactively.
WEKA allows the user to handle cluster and their instances.Inaddition it provides access to several methods for attribute selection that might involve either the full data set or a cross validation.

Working with WEKA is very simple. This is mainly because it uses a GUI Explorer. The classification process is not cumbersome, as it involves the selection of the attributes to be related and the algorithm to be used, by the user. The results provide a matrix of both the classified and the misclassified data. The classification error mentioning the mean and the standard deviation are also displayed. WEKA helps in realizing the goal of data mining; by predicting missing values and validating that the predicted values are correct. WEKA is a tool that permits users to develop and analyze new machine learning algorithms to make their job easier.

What can weka do?

reprocess – Weka has file format converters for spreadsheets, C4.5 file formats and serialized instances. It can also open a URL and use HTTP to download an ARFF file from the Web or open a database using JDBC, and retrieve instances using SQL. It also provides a list of filters to delete specified attributes from a dataset.

Direct Hit!

Applies To: Researchers in Data Mining and Artificial Intelligence

USP: Applying machine learning algorithms for data mining

Primary Link: www.cs.waikato.ac.nz/ml/weka Search Engine

Keywords: Machine Learning, Data Mining

Classify – Weka trains and tests learning schemes that perform classification or regression. The classifiers can be divided into Bayesian, trees, rules, functions and lazy. It also builds a linear regression model and allows the user to build their own classifiers interactively. It also provides options for a number of meta learners.

Cluster – Weka shows the clusters and the number of instances in the cluster. Thereafter it determines the majority class in each cluster and gives the confusion matrix.

Associate – Weka contains three algorithms for determining

association rules-apriory, predictive apriory and filtered associators. It has no methods for evaluating such rules.

Attribute Selection – Weka gives access to several methods for attribute selection, which involves an attribute evaluator and a search method. Attribute selection can be performed using the full training set or cross-validation.

In the Preprocess tab, you can view attributes in the input file, properties of the selected attribute, and visualisation of class distribution for each attribute. Building a Naïve Bayes Classifier with 10 fold cross-validation. The correctly classified instances can be viewed by right clicking on Classifier in Results Window.

Visualization – It displays a matrix of two-dimensional scatter plots of each pair of attributes.

Preparing input

Major effort in the process of data mining/machine learning goes into the preparation of input. In order to analyze data using Weka, you need to prepare it in the Attribute Relation File Format (ARFF) and then load it in its Explorer. Spreadsheets, Comma Separated Value (CSV) files and databases can be converted to ARFF. In ARFF, there is an @relation tag, @attribute tag and @data tag to represent the dataset name, attribute information and values respectively.

Classifying data

Weka should preferably be used through a graphical user interface called ‘Explorer’ than the command-line interface. The other two interfaces are ‘Knowledge Flow Interface,’ which supports design configuration for streamed data processing and ‘Experimenter,’ which helps users compare a variety of learning techniques. In this example, we use an ARFF named age.arff which contains a few selective words in the attribute and @data contains their number of occurrences per 10,000 words in a blog dataset written by bloggers belonging to various age groups.

1. Open the file you want to analyze using the Open file option in the Preprocess tab in Weka explorer, ie open the age data file, age.arff.

2. Once the input file has been opened, all attributes in the input file are shown in the Attributes Window. Properties of the selected attributes like Attribute Name, Attribute Type, number of missing values, etc are displayed in the ‘Selected Attribute’ window. Here, you can select attributes that you want to include in working relations, eg age prediction.

3. Select the classifier algorithm in the Classify tab. In this example, we selected Naïve Bayes with 10 fold Cross-Validation. Next, click on Start. The result is displayed in the Classifier Output window as shown in figure on the left.

Output of the Naïve Bayes Classifier in terms of errors, accuracy by class and confusion matrix, on Age dataset. View of an ARFF dataset which consists of a list of instances, and the attribute values for each instance separated by commas.

Analyzing the result

The result displays the summary of the data set followed by the algorithm used to analyze it. It also gives the predictive performance of the machine-learning algorithm applied on the dataset. Thereafter the confusion matrix displays the number of instances classified properly and those misclassified. The classification error is displayed mentioning the mean absolute error and the root mean squared error of the class probability estimates.

Processing huge datasets

If the dataset is too huge, running to a few thousand attributes and a few lakh records, it can happen that Weka runs into an ‘OutOfMemory’ exception. Most Java virtual machines allocate a certain maximum amount of memory which is much less than the amount of RAM to run Java programs. However, we can extend the memory available for the virtual machine by setting appropriate options. Alternately, Weka offers several filters for re-sampling a dataset and generating a new dataset reduced in size. Besides, there are schemes that can be trained in an incremental fashion, not just in batch mode unlike most classifiers which require all the data before they can be trained. Such a classifier will load the dataset incrementally and feed the data instance by instance to the classifier.

Preprocess – Weka has file format converters for spreadsheets, C4.5 file formats and serialized instances. It can also open a URL and use HTTP to download an ARFF file from the Web or open a database using JDBC, and retrieve instances using SQL. It also provides a list of filters to delete specified attributes from a dataset.

Cluster – Weka shows the clusters and the number of instances in the cluster. Thereafter it determines the majority class in each cluster and gives the confusion matrix.

Associate – Weka contains three algorithms for determining association rules-apriory, predictive apriory and filtered associators.

Visualization – It displays a matrix of two-dimensional scatter plots of each pair of attributes.

Other ML/DM software (R, Weka, Yale)

Lluís Belanche

Processing huge datasets

If the dataset is too huge, running to a few thousand attributes and a few lakh records, can lead Weka into ‘OutOfMemory’ exception. Most Java virtual machines allocate a certain maximum amount of memory which is much less than the amount of RAM to run Java programs. However, we can extend the memory available for the virtual machine by setting appropriate options. For large data processing we can take a look at Mahout, an open source scalable, Apache licensed machine learning libraries

KDD 2009: Registration Opens

Vikramaditya Jakkula — Fri, 29 May 2009 19:05:22 +0000

The annual ACM SIGKDD conference is the premier international forum for data mining researchers and practitioners from academia, industry, and government to share their ideas, research results and experiences. KDD-09 will feature keynote presentations, oral paper presentations, poster sessions, workshops, tutorials, panels, exhibits, demonstrations, and the KDD Cup competition.

The registration is now open. The program is made available. Students make sure to visit travel grants section to apply for awards. And for all those who plan to travel check out the deals posted.

For those intrested to see some interesting talks from KDD 2008, videos are available.

Venue: 17 Boulevard Saint-Jacques, 75014 Paris, France