Posts Tagged ‘Market Research’

The datamining journey so far ..

Thursday, December 31st, 2009

This new year, let us go through all the major developments that have taken place in the Data Mining industry over the years. Here is a quick glimpse:

datamining journey so far

A description:

1993
  • Development of WEKA begins:
    • In 1993, the University of Waikato in New Zealand started development of the original version of Weka.  Weka (Waikato Environment for Knowledge Analysis) is a popular suite of machine learning software written in Java, developed at the University of Waikato. WEKA is free software available under the GNU General Public License.
1996
  • CRISP-DM is conceived
    • CRISP-DM stands for CRoss Industry Standard Process for Data Mining. It is a data mining process model that describes commonly used approaches that expert data miners use to tackle problems. Polls conducted later in 2002, 2004, and 2007 show that it is the leading methodology used by data miners
1998
  • KXEN  established
    • Founded in 1998, KXEN has corporate offices in San Francisco, California and Paris, France, with Fortune 1000 customers around the world.
1999
  • CRISP-DM 1.0 released
    • After it was conceived in 1996, in 1997 CRISP-DM got underway as a European Union project under the ESPRIT funding initiative. The project was led by four companies: ISL, NCR Corporation,Daimler-Benz and OHRA. The first version of the methodology was released as CRISP-DM 1.0 in 1999.
2000
  • The ‘R’ Project considered stable for production
    • R is an implementation of the S programming language with lexical scoping semantics inspired by Scheme. R was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand, and is now developed by the R Development Core Team.
2003
  • Appricon established
    • In order to provide a better data mining solution, Analysis Studio® and the Analysis Studio® end-to-end logistic regression modeling solution were weaved into enterprise data mining projects in 2003.
    • SAS 9.1 was released in 2003
2004
  • Rapidminer distributed with GNU license
    • The initial version has been developed by the Artificial Intelligence Unit of University of Dortmund since 2001. It is distributed under a GNU license, and has been hosted by SourceForgesince 2004.
    • SAS 9.1.2 was released in 2004.
2005
  • Amazon launches Mechanical Turk
    • The service was launched publicly on November 2, 2005. In early- to mid-November 2005, there were tens of thousands of HITs, all of them uploaded to the system by Amazon itself for some of its internal tasks that required human intelligence. Most of these were related to music CD items.
    • The number of Amazon’s Mechanical Turk HITs in the system soon decreased after its launch in november, and by December 20, there were less than 100 groups of HITs on the average page load
    • Weka receives the SIGKDD Data Mining and Knowledge Discovery Service Award
    • SAS 9.1.3 was released in 2005.
2006
  • Work on CRISP-DM 2.0 begins
    • In July 2006 the consortium of CRISP-DM announced that it was going to start the process of working towards a second version of CRISP-DM. On 26 September 2006, the CRISP-DM SIG met to discuss potential enhancements for CRISP-DM 2.0 and the subsequent roadmap.
    • Pentaho acquires exclusive …..
      • In 2006, Pentaho Corporation acquired an exclusive license to use Weka for business intelligence. It forms the data mining and predictive analytics component of the Pentaho business intelligence suite.
2008
  • COGNOS acquired by IBM
    • Cognos (Cognos Incorporated) was an Ottawa, Ontario based company making business intelligence (BI) and performance management software. On January 31, 2008, Cognos was officially acquired by IBM. The Cognos name continues to be used, being applied to IBM’s line of business intelligence (BI) and performance management products.
    • SAS 9.2 is the latest release (March 2008) and was demonstrated at SAS Global Forum (previously called SUGI) 2008.
2009
  • PASW/ SPSS
    • PASW (formerly SPSS) is a computer program used for statistical analysis. Before 2009 it was called SPSS, but in 2009 it was re-branded as PASW (Predictive Analytics Software). The company announced July 28, 2009 that it was being acquired by IBM for US$1.2 billion.

Microsoft:

1996
  • Microsoft opens new team to build an OLAP product, codenamed Plato (permutation of letters from OLAP)
  • Panorama Software delegation meets with Microsoft
  • Microsoft announces acquisition of Panorama Software development team
1997
  • OLAP Services 7.0 (codename Sphinx) ships
2000
  • Analysis Services 2000 (codename Shiloh) ships
    • Microsoft Analysis Services is part of Microsoft SQL Server, a database management system. Microsofthas included a number of services in SQL Server related to Business Intelligence and Data Warehousing. These services include Integration Services and Analysis Services. Analysis Services includes a group ofOLAP and Data Mining capabilities.
    • Microsoft Corp. announces the beta release of the OLE DB for Data Mining specification, a protocol based on the SQL language, that provides software vendors and application developers with an open interface to more efficiently integrate data mining tools and capabilities into line-of-business and e-commerce applications.
2001
  • XML for Analysis SDK 1.0 ships
2004
  • ADOMD.NET and XML for Analysis SDK 1.1 ship
2005
  • Analysis Services 2005 (codename Yukon) ships
2008
  • Analysis Services 2008 (codename Katmai) ships
2009
  • Microsoft has decided to make the BI Conference into a biennial event, with the next conference in 2010. For 2009, we are excited to team with the Professional Association for SQL Server (PASS) to expand the BI tracks at PASS Summit 2009 and help deliver the content that BI architects, developers, and administrators need to get the most value from their Microsoft SQL Server and BI-based solutions.
  • PowerPivot gives users the power to create compelling self-service BI solutions, facilitates sharing and collaboration on user-generated BI solutions in a Microsoft SharePoint Server 2010 environment, and enables IT organizations to increase operational efficiencies through Microsoft SQL Server 2008 R2-based management tools.

Amazon:

2003
  • “Search Inside the Book” is a feature which allows customers to search for keywords in the full text of many books in the catalog. The feature started with 120,000 titles (or 33 million pages of text) on October 23, 2003. There are currently about 250,000 books in the program. Amazon has cooperated with around 130 publishers to allow users to perform these searches.
2005
  • In November 2005, Amazon.com began testing Amazon Mechanical Turk, an application programming interface (API) allowing programs to dispatch tasks to human processors.
2006
  • Amazon launched an online storage service called Amazon Simple Storage Service (Amazon S3). An unlimited number of data objects, from 1 byte to 5 gigabytes in size, can be stored in S3 and distributed via HTTP or BitTorrent .In April 2006, Amazon introduced Amazon Simple Queue Service (Amazon SQS), a distributed queue messaging service.
2007
  • In January 2007 Amazon launched Amapedia, a collaborative wiki for user-generated content to replace ProductWiki
  • In December 2007, Amazon introduced SimpleDB, a database system, allowing users of its other infrastructure to utilize a high reliability high performance database system.
2008
  • Amazon Web Services launched a public beta of Amazon Elastic Compute Cloud running Microsoft Windows Server and Microsoft SQL Server.

Yahoo!:

2002
  • Yahoo! HotJobs, previously known as hotjobs.com, is an online job search engine. It has been known as Yahoo! HotJobs since being acquired by Yahoo! in 2002. Yahoo! HotJobs provides tools and advice for job seekers, employers, and staffing firms.
2003
  • Yahoo! Introduces Smartsort Technology: Personalized Product Recommendation Tool
    • The new Yahoo! Product Search powers the redesigned Yahoo! Shopping, providing consumers with the most comprehensive and relevant comparison-shopping site on the Web. The redesigned Yahoo! Shopping now boasts a variety of comparison-shopping features including: side-by-side product comparison, detailed buyer’s guides, tax and shipping calculator tool, consumer product and merchant ratings, unbiased expert product reviews etc. Yahoo! Shopping is the third largest multi-category commerce destination on the Web. (Nielsen//NetRatings, August 2003)
2004
  • Yahoo! Launches SmartView Technology: New Mapping Feature Creates Customized Visual Search Capability
2005
  • Yahoo! Search Launches Search Subscriptions Beta, Providing Select Deep Web Content to Users
2006
  • Yahoo! Opens Internet Time Capsule to Capture Life in 2006
    • SUNNYVALE, Calif., October 10, 2006 – Yahoo! Inc. (Nasdaq:YHOO) today announced the launch of what is expected to be the world’s largest time capsule in history. Starting today, Yahoo! is encouraging people from around the world to contribute personal photos, stories, thoughts, ideas, poems, home movies and art to this first-ever electronic…
2007
  • Yahoo! pipes: Yahoo! Pipes was released to the public in beta on 7 February 2007.Yahoo! Pipes is a web application from Yahoo! that provides a graphical user interface for building data mashups that aggregate web feeds, web pages, and other services, creating Web-based apps from various sources, and publishing those apps. The application works by enabling users to “pipe” information from different sources and then set up rules for how that content should be modified (for example, filtering).
2008
  • The software, called Hadoop, is part of Yahoo’s massive computing grid and is transforming the way Yahoo and corporate giants such as IBM extract meaning from enormous streams of data. Universities are also using the code – an open-source version of software Google relies on for daily operation – to train a new generation of computer scientists and engineers. On February 19, 2008, Yahoo! launched what it claimed was the world’s largest Hadoop production application. The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.
  • Yahoo joins OPEN SOCIAL: On Mar 25, 2008 Yahoo! also announced it has joined the initiative . OpenSocial is a set of common application programming interfaces (APIs) for web-based social network applications, developed by Google along with MySpace and a number of other social networks. It was released November 1, 2007. Applications implementing the OpenSocial APIs will be interoperable with any social network system that supports them, including features on sites such as Hi5.com, MySpace, orkut, Netlo], Sonico.com, Friendster, Ning and Yahoo!.
  • Yahoo! Inc. announces the general availability of Fire Eagle (http://fireeagle.yahoo.net), an open platform that helps users take their location to the Web while giving them the ability to easily control how and where their location data
  • Yahoo! Opens Up Search Technology Infrastructure for Innovative, New Search Experiences, Providing Third Parties with Unprecedented Access, Re-Ranking and Presentation Control of Web Search Results:
    • BOSS: Build your own search service:   The main goal and idea of BOSS is to give users, in this case developers, free access to the Yahoo! Search index. The results can be supplied into the developer’s website or program so that they can manipulate the resources according to their product’s requirements. BOSS allows the results to be returned back in XMLJSONHTMLtext and also allows the comprehensive search feature allowed in Yahoo like pulling the results by pages, searching inside PDF, etc. The ranking of the websites for a search term is same as the Yahoo! Searchranking since both of these are pulling from the same index and ranking.

2009
  • On June 10, 2009, Yahoo! released its own distribution of Hadoop.

Google:

1998
2000
2001
  • Image Search launches, offering access to 250 million images.
  • Google is available in 26 languages
  • Search index reaches 3 billion mark.
2002
  • The first Google hardware is released: it’s a yellow box called the Google Search Appliance that businesses can plug into their computer network to enable search capabilities for their own documents.
  • Google releases a major overhaul for AdWords, including new cost-per-click pricing.
  • Google releases a set of APIs, enabling developers to query more than 2 billion web documents and program in their favorite environment, including Java, Perl and Visual Studio.
  • Users can search for stuff to buy with Froogle (later called Google Product Search).
  • Partnership with AOL
  • Google Labs is launched
2003
  • Google announces a new content-targeted advertising service, enabling publishers large and small to access Google’s vast network of advertisers. (Weeks later, on April 23, we acquired Applied Semantics, whose technology bolsters the service named AdSense.)
  • Google acquires blogger.com
2004
  • Search index reaches 8 billion
  • Orkut released
  • Keyhole Acquired
2005
  • Urchin acquired
  • Google Maps, code.google.com launched
  • Google image search boasts of 1.1 billion images.
  • iGoogle launched
  • Google Earth, Google talk launched
2006
  • YouTube acquired
  • Jotspot acquired
  • Google docs and spreadsheets launched
  • Google custom search launched
2007
  • Google hot trends launched
  • Partnership with salesforce.com
  • Postini acquired
  • Joint supercomputing project with IBM
2008
  • DoubleClick acquired
  • Google index: 1 trillion
  • Google Chrome browser launched
  • Google tracks flu trends

IBM:

1995
  • IBM acquires Lotus
1996
  • IBM launches its DB2 relational database.
  • IBM acquires Tivoli.
1998
  • IBM launches the PowerPC 740/750 processors, the world’s first manufactured using IBM’s copper manufacturing technology.
  • Two new AS/400s are introduced, as well as new products in the Aptiva, PC, and Thinkpad series.The IBM S/390 computing system for business is also launched.
1999
  • The S/390 G6 server, using IBM’s cop per technology, is introduced.
  • IBM and Dell sign a $16 billion technoogy agreement, where Dell will purchase IBM components for use in Dell systems.
  • IBM and Lotus found the Institute for Knowledge Management.
2000
  • IBM launches the NetVista line of PC devices.
  • IBM launches the eServer line.
2002
  • Product offerings during 2002 include the eServer p650 eight-way UNIX server, the eServer i890, and the IBM eServer xSeries 440.
  • IBM acquires Price Waterhouse Coopers’ business consulting and technology services unit for $3.5 billion in cash and stock.
2003
  • IBM and Cisco announce a set of open software technologies designed to advance the development of “self-healing” computer systems and networks.
  • IBM and Siebel launch CRM OnDemand.
  • IBM launches its WebSphere business integration software.
  • Japan’s largest research organization orders an AMD Opteron based eServer 325 supercomputer, running Linux.
2005
  • IBM plans to expand its data-integration product line through a $1.1 billion acquisition of Ascential Software Corp.
2007
  • Google and I.B.M. Join hands  in ‘Cloud Computing’ Research
2008
  • Researchers with IBM have developed a new set of software applications designed to improve the human memory. The software is designed to run on a smartphone or mobile handset and analyze collected pieces of data. The collected data is then used to help the user better remember faces and other information such as conversations.
2009
  • IBM boasts that its so-called Sequoia system will be capable of crunching numbers 20 times faster than IBM’s last record-breaker and 15 times faster than the current fastest machine.

Sources:

–  SAGAR JAUHARI, SDE Intern.

  • Share/Bookmark

The path of Business Intelligence

Sunday, October 4th, 2009

What is Business Intelligence(BI)?

“Business Intelligence (BI) helps business people make more informed decisions by providing them timely, data-driven answers to their business questions. BI analyzes data stored in data warehouses, operational databases, and/or ERP systems (i.e. SAP®, Oracle, JD Edwards, Peoplesoft) and transforms it into attractive and easy to understand dashboards and reports. BI delivers the insight needed to make strategic planning decisions, improve operational efficiencies, and optimize business processes.” -Microstrategy.

What are BI tools?

Business Intelligence (BI) tools  are a set of software systems and practices that enable organizations to analyze data, and make better decisions based on the insight from that information. Companies can use this insight to take the following steps to improve overall corporate performance:
Enhance cost-efficiency and productivity
Build strong customer relationships
Optimize revenue-generating strategies
Increase revenue and maximize profitability
Monitor trends and discover anomalies
Forecast business opportunities
Maintain compliance and perform risk management

Business Intelligence (BI) tools  are a set of software systems and practices that enable organizations to analyze data, and make better decisions based on the insight from that information. Companies can use this insight to take the following steps to improve overall corporate performance:

  • Enhance cost-efficiency and productivity
  • Build strong customer relationships
  • Optimize revenue-generating strategies
  • Increase revenue and maximize profitability
  • Monitor trends and discover anomalies
  • Forecast business opportunities
  • Maintain compliance and perform risk management

Categories of BI Tools:

State of art of BI in Industry today:

In one word, most companies do not use BI to their fullest potential according to market research.

Some stated reasons include:

  • Lack of proper training, followed by limited staffing resources.
  • Most custom reports remain very sophisticated
  • IT staffs still create the majority of BI reports, followed by  business analysts.
  • Dissatisfaction with BI technology is found among all user communities.
  • The idea that BI applications can create more work

Now the latest of all is the move of BI from industry to everyday life.

  • Share/Bookmark

Advertise the data mining way

Wednesday, September 23rd, 2009

office

A discussion room in an advertising company will now comprise of a few data miners as well. Yes, this is the latest trend. The question of “why data miners in an ad agency?” will be the first one which would occur to our  mind. The simplest answer that can satisfy our ask is, the company wants to market ads based on user needs. Based on the user data analysis, the marketing would bare more tastier fruit. But why not we have a data analysis done by a market researcher? The ad agencies are looking ahead and taking one step further. They are paying close attention to user needs. Does optimization of ads ring a bell?  Also, with the recession taking up the market, companies sought to take every advantage they can leverage, even combining marketing with digital technology for better ads and better ROI.

What has data mining got to do with the marketing?

The digital advent has highly increased the necessity for smarter ways of advertising through internet. Advertisers have started to thoroughly examine and debate data mining and other new sciences that will shape the interactive marketplace. It is sure to be broached in discussions about consumer trust, and maybe even in a Facebook session aptly titled “Knowing is Better.”

An instance of the same…..

An ironic prelude to the week-long fete of advertising’s digital future was the Sept. 18 settlement of a privacy lawsuit related to Facebook’s social ad experiment, Beacon. The short-lived, poorly executed program riled online consumers, whose purchase information with off-site retailers such as Zappos and Blockbuster was unexpectedly shared with their Facebook friends. Their only recourse was to “opt out” of the program after the damage was done. While there are mounting examples of online consumers trading their personal information and privacy for more targeted interactive results, Beacon assumed too much with its initial tacit user approval.

Going the giant’s way…

All major internet giants like  Google, Yahoo, Microsoft, etc. have already gone far and wide in dealing with user data. They have reports saying how their research with user data has proven to be fruitful. User insights, social connections, personal preferences and buying history — which Amazon and Google already masterfully manipulate — are building blocks for an interactive economy that relentlessly exploits links to generate revenues.

Forrester Research shows that even as digital grows from 12 percent of existing overall advertising spend to 21 percent (or $55 billion) by 2014, there is a pressing necessity for companies to master constructive interactive relationships with consumers and each other to generate many times that in digital sales and other transactions.

So why this sort of marketing now?

Though recession has considerably reduced the advertising budgets of the company and the consumer spending allocations, there are signs of few marketers who drift consumers and technology into the interactive future.

What exactly will advertisers and media do with those interactive connections, and the insights and information they yield? What do you think?

-Vidhya, Student Intern

  • Share/Bookmark