As customers interact with the site, they generate data about their liking towards certain products or feature. Companies like amazon.com and LinkedIn clearly understand how to leverage this information to make the interaction between the customer and the site even more valuable & relevant. Users who are ready to share more data with site about their likes/dislikes, the better would be the site’s recommendation for the user.  The companies need to instil this confidence in the customers mind, and hence have the users share data by will.

               Amazon.com & LinkedIn makes available every little fact about the consumer’s behaviour and interaction with other users or products to help change their behaviour in terms of decision they make to buy or not-buy a product or whether to look for an employer change, etc.

LinkedIn has amazing insights about the companies, profiles which is all available to the users freely. In a interview with linkedIn CEO, Reid Hoffman by Andreas weigend, Reid talks about every individual as a small business and every individual thinks of their reputation in terms of number of new connections, who viewed their profiles, how many times their profile came up in the search results and stats of similar kind.  Andreas Weigend, a social data expert talks behavior change brought about by features like ‘who viewed your profile in the last 15 days’ in end users and in the way companies like LinkedIn treats the users.

(i)                  Insights about companies( lets say we are researching the company mu-sigma):

• Employee switching patterns between companies. Employees moved from ‘xyz’ to mu-sigma.
• Employee switching patterns between companies: Employees  moved from mu-sigma to “abc”.
• Gender distribution: M to F ratio at mu-sigma.
• By years of experience, how does mu-sigma differ from other companies. Similar statistics is available by job function, educational qualification & university. Similar company benchmark is available for comparison.
• People who looked at “mu-sigma” also viewed – other list of companies?
• Where employees of mu-sigma call home
• Most recommended at mu-sigma.
• Time trend of employees who got a change in title.

(ii)                Insights about profiles/users

• Who viewed my profile in the last 15 days?
• How many times did your profile show up in search results?
• Recommendation about other profiles/users you  might know.
• Companies which user might be interested in following.
• Relevant jobs for every user with functionality to apply for it.
• Work recommendations by colleagues and customers.

LinkedIn

Here’s a look at what amazon.com offers. When purchasing a product at amazon.com, the user would be presented with stats related to
• How many users who searched for the book “The outliers by Malcolm Gladwell” (say) ended up purchasing it or ended up purchasing “The tipping Point” , “The Blink” , “What the dog saw”, etc.. in the same order. However I feel the need to quantify the same would help. I mean calling out that 80% of people who searched for “A” ended up purchasing “B”. Or 80% of people who searched for “A” ended up purchasing “A”.
• “Frequently brought together items” for a given product.
• Review statistics: How many rated 5-star, 4-star and so on, as a bar chart.

We are moving towards an era of socializing data with end users to make every little decision they possibly make is data driven. WordPress, Netflix, glassdoor, etc are some of the other companies geared towards this trend. The intention of collecting data has truly gone beyond marketing purpose.

London Cholera Outbreak visualized

Going back to history of visualization. The most famous, early example mapping epidemiological data was Dr. John Snow’s map of deaths from a cholera outbreak in London, 1854, in relation to the locations of public water pumps. The original (high-res PDF copies from UCLA), spawned many imitators including this simplified version by Gilbert in 1958. Tufte (1983, p. 24) says,”Snow observed that cholera occurred almost entirely among those who lived near (and drank from) the Broad Street water pump. He had the handle of the contaminated pump removed, ending the neighborhood epidemic which had taken more than 500 lives.”
The following pointers should help anyone analyze data and socialize finding by effective newer visualizations techniques:

1. Fusion Charts – involves basic chart types, all it needs is a data file, configuration file and can link the chart to the data file, flash based  & supports interactive charts, web supported.
2. Fusion Maps – contains maps of all counties and major cities world wide, interactive, flash based, involves data file and configuration file, web supported.
3. Fusion Widgets – involves coolest visualization techniques like angular gauge, spark line/column,gant chart, pyramid, cylindrical & thermometric gauge & bulb gauge. Some of these charts have power to do real time streaming generally used in stock market analysis.
4. Power Charts – contains some of the rare chart types like node chart, heat map, waterfall chart, multilevel pie chart, candlestick chart,etc. again flash based and hence web supported.
4. R – Revolution Computing – a powerful open source data mining/stat language which can generate stacked multi-combinatorial charts using a single line of command.
5. Google Visualization – javascript based, web supported, involves some of the coolest viz techniques like motion chart which can display data in 5 dimensions, geomap, word cloud, money pile, 3D chart, QR code, etc.
6. Google Charts – contains all basic chart types, from google.
7. Custom Flex Charts – Using customer written flex code and action script code.
8. Microsoft Excel – famous for its quick and ease of chart creation , latest version now has spark line chart support.
9. Tableau – Data Exploration- would recommend this tool for rapid fire analytics involving various dimension, it is just as easy as drag and drop to change views of the metrics by dimension hierarchy.
10. BI Report Tools – BOXI, Cognos – commercial BI tools with support for creation of various report type based on charts and tabular layouts.

Industry Trends involve real time streaming of charts – used in supply chain analytics, interactive charts, mobile supported charts, Creating alerts in charts(for example alert biz. users sending an email, as the sales of any product goes below $x on three consecutive days and so on..), video & audio supported charts.

1. Customers shopped online returned via stores Randy Lea, VP product & service marketing Teradata talks about one of their clients, who had tagged their e-com customers as best customers based on web sales they were generating and reaching out to them with various promotions. However, on integrating their web data with Enterprise data( store data) they found most of the customers were buying things online in multiple units and returning them through stores.

        For example, some customers brought 4-5 shirts of different colors, however they reatined one of them they liked the most and returned the rest of them visiting the stores. Effectively customers were buying through one channel(web) and returning them through another channel(store).Hence the web customers, whom they believed best not actually best rather average shoppers and shouldnt have been sent offers.

Source: Teradata ( Video) 

2. In the United States, if you live more than two miles from a pharmacy store, you probably don’t shop there!In the book data-drien marketing , Mark Jeffery talks about the case of how walgreens optimized their marketing spend using simple geo-spatial visualization. The pic on the right, is a picture of three stores of the Walgreens pharmacy chain on a map.Walgreens is a $59 billion annual revenue pharmacy company with 6,850 stores throughout the United States.

Geo spatial visualization of Walgreens stores

This geospatial picture shows dots that are the customers and where they live and are coded by shape depending on which of the threeWalgreens stores they shop. The ‘‘diamond’’ customers shop at Store 1; the ‘‘square’’ customers, at Store 2; and the ‘‘star’’ customers, at Store 3. This pharmacy retail chain predominantly markets using flyers in newspapers. The way they pay for the marketing is by zip code, denoted by the dashed line, for example, in the picture. Mike Feldner, the marketing manager who first created these pictures, noticed something interesting: the circle on the picture is two miles in radius, and after looking at many pictures throughout the United States, he noticed that there are no dots (customers) for a store more than two miles from the store. He concluded that in the United States, if you live more than two miles from a pharmacy store, you probably don’t shop there. At that time,Walgreens treated each U.S. locale equally; allocating equal dollar amounts for newspaper advertising in each zip code across the United States. But the data show that if there is no store within two miles of the zip code, customers do not shop at the store. Based on these data, Walgreens ultimately stopped spending advertising dollars in all zip codes without a store within two miles of the zip code. As you might guess, the impact to sales revenues was exactly zero. The impact to marketing, however, was a cost saving of more than $5 million, for a total cost of collecting the data and creating the plots of approximately $200,000. This multimillion-dollar saving in marketing did not require a lot of money, and the analysis was done on a personal computer (PC). This is yet another example of being simple in approach, yet making the impact.

Source: “Data-Driven Marketing” by Mark Jeffery

3. We won because we understood the science of incentivizing people to cooperateLate last year the Pentagon’s mad-scientist research wing, Darpa, announced the Network Challenge, a $40,000 prize for the first group to find and report the locations of ten red weather balloons that the agency would set aloft one day in secret locations around the country. Most of the thousands of groups that signed up quickly realized that crowdsourcing was the way to find the 8-foot spheres. So, naturally, they offered bounties to balloon hunters. But Pentland’s crew at MIT’s Human Dynamics Lab–part of the MIT Media Lab–took their crowd control a step further. “It was trivial for us to slap together the balloon thing,” says the 58-year-old Pentland. That’s because other groups’ tactics were based on guesswork, he argues. His were based on lessons learned through data-mining research. “We won because we understood the science of incentivizing people to cooperate.”

Read the entire article here: Mining Human Behavior at MIT

 [contact-form]

R Source Code:

#To set the working directory to folder where source files are placed.(set this to directory as per your needs)
setwd(“C:/Documents and Settings/deepak.babu/Desktop/output”);

#Install the R package arules
install.packages(“arules”);

#load the arules package
library(“arules”);

# read the transaction file as a Transaction class
# file – csv/txt
# format – single/basket (For ‘basket’ format, each line in the transaction data file represents a transaction
#           where the items (item labels) are separated by the characters specified by sep. For ‘single’ format,
#           each line corresponds to a single item, containing at least ids for the transaction and the item. )
# rm.duplicates – TRUE/FALSE
# cols -   For the ‘single’ format, cols is a numeric vector of length two giving the numbers of the columns (fields)
#           with the transaction and item ids, respectively. For the ‘basket’ format, cols can be a numeric scalar
#           giving the number of the column (field) with the transaction ids. If cols = NULL
# sep – “,” for csv, “\t” for tab delimited
txn = read.transactions(file=”Transactions_sample.csv”, rm.duplicates= FALSE, format=”single”,sep=”,”,cols =c(1,2));

# Run the apriori algorithm
basket_rules <- apriori(txn,parameter = list(sup = 0.5, conf = 0.9,target=”rules”));

# Check the generated rules using inspect
inspect(basket_rules);

#If huge number of rules are generated specific rules can read using index
inspect(basket_rules[1]);

#############################################################################
##############  SUPPLEMENTARY  INFO  ########################################
#############################################################################
#To visualize the item frequency in txn file
itemFrequencyPlot(txn);

#To see how the transaction file is read into txn variable.
inspect(txn);

Output:


parameter specification:
confidence minval smax arem  aval originalSupport support minlen maxlen target
0.9    0.1    1 none FALSE            TRUE     0.5      1      5  rules
ext
FALSE
algorithmic control:
filter tree heap memopt load sort verbose
0.1 TRUE TRUE  FALSE TRUE    2    TRUE

apriori – find association rules with the apriori algorithm
version 4.21 (2004.05.09)        (c) 1996-2004   Christian Borgelt
set item appearances …[0 item(s)] done [0.00s].
set transactions …[6 item(s), 7 transaction(s)] done [0.00s].
sorting and recoding items … [2 item(s)] done [0.00s].
creating transaction tree … done [0.00s].
checking subsets of size 1 2 done [0.00s].
writing … [1 rule(s)] done [0.00s].
creating S4 object  … done [0.00s].

As we see from the output, Number of rules generated are 1, with support = 50% and confidence = 90%. The generated rules can be checked using inspect(basket_rules) command:
lhs                              rhs            support               confidence     lift
1 {Choclates} => {Pencil}  0.5714286          1                        1.166667

The above rule means “If a chocolate is brought then there is 90% likelihood of purchase of pencil”. The support 0.57 indicates that 57% of the transaction in the data involve chocolate purchases.  The confidence of 90% indicates out of the transactions which involve chocolates, 90% of them also involved purchase of pencils. Hence the support indicates goodness of the choice of rule and confidence indicates the correctness of the rule.

Also we can see the distribution of items within transactions using image(txn) and  itemFrequencyPlot(txn).

Transaction.csv
===========
1001,Choclates
1001,Pencil
1001,Marker
1002,Pencil
1002,Choclates
1003,Pencil
1003,Coke
1003,Eraser
1004,Pencil
1004,Choclates
1004,Cookies
1005,Marker
1006,Pencil
1006,Marker
1007,Pencil
1007,Choclates

Item Frequency Plot

Image(txn) showing density

• Data source explosion: Business Problems are gaining complexity day by day, hence there is a huge demand for analyzing data from multitude of sources to help companies frame strategies for growth.  GPS data accumulated by Telecom companies offer insights into customers current location and provide context aware recomendations. Infact, some of the telecom companies have introduced location based pricing. Sensor data helps identify security threats to secure networks. Social network data has opened up as a channel for marketing services/product. Analysis of such closely knit data leads to behavioral & Contextual targeting. Traditional data analysis tools/algorithms fail to perform efficiently because such data are of huge sizes and needs newer datastructures for efficient analysis.
• Databases going beyond relational is gaining popularity. NoSQL dbs and Graph/Tree/XML based databases.
• Open Source tools continue to emerge.(R, RapidMiner, Weka)
• Growing need for massive dataset analysis.
• Artificial Intelligence(AI) and NLP gaining popularity among data analysts( in additional to ML techniques)
• Multimedia Analytics: Need for gathering critical metrics like customer footfalls, quantifying customers satisfaction by using facial expressions. All these applications demand high end signal processing( both Image & Video). There is a lot of scope for innovation in this area.
• Privacy preserving techniques for data analysis. This in turn encourages companies to outsource some of the critical data analysis to third parties.
• Agile Methodologies for Analytics Project to cope up with rapidly changing customer/business needs.
• Bio-Inspiration/Bio-Imitation: To learn from nature/natural processes and develop analogous techniques which could potentially solve a real-world problem. Some classic examples are development of Neural network inspired by working of a human brain, solving path optimization problem from Ant colonies, 280 degree view of honey bee(vision) etc.
• More and more data are made publicly available.
• Real Time data integration, insight generation and business decision.
• Complex visualization techniques through new technology like Adobe Flex , MS Silverlight,etc which are known for generating RIA.(Rich Internet Applications)

And I am sure these are just few items in the list and really not exhaustive. Feel free to share your comments.

  (i) From David Mease who teaches DM at Google:You can access approximately 11 hours of video(11 parts) on the semester topic “Statistical Aspects of Data Mining” here http://video.google.com/videosearch?q=mease+stats+202&sitesearch=# and also you can get pdf version of lecture slides and assignments, try to solve them and master them. I guess the author has also some blogs to discuss problems in this topic. The best thing about this video tutorial is that David has demonstrated implementation of each of these techniques using open source data mining tool – R (short for Revolution).

Videos: http://video.google.com/videosearch?q=mease+stats+202&sitesearch=#
Lecture notes -pdf : http://www.stats202.com/original_index.html
Course Home: http://www.stats202.com/

  (ii)From Stanford University as Andrew Ng. teaches “Machine learning”: This is another very usefull video course. The semester course is covered in 20 parts and hence approx. 20 hours of quality knowledge. The best thing about Andrew is he teaches the mathematics so good, you start visualizing equations and that is one good way to learn maths. Its not just about maths, he also demonstrates the video demos on Machine learning projects implemented by his students like autonomous car driving, autonomous flying, converting a picture to a 3-d experience,etc…that way you dont get bored anytime during the lecture.I loved it a lot.Hope you enjoy it too.

videos: http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599&search_query=stanford+%2B+machine+learning
Lecture Notes:  http://www.stanford.edu/class/cs229/materials.html
Course Home: http://cs229.stanford.edu/

   I am sure you will find more content than what i have mentioned here. Feel free to explore the course page. I personally believe anything can be learnt best only by first learning its applications,which in process gets you motivated and the rest is assured. I would like to thank Andrew Ng. and David Mease for sharing their expertise. A good initiative by stanford. Expecting more from top educational schools.

Association Rule mining is one of the classical DM technique. Association Rule mining is a very powerful technique of analysing / finding patterns in the data set. It is a supervised learning technique in the sense that we feed the Association Algorithm with a training data set( as called Experience E in machine learning context) to formulate hypothesis(H) . The input data to a association rule mining algorithm requires a format which will be detailed shortly.
Ok let me first introduce the readers with some of the application areas of this DM technique and motivation for the study of Association analysis. The classic application of the association rule mining is to analyse the Market Basket Data of a retail store. For example, Retail stores like Wal-Mart, Reliance fresh, big bazaar gather data about customer purchase behaviour and they have complete details of the goods purchased as part of a single bill. This is called Market basket data and its analysis is termed “market basket analysis”. It has been found that customers who buy diapers are more likely to buy beer. This is a pattern discovered by association analysis. Other applications include but not limited to scientific data analysis (earth science to study ocean, land and atm. Processes) and in the field of bioinformatics (genome sequence mining, etc.) Also it is used in document analysis for determining the words that often occur together and weblog mining temporal data for any pattern in online behaviour and website navigation. There are numerous other examples of association analysis which is only bounded by human imagination and capability.
Let’s start with Association mining with market basket data as the example. An itemset is the group of items. A k-itemset indicates the no. of items under study is K numbers. As part of a transaction (purchase by customer) one or more items from the itemset may be included. The occurrence/purchase of an item is indicated by a value 1 while non-inclusion is indicated by a value 0. Hence a typical market basket data like the one below:

Hence it means
T1 – {Book,Pen,Eraser,Crayons}
T2 – {Pen,Eraser,Crayons}
T3 – {Book,Eraer,Crayons}
In the above example, we call it an 8-itemset.

If you see the above representation of market basket data, one may think there are few additional info which are missing like the quantity purchased, Amount involved in the transactions/purchase. Of course, the association analysis can be extended to involve such detail.
The application of Association rule mining algorithm results in the discovery of rules/patterns of the following form:
{Pencil} – > {Eraser}
{Book,Maps}- > {Pen}
What it simply says is “If a customer bought a pencil he is more likely to buy an eraser”. Mathematically, it says “Purchase of Pencil implies purchase of eraser”. Once a pattern is discovered, it can used/integrated into decision support system to form strategies based on the rule. In the above case, the company may use this rule to do cross-selling i.e place pencil and eraser as close to each other which increases the sales and hence profit. Just imagine, if a number of such strong rules are disovered in a jewellery shop it would result in tremendous value. Now let me answer what “strong” rule means?
Now that we have defined what a rule is, we are posed with two important questions. Are all rules discovered by my algorithm is really useful / meaningful? How confident i am about the rule? To answer these questions, we use some mathematical measure to quantify the usefulness and confidence.
Most common evaluation measures for a rule are support and confidence measures. There are other measures namely lift, interest factor, correlation. We will talk about it a bit later. A support measure answers the first question, the interestingness measure. It is represented in percentage. It defines how many of my transactions support this rule. If it is say 4/100 it means just 4 out of 100 transactions involve this rule, then probably this is uninteresting so we may choose to ignore it. Hence our Association rule mining algorithm sets some threshold/min value for the support to eliminate uninteresting rules and retain the interesting ones. An example of uninteresting rule could be {pen} – > {eraser}, this could be an uninteresting rule as pen and eraser might be purchased as a matter of chance, i.e it has lower support.
Now having answered the interestingness criteria, we are left with determining the confidence of the rule. A confidence measure quantifies the confidence as a ratio of no. of transaction holding this rule valid against the no. of transactions involving this rule. Higher the value, more reliable is the rule. A strong rule indicates a rule with higher confidence value.

Lets quickly jump into details of the algorithm. The Association rule mining is carried out using the famous Apriori Algorithm. We will also talk about the variations of this algorithm to apply it for continous data and hierarchial data. Before that, let’s formalize the definition of the association analysis problem:
“Given a set of transactions, the problem is to find all rules/patterns with support >= minsup and confidence >= minconf”

The Apriori Algorithm:
A brute force approach is very expensive task. Hence the approach followed by apriori algorithm is to break up the requirement of computing support and confidence as a two separate tasks. In the first step, frequent itemsets are generated i.e those itemsets which holds the criteria of minimum support. In the second and final step, Rule generation is made possible by evaluation the confidence measure. Let’s visualize the approach diagrammatically as shown below:

Apriori Algorithm

Measures could be classified into two categories – subjective and objective. A subjective measure often involves some heuristics and involves domain expertise to eliminate un interesting rules while objective measure are domain independent measures. Support and confidence are good examples of objective measures. Objective measures could be either symmetric binary or asymmetric binary. The choice of measure depends on the type of application and it must be carefully chosen to get quality results.
Simpson’s paradox states that there is a possibility of misinterpretation due to the hidden variable not as part of the analysis influencing the rules/patterns.

The apriori algorithm can be extended to solving various other problems by making little modifications to the data representation methods, Data structures and algorithm.

1. To handle categorital and continous data. For example gender is categorical attribute and can be represented using two items namely gender=’M’ and gender = ‘F’.
2. To handle concept of hierarchy in itemsets. For example, if IPod, Smartphone are two specific itemsets, then we can define a hierarchy item called electronic goods as a parent item.
3. To handle sequential pattern mining. Example: weblog mining, genome sequence mining, customer purchase behaviour.
4. Graph and sub-graph mining: eample – weblogs to identify navigation patterns, chemical structure analysis.
5. To identify infrequent patterns, negatively correlated patterns, etc.

You can also download a copy of the above material here: Association Rule Mining. Please feel free to comment on the topic. You can also subscribe to this blog by clicking on the subscribe button on the right side of the page.

1. Uplift modelling: The true effectiveness of a marketing campaign isn’t response rate! It’s the incremental impact – that is, additional revenue directly attributable to the campaign that would not otherwise have been generated. Yet traditional targeting criteria are often designed to find clients that are interested in the product, but would have bought it whether or not they received a promotion. In such cases, the incremental impact is insignificant and the marketing dollars could have been spent elsewhere.

Net Lift Models are designed to maximize incremental impact by targeting the undecided clients that can be motivated by marketing. These “swing customers” are akin to the swing states of a presidential election; data miners could learn a lot from presidential campaign
More Here: http://www.predictiveanalyticsworld.com/sanfrancisco/2010/agenda.php#day2-2

2. Social Data Mining: There are many networking sites, there’s lot of data out there in the form of tweets, status messages, etc all of which have information. Be it a product related, customer feedback, complaints, oppurtunities, etc. Such data can prove to provide valuable insights about the subject under study.
           In one of the blogs, Eric Siegel talks about interesting facts about social data analysis.
(i)  Health care industry had identified that quitting smoking is contagious.
(ii) Risk of obesity increases if you have a obese friend.
                   So the above facts prove that social connections can reveal more predictive data about the customers.

3.Unstructured Data handling: IBM is working on a project called ‘Avatar’ offer users a mechanism to deal with unstructured data. Nearly 80% of data is unstructured in nature. Traditional BI tools are known to work best with structured data only. But practically most of the data is in the form of mails, documents, blogs,etc which is unstructured in nature. I hope unstructured data handling come to the commercial levels.

4. Real time BI: Now most of the mobile users are GPS enabled, due to its low price offering. This data about customer where-about information can bring out lot of interesting applications. Based on the rate of change of GPS location, we can ascertain the speed of movement of the user( based on this value, we can decide whether the customer is walking or using a vehicle). This data can help in traffic congestion management there by help the city authorities plan better. Analysis of GPS data might give insights on building systems which recommend routes based on current traffic conditions. It’s not just the only use, sky is the limit for the imagining creative ways of using GPS data. However, this raises privacy concerns as this data reveals confidential data about the customer behaviour. It is to be noted that we are in the stage where researchers are developing privacy-preserving data mining algorithms.  But still we have a long way to go.

                I was just thinking why companies don’t model the employee attrition as this may help in predicting the likely chances of employee planning for a job change and take preventive measure to retain him/her if their loss is significant. In fact i know companies which rate their employees during appraisal cycle on a scale of 1 to 5 which in turn decides salary and promotion, this rating is one of the strong predictors of attrition modelling. I promise to bring you more info about this subject as and when i get something interesting to blog about.

If you find my blog interesting, please subscribe here by entering your mail id in the right side subscribe box. Please feel free to comment and share your thoughts.