A Three-Pronged Approach to Enterprise Analytics
Over the past 3 years, I have spoken with hundreds of people at many large enterprises doing analytics of one kind or another. While some of these conversations focused on reporting and Business Intelligence, many expressed the desire to move toward data science and Big Data Analytics.
These conversations have revealed that many companies don’t know how to make the shift toward this end of the analytics spectrum. They may know they want to do something, but not what to do or how to get there – only that they have a need. As a colleague of mine remarked, this is akin to a five year old asking for lunch. They don’t know what they want, they don’t know what you have to offer, they just know they are hungry.
Similarly, companies are becoming hungry for data science and predictive analytics, but don’t always know what they want or when they have completed their quest. Occasionally, groups hire a data scientist or two into their reporting teams. I have seldom seen this approach succeed. The Data Scientist tends to be underutilized or is asked to do reporting work (this doesn’t last long). Or the Data Scientist is managed by someone with expertise in Reporting but not in Data Science and thus can provide little guidance on their unique set of challenges.
After speaking with numerous companies, I’ve noticed a trend where many organizations now take a 3-pronged approach to analytics. Many companies deliberately create three separate groups, with some coupling or coordination among them as follows.
1. Reporting team. This is a centralized reporting team that deals with regular reporting to the business on key metrics of interest. Enterprises seem to have a nearly insatiable appetite for reports and dashboards. I believe part of this is because Business Intelligence artifacts are understandable and familiar to lay people. These teams generally use some kind of RDMBS, OLAP, or similar, and push data to tools such as Crystal Reports, Business Objects, or more recently tools like Tableau, Qliview, or Spotfire.
- A sample project might be to create a report of the 10 customers that spend the most.
2. Advanced Analytics Team. Separate from #1 above, this group is comprised of quantitative analysts, statisticians, operations researchers, and the like, and generally is managed by someone with expertise in these domain areas. These people tend to work with structured data (though don’t have to), often using data stores of manageable, though large, size. Often, these people use desktop analytical tools, such as SAS, R, SPSS, Python, and Weka to create predictive models, propensity models, attrition models, clustering, and execute data mining methods.
- A sample project might be to run a Kolmogorov-Smirnov test to identify a threshold for marketing to customers most likely to buy your products, and not wasting money on those who probably won’t.
3. Big Data Analytics Team. While this team may employ similar methods as #2 above, a key difference is they work with fundamentally different kinds of data. The data is messier, largely unstructured, and require different tool sets to manage, often using HDFS as the data store or other NoSQL data stores (Hbase, Neo4j, Couchbase, and others). As such, this team must employ some of the advanced analytical methods mentioned in #2 above, but in a new context (Big Data). Therefore, this team must be the most technically savvy of the three groups and be comfortable writing code (SQL, Java, Python, R, Scala, and others are popular), as they may deal with applying advanced quantitative methods to messier data, or use entirely different sets of techniques, such as those in natural language processing.
- A sample project might be to identify your company’s most valuable customers. Beyond just reporting on the customers that spend the most, or identifying customers most likely to purchase, a Big Data Analytics team can uncover customers who exert the most influence over your other customers and encourage them to buy your products.
Using Big Data tools (i.e. Hadoop, Pivotal Greenplum) and Linked Data, one could connect customer reference data and tweets to determine which customers are influencing other customers to speak at your events, act as references for your customers and prospects, and determine which referrals result in sales.
It should not be automatically assumed that your most valuable customers—as in the Customer Top 10 list I allude to in #1 above—are those who spend the most in a year. Often, it is customers who cause more people to buy your product that may actually prove to be most valuable. A Big Data Analytics team can enable your organization to develop an app for comparing data structures, perform social network analysis at large scales very quickly, and render a result that will help you gain a more accurate picture of your most valuable customers.
Most organizations I’ve spoken with are mainly doing #1 (reporting), a little bit of #2 (advanced analytics), and still getting their feet wet with #3 (Big Data Analytics). How would you assess your organization’s progress in these 3 areas?