Crossing the Chasm with Big Data
“The Business Potential of Big Data!!” Over the past few months, I’ve talked to many folks at several conferences about this. Every conversation, I hear the enthusiasm about the Big Data technology enablers. Every conversation, I see the excitement regarding how Big Data can power key business initiatives. I see clients across many industries using Big Data in areas such as customer lifecycle management, predictive maintenance, customer satisfaction, claims adjudication, cross-media campaign attribution, patient treatment effectiveness, inventory and supply chain effectiveness, network operations optimization, and fraud and network theft detection.
However, many companies are hesitant to jump at this game-changing opportunity. Why are they still waiting on the sidelines? Let’s take a moment to review the market dynamics of new technology adoption, and the applicability of those lessons to the “Big Data” market of today.
Geoffrey Moore and Crossing the Chasm
In 1991, Geoffrey Moore wrote the seminal book on technology adoption called “Crossing the Chasm.” If you work in the technology industry, or your job involves working with technology, then this book is a must read. Below is an edited description of the theory behind the book.
In “Crossing the Chasm,” Geoffrey Moore postulates that there is a chasm between the Early Market Adopters of technology and the Mainstream Market (see chart below). Moore believes that the Visionaries and Pragmatists that comprise the Early Market Adopters have very different expectations of new technology innovations, and are more willing to work with “incomplete” technologies in order to gain competitive advantage. However the Mainstream Market wants a technology that is more complete and proven before adopting. It is this interplay between the technology and business benefits and risks for the Early Market adopters versus the Mainstream Market that sets the stage for the technology to “cross the chasm” and become a de facto standard.
Figure 1: Geoffrey Moore’s “Crossing the Chasm”
How Does Big Data “Cross the Chasm?”
The encouraging signs are there regarding big data “crossing the chasm.” Visionary companies like Google, Facebook, Amazon, and Yahoo have proved the technology and built out big data architectures primarily because A) existing technologies could not handle the volume, variety, and velocity of their data, and B) their businesses are ALL about data; at the end of the day, that’s all they sell (with the exception of Amazon). So we already have a good idea as to what the final Big Data end point will look like.
1) We have a good understanding of the modern, Big Data-ready architecture requirements:
2) We also have a good understanding regarding the technology components of the Big Data architecture (see graphic below) (note: modern Big Data Fast Data architecture components are reflected above the dotted line, while traditional BI/DWH architecture is reflected below).
Figure 2: The Modern, Big Data Architecture Components
3) Finally, we are starting to understand the organizational and human requirements of a Big Data organization. Chuck Hollis’s recent blog – The Role of the CIO in Big Data Analytics – lays out many of the non-technology areas of Big Data adoption that need to be addressed (organizational alignment, training, hiring, governance).
So why are people waiting? What more do they need to see in order to start their Big Data, game-changing journey?
Perceived Value Versus Risk Assessment
I get the impression that some companies feel that the risks associated with Big Data technology and skill sets today over-shadow the potential business benefits enabled by Big Data (see chart below).
Figure 3: Pre-chasm Risk-to-Reward Challenge
In particular, many folks are “uncomfortable” with Hadoop and the Hadoop ecosystem (e.g., MapReduce, Pig, HBase, Hive). They have little to no experience with it, they don’t have any skilled people, and hiring skilled people is expensive and time consuming. There are small Hadoop consulting shops to help address the skills shortage in the short run, but organizations don’t feel comfortable with pursuing new technologies until they have their own, trained people. And until they start to see more compelling business cases within their own companies, there is going to continue to be a reluctance to “cross the Big Data chasm” to mainstream adoption.
So the key is to 1) reduce the technology and skills risks associated with Big Data while 2) identifying compelling organization-specific business use cases.
Shifting the Risk-to-Reward Balance
There are five steps in the Big Data journey that address this Big Data risk-to-reward dilemma. Each of these steps either reduces the risk associated with Big Data technology adoption, or increases the business value and benefits to the organization.
- Step 1: Identify a key business initiative where big data (large volumes, wide varieties, and/or high velocity of data) and new technology innovations (Hadoop, MPP databases, in-database analytics, in-memory computing) can deliver compelling competitive advantage. This might be in the area of adding unstructured internal data (e.g., consumer comments, emails, notes) and social media data to your customer acquisition, maturation, and retention processes. Or it might be in leveraging machine or sensor generated data to optimize network operations, reduce fraud, and improve predictive maintenance.
- Step 2: Conduct envisioning workshops to help the business users to see the “realm of the possible” with respect to how these new sources of data and innovative technologies can yield valuable and actionable insights in support of their key business initiatives. Use group dynamics to tease out and prioritize the different business opportunities where Big Data can deliver new insights about customers, products, operations, markets, and competitors.
- Step 3: Deploy an analytics lab in a fail-safe environment where you can prove out the business case for big data to power a key business initiative. This also provides an opportunity to begin the skills transfer and best practices learning as the organization works to become self-sufficient on these new data sources and technologies.
- Step 4: Operationalize the results of the analytics lab into the appropriate systems (e.g., call centers, direct marketing, procurement, inventory, marketing). To steal from Stephen Covey, “begin with an end in mind” as you identify the operational parts of the business where the analytic insights can deliver compelling and actionable value to the business.
- Step 5: Throughout the process, conduct data scientist training and certification, and ensure that your Big Data projects with outside consultants focus equally on skills transfer as well as overall project delivery success.
These five steps are simple, logical, low-risk, and provide an immediate way to get started down the Big Data path (see chart below).
Figure 4: Shifting the Risk-to-Reward Balance of Power
The best way to drive Big Data across the big data chasm in your organization is to focus on a business solution. So there is no need to stand on the sidelines waiting for whatever to happen. Jump in! The water is fine!!