Big Data

App Integration vs Data Integration – Who Will Win?

Bill Schmarzo By Bill Schmarzo June 23, 2014

I was at Starbucks the other day (surprise, surprise) with my friend Matt whose company was getting ready to invest significant time and money to integrate the ERP systems of a company that they had just acquired. I asked him why bother integrating operational systems (except to ensure that the different operational systems are using the same customer, product, store, and other master files), unless there is significant cost savings.

rocky bill pic 1

Instead of trying to integrate the operational systems, why not invest in integrating the data that comes out of those operational systems? As long as you can enter orders and pay people in a timely manner, who cares if you can capture an order a sub-second faster or pay someone seconds faster than before? Is integrating your transactional data capture really the best place for IT to invest their precious resources in today’s competitive world?

ERP is so old school. I wish I had known back in the 1990’s and early 2000’s what I know now:  that trying to create competitive differentiation within or across packaged, monolithic ERP, MRP, CRM, SFA, and other operational systems only benefits the ERP vendors and the systems integrators whose business models are built on the endless customizations to those ERP systems.

If you’re an organization considering an ERP system upgrade or integration, you seriously need to consider how much you want to invest in customizing that ERP system that, at best, just delivers business parity. Or should you invest your time, money, and human resources in building customer-facing apps that provide unique customer value, business differentiation, and competitive advantage?

Integrate the Data, Not the Applications

The graphic below nicely summarizes the value creation transformation occurring within IT organizations (see Figure 1). Organizations are realizing that the business value of their operational systems doesn’t lie in their ability to capture an order faster than their competitors, but instead lies in the depth and breadth of data that can be integrated and mined to capture new insight into customers, products, and operations.

bill blog pic 2 6.23.14

Figure 1: Value transition from app-centric to data-centric

It’s a transformation from application-centric mentality (trying to create value in the deployment and customization of monolithic operational application) to a data-centric mentality (mining value out of the wealth of data held captive in those systems).

Figure 2 below shows a typical IT operational environment. Multiple operational systems manage the transaction processing for various business functions like manufacturing, distribution, inventory, payroll, human resources, finance, call centers, sales force automation, etc.

bill blog pic 3 6.23.14

Figure 2: Traditional monolithic operational apps

You can buy these applications from a mega-vendor (who has probably acquired numerous other vendors in order to create a “ransom note” of loosely connected applications), or you can select a best-of-breed approach where a systems integrator tries to tie these applications together. Either approach leads to a brittle, hard-to-scale, expensive-to-maintain architecture and a significant investment in systems integration and consulting resources to keep these “Franken-architectures” running. And what do you get in the end? Nothing more than business parity.

This quote from a Business Week article titled “Plex Systems: Detroit’s New Dashboard” summarizes the ERP value challenge quite well:

Inteva Chief Information Officer Dennis Hodges explains that because each [of their] offices had its own ERP system running on a local server, managers in Michigan had no way of knowing what was happening in Alabama, Mexico, or Poland. The company was spending more than half a million dollars a month on an ERP product that didn’t allow management to look at revenue and margins across the company.

See my blog “Developing Competitive Differentiation” for more thoughts about where best to invest your precious IT resources to deliver competitive differentiation.

The Role of the Data Lake

Don’t invest (waste?) time and money to integrate your disparate operational applications. Instead, invest in a data architecture (see Figure 3) that allows you to integrate all of the data across those disparate operational applications and is able to capture the other 90%+ of the corporate and external data needed to achieve business differentiation.

That investment in data architecture will enable you to differentiate with superior customer service, successful new product introductions, campaign marketing excellence, fraud elimination, predictive maintenance, revenue loss minimization, increasing market basket margins, reducing the number of hospital-acquired infections, lowering hospital readmissions, etc.

Bill blog pic 4 6.23.14

Figure 3:  Integrate all of your internal and external data in the data lake

See my blog “How I’ve Learned To Stop Worrying And Love The Data Lake” for advice about how to leverage Hadoop to create a data lake. The data lake not only supports the integration of data across your operational applications, but also enables the integration of other internal data sources (consumer comments, email conversations, clinical studies, technician notes, prescriptions, web logs, etc.) with external data sources (social media, mobile, blogs, newsfeeds, third-party data,, etc.).

Embracing an Analytics (Data Science) Culture

But collecting the data isn’t enough. You also need a corporate culture that seeks to deploy data science within your key business functions; analytics integrated into your key business processes to uncover new insight into the “strategic nouns” of your business—your customers, products, partners, campaigns, stores, wind turbines, jet engines, ATMs, trucks, etc.

You need a modern architecture that supports your traditional data warehouse and business intelligence environment, while expanding your data and analytic assets to include advanced analytics and data science capabilities.

Bill blog 5 6.23.14

See my blog “Modernizing Your Data Warehouse Part 2” for more details about how to leverage Hadoop to modernize your data warehouse environment while adding a complementary, advanced analytics sandbox architecture.

Monetizing Customer, Product, and Operational Insights

In the end, the best way to achieve competitive differentiation and uncover new monetization opportunities lies in how you are delivering the insights that you gain from your data lake and advanced analytics environment (see Figure 4).

Bill blog 6 6.23.14

Figure 4: Analytics powering the Third Platform and the Internet of Things

The rise of the “Third Platform,” those pervasive smartphones and mobile tablets, are enabling organizations to deliver actionable insight to customers, partners and front-line employees alike. It’s enabling organizations to optimize key business processes and capitalize on new monetization opportunities. And for many leading organizations, it’s the culmination of IT becoming a strategic partner to the business. Instead of replicating existing business processes within your transactional systems, it enables IT to transform those key business processes and empower new business models.

For an example, see my blog “The Actionable Retail Manager Dashboard:  Next Generation BI,” which talks about how to integrate the insight gleaned from your advanced analytics system to create the next-generation dashboard—a dashboard that not only delivers business insight, but transforms the dashboard from a passive monitoring tool to a prescriptive recommendation engine to help empower front-line employees and management.

Summary of Best Practices

  1. Leave your operational systems in their silos. Don’t waste time and effort trying to integrate your disparate monolithic operational applications, except to ensure that they are using the same product, customer, store, and other master files.
  2. Integrate the data from your operational applications into a data lake that simplifies the integration problem (it’s easier to integrate data than applications). Focus your IT resources (people, time, and money) on those areas of data integration that create business differentiation, not just business parity.
  3. Augment the value of your operational data by adding new structured and unstructured data (both internal and external) to your data lake. And in the process, develop a corporate hunger for grabbing and integrating data into the data lake, even if you’re not yet sure how you might leverage that data.
  4. Finally, focus on:  building differentiated products; optimizing key business processes; monetizing key customer, product, and operational insights; delivering a more compelling, more engaging customer experience; and empowering front-line employees to make decisions that drive business value.













Bill Schmarzo

About Bill Schmarzo

Read More

Share this Story
Join the Conversation

Our Team becomes stronger with every person who adds to the conversation. So please join the conversation. Comment on our posts and share!

Leave a Reply

Your email address will not be published. Required fields are marked *

4 thoughts on “App Integration vs Data Integration – Who Will Win?

  1. Bill, great content and pleasure to read as always. I agree with the presented concepts but wonder if a data lake is the exhaustive solution for all integration needs. I agree that for all data and analytics discovery purposes the data lake offers a very flexible integration pattern. But what about operational analytics? Wouldn’t we need integration structures like MDM to make sure that data is consistent between applications? If the lake is fed on a batch basis it would not be possible to use it e.g., for cross-channel integration of online interactions. Also, for operational processes where data moves across multiple applications (e.g., order management) a data lake would not help you in ensuring a single source of truth.
    Looking forward to your thoughts.

  2. Matt, you make many excellent points. The Data Lake isn’t the cure-all for all our reporting and analytic needs. You raise a couple of use cases that are probably best addressed outside the Data Lake:

    1 – Operational reporting. Probably best to do this off of the operational systems since most operational reporting only requires current data (not historical data). The operational data would eventually find its way into the Data Lake for other analytic needs, but you can probably use products like Crystal Reports for developing operational reports directly off of the operational systems.

    2 – Real-time data analysis. There are several use cases (fraud detection, money laundering, ad serving, real-time bidding, algorithmic trading) where you are going to want to stream the real-time data through a real-time analysis environment first before shuffling the data off to the Data Lake. Environments like GemFire are ideal for this type of analysis that in reality sits in front of the Data Lake. I wrote a blog previously on this particular use case (

    BTW, another key point that you make is the importance of MDM and data governance in a data lake world. Again, the data lake isn’t the “Magic Kingdom” so there is a significant amount of work necessary to ensure that the data in the data lake is accessible, usable and accurate. My colleague Rachel Haines is preparing a blog or white paper that goes into much more detail on this topic. A most piece of excellent work!

    Thanks again Matt!

  3. Great blog and also very informative related app integration vs data integration. This is the exact data which I was looking for. I was bit confuse regarding this and now got clear idea with your content. Thanks for sharing