AI/IoT/Analytics

Are Your Big Data Projects Set Up to Fail?

Frank Coleman By Frank Coleman Senior Director, DELL EMC Services April 30, 2015

Are Your Big Data Projects Set Up to Fail?

I recently read an article called “Where Big Data Projects Fail” by Bernard Marr that resonated with me. Bernard summarized his experiences by saying, “One thing they have in common is they are all caused by a lack of adequate planning.”

He predicts that half of all Big Data projects will fail to meet expectations and highlights 5 major causes of failure:

  1. 1. Not starting with clear business objectives
  2. 2. Not making a good business case
  3. 3. Management Failure
  4. 4. Poor Communication
  5. 5. Not having the right skills for the job

Bernard goes in to detail on each cause of failure and I highly recommend you give it a read.

When I think about my experience this list hits home. Many companies / people don’t fully grasp Big Data and get distracted by the technical component of it rather than defining an objective.

I’ve been fortunate…or misfortunate depending on how you look at it…to have several project failures under my belt. Luckily, they haven’t been multi-million dollar failures. Failure is part of the process but you don’t want to have a colossal failure.

Below are a few questions I always now use to reduce my failure rate.

  • Who is my customer?
  • Executive sponsor – They can help you navigate any management barriers and ensure alignment if you have a cross-functional team. Action takers – Get “buy in” from the group who will take action. This is often overlooked. Our team may agree but if no one has talked to the people who have to use the solution, you have failure just waiting to happen.
  •  What will they do with what I’m building?
  • Understanding what they will do with this solution to achieve their business objective is key.
  • Business Intelligence (BI) layered on top of a Big Data solution is often dumped on the end users without an understanding of the action takers’ needs. This is another failure waiting to happen.
  •  Can I define clear actions items and owners of those items?
  •  You should be able to define and assign a few very specific action items. For each action understand how the owner of this action would best like to use this solution.
  • Building this into workflow may be a better solution than forcing a process change and leveraging a BI solution. BI is often easier and faster so we often jump to this as the answer. But adding a flag or action right into the workflow may be the way to success. This gets back to what Bernard was talking about in lack of planning.
  • Are your owners signed up?
  • Measure Success – Once you have all these questions answered go back to your customer, review this list, and make sure they are signed up.
  • Wherever possible quantify the impact of their actions so you have success metrics defined and agreed to by your action takers. Ensure they know the Executive Sponsor is expecting readouts on this Success metric. Don’t do this to instill fear but to make sure they truly buy in.
  • Many times this conversation will expose potential failure areas for the project. I like to call this the Show me the Money step. If you don’t have a good business case, keep at it or maybe this isn’t the right project to work.

Poor communication and not having the right skills for the job are about proper planning. Run your project with a Project Manager. Don’t just throw a few techs and a Data Scientist together and expect results.

As I’ve said before this is a team sport. Having people on your team that can translate the technical jargon to business speak is essential. Mapping out clear roles and responsibilities of the team will also ensure you don’t have a Data Scientist wasting their time setting up joins on tables when they should be building out a predictive model on another project.

I completely agree with Bernard that Data Scientists are rare and expensive. You don’t want them to quit because of your poor planning and having them do tasks that a Data Engineer should be doing.

If you have any other lessons learned from your failed projects please share them in the comments below or with me directly.

Frank Coleman

About Frank Coleman


Senior Director, DELL EMC Services

Frank is a Senior Director of Business Operations for Dell EMC Services. He is living the world of Big Data in this role, as he is responsible for using advanced data analytics to improve the customer experience with Dell EMC’s services organization.

This role keeps Frank immersed in Big Data, and he is at the cutting edge of using Big Data to solve real business problems. Frank has a strong blend of technical knowledge and business understanding, and has spent the last nine years focused on the business of service.

Under his leadership, EMC was honored in mid-2012 for the third consecutive year with the Technology Services Industry Association (TSIA) STAR Award for “Excellence in the Use of Metrics and Business Intelligence.” Prior to joining EMC, Frank worked in various fields and remote technical support roles.

Read More

Share this Story
Join the Conversation

Our Team becomes stronger with every person who adds to the conversation. So please join the conversation. Comment on our posts and share!

Leave a Reply

Your email address will not be published. Required fields are marked *

3 thoughts on “Are Your Big Data Projects Set Up to Fail?

  1. Nice article, worth a read.
    clear requirements, Business case, Right Skills, communication and good management is very important for any project to be successful.

    An incomplete or improper requirements can directly lead the whole project into a failure state.

  2. This article would be a good tool to consider having each team member read and understand before developing the all important hypothesis for the business problem they have been challenged to fix. Further,e the basic fundamentals called out in this article can be referred to as the true “keys to success”

  3. Great article from Bernard Marr and excellent observations by you. I agree that this is a team sport as well as the importance of mapping out clear roles & responsibilities.