Big Data in Automotive and Machinery: Using Analytics to Deliver Better Products and a More Fulfilling Driver Experience

Bill Schmarzo By Bill Schmarzo September 10, 2012

“Our vehicles are very well instrumented. They’re closed loop control systems. There are many many sensors in each vehicle… Until now, most of that information was [just] in the vehicle, but we think there’s an opportunity to grab that data and understand better how the car operates and how consumers use the vehicles and feed that information back into our design process and help optimize the user’s experience in the future as well.”

John Ginder, Ford Motor Company
Ford’s Big Data Chief Sees Massive Possibilities
ZDNet, July 5, 2012

The automotive industry (I have included cars, trucks, buses, semi-trailers, farm equipment, and construction equipment in this category) is on the cusp of a Big Data revolution.  Automotive companies have the opportunity to leverage new sources of Big Data to accelerate product design, improve vehicle performance and enhance the driver experience.  These sources of data include:

  • Internally-generated unstructured data (emails, consumer comments, design notes, service and maintenance notes)
  • Externally-available unstructured data (social media, blogs, product reviews)
  • Sensor-generated data (sensors, onboard communications, GPS, telematics) especially as manufacturers move towards the “connected vehicle” with upwards of 10,000+ sensors per vehicle generating data including MPH, MPG, RPM, oil pressure, water temperature, engine temperature, tire wear, oil viscosity, fuel efficiency, etc.

Let’s start a great example of how Big Data is transforming one of the world’s oldest industries (no, not that industry) …farming.

John Deere:  The Big Green “Connected” Profit Machine

“There are various versions of AutoTrac, the GPS-linked system that lets the tractor do the driving itself, and of JDLink, which allows farmers and dealers to monitor tractors remotely. Used along with onboard sensors that track crop yields, these tools can help a farmer decide, for instance, which parts of a field need more or less fertilizer—so-called precision farming. And JDLink can save farmers money by using cellular technology that alerts Deere dealers about equipment problems before they become costly.”

Deere’s Big Green Profit Machine
BusinessWeek, July 26 2012

I love the term “Precision Farming.”  Being from Iowa (Charles City), I’m especially proud of examples where middle American companies are not getting caught up in the hype of Big Data, but are instead making practical use of advanced technologies and new sources of data to improve the quality and reliability of their products, enhancing the driver or operator experience, and optimizing the effectiveness of their dealer networks.

Enhancing “Traditional” Automotive-based Analytics

New detailed data sources, coupled with advanced analytics and new data preparation and enrichment capabilities, can yield new insights on some of the automotive industry’s traditional business areas including:

  • Warranty Analysis.  Automotive companies can improve their ability to predict warranty costs and forecast warranty liabilities by integrating consumer comments (both internally generated as well as comments from external social media, blogs, and product review sites), dealer service notes, and existing warranty claims data.
  • Predictive Maintenance.  The bevy of data being generated from the 1,000’s of on-board sensors provides the potential to flag abnormal events more quickly and to proactively take corrective actions on potential product performance problems.  Wouldn’t it be great to know that your right rear tire is ready to blow out before it actually does!
  • Product Performance.  Product Development should want to be able to monitor the performance of their vehicles and each vehicle subsystem in order to guide future product design and development efforts.  Being able to monitor specific product performance holds the potential to shorten the product development process by quickly identifying those areas where product design and development needs to prioritize.
  • Parts Forecasting and Distribution.  If you understand warranty trends (by car by component by geography by season), predictive maintenance trends and product performance problems, then one can leverage that data to greatly improve the effectiveness of inventory, distribution, and supply chain systems while driving out redundant costs.  That’s a winner for customers, dealers, and manufacturers alike!
  • Dealer Satisfaction.  Being able to monitor social media and the blogosphere are a couple of effective ways of monitoring dealer satisfaction.  In fact, it’s also a great way to monitor customer satisfaction with competitive dealers in order to identify potential win-back opportunities.

The Connected Vehicle:  Improving the Driver Experience

In addition, automotive companies have the opportunity to leverage additional sources of data to understand and analyze driver behaviors in order to improve the overall driver experience.  These insights can be fed into the product design processes and used to enhance the customer’s driving experience.  Some of these new sources of data for driver behavioral analytics include:

  • Embedded car sensors
  • On-board navigation (GM’s OnStar, Ford SYNC)
  • Telematics (like Progressive Snapshot)
  • GPS-enable smartphone apps
  • Weather conditions

Data enrichment techniques need to be incorporated into the driver behavioral analytics to create new metrics that might be better predictors of product performance and driver behaviors.  Think Moneyball[1] in creating and testing new composite metrics (such as frequency, recency, and sequencing) and their ability to improve behavioral and product performance predictive models.

All of this highlights the need for a well orchestrated, tightly coordinated data acquisition and instrumentation strategy to further automotive companies’ insights into driver behavioral analytics.

Digital Media Attribution Example

There are relevant learnings from other industries that automotive companies can leverage to support this Driver Behavioral Analytics initiative.  The digital media industry uses advanced attribution analysis to credit the appropriate stream of activities that lead to an online conversion event.  These same techniques can be used to understand the series of driver actions that lead to certain driving outcomes.

Digital Media Challenge:  Across myriad different digital treatments (keyword searches, display ad impressions, display ads clicked), to what activities do you attribute credit when a conversion event occurs?

Hypothesis:  It’s not only the online behaviors that drive conversions, but it is also the frequency, recency, and sequencing of these activities that contribute to conversions, and that certain combinations of activities (across frequency, recency, and sequencing) are more effective in predicting certain behaviors.

Solution:  Create an attribution model that analyzes all the visitor’s activities (keyword searches, impressions viewed, display ad clicks) that occurred prior to a conversion event, and create new digital media metrics (frequency, recency, and sequencing) that quantify the effectiveness of the different digital media activities.  Use these new metrics to optimize media spend across multiple digital campaign dimensions including audiences, keywords, websites, display ads, ad locations, ad types, time of day, day of week, day, week, etc.

This same attribution modeling can be used in driver behavioral analysis to identify and quantify the series of driver actions that take place prior to some defined outcome (hard braking, spin out, crash, product failure).

Driver Behavioral Business Applications

There are several business areas where identifying, analyzing, and quantifying driver behavioral tendencies can drive product differentiation and improve the driver experience including:

  • In-flight forecasting of driving range and recommendations of optimal driving behaviors for limited range cars, taking into account variables like passenger and cargo weight, vehicle operating conditions (air pressure, oil pressure, air filter throughput, oil viscosity) and current weather and traffic conditions.
  • Flagging unusual driving behaviors that might indicate a potentially stolen vehicle or impaired driving capabilities (driving while impaired, falling asleep, texting, talking on the phone).
  • Support an enhanced in-car driver experience by providing driving performance improvement recommendations directly on the dashboard and/or the driver’s smartphone app.


Big Data can change the game for automotive companies and in the process create a more sticky, longer-term customer relationship.  Intelligent vehicles can provide a platform for not only improving product performance and providing predictive maintenance, but can enhance the driver experience by modifying vehicle performance based upon driver behaviors and make driving improvement recommendations.

Lastly, don’t miss the opportunity to join me this Wednesday, 9/12 for webcast Analyze This! Best Practices For Big And Fast Data. We will discuss one of the most important trends in data management – the emergence of big and fast data!

[1] “Moneyball, The Art of winning an Unfair Game” by Michael Lewis

Bill Schmarzo

About Bill Schmarzo

Read More

Share this Story
Join the Conversation

Our Team becomes stronger with every person who adds to the conversation. So please join the conversation. Comment on our posts and share!

Leave a Reply

Your email address will not be published. Required fields are marked *

5 thoughts on “Big Data in Automotive and Machinery: Using Analytics to Deliver Better Products and a More Fulfilling Driver Experience

  1. Pingback: Big Data en automoción | Big data

  2. Hi Bill, Great article I must say! Lots of useful information.
    In fact, I’m also doing a mini project on Automotive-based Analytics you have mentioned but I need some sample vehicle sensors data.
    Are you aware of any such source where I could get this data? Please advise.


  3. Hey Prashant, I’m not aware of such a data source, but man, that would be fun to play with! Maybe trying exploring some of the Michigan Universities to see if they might have gotten such data. They might have an “open data” relationship with the local automotive industry.


  4. Pingback: How is big data driving the automotive industry? | CODE_n Blog

  5. Big data has a lot of capacity to profit organizations in any kind of industry, ubiquitously in world. it is a useful to decision-making and helpful to improve the financial position of any organization.