Bill Schmarzo’s Top 2017 Big Data, Data Science and IOT Blogs
To put us on the path for a successful and engaging 2018, here is a quick review of my top 10 blogs from 2017.
Science works within systems of laws, such as the laws of physics, thermodynamics, mathematics, and many others. Scientists can apply these laws to understand why certain actions lead to certain outcomes or why something is going to occur.
While there may never be “laws” that dictate human behaviors, in the world of IOT where organizations are melding analytics (machine learning and artificial intelligence) with physical products, we will see “data science” advancing beyond just “data” science. In IOT, the data science team must expand to include scientists and engineers from the physical sciences so that the team can understand and quantify the “why things happen” aspect of the analytic models. If not, the costs could be catastrophic.
Note: I’m adding Figure 1 to this blog to highlight the importance of the Scientific Method and understanding basic statistical techniques to ensure that one is building their analytics on unbiased data against unbiased hypotheses.
While there is a high probability that machine learning and artificial intelligence will play an important role in whatever job you hold in the future, there is one way to “future-proof” your career…embrace the power of design thinking.
Design thinking is defined as human-centric design that builds upon the deep understanding of our users (e.g., their tendencies, propensities, inclinations, behaviors) to generate ideas, build prototypes, share what you’ve made, embrace the art of failure (i.e., fail fast but learn faster) and eventually put your innovative solution out into the world. And fortunately for us humans (who really excel at human-centric things), there is a tight correlation between the design thinking and the machine learning (see Figure 2).
“The problem is that, in many cases, big data is not used well. Companies are better at collecting data – about their customers, about their products, about competitors – than analyzing that data and designing strategy around it.” – “Companies Love Big Data but Lack the Strategy to Use It Effectively,” Harvard Business Review
Build a business strategy that incorporates big data. Build a business strategy that uncovers detailed customer, product, service and operational insights serve as the foundation for optimizing key operational processes, mitigating compliance and cyber-security risks, uncover new revenue opportunities, and create a more compelling, more differentiated customer or partner experience.
Much of what comprises “Machine Learning” is really not all new. Many of the algorithms that fall into the Machine Learning category are analytic algorithms that have been around for decades. These include clustering, association rules, and decisions trees. However, the detailed, granularity of the data, the wide variety of data sources, and a massive increase in computing power has re-invigorated many of these mature algorithms.
Machine learning is a type of applied artificial intelligence (AI) that provides computers with the ability to gain knowledge without being explicitly programmed. Machine learning focuses on the development of computer programs that can change when exposed to new data (see Figure 4). How can businesses, and business leaders, take advantage?
Blockchain is a data structure that maintains a digital ledger of transactions among a distributed network of entities. Think of a “distributed ledger” that uses cryptography to allow each participant in the transaction to add to the ledger in a secure way without the need for a central authority or central clearinghouse (see Figure 5).
Is blockchain the ultimate enabler of data and analytics monetization; creating marketplaces where companies, individuals and even smart entities (cars, trucks, building, airports, malls) can share/sell/trade/barter their data and analytic insights directly with others?
The impact that has on a company’s financials could be overwhelming, or devastating, depending upon what side of business model transformation you sit.
When you insert something, a new demand, into a circular flow,you create an economic concept called the Multiplier Effect. It is a concept that countries use to consider how to invest money and how that investment, by having it distribute though a supply chain, like the example above, will impact the economy of their country.
Multiplier Effect Definition: “An effect in economics in which an increase in spending produces an increase in national income and consumption greater than the initial amount spent.”
Data exhibits a Network Effect, where data can be used at the same time across multiple use cases thereby increasing its value to the organization. I would contend that this network effect is in fact the same thing principally as the Multiplier Effect.
I had the opportunity in 2017 to give a 10-minute keynote at DataWorks Summit 2017. What sort of keynote could he give in just 10 minutes? Ten minutes is not long for a keynote, and to be honest, I too struggled with what to say.
But after some brainstorming with my marketing experts, we came up with an idea: Pose five questions that every organization needs to consider as they prepare themselves for digital transformation. And while I didn’t have enough time in 10 minutes to answer those questions in a keynote, I certainly did in a blog!
You can also check out a video of my DataWorks Summit keynote presentation, complete with air guitar at the end so that I could embarrass my daughter (my presentation starts around the 39:30 mark)!
Design Thinking, or human-centered design, is all about building a deep empathy with the people you’re designing for; generating tons of ideas; building a bunch of prototypes; sharing what you’ve made with the people you’re designing for; and eventually putting your innovative new solution out in the world (see Figure 8).
There is a good reason why Stanford’s d.school does not sit within one of their existing schools. Design thinking is used in almost all of Stanford’s schools including business, computer science, electrical, mechanical, and even healthcare. Design thinking appears to be one of the secret sauces to Stanford’s success and cultivating the entrepreneurial spirit of its students and faculty (and neighbors, in my case).
I have seen the future! The future is a collision between big data (and data science) and application development that will yield a world of “intelligent apps.”
These “intelligent apps” combine customer, product, and operational insights (uncovered with predictive and prescriptive analytics) with modern application development tools and user-centric design to create a more compelling, more prescriptive user experience.
Intelligent apps will support or enable key user decisions, while continually learning from the user interactions to become even more relevant and valuable to those users.
The journey to building intelligent applications starts by understanding the decisions that key business constituents need to make in supporting their business and operational objectives.
And my #1 blog of 2017 (drum roll please)…
What are the differences between big data and IOT analytics? Big data analyzes large amounts of mostly human-generated data to support longer-duration use cases. IOT aggregates and compresses massive amounts of low latency / low duration / high volume machine-generated data coming from a wide variety of sensors to support real-time use cases.
I don’t believe that loading sensor data into a data lake and performing data science to create predictive analytic models qualifies as doing IOT analytics. To me, that’s just big data (and potentially REALLY BIG DATA with all that sensor data). In order for one to claim that they can deliver IOT analytic solutions requires big data (with data science and a data lake), but IOT analytics must also include:
- Streaming data management with the ability to ingest, aggregate (e.g., mean, median, mode), and compress real-time data coming off a wide variety of sensor devices “at the edge” of the network.
- Edge analytics that automatically analyzes real-time sensor data and renders real-time decisions (actions) at the edge of the network that optimizes operational performance (blade angle or yaw), or flags unusual performance or behaviors for immediate investigation (security breaches, fraud detection).
Figure 1: Scientific Method Beliefs and Biases