Executive Summary
We are in the midst of a data arms race. Due to a handful of disruptive societal trends and emerging technologies, it has never been easier and cheaper to leverage data to support business outcomes. This is good news for proactive firms, and disastrous news for firms who move slowly.
In this post, I share a quick overview of what I believe to be the most disruptive societal trends behind the data arms race, emerging technologies which are upping the ante, and, ultimately, what capabilities I believe to be critical for competing and winning in the data arms race.
Disruptive Forces
-
Big Meaning, Not Big Data
-
The Gig Economy
-
Quantified Action
Emerging Technologies
-
Open-Sourced Tools
-
Infra as a Service
-
Knowledge Graphs
Core Capabilities
-
Data Support Teams
-
Rapid Onboarding
-
Rapid Deployment
-
Leveraging IaaS
Disruptive Forces
A disruptive force is a known or emerging trend with potential to impact all industries.
Disruptive forces represent shifts in customer behavior, markets, and/or preferences or advances in technology that create new standards for products, services, and experiences. The future will be shaped and formed not by a single trend but a combination of disruptive forces.
Savvy firms should therefore consider future initiatives in light of these trends, harnessing them like wind behind the sails of a ship.
1. Big Meaning, not Big Data
The increased digitization of work processes and the proliferation of sensors have laid the groundwork for the big data revolution. Making the leap to ‘big meaning,’ however, requires organizations to selectively identify and decipher nontraditional sources of information. The focus of analytics will shift from conducting analysis using a broad set of data to pursuing data thoughtfully and deliberately.
2. The Gig Economy
Dynamic networks of shared, coordinated resources are reshaping the way we live and work. As technological innovation assumes more functional and administrative roles, top-tech talent will be incentivized to work in ‘freelance,’ independent roles. Companies will need to be able to tap into the gig economy to meet their tech needs with a flexible, efficient talent pool.
3. Quantified Action
Consumers have been increasingly collecting, tracking, and sharing information about their everyday activities and environment. In doing so, they are setting the conditions to use not just traditional data, but quality of life information to inform business and marketing decisions. Companies who are able to take advantage of these conditions will outperform those who cannot process the vast amounts of information available.
Emerging Technologies
For sake of this perspective, I use the term ‘emerging technology’ to indicate any data or computer science specific technology which will fundamentally change the landscape in which businesses operate.
The technologies included herein have been specifically called-out because they represent decision points. These technologies represent a ‘rising tide’ for the power of data. But they are contrary to the notion of the old adage ‘a rising tide lifts all boats.’
To be clear – these technologies will only ‘lift the boats’ of firms who can harness them. By contrast, firms who cannot harness these trends will be significantly less competitive.
1. Open-Sourced Tools
There has been an explosion of Free/Libre Open-Sources Software (FLOSS) over the last decade. Top-notch open-sourced AI and Machine Learning libraries (e.g. TensorFlow, PyTorch) are freely available to public, along with the tools needed to support data infrastructure (e.g. Apache) and quickly develop data models (e.g. Dash, Streamlit). This explosion has been met with a democratization of learning platforms for computer and data science.
The open-sourced trend will likely continue to grow over the coming years. As a result, there are virtually no barriers-to-entry for firms who wish to harness data. This is good news for proactive firms, and disastrous news for firms who move slowly.
These free and open sourced tools collectively create a data ‘arms race’ of sorts. All companies now have access to the tools and expertise needed to gain competitive advantages through use of data. Those who fail to do so will be left behind.
2. Infrastructure-as-a-Service
Similar to the point on open sourced tools, advances in infrastructure services have reduced barriers and radically transformed the capabilities and capacity available to small and medium businesses.
Not long ago, managing data infrastructure once required significant overhead costs. Companies had to rent or buy physical servers and meet complicated energy and cooling requirements to keep them running. Moreover, they had to constantly update the servers to keep up with software specifications. As a result, only fairly large companies had access to the infrastructure needed to support ‘big data’ requirements.
Now, all companies have the ability to rent massive infrastructure at a remarkably cheap price. Moreover, the features for this infrastructure (e.g. serverless compute) are improving at a dizzying pace. Companies who can take advantage of this trend will find themselves in a position of advantage over those who cannot.
3. Knowledge Graphs
Databases are organized collections of data stored in a computer system such that they can be quickly accessed and queried. Over the last two decades, there have been amazing advances in database technologies, specifically with graph databases and, in turn, knowledge graphs.
The most recent frontier in databases are Graph Databases. These databases are specifically designed to make relationships among data easy to identify and explore. In graph databases, the relationships among data is just as important as the data itself. Unsurprisingly, graph databases are being implemented by many social media platforms and emerging CRM providers. They use such databases to bolster knowledge graphs, among other things.
Graph databases are currently in a nascent stage. As a result, early explorers and adopters of this technology will discover meaningful knowledge faster than their competitors.
Core Capabilities
I use the term ‘Core Capabilities’ in this post to refer to capabilities that businesses will need to develop & strengthen to compete and win in the data arms race.
I consider a capability to be the answer to a simple questions - what can you do? We differentiate this from the notion of capacity, which we consider to the answer to the question – to what extend can you do it?
The capacity behind each of these capabilities is dependent on the needs of each specific business, and will likely change over time for each firm. But I am resolute in my belief that these capabilities are fundamental to gaining a competitive advantage through use of data and technology.
1. Data Support Teams
Data support teams are the most important capabilities to develop for businesses who want to win in the data arms race. These teams should be comprised of, at a minimum, Data Engineers, Analysts, and Programmers. For small businesses or teams who don’t have the resources to hire many full-time employees, data engineers are arguably the most crucial role. By setting up resilient infrastructure and user roles, data engineers can help a small firm set the conditions to leverage additional tech talent through the gig economy. This can significantly support the growth of a small firm who otherwise wouldn’t have access to such talent.
2. Rapid Onboarding
Businesses must also develop a capability for rapidly onboarding new teams and data to a set of common infrastructure. There are enumerable benefits to having all teams and partners on such a common infrastructure. But to realize this potential, firms must first develop the people, process, & tools necessary to onboard data and supporting personnel in a timely manner. Companies simply will not be able to keep up with the pace of technology if it takes multiple months for every onboarding effort.
3. Rapid Deployment
Similar to the comments on rapid onboarding, firms must develop the capability to rapidly code, test, release, deploy, and maintain tools. As firms continue to onboard data and new employees, they will realize a tremendous amount of opportunities for the development of tools specific for their needs (e.g. dashboards, workflow tools, etc.). In order to capitalize on these opportunities, however, firms must be able to deploy code quickly.
4. Leveraging IaaS
Advances in infrastructure ability and affordability will enable firms to be more and more aggressive in working with data. That is, of course, if the firm is positioned to take advantage of such advances. Going forward, companies will need to develop a capability for making best use of cheap, emerging, infrastructure features like serverless computing, and container management tools. Not only will this help them operate better, it will save them money.