Every year, we sit down for an internal strategy session to explore the upcoming trends in data technology. As this is our 10 year anniversary (hard to believe!), its been useful to reflect on the broad progression of the industry since we started in 2009.
Data and data technologies have reached the plateau of productivity - almost every organization relies on data to support and inform core business activities, and many now look at staffing and investment in data as a key part of long-term strategy.
And we see this going beyond just for-profit businesses; we’ve seen NGOs and governments adopting the same forward looking approach to data as the private sector. For example, we recently worked with a large national foundation that was spending a fifth of their total grant giving (hundreds of millions of dollars) on data and data infrastructure capacity grants.
We recently worked with a large national foundation that was spending a fifth of their total grant giving (hundreds of millions of dollars) on data and data infrastructure capacity grants.
So what does this mean for 2019? We see three broad trends that you should pay attention to.
Rise of Synthetic Data
Data is everywhere, but its not always useful. One of the most common frustrations we hear from clients is that their data isn’t detailed enough (we use the term ‘granular’) to inform the types of decisions they’d like to make. Whether its information about customers, or data about a specific neighborhood, or daily rather than quarterly updates, people are more aware than ever of the gaps in the availability of the data they want. The solution to this problem is synthetic data.
For a long time, the only way to address missing data was to just collect more: e.g. if you don’t have it, go get it. The problem is that that cost of data acquisition typically scales linearly with the amount of data you want to collect. The more data you need to source, the greater your cost. If you want to increase the amount of data you have by an order of magnitude, it often isn’t feasible to increase your data cost by the same amount.
In the last couple of years, we’ve seen a big shift in a new direction: synthetic data. What is synthetic data? Basically, its a fancy word for estimates. Though niche industries have relied on synthetic data for decades (think of weather forecasting), its only been in the last few years that the technology, algorithms and research validation have advanced to make synthetic data practical for many new categories of problems. Because the data is generated using fancy statistical techniques (hence, ‘synthetic’), the cost of data acquisition is basically flat.
Because the data is generated using fancy statistical techniques (hence, ‘synthetic’), the cost of data acquisition is basically flat.
For example, the health care companies are increasingly relying on synthetic data (see the 500 cities data set) to make decisions about where to invest money to address public health issues like opioid addiction. Sales and marketing teams are increasingly relying on synthetic data to better target advertisements at users.
There are some very big caveats to using synthetic data (how often is the 7 day forecast 100% correct?), however for many types of data gaps, synthetic data offers a compelling solution.
AI Becomes A Tech Commodity
Of all the tech trends in the last 10 years, AI certainly seems to be the one that has delivered on much of the hype. From typical use cases like AI powered chatbots and virtual assistants, to new technologies like FaceID and autonomous vehicles, AI has reached the ‘tech commodity’ phase.
In 2018, we saw a number of large providers roll out AI solutions as part of their cloud technology offerings; Amazon, Google and Microsoft all launched very similar AI products for voice/speech recognition, natural language processing, image recognition and categorization. For the first time, powerful AI tools and techniques are available with the same ease of use as spinning up a new EC2 server.
Amazon, Google and Microsoft all launched very similar AI products for voice/speech recognition, natural language processing, image recognition and categorization
In 2019, AI technology will become much more prevalent and integrated into products that don’t have an exclusive AI focus. We’re already seeing many websites integrating in AI powered support chatbots to improve user experience. Apple’s FaceID is a great example of an incredibly complex AI solution that most users would never guess has anything to do with AI.
In the same way that could computing created an explosion of new products by dramatically lowering the cost of development, we’ll likely see another similar explosion in new products and services that leverage AI to provide a unique user experience that so far has been limited to companies that can afford billion dollar R&D investments.
Businesses focus on data workflow automation
If 2018 felt like a roller-coaster ride, we’re certain 2019 will be even more crazy. From politics to macro-economic factors, it feels like anything could happen. Generally though, the odds of some sort of economic downturn are going up, and geo-political uncertainty doesn’t help build business confidence.
The trickle-down effect is that companies are looking for ways to protect against downside risk, and a great way to do that is to increase productivity; with so much uncertainty, businesses want to be able to ramp production up or down with lower fixed labor costs.
With so much uncertainty, businesses want to be able to ramp production up or down with lower fixed labor costs.
Because of this, we see overall IT investments in 2019 going towards automation, and for our customers that means automating data workflows. Gathering, cleaning and analyzing data for most organizations is a highly manual process - data analysts are easy to find, and provide a great way to make an initial investment without long-term commitment.
But data is now integral to most organizations, and rather than increase the fixed labor cost related to data, many are looking to automate their data workflows to free up labor costs.
This type of investment tends to have a very clear ROI which resonates even more when the broader business climate is so uncertain. Because of this, we expect to see a lot more businesses investing in workflow automation in 2019.
Making the most of 2019
Every organization is unique, as are the impacts and opportunities of each of these trends. We’re here to help - we provide technology and data strategy support to companies who want to explore how they can leverage these broad trends to achieve results. Contact us for more information.
