Leveraging the Value of Big Data with Automated Decision Making

It is a widely accepted fact that we are living in the era of Big Data. Many traditional companies are looking for ways to improve their business through the virtues of Big Data and Data Science. While matured startups born in this era like Facebook and Twitter seem to naturally exploit the value of their data, many traditional companies struggle to find new ways of utilizing their data to leverage its value for their classical businesses.

In this post I elaborate on the specific domain of decision making where Big Data and Data Science can help to improve the efficiency of conventional businesses. Proving the benefits of Big Data in a lighthouse project is of utmost importance in long-established companies with regard to overcoming initial resistance in the digital transformation of business processes. We will see that the automatization of operational decisions, i.e. routine decisions related to the day-to-day running of the business, are especially suitable candidates for lighthouse projects to prove the value of Big Data in a company.

The notion of automating data-driven decisions with the help of Data Science is often denoted with the term Prescriptive Analytics, which can be regarded as the conclusive step after Predictive Analytics. In other words, the predictions generated with the help of Predictive Analytics are used to optimize a predefined metric under consideration of side conditions, strategic direction, business processes etc. to derive excellent business decisions. The predictive analytics diagram from Gartner illustrates the business value compared to the difficulty of different analytical approaches.

In many businesses repetitive operational decisions consume lots of working time. For instance pricing of articles and services, replenishment of stores or stocks, demand forecasts and customer services involve operational decisions which are often conducted in a manual process supported by traditional, rule based decision support systems. Automating these decisions with the help of data-driven decision systems has several benefits:

1. Labor costs are reduced and scarce expertise can be leveraged for non-routine, exceptional decisions. Less routine decisions means having more time for decisions in extraordinary circumstances as well as decisions that are of a more tactical or strategical nature. This encompasses also decisions in situations where data is lacking as well as decisions about creative and visionary solutions. For instance, no machine learning algorithm could have ever predicted the success of the first iPhone since it was something completely new and its success was a consequence of not only that but also many other soft factors.

2. The quality of decisions is improved given that all information sources used in the manual decision process are available as machine readable data. Modern machine learning algorithm are able to quickly analyze huge amounts of data that a human being could never even read in a lifetime. This plethora of data allows the inference of patterns that lead to fast, consistent, high quality decisions which are resistant to the long list of cognitive biases or decision fatigue.

3. Prescriptive Analytics allows to scale the number of decisions. Too often in traditional businesses, decisions are made by not actually taking a decision which consequently leads to idleness and thus unexploited potential. Being able to scale the number of decisions, enables this untapped potential to be fully realized and can also generate new services. Imagine for instance that one marketing tool of a company is to give special offerings and product recommendations based on different market segments, not single customers. Being able to scale the number of decisions due to automation would allow special offerings and recommendations for individual customers, just like Amazon’s recommendation system.

To justify our statement that the automation of routine decisions with Prescriptive Analytics is exceptionally well suited as a pioneer project in a traditional company, it is necessary to elaborate on certain characteristics that many operational decisions hold.

While a single operational decision, e.g. a small change in the price of a single article, may have an insignificantly small but direct impact on the revenue of the whole business, the sum of all decisions quite often has great economic impact. This is due to the fact that the frequency of operational decisions is often huge, meaning that a small overall improvement in decision quality is highly profitable. Obviously, candidates for a Prescriptive Analytics project should have exactly these properties of high and direct economic impact. The ability to measure such an impact requires that a performance metric or key performance indicator (KPI) is already established. This is another important prerequisite for a successful Prescriptive Analytics project.

Since operational decisions are often related to the core of the business even in traditional companies huge amounts of data are already collected and available. Often, routine decisions that are taken by analyzing spreadsheets and personal experience are based on data with high predictive power. A quote, often attributed to Mark Twain, says that “history doesn’t repeat itself, but it does rhyme” which captures the essence of what automated decision making is about. Having lots of data about past events allows us to find patterns and relationships which can predict future events to some extent. The goal is to develop a model that describes what happened in the past without being bound to the past and thus allowing to apply the model to the future. In just the same way as our brain learns from experiences and infers future outcomes in similar situations.

Consequently, the high frequency of routine decision with a direct economic impact combined with an abundance of data and a metric to measure performance are favorable characteristic of a business process that can be successfully automated. In order to quantify the added value of Prescriptive Analytics an estimation of the gain in decision quality and its impact on revenue is needed with the help of the predefined metric or KPI. For this complex estimation it is recommended for traditional companies to have an experienced partner alongside and optionally a proof of concept to evaluate the predictive power of the data and the business case as a whole.

We should not ignore the fact that automation also includes costs encompassing the maintenance, licence & support etc. of an established automated decision system. The automation costs depend mainly on the order of magnitude of performed decisions as well as the time frame. For instance the IT setup of a decision system for one million decisions per day will be much smaller than the setup for one billion that need to be determined in real- time. The initial costs for the implementation of an automated decision system varies largely depending on the domain of application, the necessary changes in the business processes and other factors. An estimation of these costs is needed to determine the time-to-value. Since the added value of an automated decision system quite often heavily surpasses automation costs by at least one order of magnitude time-to-value is often low.

We conclude that the scaling in the number of decisions and the improved effectiveness of the decisions are the main drivers of the added value in automated decision making. Subsequently, operational decisions that are ubiquitous and directly influence the business value are well suited for a Predictive Analytics light-house project.

Following mnemonic recaps the main qualities of a successful Prescriptive Analytics project. It consists of the following questions that should be answered positively:

• Are decisions taken Frequently?
• Does the business process allow the Automation of decisions?
• Are Metrics defined to determine the quality of a decision?
• Do decisions have a direct Economic impact?
• Is enough and suitable Data available to base decisions on?

Prescriptive Analytics projects with these properties are very likely to become famed in your company. The successful implementation of a lighthouse project in the business process generates momentum for new projects. This drives the digital transformation of a classical business in an iterative way.

This article was originally posted on the “CSC 21st Century IT blog”, which no longer exists.