KPI Battlegrounds...

What happens when business science challenges gut instinct in success metrics?

Key Performance Indicators… KPIs. The magic telltale signs of success used to govern and direct the machinations of any organization. The last several decades has seen an explosion of new KPIs driven by more granular data available and the powerful model building courtesy of data science.

KPIs used over the centuries, profit, number of customers, sales revenue have given way to Net Promoter Scores, percentage of repeat site visits and Customer Acquisition Costs. An old guard forced to yield.

The issue isn’t that something like profit isn’t worthwhile but that it doesn’t relay actionable intelligence. Profit increasing or decreasing doesn’t tell you the how, only the what. Business leaders are driving to find objective identifiers of specific levers that allow them to make changes within the business leading to a specific outcome. These new KPIs provide information that 200 years ago would be unheard of.

While the history of KPIs and their use to enact change goes back almost to the third century, it is only within the last few decades that data collection and processing has reached a maturity where businesses can objectively tie KPIs to levers of change within an organization.

Prior to this, KPIs and thresholds of success were the guarded secrets of business Jedi; managers and directors with years of experience in interpreting the tea leaves of KPIs, able to enact change through shear instinct.

But is this foundational shift from subjective interpretation to objective identification causing a rift between the experienced business leader and the new art of business analytics and data science?

Strategic Measurements to Nebulous Questions

The idea of a Key Performance Indicator is fairly simplistic. Its composed of two parts. The question… How do we measure success. Is success measured by customer base, how many customers are we serving? Maybe its aimed at something like profit, is our profit up?

The second part, sadly oft forgotten, context. What is the context involved in our question. Is our profit up compared to last month… last week…. last year? Has our profit increased as compared to our customer base?

Certainly worthwhile questions but without an ability to interpret the results these KPIs become just talking points. Imagine a business owner, looking at the company’s customer base and seeing it is up (That’s good) and then looking at the company’s profit and that is also up (seems good too) but then looking at the profit per customer and that is down (that’s bad). In this scenario the KPI doesn’t provide information on the lever. What changed causing the average customer to generate less profit?

More mature KPIs suffer from this strategic limitation. Important to the health of the business but missing the connection to the actionable lever. In comes the industry expert and together with strategic KPIs is able to rejigger “this” and manipulate “that“. Now we wait to see if the success metric goes up or down.

New KPIs on the Block

Technology, a truly great equalizer. A powerful tool that when applied to virtually any domain can make processes faster, speed to customer cheaper, product specs better. This is certainly true for business analytics and data science. We collect more data, faster and with depper grainularity than ever before. A search for a down jacket on my computer almost instantly modifies the advertisements in my Instagram feed. Intention, engagement, desire… all measured and accounted for.

Organizations now have the ability to assess success at the individual action level and tie it to a tactical decision, real life levers.

A/B testing, an absolute gem of a process can be planned, executed, quantified and rated all with a few mouse clicks and key strokes. There is no guessing as to whether marketing page A or marketing page B has a stronger affect on the customer. We gauge success down to the placement of buttons and slight changes to word composition.

How granular? I recently saw an article that discussed metrics of a branding campaign tied to the usage of the term “writer” vs the term “author”. KPIs identified at the tactical level and tied back specifically to the use of a single word. Complete awesomeness. Robust tactical KPIs are here to stay. Magic to those who operated 150 years ago.

But what about all those gut instincts?

Round 1…

What happens when Gut Instinct disagrees with Business Science over a KPI?

Imagine the following scenario. A business analyst discusses a question that a team is trying to answer. The group establishes success criteria (a KPI), collects significant amounts of data, and crunches the numbers. The BA assess the question and in conjunction with the analysis confidently identifies a value for the KPI and a change lever. Change can now be affectuated.

Except change does not get affectuated. Nothing changes, no levers are pulled. Domain experts “know” that lever doesn’t do anything. After all, domain experts know best. Their “right” because that’s the way it has always been done. The data must be wrong. Or maybe it is the process the analyst used; something is wrong because the expert can’t be…

These types of scenarios play out more than most people think. The move to a data driven culture displaces much of the “gut instinct” culture prevalent in organizations operating on guesswork and hunches. Leaders and managers who are accustomed to being unchallenged when making decisions are easily put off by decision making with strong data driven foundations. KPIs get turned on their head and that is unsettling.

So is there a middle ground?

The Pathway Forward

Data backed decision making paired with tactical KPIs are powerful arrows in the organizational quiver. However, these are just tools. No magic or wizardry involved. The usefulness of the tool diminishes when leadership clings to deferring to gut instinct.

The dreaded, “we’ve always done it this way”.

This isn’t to say that gut instincts are always bad or shouldn’t have a seat at the decision making table. Leadership and experience play a vital role in driving organizational success.

However, if business science is going to be utilized to its fullest potential, organizations that cling to gut instincts need to take a hard look at decision making. Those used to managing by gut instinct need to find common ground with business science. If analytical work is only used to fill time then it is pointless to invest in something that only serves a contrarian purpose, a punching bag for domain experts to practice some jabs against.

Start small. Find little tactical wins that involve minor KPIs and managers accustomed to doing something the same way. Educate on the benefits of balancing the discussion between the two viewpoints and not on pairing the viewpoints against each other.

It takes skill and art to ensure business science has informed domain knowledge and asks the right questions. Analytical teams are empowered by leaders and domain experts who have knowledge about the nuances of process and the context of KPIs. The groups are stronger together. Domain experts have an opportunity to deepen their knowledge even more and back it with empirical evidence.

---

Baltimore-Pittsburgh game postponed…Denver-New Orleans game on as usual?

The last 9 months for sports, including the NFL, has not been good. A recent AP article had the most recent case of what I see more and more these days, stupid statements unsupported within the same article.

Marc Ganis’ statement: “I know there have been lots of, well, ‘NFL ratings are down.’ Not really. Election years, they’ve been going down for the last three election cycles, not just the last one,” Ganis said. “The NFL ratings have actually been quite good.”

From the same article. NFL television ratings see 6% decline. Thursday night and sunday night down 16%.

Turns out, Yes really. We don’t even have to guess at the amount.

So why disenfranchise a team and force them to play with all 4 of their quarterbacks sidelined by the league? Because the NFL felt the situation was contained.

Mmmmmm, the feeeeeelssssss.

Interesting medical benchmark. Imagine if the entire Kansas City Chiefs defense goes out for dinner and comes into contact with a positive case at the restaraunt. If they quarantine, technically I guess the virus is contained and KC would have to play… right?

When your ratings are already down, why negatively impact a team, leading to more potential abandonment from that team’s fans?

---

Unless you’re in the industry..

Software development industry quote: “After you finish the first 90% of a software project, you have to finish the other 90%.”

Why is that so crazy and yet so true?

---

Hey all, thanks for reading and being a subscriber to the newsletter. This first full issue was some work but invigorating work! Help me get better. Ideas for topics, notes on questions… please send them my way!

---

Looking for more, excited to support this endeavor? Feel free to follow me on Twitter. As Johnny Rose would say, Tweeter me @JWilcoxData.

Thanks to Hasan Almasi & Unsplash for the cool battle photo!