Measuring Success – Part 2/3: The Unmeasurable Intangibles

In the previous post, I wrote about the concept of measurement. We saw how measurement is not about capturing real-world observations as a single value but rather about reducing our uncertainty about the range of possible values.

While that is a critical component in understanding how to interpret measurements, one of the biggest impediments for getting value from measurements and metrics is that we often think the things we care about are not measurable.

How does one measure things like “public image,” “innovation,” or “IT security”?

These concepts are often fundamental drivers of value for our organizations. Yet, because we deem them hard to measure, they become all too easy to ignore in favour of vanity metrics that are easy to report.

Rest assured, these “fuzzy” concepts are not unmeasurable, and the process is simpler than you think.

Why – The Critical Context

Before we dive in and see how to measure the “intangibles,” I want to gently remind you not to put the cart before the horse:

Any measurement you plan on doing should be in support of a decision you need to make. Without a dilemma, there are no data-driven decisions, just data-driven theatre.

If you don’t know what action to take when presented with measurement results, you might as well skip the measurements and save some time.

That’s why the first thing you should ask yourself (or others) is why do you want to measure this?

Asking this simple question upfront provides the context needed to frame the problem, which can aid in finding other relevant information to support the decision in question and give clues on how we might measure them.

Asking why can also prevent wasted effort, as the reason behind the measurement can be based on false assumptions or bad strategy, opening doors to more useful discussion.

What – From Concepts To Observations

Why might something be perceived as being “impossible to measure”?

In my experience, it’s usually because we simply have not defined it.

Language is ambiguous. Everyone interprets words differently.

When you use a word like “quality,” even when everyone nods vigorously, I will wager that no 2 people in the room would describe “quality” the same way when asked to elaborate.

That’s why we have to ask the clarifying question: “What do you mean by X?”

It’s not always easy to get to an answer. The idea that something is “intangible” is sometimes so ingrained in our minds that we really need to slow down and consider the basics:

1. If I care about X at all, I must have had some way of detecting that it exists.
2. If it’s detectable, then I should be able to detect more of X or less of X.
3. If I can detect more of X or less of X, then it must be measurable

It is often not the thing itself, but its consequences that you care about and can measure.

For example, how would you measure “employee empowerment?”

If I cloned your organization, and the copy had more “employee empowerment” what different things would you observe in the copy?

• There would be fewer approval steps for each decision
• Decisions would be made faster, decreasing time-to-value
• There would be more ideas coming from employees rather than management
• There would be a rise in job satisfaction, and thus a decrease in employee churn

What would “improving security” mean?

• Would attacks become less likely?
• Would we decrease the impact of attacks? By what measure? Surface Area? Dollars lost?

Notice how getting specific on what we mean gives us clues about what to measure.

Decomposition – The Art of Guesstimation

Sometimes the thing you’re trying to measure is not vague but is hard to observe directly. These things are often composed of smaller measurements that aggregate into a single measure.

Tech companies were notorious for asking “guesstimating” and “market sizing” questions during technical interviews to assess a candidate’s ability to break down tough problems (I don’t think this is a recommended interview question for software engineers anymore, so please don’t torture the poor candidates).

For example, they might have asked you: “What is the current size of the mobile phone market in Canada?”

After the initial panic wears off, you might start to think about this problem in terms variables that are easier to estimate:

# Phones Sold in Canada =
  Canada's Population ⨉
  % of the population that owns a phone ➗
  Average Lifespan of a Smartphone in Years

Although the value of the number of phones sold in Canada is hard to observe directly, breaking the problem down into its components allows us to get very close to the true value.

Conclusion

If you remember one thing from this post, let it be this:

If you think something is not measurable, you just gotta break it down.

Start with why.

Then ask what it is you’re really measuring.

When applicable, decompose the problem.

It really is that simple.

Still not convinced? Drop a line in the chat to discuss any measurement challenges you’ve been facing and we can try to solve them together!

Don’t forget to check out Part 1, where I uncover a shift in perspective (for most) about the fundamental concept of measurement.