We’ve all heard the argument: if we can save staff 2 minutes a day looking for information on the intranet, we can multiply this out by the number of staff and the days in the year to get a huge productivity benefit. This can then be used to justify the intranet redevelopment, and to demonstrate the benefits of improving intranet usability.

The time saved can be multiplied by the average staff salary, to get dollar savings of millions of dollars:

Thus, a company with poor intranet usability would save $3 million per year if it improved its intranet usability to an average level. And a company with average intranet usability would save $2.4 million per year if it improved its intranet to the usability level found in the best 25%.

Going still further, we can then compare the savings between “bad” and “good” intranets, and even calculate economic benefits for the world as a whole:

If we improved all the intranets in the world to the usability level achieved by the best 25% in our study, the world economy would save $311,294,070,513 per year for the sixteen test tasks alone. Adding the likely savings from company-specific tasks leads us to an estimated $600 billion in total annual productivity improvements. This level of intranet productivity is a modest goal; we could realistically achieve it with an investment of about $31 billion per year in intranet usability.

Assuming further usability improvements — up to the very best level we found for each study task — could save the world economy $1.3 trillion per year when we include estimated improvements in company-specific tasks.

We hear this from intranet teams frequently, and are asked questions about it from our clients.

The problem is that these metrics are fatally flawed, in almost every respect. They are a bad basis for intranet teams to justify their existence or demonstrate their value, and are dangerous in ways that aren’t properly recognised. So as a service to intranet teams, I’d like see if I can demolish this once and for all.

To start with, here’s my “big five” problems with this metric:

1. We’re not measuring end-to-end task completion. This metric focuses solely on the time needed to find information on the intranet, not the task or activity itself. When establishing a new customer credit card, how much of that time is spent on the intranet, versus working in the front-line systems? Much more valid productivity gains can be argued from improving end-to-end performance, not just “finding stuff”.
2. Can we realise the value? In financial jargon, financial benefits must be “realised”, before they count. In other words, if we save $1.7 million, can we get this as cash or equivalent? In practice, there are only two real ways of concretely realising the benefits: increase the number of tasks that can be done by the same number of staff, or reduce headcount (sack people). As intranet teams, we’re rarely in a position to use either of these approaches (and maybe wouldn’t want to).
3. Time saving and productivity is complex. So we’ve saved each staff member 2 minutes a day, do they now spend all that time in productive ways? Or do they spend it reading the newspaper or chatting on Facebook? Some while back I read research on how different types of staff make use of time savings, and it highlighted big differences between blue collar, white collar and senior managers. Our calculations make a lot of assumptions and simplifications. (I now can’t find this research, can anyone help?)
4. We are multiplying a lot of assumptions. How valid is a figure when we have to make a dozen different assumptions, and then multiply them all together? The more we look at the calculations, the more the details squirm in our fingers. At their most extreme, the numbers obtained are simply wacky.
5. Many financial people won’t accept it. In many organisations, CFO’s and their staff simply won’t accept these types of metrics, for the intranet or any other system. Better check first!

And then to round it out, here’s another 20 problems worth considering:

1. It’s only about usability. Measuring time spent on the intranet is easy, but not very relevant. Why are we not demonstrating actual business benefits, rather than just improved findability?
2. How many staff are using the intranet? We comfortably multiply out the time saved by the number of staff in the organisation, but this seems somewhat optimistic. How many staff are actually using the intranet? In many cases, the problems with the intranet mean that few are using it, and so the nominal benefits will be small.
3. Increasing usage now costs us money. We’ve made the argument that reducing time spent on the intranet will save the organisation, yet simultaneously we are aiming to increase intranet adoption. By this argument, the greater they number of intranet users, the more money we lose. We can’t have it both ways.
4. Tell me again, are we trying to increase or decrease usage? One of the most common metrics of intranet success is usage: visits, pages and hits. On the other hand, we’re trying to demonstrate a reduction in time spent on the intranet, and the number of pages visited per task. Pick one.
5. The intranet is considered in isolation. The intranet sits alongside (and competes) with every other way of completing a task or getting an answer. This includes asking the person next to you, ringing someone up, or using a bit of paper. To make our calculations valid, we have to assume that the intranet is the only source of information, which may be the case in some situations, but certainly not universally.
6. We don’t consider experience. Experienced staff just know how things work, as well as the answers to common questions and issues. They don’t need to look up the intranet. So reliance on the intranet will naturally change over the lifetime of a staff member, but this is not factored in.
7. What about learnability? No matter how bad an intranet is, given enough time, staff will often learn how to use what they need (or will give up entirely). This is not considered in the comparative usability studies.
8. How often are these tasks done? Savings are generally calculated based on the before-and-after testing of a number of “common” tasks on the intranet, such as looking up a phone number, or applying for leave. How often does an individual staff member actually do these tasks? In the case of applying for leave, it may only be once or twice per year, certainly not daily.
9. Are the usability testing results valid? To determine time savings, quantitative usability testing is required, which obviously can’t be done at the same time as qualitative testing. Even when run carefully, it is widely recognised that usability testing is somewhat artificial, with user behaviour different to that in the real world (would we really spend 15 minutes looking for the leave form, or would we be on the phone asking for help within 2 minutes?).
10. Are we testing the right tasks? In order to be relevant across a wide range of staff, tests tend to focus on general intranet activities. These generic tasks often relate to the staff directory, HR, finance, news, etc. Most are “corporate services” activities that are not directly related to the day-to-day business of staff, making them much less valuable and relevant.
11. Can we generalise out to the whole intranet? We’ve tested a few “representative” tasks, and have found before-and-after time savings. But this is just for 6-12 tasks, can this be generalised out to time savings across the whole intranet? This is a big leap to make.
12. All staff are assumed to be the same. The information needed by someone in HR is not the same as someone in Finance, the call centre, or out in the sales team. That makes it impossible to apply a single set of results and metrics to the organisation as a whole.
13. What are we actually testing? A typical task may be to find out how to apply for leave. Should that be a quick or slow task? Do we want people to carefully read the policy (perhaps), or are we just measuring the time taken to find it? User intention varies so greatly between tasks that simply aiming to reduce the time taken is simplistic.
14. We oversell the value of these metrics. I’m completely fine with using before-and-after usability testing to show that improvements have been made, and to demonstrate progress. As indicative findings, they have great value. But many organisations go way beyond this, claiming millions of dollars of savings and 100% “performance improvements”. This an overstretch, and dangerous territory.
15. We are treating our managers as fools. The many numbers involved in justifying time savings smacks of “blinding with science”. It all seems very plausable and detailed, until you look a little closer. In taking this approach, we risk treating our managers and stakeholders as fools, selling them vague figures as concrete business cases. If they call our bluff, we’re in a lot of trouble.
16. At their most extreme, these approaches make us look like idiots. I’m sorry, improving intranet usability will save the world economy trillions of dollars? Even assuming this is said half in jest (and I’m not sure it is), this makes us look pretty foolish, and does nothing to build our reputation with business managers or other professionals.
17. Show me the real figures. A lot of what is published is hypothetical: “we could save $3 million a year in an organisation with 10,000 staff”. Theory is good, but show me the actual case studies with realities rather than assumptions. There are a few examples, but nowhere near enough to demonstrate that the theory can reliably be put into practice.
18. Negative business cases are dangerous. Once we start down the road of negative business cases based on money saved, we can get ourselves in a lot of trouble. What if the cost of all the intranet authors is measured, wouldn’t it be a cost saving to reduce the number of writers? How do we justify an expensive new project, if we’ve started talking about costs? Wouldn’t the quickest and easiest way to save money be to sack the intranet team and fully devolve intranet management? We always recommend intranet teams focus on positive business cases (increased capability or capacity) wherever possible.
19. We are setting ourselves up for failure. Using these types of metrics can rush us into quoting substantial productivity benefits, even as we are trying to fix the basics of the intranet. It’s better to wait until the intranet is delivering direct business benefits before claiming such successes.
20. It doesn’t give us a roadmap for future development. Once we improve usability, what then? The narrow focus on usability rather than functionality leaves intranet teams constantly tuning their existing sites, rather than looking for ways to deliver new business value.

Have I missed any?

Addressing the problems

Depending on the rigor of the approach taken, some of these issues can be addressed. Many of the individual concerns can be addressed one at a time, although we can still end up multiplying together a lot of assumptions.

In the real world, of course, nothing is black and white. There are going to be situations where these metrics are the right approach, particularly when they match the organisation’s culture or the expectations of senior managers. As part of broader business case, these metrics could be used in a restrained way.

There are also business environments where it is easier to use these metrics. In a call centre, for example, staff productivity is extremely closely monitored and managed. Here it should be straightforward to demonstrate improvements in call handling times, or call volume.

That being said, intranet teams should always be aware of the limits and dangers of these metrics, even when used in the best circumstances. Tread with care!

In the broadest sense, perhaps my biggest concern is that these metrics are holding back the intranet community, leading intranet teams down a blind alley of “nothing but usability” when there are so many other approaches to consider as well. (More on this below.)

(And yes, I’m aware that a defense has been published, but sorry, I’m just not convinced.)

Finding a better way

As I’ve discussed in a slidecast on the four purposes of an intranet, most intranets focus on content and communication as their primary goals. If this is all that intranets do, there can be no concrete metrics or ROI, and the best we can hope for is this flawed “time saving” measure.

The reality is that intranets simply aren’t important enough when all they do is provide corporate policies and news. In order to be able to demonstrate value, we need to find new uses for intranets, that are directly tied into core staff activities and tasks.

Intranet must also become a “place for doing things”, not just a “place for reading things”. It then becomes much easier to demonstrate real benefits, including quantifiable dollar savings.

Focus on demonstrating:

• improvements to end-to-end tasks
• direct solutions to business problems and business needs

At the end of the day, useful is more important than usable when it comes to intranets. So let’s aim higher than just making it a bit easier to find things.

I know I’m not the only one to mock the “minutes saved” approach, and I’d love to see a broader (and more visible) discussion about this. Not just on what not to do, but on what we should be doing instead…