The authors used a sample base of around 150 managers to try to establish what the reasons for intranet user acceptance might be, using some classic user acceptance models. The paper is exceptionally well written, and is one of a series of papers that these authors have published. Their main conclusions (as set out in the paper) are that the top management of the port industry must continue to commit to intranet implementation as its use has been proven to significantly affect job performance among the managers in general irregardless of their backgrounds.

Below is a short list of the KPIs that I put together for them on the basis that their home page functions very much as a digital newspaper. These KPIs are intended to provide an indicative view of quality in addition to drawing attention to quick wins and improvement opportunities. Importantly, as an intranet evolves into a more collaborative, digital workspace KPIs such as these would need to evolve quite dramatically.

In a previous article we explained How intranet teams should spend their time. This article outlines some metrics that teams can use to measure their own. These are not the same as the ways to measure the effectiveness of the intranet itself.

Intranet team responsibilities fall into three main areas:

• day-to-day maintenance
• projects and new initiatives
• managing relationships with staff and stakeholders

Day-to-day maintenance

Day-to-day maintenance of the intranet is core business for any intranet team. Most teams will have a variety of valid measures for this, including:

• timeliness of content updates
• number of complete support calls
• number of training sessions run for the publishers
• adherence to service level agreements
• number of content pages reviewed
• up time for the intranet

[CM Briefing 2010-03 written by Catherine Grenfell, read the full article]

What follows is an extract of 2 pages from the report, pages 96 for adoption indicators and page 102 for core business indicators. Note that the number (n =) at the bottom left of each chart shows how many participants out of the 283 organizations are using this type of indicator.

This leads to the requirement for an intranet business case, ideally with clear measures that quantify the value of the intranet and the benefits it delivers.

In tough economic times, a very bright spotlight is shone on the intranet and the resources it requires. All projects will struggle to proceed without demonstrating how they tangibly improve the bottom line.

This intensifies discussions around intranet measures and metrics. It also produces a new round of intranet business cases being put forward to senior management.

In practice, creating an intranet business case can be hard. In all too many cases, intranet teams put forward their existing plans, and then try to justify them after the fact.

The difficulty is that many of the ‘typical’ intranet improvements are hard to quantify. What is the real dollar value of improving intranet navigation, or cleaning up content?

The arguments than can be put forward to justify these projects tend to be weak, and what is needed is a fresh look at how to plan intranet improvements.

This article will put forward the case that intranet teams can plan from the outset to demonstrate success. By targeting and packaging intranet improvements in the right way, identifying concrete benefits and metrics becomes easy and effective.

[Read the full article]

Studies done by different organizations, in different parts of the world, with different populations and very different survey strategies are finding similar results.

I wrote about how JMC and Nielsen Norman found the same figure when looking at intranet resources: Size of intranet teams: Nielsen Norman and JMC agree. It has happened again with Prescient Digital Media and JMC.

I think we overemphasize metrics when it comes to usability. They often introduce problems in terms of the validity and reliability of the data. First, ask whether a metric is a valid measure of usability. For example, the amount of time it takes to perform a task is often a usability goal—as in your example Six out of ten novice users shall perform task X in Y minutes. What if all of the participants took more than Y minutes to complete the task, but everyone said, ‘This is the best application for doing this task I have ever experienced. I’m recommending this product to all my friends.’ Does the product fail to meet the requirements regarding that task? What if all participants perform the task in less than Y minutes, but everyone says, ‘That was a horrible experience. I would never buy this product.’ Does the product satisfy requirements?

Let’s say that Organization A has 30,000 employees. Let’s say each employee does a particular task on average 50 times a year. Let’s say it takes 10 minutes longer than it should because it’s badly managed. This is costing the organization 15 million minutes a year in lost productivity. That’s 250,000 hours. Or 33,333 days. Or 150 person years. It’s significant.

Now, I wrote a piece titled 25 reasons why saving time on your intranet is a bad metric, so I’m obviously not a fan. I’ve also commented on Gerry’s piece, in an effort to shift the discussion to more productive topics.

Gerry and I agree on so much, it’s nice to have a good debate online for a change. Keeps the internet interesting.

Where do you stand on this perennial debate?

If you are running a knowledge management programme then you know that you do good work. Whether it is involves communities of practice, learning from projects and operational activities, or documenting organisation knowledge in a myriad of ways (from traditional databases to blogs and wikis), you know what you do is important for your organisation.
However simply knowing this is not enough. At some point, you will have to justify what you do to those above you. If you don’t, you may not get to do it any longer. Often these justifications have to take a financial form.

We’ve been talking about this for a while now. I recently argued that “minutes saved” on the intranet is meaningless, unless it’s backed up by real numbers, and not just theory. 40% of the scoring for the Intranet Innovation Awards comes from business value, with only 30% relating to novelty.

One of the most concrete ways of demonstrating improvements is to show before-and-after figures. Now, we’re reasonably good at measuring things after the project, such as via usability testing of new designs. How often do we measure the intranet before we make changes?

Without the before figures, it is very hard to quantify the concrete benefits. While this is “science 101″, the pressures of the typical workplace makes it difficulty to find time, and to justify, conducting research before the project actually starts.

Maybe if our senior managers don’t expect these kinds of metrics, we can save the time and effort involved in measuring both before and after. I would still argue, however, that we will be super-stars if we really can quote figures that really make an impact.

Once we demonstrate these types of returns once, getting resources will be dramatically easier. With greater management confidence in our performance, perhaps we won’t have to measure to this depth again, at least not for a little while.

So how can we do these types of measurements in practice? I’ve listed a few ideas…

Some lightweight approaches:

• Measure usage levels before the change, putting in a stats package if required.
• Conduct a “user satisfaction survey” before any major change, and then repeat 6 or 12 months after.
• Use the Intranet Review Toolkit to show improvements to core intranet heuristics.
• Collect anecdotes and examples of staff frustrations with the current solution, and then demonstrate increased satisfaction after the improvements.
• Assemble a before-and-after “portfolio”, thereby ensuring that the old designs are not forgotten, and the new designs not taken for granted.
• Record videos of users struggling with the current system, as part of lightweight usability testing.

More robust approaches:

• Measure end-to-end task completion, before and after the improvement (“it took 3 people a week of elapsed time to process a loan application, now it can be done in two hours”).
• Conduct quantitative before-and-after usability testing, focusing on key tasks that operational staff use the intranet for (and not just generic tasks).
• Find ways of saving actual money (although this is often extremely hard).
• Demonstrate improvements to key business metrics that are already being measured (improved product sales, increased customer satisfaction, reduced helpdesk calls, reduced call handling time in a call centre).

Also see our earlier article Metrics for KM and CM for more ideas on intranet metrics and ROI.

Any other lightweight or robust approaches you’d recommend?

I met up with some intranet managers the other day and there was a lot of talk of metrics. How many page views, how many users, who had clicked what and so on.

The conversation swiftly turned to which software each of us was using. However, I am sure that this was missing the point of metrics.

(British Airways won a Gold Award and Commended in this year’s Intranet Innovation Awards, so Allen’s blog is definitely worth following!)

The time saved can be multiplied by the average staff salary, to get dollar savings of millions of dollars:

Thus, a company with poor intranet usability would save $3 million per year if it improved its intranet usability to an average level. And a company with average intranet usability would save $2.4 million per year if it improved its intranet to the usability level found in the best 25%.

Going still further, we can then compare the savings between “bad” and “good” intranets, and even calculate economic benefits for the world as a whole:

If we improved all the intranets in the world to the usability level achieved by the best 25% in our study, the world economy would save $311,294,070,513 per year for the sixteen test tasks alone. Adding the likely savings from company-specific tasks leads us to an estimated $600 billion in total annual productivity improvements. This level of intranet productivity is a modest goal; we could realistically achieve it with an investment of about $31 billion per year in intranet usability.

Assuming further usability improvements — up to the very best level we found for each study task — could save the world economy $1.3 trillion per year when we include estimated improvements in company-specific tasks.

We hear this from intranet teams frequently, and are asked questions about it from our clients.

The problem is that these metrics are fatally flawed, in almost every respect. They are a bad basis for intranet teams to justify their existence or demonstrate their value, and are dangerous in ways that aren’t properly recognised. So as a service to intranet teams, I’d like see if I can demolish this once and for all.

To start with, here’s my “big five” problems with this metric:

1. We’re not measuring end-to-end task completion. This metric focuses solely on the time needed to find information on the intranet, not the task or activity itself. When establishing a new customer credit card, how much of that time is spent on the intranet, versus working in the front-line systems? Much more valid productivity gains can be argued from improving end-to-end performance, not just “finding stuff”.
2. Can we realise the value? In financial jargon, financial benefits must be “realised”, before they count. In other words, if we save $1.7 million, can we get this as cash or equivalent? In practice, there are only two real ways of concretely realising the benefits: increase the number of tasks that can be done by the same number of staff, or reduce headcount (sack people). As intranet teams, we’re rarely in a position to use either of these approaches (and maybe wouldn’t want to).
3. Time saving and productivity is complex. So we’ve saved each staff member 2 minutes a day, do they now spend all that time in productive ways? Or do they spend it reading the newspaper or chatting on Facebook? Some while back I read research on how different types of staff make use of time savings, and it highlighted big differences between blue collar, white collar and senior managers. Our calculations make a lot of assumptions and simplifications. (I now can’t find this research, can anyone help?)
4. We are multiplying a lot of assumptions. How valid is a figure when we have to make a dozen different assumptions, and then multiply them all together? The more we look at the calculations, the more the details squirm in our fingers. At their most extreme, the numbers obtained are simply wacky.
5. Many financial people won’t accept it. In many organisations, CFO’s and their staff simply won’t accept these types of metrics, for the intranet or any other system. Better check first!

And then to round it out, here’s another 20 problems worth considering:

1. It’s only about usability. Measuring time spent on the intranet is easy, but not very relevant. Why are we not demonstrating actual business benefits, rather than just improved findability?
2. How many staff are using the intranet? We comfortably multiply out the time saved by the number of staff in the organisation, but this seems somewhat optimistic. How many staff are actually using the intranet? In many cases, the problems with the intranet mean that few are using it, and so the nominal benefits will be small.
3. Increasing usage now costs us money. We’ve made the argument that reducing time spent on the intranet will save the organisation, yet simultaneously we are aiming to increase intranet adoption. By this argument, the greater they number of intranet users, the more money we lose. We can’t have it both ways.
4. Tell me again, are we trying to increase or decrease usage? One of the most common metrics of intranet success is usage: visits, pages and hits. On the other hand, we’re trying to demonstrate a reduction in time spent on the intranet, and the number of pages visited per task. Pick one.
5. The intranet is considered in isolation. The intranet sits alongside (and competes) with every other way of completing a task or getting an answer. This includes asking the person next to you, ringing someone up, or using a bit of paper. To make our calculations valid, we have to assume that the intranet is the only source of information, which may be the case in some situations, but certainly not universally.
6. We don’t consider experience. Experienced staff just know how things work, as well as the answers to common questions and issues. They don’t need to look up the intranet. So reliance on the intranet will naturally change over the lifetime of a staff member, but this is not factored in.
7. What about learnability? No matter how bad an intranet is, given enough time, staff will often learn how to use what they need (or will give up entirely). This is not considered in the comparative usability studies.
8. How often are these tasks done? Savings are generally calculated based on the before-and-after testing of a number of “common” tasks on the intranet, such as looking up a phone number, or applying for leave. How often does an individual staff member actually do these tasks? In the case of applying for leave, it may only be once or twice per year, certainly not daily.
9. Are the usability testing results valid? To determine time savings, quantitative usability testing is required, which obviously can’t be done at the same time as qualitative testing. Even when run carefully, it is widely recognised that usability testing is somewhat artificial, with user behaviour different to that in the real world (would we really spend 15 minutes looking for the leave form, or would we be on the phone asking for help within 2 minutes?).
10. Are we testing the right tasks? In order to be relevant across a wide range of staff, tests tend to focus on general intranet activities. These generic tasks often relate to the staff directory, HR, finance, news, etc. Most are “corporate services” activities that are not directly related to the day-to-day business of staff, making them much less valuable and relevant.
11. Can we generalise out to the whole intranet? We’ve tested a few “representative” tasks, and have found before-and-after time savings. But this is just for 6-12 tasks, can this be generalised out to time savings across the whole intranet? This is a big leap to make.
12. All staff are assumed to be the same. The information needed by someone in HR is not the same as someone in Finance, the call centre, or out in the sales team. That makes it impossible to apply a single set of results and metrics to the organisation as a whole.
13. What are we actually testing? A typical task may be to find out how to apply for leave. Should that be a quick or slow task? Do we want people to carefully read the policy (perhaps), or are we just measuring the time taken to find it? User intention varies so greatly between tasks that simply aiming to reduce the time taken is simplistic.
14. We oversell the value of these metrics. I’m completely fine with using before-and-after usability testing to show that improvements have been made, and to demonstrate progress. As indicative findings, they have great value. But many organisations go way beyond this, claiming millions of dollars of savings and 100% “performance improvements”. This an overstretch, and dangerous territory.
15. We are treating our managers as fools. The many numbers involved in justifying time savings smacks of “blinding with science”. It all seems very plausable and detailed, until you look a little closer. In taking this approach, we risk treating our managers and stakeholders as fools, selling them vague figures as concrete business cases. If they call our bluff, we’re in a lot of trouble.
16. At their most extreme, these approaches make us look like idiots. I’m sorry, improving intranet usability will save the world economy trillions of dollars? Even assuming this is said half in jest (and I’m not sure it is), this makes us look pretty foolish, and does nothing to build our reputation with business managers or other professionals.

17. Show me the real figures. A lot of what is published is hypothetical: “we could save $3 million a year in an organisation with 10,000 staff”. Theory is good, but show me the actual case studies with realities rather than assumptions. There are a few examples, but nowhere near enough to demonstrate that the theory can reliably be put into practice.
18. Negative business cases are dangerous. Once we start down the road of negative business cases based on money saved, we can get ourselves in a lot of trouble. What if the cost of all the intranet authors is measured, wouldn’t it be a cost saving to reduce the number of writers? How do we justify an expensive new project, if we’ve started talking about costs? Wouldn’t the quickest and easiest way to save money be to sack the intranet team and fully devolve intranet management? We always recommend intranet teams focus on positive business cases (increased capability or capacity) wherever possible.
19. We are setting ourselves up for failure. Using these types of metrics can rush us into quoting substantial productivity benefits, even as we are trying to fix the basics of the intranet. It’s better to wait until the intranet is delivering direct business benefits before claiming such successes.
20. It doesn’t give us a roadmap for future development. Once we improve usability, what then? The narrow focus on usability rather than functionality leaves intranet teams constantly tuning their existing sites, rather than looking for ways to deliver new business value.

Have I missed any?

Addressing the problems

Depending on the rigor of the approach taken, some of these issues can be addressed. Many of the individual concerns can be addressed one at a time, although we can still end up multiplying together a lot of assumptions.

In the real world, of course, nothing is black and white. There are going to be situations where these metrics are the right approach, particularly when they match the organisation’s culture or the expectations of senior managers. As part of broader business case, these metrics could be used in a restrained way.

There are also business environments where it is easier to use these metrics. In a call centre, for example, staff productivity is extremely closely monitored and managed. Here it should be straightforward to demonstrate improvements in call handling times, or call volume.

That being said, intranet teams should always be aware of the limits and dangers of these metrics, even when used in the best circumstances. Tread with care!

In the broadest sense, perhaps my biggest concern is that these metrics are holding back the intranet community, leading intranet teams down a blind alley of “nothing but usability” when there are so many other approaches to consider as well. (More on this below.)

(And yes, I’m aware that a defense has been published, but sorry, I’m just not convinced.)

Finding a better way

As I’ve discussed in a slidecast on the four purposes of an intranet, most intranets focus on content and communication as their primary goals. If this is all that intranets do, there can be no concrete metrics or ROI, and the best we can hope for is this flawed “time saving” measure.

The reality is that intranets simply aren’t important enough when all they do is provide corporate policies and news. In order to be able to demonstrate value, we need to find new uses for intranets, that are directly tied into core staff activities and tasks.

Intranet must also become a “place for doing things”, not just a “place for reading things”. It then becomes much easier to demonstrate real benefits, including quantifiable dollar savings.

Focus on demonstrating:

• improvements to end-to-end tasks
• direct solutions to business problems and business needs

At the end of the day, useful is more important than usable when it comes to intranets. So let’s aim higher than just making it a bit easier to find things.

I know I’m not the only one to mock the “minutes saved” approach, and I’d love to see a broader (and more visible) discussion about this. Not just on what not to do, but on what we should be doing instead…

Your thoughts, comments?