Column Two

Intranets: measuring before and after

Written by James Robertson, published October 30, 2008

Categorised under: Intranets, Metrics & ROI

Intranets aren’t owed a living, and we must demonstrate their value if we’re to get the resources and support we need. As the technology landscape changes, this will become more important, not less.

We’ve been talking about this for a while now. I recently argued that “minutes saved” on the intranet is meaningless, unless it’s backed up by real numbers, and not just theory. 40% of the scoring for the Intranet Innovation Awards comes from business value, with only 30% relating to novelty.

One of the most concrete ways of demonstrating improvements is to show before-and-after figures. Now, we’re reasonably good at measuring things after the project, such as via usability testing of new designs. How often do we measure the intranet before we make changes?

Without the before figures, it is very hard to quantify the concrete benefits. While this is “science 101″, the pressures of the typical workplace makes it difficulty to find time, and to justify, conducting research before the project actually starts.

Maybe if our senior managers don’t expect these kinds of metrics, we can save the time and effort involved in measuring both before and after. I would still argue, however, that we will be super-stars if we really can quote figures that really make an impact.

Once we demonstrate these types of returns once, getting resources will be dramatically easier. With greater management confidence in our performance, perhaps we won’t have to measure to this depth again, at least not for a little while.

So how can we do these types of measurements in practice? I’ve listed a few ideas…

Some lightweight approaches:

• Measure usage levels before the change, putting in a stats package if required.
• Conduct a “user satisfaction survey” before any major change, and then repeat 6 or 12 months after.
• Use the Intranet Review Toolkit to show improvements to core intranet heuristics.
• Collect anecdotes and examples of staff frustrations with the current solution, and then demonstrate increased satisfaction after the improvements.
• Assemble a before-and-after “portfolio”, thereby ensuring that the old designs are not forgotten, and the new designs not taken for granted.
• Record videos of users struggling with the current system, as part of lightweight usability testing.

More robust approaches:

• Measure end-to-end task completion, before and after the improvement (“it took 3 people a week of elapsed time to process a loan application, now it can be done in two hours”).
• Conduct quantitative before-and-after usability testing, focusing on key tasks that operational staff use the intranet for (and not just generic tasks).
• Find ways of saving actual money (although this is often extremely hard).
• Demonstrate improvements to key business metrics that are already being measured (improved product sales, increased customer satisfaction, reduced helpdesk calls, reduced call handling time in a call centre).

Also see our earlier article Metrics for KM and CM for more ideas on intranet metrics and ROI.

Any other lightweight or robust approaches you’d recommend?

Tags: Intranets, metrics