Articles by Category: Metrics & ROI

CMS business case

Seth Gottlieb has written a post on creating a CMS business case. To quote:

In my opinion, the business case discussion should be around the content itself - not the technology used to manage it. This is a difficult conversation to have for a number of reasons. First of all, the human effort required to manage content (no matter what tools you have), while very costly, does not have a big spending event that triggers an ROI conversation. The expenditures from managing content (or the cost of not managing it) happens in drips and drops but it can really bleed a company. Secondly, there is a misperception that all content is good and worth managing.

Posted by jamesr at 10:52 AM | Permalink
Categories: Content management, Metrics & ROI

Panorama across the desert (IV)

Panorama across the desert (IV)

On the last day of my San Diego trip I drove up to Joshua Tree National Park. While the cloudy weather didn't help the photos, the landscape was truly extraordinary. Like nothing I've seen before...

Posted by jamesr at 08:26 PM
Categories: Metrics & ROI

Managers say the majority of information obtained for their work is useless

An Accenture study reports on staff's ability to find useful information. To quote:

Half (51 percent) of customer service managers -- more than managers in any other area -- are likely to encounter challenges when trying to obtain information about other parts of the company. In addition, 40 percent of customer service managers -- more than those in any other function -- said that the most difficult aspect of managing information for their job is going to numerous sources of information.

[Thanks to Jack Vinson.]

Posted by jamesr at 08:20 AM | Permalink
Categories: Information management, Intranets, Metrics & ROI

The return on investment (ROI) for personas

Usability News has posted an extract from The Persona Lifecycle: Keeping People in Mind throughout Product Design on ROI for personas. To quote:

One of the most common questions we are asked about personas is if they actually work. And if they do, how can you tell if they were worth the effort? Answering these questions is difficult. What counts as proof of the method is different for different people, products, and companies. There is no single case study or research study that proves their effectiveness rigorously. However, as you have seen throughout this book, there are hundreds of little examples of their effectiveness and value scattered about the industry. This is why this section includes so many stories from the field describing success stories from persona practitioners.

Posted by jamesr at 11:59 AM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

Measure twice, cut once: content management metrics

Brian Manning & Caleb Brown have written an article that discusses common content management metrics. To quote:

Often, when companies choose to implement a content management (CM) strategy they do so in an attempt to alleviate the operational inefficiencies associated with maintaining multiple content repositories. The documents in these repositories typically overlap each other, or fail to accurately represent true business processes. So it's not surprising to find that most businesses strive to minimize these effects and, when they think about measuring the impact of their CM, tend to favor operational efficiency. But the way User Experience Consultants, Brian Manning & Caleb Brown see it, productivity wins are mostly illusory. Businesses are only looking at half of the CM metrics picture.

[Thanks to Scott Abel.]

Posted by jamesr at 09:30 AM | Permalink
Categories: Content management, Metrics & ROI

Case Study: intranets, usability, and value

Jeff Veen presents a mock case-study exploring the redesign of an intranet, and the ROI it may deliver. To quote:

As Tracy sits down to write a project plan, she considers the problems the company is facing: Almost all the information employees are looking for exists somewhere, but nobody can find anything. She could try to solve this problem with a content management system, but will employees adopt the new tools? And will departments want to give up the control they have over the local servers they have? And how will she show measurable change if she does implement new systems? After all, she wants credit for saving HR all that money at the call center.

Posted by jamesr at 05:33 PM | Permalink
Categories: Intranets, Metrics & ROI, Usability & user-centered design

ROI is not a silver bullet

Scott Hirsch has written an article on the ROI of user experience. To quote:

For years now, the "ROI of User Experience" has been sought as a means to justify larger corporate investments in web design. Although ROI methodology can be a useful tool for prioritizing possible web development projects, by itself ROI is not the answer to building a stronger user experience design competency.

Scott then lists five key steps:

• Get an Understanding of Business Value
• Look for Quick Wins
• Determine the User Behaviors and Measure Baseline Data
• Measure Outcomes Post-launch
• Evangelize Success

[Thanks to InfoDesign.]

Posted by jamesr at 10:53 AM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

The tao of ROI

Sandra Winters has written an article on the issues surrounding ROI. To quote:

ROI analysis is an important part of your due diligence for new IT investment. How could you even think of participating in a new business venture or investing in another company without it? While ROI analysis is a must-do for any self-respecting management team, ROI is often misused and little understood.

[Thanks to the eGovernment Resource Centre.]

Posted by jamesr at 05:35 PM | Permalink
Categories: Metrics & ROI

Identifying the components of your ROI

Ann Rockley has written an article on determining CMS ROI. To quote:

Identifying Return on Investment (ROI) for your content management business case begins with a thorough analysis. This article reviews the information you need to gather to identify ROI for an effective business case for content management.

Posted by jamesr at 07:38 PM | Permalink
Categories: Content management, Metrics & ROI

The high cost of not finding information

Susan Feldman reports on IDC research into the high cost of not finding information. To quote:

Knowledge workers spend from 15% to 35% of their time searching for information. Searchers are successful in finding what they seek 50% of the time or less, according to both Web search engines and our own surveys. An IDC study in 2001 ("Quantifying Enterprise Search," IDC, May 2002) found that only 21% of respondents said they found the information they needed 85% to100% of the time (see Figure 1). 40% of corporate users reported that they can not find the information they need to do their jobs on their intranets.

[Thanks to elearningpost.]

Posted by jamesr at 10:19 AM | Permalink
Categories: Metrics & ROI

K-Log productivity: Time to find and availability

John Robb has written another weblog entry on K-log metrics (k-logs are "knowledge weblogs"). This one focuses on the time taken to find information in an organisation. To quote:

An oft quoted statistic is that "knowledge workers spend 35% of their productive time searching for information, while 40% of the corporate users report that they cannot find the information they need to do their jobs on their Intranets" (source: Working Council of CIOs). The Delphi group estimates that this costs the average 20,000 person organization $720 million a year ($120,000 all in cost per employee equates to $36,000 per employee spent searching).

Posted by jamesr at 03:00 PM | Permalink
Categories: Intranets, Metrics & ROI, Weblogs

Report Review: Nielsen/Norman Group's Usability Return on Investment

Peter Merholz and Scott Hirsch have written a review of the Nielsen/Norman Group's report on usability return on investment (ROI). To quote:

Although it does make a reasonable anecdotal case for investing in usability, the report methodology is so fundamentally flawed that any financial analyst worth her salt would immediately question its findings. Very simply, the authors do not make a strong business case for usability—a requirement for passing the muster with the accountants and senior managers who have ultimate accountability for profit and loss in a business.

Posted by jamesr at 11:05 AM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

ROI calculations: K-Logs vs. traditional Intranet Portals

John Robb has written a weblog entry that compares the ROI of enterprise portals and k-logs (k-logs are weblogs used inside an organisation). In it, he finds that k-logs provide similar benefits at a fraction of the cost.

What is also very interesting are the ROI figures he has obtained from PlumTree for their enterprise portal software ($1.2mil per year in admin costs!).

Posted by jamesr at 10:01 AM | Permalink
Categories: Intranets, Metrics & ROI, Weblogs

ROI of Usability: A Collection of Links

Rashmi Sinha has posted an excellent set of links to articles on the ROI of usability. To quote:

It is hard to directly estimate the economic impact of usability. However, it is important to do so, both at the macro and the micro level. Products with bad usability can have impact in terms of wasted time, erros, unfinished tasks, frustration, and even medical problems such as RSI. A concrete calculation of usability impact is difficult, but is often needed by usability advocates within companies, independent consultants etc. Below, I am collecting links to articles and cost-benefit calculators for estimating the economic impact of usability.

[Thanks to Katie Harter.]

Posted by jamesr at 07:06 PM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

New mailing list: Aifia-metrics

A new mailing list has been established: Aifia-metrics, with the goal of establishing common metrics for assessing information-rich sites. To quote:

This list is for discussions of evaluation methods and metrics for information-rich websites and non web-based applications. The goal of the project is described below. Please join in the conversation!

Project Goals:

Currently, there are no standard methods and materials available for evaluation of information architecture. Usability questionnaires such as WAMMI (Website Analysis and Measurement Inventory) or QUIS (Questionnaire for User Interaction Satisfaction) are not suited to information-rich domains. Moreover, neither of the both can be used only after payment of a fee. Our goal is to propose some standard methods and questionnaires suitable for the use by Information Architects.

I think this is a great initiative, and I've just joined. (Metrics is something of a passion for me, as evidenced by my article on the topic...

Posted by jamesr at 03:15 PM | Permalink
Categories: Information architecture, Metrics & ROI

Popular paper: metrics for KM & CM

I received an e-mail today from ITpapers.com, informing that I have won some sort of monthly prize for having written one of their most-downloaded articles.

Apparently, the article Metrics for knowledge management and content management is much in demand. I'll even get a plaque mailed to me listing my achievement, although I don't know that they realise that I'm in Australia, not the US.

All very strange...

Posted by jamesr at 03:05 PM | Permalink
Categories: Content management, Knowledge management, Metrics & ROI

Jakob Nielsen defends productivity gains

Jakob Nielsen's latest Alertbox goes to some length to defend his popularisation of productivity gains as a basis for calculating ROI. To quote:

Usability improvements can save time-on-task, but critics argue that this is not the same as saving money. Others worry that productivity gains cause unemployment. Neither is correct: usable design saves money and saves jobs.

I'm just not convinced. I'm not arguing that improving usability doesn't saves staff time, quite the opposite. Clearly, this is a good thing, in general terms. What I have a problem with is the calculation of every project saving "millions of dollars", by making an individual task a few seconds quicker.

These sort of "rubbery figure" estimates are meaningless, and devalue the reputation of our industry.

This was brought home very clearly to me in a recent conversation I had with Tina Calabria, a usability colleague. She highlighted that this was meaningful when you had staff using a single dedicated application all day, every day. In this situation, time saved directly equalled increased transactions, etc. However, in these days of general staff activities, productivity savings are more an illusion than reality.

Clearly we can't yet bury this "metric"...

Posted by jamesr at 08:35 AM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

Projects that defy ROI

David Joachim has written about projects that have hard to quantify ROI figures. To quote:

It's one thing for your bosses to insist on a business justification for IT projects that clearly cut costs or generate revenue. But what about those initiatives that do neither?

We're alluding to technologies that wouldn't exist in a world in which nothing went wrong. They include firewalls, intrusion-detection systems, high-availability servers and hot-backup systems. These categories defy conventional ROI analysis because on a typical day they don't produce any returns.

[Thanks to ITtoolbox Knowledge Management.]

Posted by jamesr at 01:44 PM | Permalink
Categories: Metrics & ROI

Time saved: a misleading justification for KM

Dr. Michael Koenig has written a brief article about the dangers in justifying KM efforts on the basis of time saved. To quote:

The problem with time saving as a justification is that even if the user's time is saved by a KM system, management probably won't see any difference in the user's behavior, and the justification could backfire. Why won't management see a difference? The answer is a surprising one. Over the years, a number of studies of the practices of white-collar professionals have been conducted and their findings are uniform and corroborative. White-collar professional employees spend a consistent 20% to 25% of their time information seeking. A recent study arrived at the same conclusion (Normier, Bernard, report of a 2001 study conducted by Lazard Freres & Co., presentation on "Natural Language Advances," KMWorld Conference, 2001). Workplace technology may have changed, but that 20% to 25% hasn't. The proportion is surprisingly independent of the apparent information intensity of the job domain. Line business managers and administrators spend as much of their time information seeking as do research scientists. What is going on here?

[Thanks to CMSWatch.]

Posted by jamesr at 08:32 AM | Permalink
Categories: Knowledge management, Metrics & ROI

The "proof" for usability's ROI: statistics and examples

The Usability Professionals' Association (UPA) has released a summary article outlining the ROI for usability. This brings together a lot of interesting stats and resources into the one location. To quote:

Because there have been many well-documented examples of cost savings with usability engineering, sound statistics can be applied generally to UI development. These statistics serve as benchmarks.

Posted by jamesr at 03:21 PM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

Intuiative ranking

There is an excellent discussion on drop.org about ways of implementing intuiative ranking of web pages. To quote:

From Amazon to Google to Daypop, there is a demand for accurately ranked content.

As a result, most Content Management Solutions include a ranking feature. Typically, this involves a call to action for users to rate the content they are looking at. I believe this is a less than satisfactory solution for the following reasons:

• It is asking the users or site visitors to do work and offers nothing significant in return.
• It clutters up the user interface, making the experience less enjoyable for the user.
• Biased users can easily manipulate the system to change the ranking.
• The results are often not very satisfactory.
• I believe you need a significant amount of traffic before it offers meaningful data.

Posted by jamesr at 10:21 AM | Permalink
Categories: Metrics & ROI

KM Column: Metrics for knowledge management & content management

My latest KM Column article has just been uploaded to the website:

• Metrics for knowledge management and content management
  Metrics are an effective way of setting project targets, assessing success, and tracking ongoing health. This article summarises a range of practical KM and CM-related metrics.

Feedback or comments are always welcome...

Posted by jamesr at 11:08 AM | Permalink
Categories: Content management, James' articles, Knowledge management, Metrics & ROI

HR self-service success

Sarah Fister Gale has written a feature exploring three stories of HR self-service success, broken into a number of parts:

• Introduction
  "HR fights back against paperwork, expense, and errors by instituting employee and manager self-service systems. Most employees love it immediately. It might take the managers a little longer to adjust. "
• Open Enrollment Is a Non-Issue, Thanks to Self-Service
  "Costs have been reduced dramatically and quality has improved immensely."
• Self-Service Causes Call Volume to Drop 75 Percent
  "Tally Defense has reduced the size of the HR department without falling behind on HR obligations."
• Self-Service Eliminates 80 Percent of Paperwork
  HR had to find a new way to manage data once the company's headcount tripled.

Posted by jamesr at 08:47 AM | Permalink
Categories: Knowledge management, Metrics & ROI

Intranet cost-benefits

Martin White has written a blog entry on the difficulty of calculating intranet cost-benefits. To quote:

The minute a prospective client asks me to assist in developing an ROI justification for their intranet I suddenly become very busy! I have a huge problem with any justification for an intranet that requires a quantitative business case to be made. I am not against doing so for certain projects, but in my view if a senior maangement team does not have the vision to see what the benefits of an intranet can be, then it's time to find a job elsewhere before the intranet implodes.

Posted by jamesr at 04:28 PM | Permalink
Categories: Intranets, Metrics & ROI

ROI on professional usability testing

Charles L. Mauro has produced a 57-page whitepaper on ROI on professional usability testing. The abstract is as follows:

This paper discusses the return on investment (ROI) implications of integrating formal usability testing methods into web development projects. Online and traditional lab- based approaches are discussed and compared for their respective strengths and weaknesses. The white paper provides detailed technical descriptions of current online usability testing methods and draws conclusions about the future of this important new customer response testing methodology. It includes a comprehensive trade-off matrix useful in making decisions about important technological approaches and research benefits.

[Thanks to Peter J. Bogaards.]

Posted by jamesr at 08:06 AM | Permalink
Categories: Metrics & ROI, Usability & user-centered design

ROI for intranet communication?

Gerry McGovern has written about assessing the impact of communication via an intranet, looking specifically on the business benefits it delivers. To quote:

A way to measure return on investment (ROI) for your intranet is to answer two basic questions. How does the intranet increase the level and quality of communication? How does it replace traditional forms of communication? To develop such an ROI model, you need to be clear on the current level and type of communication within your organization.

Posted by jamesr at 08:31 AM | Permalink
Categories: Intranets, Metrics & ROI

Intranet ROI survey

Toby Ward writes about the results from a recent survey into intranet ROI, which looked at which measures were considered to be most important. These were the top five categories of benefits to be identified:

1. Enhanced competitiveness
2. Content management
3. Enhanced communications
4. Hard cost reduction
5. Enhanced customer service

Of the specific ROI metrics, these were the top six:

1. Improved information sharing (customer svc.)
2. Enhanced communications and information sharing (communications)
3. Increased consistency of info (customer svc.)
4. Increased accuracy of info (customer svc.)
5. Reduced or eliminated processing
6. Easier organizational publishing

Apparently, there will be a second phase to this research, and it will be interesting to see what comes out of that.

Posted by jamesr at 02:52 PM | Permalink
Categories: Intranets, Metrics & ROI

ROI: Results Often Immeasurable?

Norm Alster writes about the challenges in determining ROI (Return on Investment). To quote:

Little wonder, then, that most corporate buyers are searching for new ways to gauge the payback from IT investments. Eager to oblige, vendors and consultants have trotted out a variety of tools that purport to more precisely measure return on investment. The approaches range from self-service Web sites that cough up an ROI calculation based on two or three inputs all the way up to new software programs costing as much as $200,000. Indeed, the mad rush to ROI is beginning to look like a tour bus unloading blackjack players in Las Vegas: everybody's got a system.

I found this gem at the end of the article particularly interesting:

Productivity gains don't necessarily boost revenue or cut cost, because people don't necessarily apply freed-up time to work -- particularly in the executive suite. Ian Campbell, vice president of research at Nucleus Research Inc., says that, generally speaking, lower-level employees are more likely to turn extra time into more work. "It's the VP who says, 'I can close my door and practice putting,'" says Campbell. "If you're a line worker, there's no chance to goof off."

This reality, Campbell says, needs to be factored into ROI equations. Here are his rule-of- thumb calculations on the percent of freed-up time various workers convert into increased output:

• Assembly-line workers: 95-100%
• Call-center support: 90-95%
• Administrative and support help: 70-80%
• Engineering (technical): 75%
• Engineering (nontech): 65-75%
• General staff within a group (marketing, PR, accounting): 60%
• Companywide general staff: 50%
• Middle management (large corporations): 40-55%

Posted by jamesr at 08:45 AM | Permalink
Categories: Metrics & ROI

Intranet ROI case studies

Gerry McGovern has posted a few intranet ROI case studies on his site, primarily from PeopleSoft, Cisco Systems and Hewlett Packard. To quote:

An intranet can deliver return on investment (ROI) by either reducing the cost, or expanding the ability, to communicate. By shifting manual processes to the intranet, the cost of accessing and processing information is reduced. The intranet speedily delivers information to large numbers of people. This gives the organization a greater capacity to change.

Posted by jamesr at 08:23 AM | Permalink
Categories: Intranets, Metrics & ROI

How Idiots Track Success (HITS)

Martin White has written a blog entry discussing ways of measuring intranet success, such as usage stats or surveys. This is certainly a difficult area, one that I think we are all still struggling with. To quote Martin:

Web page stats for intranets can be very misleading. You can often get high page hits on pages that people have found in error as a result of poor information architecture. The hits need to be analysed in context, taking account of the paths through the site (i.e. was the page found by a search, hyperlink, or another navigation option). Often a page might be used very infrequently, but the low hit rate is not a measure of the value of that page, which might be an important policy document, for example.

What is missing from this weblog entry is reference to search engine statistics. Unlike hits, this tracks what users were looking for, not merely what they found...

Posted by jamesr at 08:38 AM | Permalink
Categories: Intranets, Metrics & ROI

Estimating productivity

Peter Edmonston writes about the challenges of estimating productivity when calculating ROI for IT projects. To quote:

A company needs to consider an array of factors, including the number of employees affected by a project, how much each is paid, and the amount of time each is expected to save. Often neglected in such calculations, however, is a key aspect of human nature: the primordial urge to waste time. In the real world, employees are likely to fritter away a sizable chunk of any newfound productivity by taking more breaks, leaving early or bidding on eBay.

(This is an interesting site that I need to set aside some time to browse.)

Posted by jamesr at 04:46 PM | Permalink
Categories: Metrics & ROI

Assessment and measurement SIG

The Knowledgeboard provides an excellent special interest group for knowledge management assessment and measurement. This includes a discussion board, and a good range of useful articles and case studies. Well worth keeping an eye on.

(My only frustration is that all the articles are delivered as PDFs, which makes it hard for me to link to them.)

Posted by jamesr at 05:54 PM | Permalink
Categories: Knowledge management, Metrics & ROI

The dollar value of an intranet?

Toby Ward has written two articles on measuring the dollar value of an intranet. In Part 1 he introduces the issues, and outlines three high-level ways of measuring intranet ROI:

1. Ascribe the intranet a value equal to the investment; dollar in equals dollar out.
2. Ascribe a non-monetary value to the intranet as a collaborative communication and knowledge management asset.
3. Measure and appraise specific benefits from implementing an intranet and/or corporate portal.

In Part 2 he explores specific metrics that can be applied, which he divides into ten main categories:

• Hard Costs
• Sales
• Productivity
• Competitiveness
• Application Access
• Infrastructure
• Collaboration
• Time To Market
• Customer Service
• Human Resources

Overall, this article looks at metrics from a fairly high-level, and is a bit short on specifics. Still, it does serve as a useful introduction.

Posted by jamesr at 04:41 PM | Permalink
Categories: Intranets, Metrics & ROI

A Culture of Numbers

Paul Cox writes about the culture of numbers at Worldcom. This is a very interesting view of a situation where metrics became the end, not the means. To quote:

Any part of our work that could be measured and put into a spreadsheet was reported and circulated and compared to others.

The problem I saw was this: The most important aspects of our job, the ones that really counted (such as having satisfied customers), could not be measured. Because they could not be measured, they went unreported, thus unacknowledged. Those with "good Numbers" were often rewarded over those that actually did their job (and satisfied their customers).

Yes, this is yet another Worldcom article. It doesn't really dwell on the failure, instead it looks at a very interesting culture in action. Worth a read for all KM people.

Posted by jamesr at 10:35 AM | Permalink
Categories: Knowledge management, Metrics & ROI

Optimising intranet ROI

Mike Parsons writes about how to determine ROI for intranets. A quote from the article:

ROI with intranets and extranets can be hard to determine. Many of the benefits fall into the 'soft' category and therefore lack the weight of clear revenue improvements. However, there is much evidence to suggest the real impact intranets and extranets can have.

Intranet and Extranet environments can create value for organisations in a number of ways. A company can increase sales, decrease costs, improve productivity, improve customer service and build intellectual capital. Understanding the ROI can inform organisations on the best way to invest in their Intranets and Extranets.

The bulk of this article consists of small case-studies, compiled from other sources. While there is little practical advice on how to determine metrics, the article is still worth a read.

[Thanks to Simon Willison's Weblog.]

Posted by jamesr at 11:41 AM | Permalink
Categories: Content management, Intranets, Metrics & ROI

Call centre measures

Melissa Solomon has written a very brief article on the call centre measures used by Pacific Gas & Electric Co. While this article doesn't really explore the topic, it is always interesting to see what metrics are being used in various settings.

Posted by jamesr at 11:43 AM | Permalink
Categories: Metrics & ROI