Practical ways to assess CMS usability

Written by , published November 25th, 2008

Categorised under: Content management, Usability & user-centered design

The usability of a content management system is paramount. If authors and site owners can’t work out how to use the CMS, you’ve got nothing. The CMS can have all the functionality in the world, but usability trumps it all.

I’ve written about this before, outlining 11 usability principles for CMS products, highlighting the importance of CMS usability, and arguing that more users = simpler CMS.

That’s all well and good, but how to we test usability in practice?

Let’s start first by highlighting how you can’t evaluate usability: by reading written tender responses. At their best, these documents only outline the functionality provided by the CMS, and it’s not possible to build up a clear picture of how the product works in practice.

A few approaches that can be used:

  1. Vendor demo

    A well-managed vendor demonstration should provide a pretty good idea about the usability of the product. The key is to use scenarios as a script for the demo, so you don’t just see the vendor’s standard “smoke and mirrors” pitch. Two hours is still not enough to get an in-depth understanding of the product, but it’s a good start.

  2. Involve end-users

    There’s no point having a CMS selected by IT if they’re not going to be the actual authors. Involve at least one representative user throughout the selection process, and value their thoughts on the products.

  3. End-user training

    Ask the vendor to run their standard end-user training for the CMS. Recruit a few typical authors (who haven’t been otherwise involved in the selection process), and see how they go with the training. This will help to answer many questions: is the training 2 hours or 2 days? how simple is the product to use? can the author actually understand the product at the end? is the training run professionally?

  4. Demo site

    Many vendors are able to provide an online demo site for the CMS. This offers a simple way of playing with the product in a hands-on way. (Note that every CMS requires training, so simply playing with a product can be potentially misleading.)

  5. Reference sites

    Checking reference sites is a standard part of many decision-making processes, whether choosing a product or hiring a new staff member. See if you can talk to someone who represents the authors (rather than the IT folks), and ask questions about the ease of use of the solution.

  6. Proof of concept

    The most intensive and in-depth way of assessing the usability of a CMS is by conducting a proof-of-concept. The vendor deploys the CMS into a realistic environment, and several weeks are set aside for working with the product. This will shed light on many aspect of the CMS, not least on its usability.

  7. Usability testing

    Once a proof of concept has been set up, formalised task-based usability testing can be used to assess the product. Carefully scripted tests with real users will quickly identify major issues (again, note the need for user training in advance of this testing.)

Even with a range of techniques, CMS usability can still be hard to assess. What is simple for one user may be perceived as difficult by another. Despite these challenges, it is still vital to conduct a rigourous evaluation of usability as part of the selection process.

What approaches have worked for you?

Tags: ,


  1. Craig Thomler commented on November 25th, 2008

    Unfortunately sometimes the only way to assess usability is to use a product for several years.

    Then you get a clear picture of what does and doesn’t work and the extra features you need (and the extraneous ones).

    On this basis I’ve found that wen implementing a replacement system, the best approach is to start by uncovering what users of the existing system really think.

    This can be hard, we all develop habits designed to get around limitations – however where it is possible to observe and identify these suboptimal workarounds they lead directly into the usability experience people would like to have but cannot under the current system.

    Note also that sometimes usability is not the goal. Witness the creation of the typewriter, which still curses us with the QWERTY keyboard of today.

    Sometimes systems are designed to slow down users, add barriers to processes and force conscious reflection before publishing. This is not always a bad thing IMHO.

  2. Yes, agree with the observation that it may take several years to fully understand the strengths and weaknesses of a system.

    That’s why I recommend that first-time purchasers of a CMS get the smallest, simplest, cheapest solution that meets their immediate needs.

    They should then plan to outgrow the solution within 3(-ish) years. If they haven’t outgrown it, they either haven’t grown (and learned) or they tried to by too much.

    That way, the second time around, they can learn from their experiences with the first CMS, and can be much more confident about many issues, including CMS usability.

    It also means, to my mind, that you don’t want to customise the CMS at the outset of the project, before you’ve understood how it’s going to work in pratice.

  3. Paul Bain commented on November 25th, 2008

    This is list is OK, but it omits two important factor. First, open source (OSS) CMS’s are destined to prevail sooner or later, so choosing a proprietary CMS is foolish. You should _not_ choose a proprietary CMS, if at all possible.

    A second factor is information security. MS Windows is insecure, and you should avoid any CMS that runs exclusively on MS Windows and not also on Linux or OpenBSD.

    — Paul Bain

  4. Actually, open-source vs proprietary is irrelevant here.

    Regardless of the licensing model: if you are going to have decentralised authors using the CMS, then the usability and simplicity of the solution is paramount.

    Of course, if you are choosing an open-source solution, the evaluation process will be different than when selecting a commercial solution.

    Nonetheless, there needs to be a way of evaluating the usability. Pick one or more of the options above, or find a different way! :-)

  5. David Hobbs commented on November 25th, 2008

    Thanks for the useful breakdown of approaches to evaluate the usability of CMSes. I would argue that getting the most “real world” as possible is best (which might argue against the standard end user training, except to support fairly simple use cases). Of course, actually using a system over time both gives users experience with a specific CMS, but, as you mention, lets an organization grow to discover other requirements/directions that no one would have anticipated at the start. So a system that may be easy to use on Day One may feel like a very difficult to use system on Year Three when it’s being pushed beyond its initial design.

  6. Paul Bain commented on November 25th, 2008

    To James Robertson:

    OSS CMS’s shall ultimately prevail over proprietary CMS’s. If you do not understand that by now, then you are stupid.

    If you do not understand the OSS Revolution by now, then you are just as clueless as Tony Byrne of Damn! Why are the CMS “analysts” so stupid?

    –Paul Bain

  7. Hi Paul, you are certainly passionate, but perhaps off-topic. I am not anti OSS, quite the opposite. I do think, however, that OSS is the “wrong question”, see my presentation on this and feel free to add your comments there:

  8. James, you are far too diplomatic, considering Paul Bain’s comments were little more than spam.

    You are, of course, correct. Whether a CMS is open source or not bears no relation to its relative usability.

    And of course, it must be said that open source solutions are *not* inherently superior in terms of usability (in fact to suggest so is quite a good indicator of how “clueless” one is). I’ll even go out on a limb and say they are typically less usable, by and large.

    Great list James, if only more people would follow your guidance when selecting a CMS.

  9. David Cain commented on December 10th, 2008

    Interesting that comments by Bain have hijacked my mind as I read your column.

    I’ve got a house and an office full of computers, Mac, PC, Linux (I use 3 different Linux distros on servers, laptops, appliances). I’ve invested thousands of hours in applications that run on them. I carry no water for folks trying to sell you an OS or application. I love and use OSS daily.

    That said, Bain’s comments smack of cant. First, the old saw “MS Windows is insecure”: I’m no fan at all of Windows (really, really hate the experience of using it), but it’s just as easy to run an insecure Linux install as it is to run an insecure Windows install. Both of them come not terribly secure right out of the virtual box. Getting any OS securely installed, provisioned and updated requires expertise, laser-like focus, experience, best practices and lots and lots of work and time.

    Anticipating Bain’s comment on this thought ‘It’s not so hard to secure Linux'; yes it is. It’s hard to secure ANYTHING, particularly production-grade applications. And it does take lots of time on any OS.

    The other comment: “open source (OSS) CMS’s are destined to prevail sooner or later” seems equally deluded on its face.

    It’s a false assumption that commercial apps offer no value over open-source apps. Hey, let me know when GIMP knocks Photoshop out of the market, OK?

    It’s a false assumption that the usability of open-source community projects designed for the whims of geeks will be well-managed for average business users who shouldn’t have to care whether the software is open-source or not in order to get through their day. Commercial vendors have more invested in making software work for business than for geeks (though it’s clear that most commercial CMS vendors have not done a great job at this)

    It’s a false assumption that there is a single kind of open-source CMS software that will win out over commercial offerings. Certainly server-side-scripted open-source offerings like Mambo/Joomla, Drupal, etc. have a great future ahead for some uses, but they can’t, simply can’t replace an enterprise-class CMS at the level of Vignette, Interwoven, et al. I don’t see an open-source enterprise-class WCMS that’s ready to step in here, and I’ve looked. Hard. Yes, I’ve had commercially-supported OSS CMS vendors in for demos. No, they’re not ready for our needs yet. When is this “sooner or later” you’re talking about going to happen?

    Finally you ask, “Damn! Why are the CMS “analysts” so stupid?”. I find this most interesting of all. In my mind, stupid+analyst comes from waving around a bunch of talking points unsupported by facts or data – much as you’ve done in your post. I find CMS Watch quite insightful and entertaining. May not be your cuppa tea, but they’re far from stupid.


  10. This list is very good. There’s a couple of bits of advise to go with it:

    Keep an open mind – quite often people come in with ideas based on old ways of doing things. Those with an open mind can take advantage and reap the benefits of the latest developments.

    Look for the best-fit solution for the next three years. Investing effort into something which will only meet your requirements for a year or so will have a long-term detrimental impact.

    The ongoing running costs will always outweigh the initial investment. Make sure the system delivers on the day-to-day tasks as well as the initial setup scenarios.

    Our experience of various CMSs is that OSS can be excellent for simple promotional sites, but once you get into complex environments and commercial projects then there is still a material gap in the functionality that they deliver.

  11. Hi Markus, I definitely agree on keeping an open mind.

    If an organisation has an existing CMS, it’s “ways of doing things” will unavoidably influence ways of thinking, and expectations.

    The new CMS will most likely work very differently. At the end of the day, you want to keep the best of the old, but take the opportunity to find new ways where they will work better.