The usability of a content management system is paramount. If authors and site owners can’t work out how to use the CMS, you’ve got nothing. The CMS can have all the functionality in the world, but usability trumps it all.

I’ve written about this before, outlining 11 usability principles for CMS products, highlighting the importance of CMS usability, and arguing that more users = simpler CMS.

That’s all well and good, but how to we test usability in practice?

Let’s start first by highlighting how you can’t evaluate usability: by reading written tender responses. At their best, these documents only outline the functionality provided by the CMS, and it’s not possible to build up a clear picture of how the product works in practice.

A few approaches that can be used:

1. Vendor demoA well-managed vendor demonstration should provide a pretty good idea about the usability of the product. The key is to use scenarios as a script for the demo, so you don’t just see the vendor’s standard “smoke and mirrors” pitch. Two hours is still not enough to get an in-depth understanding of the product, but it’s a good start.
2. Involve end-usersThere’s no point having a CMS selected by IT if they’re not going to be the actual authors. Involve at least one representative user throughout the selection process, and value their thoughts on the products.
3. End-user trainingAsk the vendor to run their standard end-user training for the CMS. Recruit a few typical authors (who haven’t been otherwise involved in the selection process), and see how they go with the training. This will help to answer many questions: is the training 2 hours or 2 days? how simple is the product to use? can the author actually understand the product at the end? is the training run professionally?
4. Demo siteMany vendors are able to provide an online demo site for the CMS. This offers a simple way of playing with the product in a hands-on way. (Note that every CMS requires training, so simply playing with a product can be potentially misleading.)
5. Reference sitesChecking reference sites is a standard part of many decision-making processes, whether choosing a product or hiring a new staff member. See if you can talk to someone who represents the authors (rather than the IT folks), and ask questions about the ease of use of the solution.
6. Proof of conceptThe most intensive and in-depth way of assessing the usability of a CMS is by conducting a proof-of-concept. The vendor deploys the CMS into a realistic environment, and several weeks are set aside for working with the product. This will shed light on many aspect of the CMS, not least on its usability.
7. Usability testingOnce a proof of concept has been set up, formalised task-based usability testing can be used to assess the product. Carefully scripted tests with real users will quickly identify major issues (again, note the need for user training in advance of this testing.)

Even with a range of techniques, CMS usability can still be hard to assess. What is simple for one user may be perceived as difficult by another. Despite these challenges, it is still vital to conduct a rigourous evaluation of usability as part of the selection process.

What approaches have worked for you?