Step Two Designs

JANUARY 2002

How to evaluate a content management system

Written by James Robertson, published January 23rd, 2002

Categorised under: articles, content management

Selecting and implementing a content management system (CMS) will be one of the largest IT projects tackled by many organisations. With costs running into the millions of dollars, it is vital that the right CMS package be selected.

This article outlines some of the lessons that we have learnt when assisting clients to chose a CMS. It offers ideas and tips, and provides an approach for identifying your business’ actual requirements for a CMS.

With so many vendors and products, it can be very hard to compare between them. Preparation, and a disciplined approach to this evaluation process is critical.

What this article isn’t

No vendors or products are mentioned in this article: this is not a survey of current commercial solutions.

Instead, it provides tools to assist you to conduct a review of suitable products. There is no “one size fits all” solution: no two organisations have the same requirements.

Assumptions

In developing these guidelines, we have made several key assumptions about the type of organisation purchasing a CMS:

• medium to large organisation
• current publishing systems will be replaced by the new CMS
• CMS will manage both the intranet and internet website
• CMS will be enterprise-wide.

A wide range of content will be published using the CMS. This can be characterised as:

• simple pages
• complex pages, with specific layout and presentation
• dynamic information sourced from databases, etc
• training materials
• online manuals (policy & procedures, HR, etc)
• general business documents
• thousands of pages in total
• extensive linking between pages.

Ask yourself: what are your business goals and needs?

Business goals & strategies

Why are you purchasing a CMS? Before identifying specific requirements, you must determine the business goals that will be achieved by implementing a CMS.

These must also reflect the long-term strategies and directions of your business.

It should be possible to succinctly outline your business goals on a single page. Make sure these are well-understood and agreed to by all stakeholders before starting the requirements gathering process.

Identifying requirements

There is no single best list of requirements for a content management system. Every organisation has unique needs.

Involve all your stakeholders in the requirements process. This includes relevant IT groups, business units, and end users.

This is particularly important if you are purchasing an “enterprise-wide” CMS.

Use structured investigation methods, to ensure that the list of requirements is both manageable and sufficient. If this process is approached in a disciplined way, there is little danger of the project suffering from “feature creep”.

Structuring requirements

The list of requirements for an enterprise-wide CMS will grow quite large. Group the items into categories, to make this list more manageable.

One classification scheme that has worked well for us is:

• Content creation
• Content management
• Publishing
• Presentation
• Contract & business

This list covers the full lifecycle of a content management system, from initially creating the content, through to delivering it to end users.

Ideas box

This section gives you some starting points for your requirements gathering process. These ideas have been distilled from the CMS projects we have been involved in.

This is far from a complete list, and is no replacement for a full requirements gathering process.

(See the Content Management Requirements Toolkit for a comprehensive list of over a hundred individual requirements.)

Content creation

This is the functionality required by the authors (content creators) using the CMS.

Without an effective authoring process, use of the CMS will wither and fail within a year of implementation.

Key requirements may include:

• Integrated authoring environment

  The CMS must provide a seamless and powerful environment for content creators. This ensures that authors have easy access to the full range of features provided by the CMS.

• Separation of content and presentation

  It is not possible to publish to multiple formats without a strict separation of content and presentation.

  Authoring must be style-based, with all formatting applied during publishing.

• Multi-user authoring

  The CMS will have many simultaneous users. Features such as record locking ensure that clashing changes are prevented.

• Single-sourcing (content re-use)

  A single page (or even paragraph) will often be used in different contexts, or delivered to different user groups.

  This is a prerequisite to managing different platforms (intranet, internet) from the same content source.

  (This is a complex requirement that warrants a whitepaper of its own.)

• Metadata creation

  Capturing metadata (creator, subject, keywords, etc) is critical when managing a large content repository.

  This also includes keyword indexes, subject taxonomies and topic maps.

• Powerful linking

  Authors will create many cross-links between pages, and these must be stable against restructuring.

• Non-technical authoring

  Authors must not be required to use HTML (or other technical knowledge) when creating pages.

• Ease of use & efficiency

  For a CMS to be successful, it must be easy to create and maintain content.

Content management

The core of most CMS solutions is a central repository, supported by a range of tools for manipulating and managing the content.

Key requirements may include:

• Version control & archiving

  Strict version control is necessary for legal accountability, backup and disaster recovery. A simple but powerful
  interface must be provided for these features.

• Workflow

  Decentralised content creation relies heavily on a powerful workflow model, that can be easily customised, and is
  resilient against organisational change.

Understand the needs of your authors: they will be doing all the hard work

• Security

  Adequate security levels and audit trails must be in place to protect the integrity of the content.

• Integration with external systems

  A CMS is typically only one of a number of systems used to present information on the intranet or website.

  An enterprise-wide CMS will only be successful if it can be cleanly integrated with existing business systems.

  The mechanisms for achieving this must be fully documented, and based on open or industry standards.

• Reporting

  The CMS must provide an extensive range of reports, for both users and administrators. Ideally, the system should pro-actively report on any issues that arise.

  Support for customised reporting is also desirable.

An enterprise-wide CMS will fail unless all stakeholders are involved from the outset

Publishing

The publishing engine takes the content stored in the repository, and generates the final pages. While this may be a dynamic or batch process, the same basic requirements apply.

Key requirements may include:

• Stylesheets

  Final appearance is controlled through the use of stylesheets. This provides flexibility and expandability.

• Page templates

  Overall page layout is specified via page templates. Ideally, a non-technical interface should be provided for managing
  this.

• Extensibility

  It must be simple to integrate code “snippets” (or equivalent) to provide additional publishing functionality. The CMS
  must support a process of “continual improvement” in interface design.

• Support for multiple formats

  The CMS must publish to multiple formats, such as: HTML (web), printed, PDF, hand-held (WAP), and more.

  It should be possible to add support for additional formats, which will be necessary as new standards evolve.

  In order to achieve high-quality in every format, it is critical that the content be separated from presentation at the
  time of authoring. This allows distinct stylesheets to be used for each output.

The most important part of a CMS is the content itself

• Personalisation

  Different information is presented based on either user profiles, or metadata in the source content. This is typically
  required for large “portal” websites.

• Usage statistics

  The CMS must allow comprehensive usage statistics to be gathered, including: most popular pages, daily usage, and
  search engine usage.

  This information allows the success of the site to be tracked, and any usability issues identified.

Presentation

The published pages must meet certain standards if they are to be of value to your users. It is important to specify these requirements if you are asking the vendor to design the appearance and layout of the web pages.

Key requirements may include:

• Usability

  This covers aspects such as ease of use, learnability and efficiency. Usability can be assured by conducting tests on the prototype designs with real users.

  Usability heuristics (guidelines) must also be followed.

• Accessibility

  The CMS must conform to standards such as the W3C Web Accessibility Initiative (WAI).

• Cross browser support

  The pages must be viewable in all major web browsers (Internet Explorer, Netscape, Opera, etc). Specify which browser versions are to be supported.

• Limited client-side functionality

  You may wish to limit which client-side technologies (Java, JavaScript, Flash, etc) are required to view the site. This is more important for a website than an intranet.

• Speed

  Page size must be limited to ensure that load times are acceptable for users. Specify the typical user access methods (LAN, modem, cable, etc).

• Valid HTML

  All pages must conform to the current HTML specification. This ensures maximum compatibility across browsers and platforms.

• Effective navigation

  Users must be provided with consistent, comprehensive and usable navigation aids.

• Metadata

  All pages must provide sufficient metadata to allow effective indexing and searching. This should conform to a standard such as Dublin Core.

The total cost of running a CMS includes the skills and resources needed within your business, not just the dollar values

Contract & business

Project management and business requirements must also be satisfied in a CMS project.

Key requirements may include:

• Training

  The vendor must list the training materials that exist for the CMS, and the training services that they can provide.

• Documentation

  The CMS must be supported by adequate documentation: for users, administrators and developers.

• Warranty

  The warranty period provided, once the software has been purchased.

• Maintenance agreements

  The vendor must outline their preferred support arrangements, including service level agreements and upgrade processes.

• Resources required

  The hardware, software and operating systems required by the CMS.

• Skills required

  What skills and knowledge will be required within your organisation to customise and maintain the CMS?

• Cost

  Both the fixed costs for the CMS, and the per-user (“per-seat”) costs. The latter is generally more significant for a large organisation.

• Scalability

  The load levels that the CMS supports, and the additional resources (hardware & software) required for increased usage.

• IT constraints

  Specify any pre-existing hardware or software that the CMS must interface with, or run on. This includes specific operating systems, databases or webservers.

• Reference sites

  The vendor must supply a number of sites where the software has been successfully implemented. These must match the characteristics of your organisation.

Avoid jargon, as there is little common understanding of content management terms, even amongst vendors

Some tips

When developing your list of requirements, keep these guidelines in mind:

• Understand your content

  Conduct a thorough survey of the information that is to be published via the CMS. The nature of this content will influence the functionality required.

  It is important not to lose sight of the content when selecting the content management system.

• Relate to business needs

  Every requirement must be associated with one or more business needs. This allows you to identify the reasons for the requirements, and to prioritise them.

• Avoid technical details

  Ensure you specify business needs, not implementation details. Vendors should be free to propose any methods or technologies that are able to meet your goals.

Avoid jargon, as there is little common understanding of content management terms, even amongst vendors

• Provide descriptions

  Your challenge is to make your needs understood, despite differences in vendor backgrounds and understanding. Avoid using jargon terms, and spell out your requirements in as much detail as possible.

• Use examples

  Examples provide concrete situations and business needs, and are an effective way of supporting your requirements. Use wherever possible.

• More not less

  It is better to have too many requirements than too few. The CMS must work for your entire business for several years at least: do not be surprised if the final list of requirements is long.

Evaluating vendor products

Once your requirements have been identified, these must be used to select from the (potentially) large list of vendors.

There are a number of approaches that may prove useful:

Formal tender

The vendors must be required to provide detailed descriptions of how each of your requirements will be met by their system.

These responses may be gathered in a variety of ways, but the most common is a formal tender process.

Using this approach ensures that the vendors are accountable for any promises or commitments they make regarding their CMS.

Demonstrations

You must ensure that vendor demonstrations are more than just a sales pitch. To be of value, they must demonstrate how the product will meet your business’ needs.

The best way to achieve this is to develop scenarios. These describe common or important tasks that will be performed using the CMS.

By presenting these in a “narrative” form, considerable scope can be covered in a relatively brief description.

Provide these to vendors at least a week before the presentations, and require strict adherence. This allows direct comparison between the vendors’ products.

Scoring

Whatever evaluation processes are followed, you must eventually chose a single successful vendor.

To do this in an impartial way, create a scoring system. Determine this before the vendors are contacted, and incorporate the results of any tenders or demonstrations.

Using a formal scoring system eliminates the potential for accusations of bias or corruption.

Conclusion

Selecting an enterprise-wide CMS is often a multi-million-dollar exercise. It is therefore critical that the new system meets your current and projected needs.

The single most important activity is to identify your business goals and requirements. This process must involve all relevant stakeholders.

Only once you have a full list of requirements are you in a position to compare vendor products.

Spending time on these initial phases reduces the business risks inherent in purchasing a CMS. The project is also likely to be more successful (it will do what the users need), and development efforts will be reduced (goals are clear).

(For further information regarding the evaluation and selection of a CMS, download the Content Management Requirements Toolkit.)

Tags: content management systems, evaluation, requirements, selection