Card sorting: online versus offline

blank cards from Shutterstock

facebooktwittergoogle_pluslinkedinmail

Card sorting is a common technique within user-centred design (UCD) methodologies. It’s also an activity that can polarise opinions; people either believe in it, or think it’s a waste of time.

There are many articles and blog posts out there that measure the value of this UCD technique. This article focuses on a more specific aspect of the technique, which comes out of experience conducting a number of projects using card sorting — the use of offline and online card sorting methods.

What is card sorting?

Card sorting is an activity which asks participants to sort a selection of content — web pages, menu options etc — into categories that make sense to the individual or group.

Conducting this activity provides researchers with a user’s perspective into the groupings, and will typically inform information architecture (IA) designs.

It’s a simple technique, and perhaps the most widely used during intranet and website redesigns. Run with multiple groups of staff, it can provide valuable insights.

For a full rundown on card sorting, read the ‘Definitive guide to card sorting‘ on Boxes and Arrows.

Card sorting is also described in the context of a full intranet redesign in the book Designing intranets: creating sites that work.

Online card sorting sits alongside other options

How useful is card sorting?

Treated as one input among several in a UCD process, card sorting is valuable in sparking ideas, understanding nuances in nomenclature, and identifying confusing, contentious and unnecessary architectural components and structures. Card sorting will also invariably produce some surprises.

Yet, on its own, and even at its most informative, card sorting will not produce a finished, perfect blueprint for an IA, particularly in the intranet space. Other research and testing phases are just as valuable, if not more so.

For the best results, card sorting should only be carried out as part of a UCD process that includes other activities such as needs analysis and contextual enquiry research, and further IA and usability tests using drafted structures, prototypes and so on.

Offline, manual card sorting (above) is typically a group activity. It provides a very different experience from online, automated card sorting, which is typically an individual activity. As such, the two should be treated and applied very differently.

Online versus offline

There are two main ways of conducting card sorting: offline card sorting workshops where participants sort physical index cards on a table (usually as a group), and online programs used on a computer (usually one participant involved at a time). There is plenty of debate on the pros and cons of each approach.

Often this discussion is one of competition; one technique is seen as better than the other. Or, the two are seen as equally useful, and online card sorting is chosen because it’s deemed easier, faster, and with broader reach capabilities.

In practice, card sorting of either type should not be used without other supporting research techniques. The following sections provide guidelines on when and how to use each technique.

Pros of online card sorting

There are a range of online card sorting tools, including Optimal Sort, Websort and Usabilitest. These are popular, well-used tools, and for good reason. They offer a number of benefits:

Accessible

Anyone with a web connection can participate in a card sort. This is arguably online card sorting’s killer feature. It allows project teams to involve participants who may be geographically too far away to be involved in person.

Fast for the tester

All the tester needs to do to set up an online card sort is export the content data as a list, paste it into the card sorting software and email a link to participants. This is much faster than manually preparing physical index cards.

Fast for the participant

An online card sort of around 100 cards can be completed in under an hour. This is also, seemingly, a compelling aspect for many project teams eager to conduct a decent amount of testing in a short space of time. A manual card sort with a group will typically take 1–2 hours.

Analysis provided

With all data included in one program, an automatic analysis of card grouping correlation is provided. Data is also made available to download in numerous formats.

Cons of online card sorting

The benefits of online card sorting seem clear, yet there are drawbacks and counter points, described below.

The analysis provided is limited

IA development is an iterative process that requires an in-depth understanding of the content involved. Creating effective, intuitive headings and subcategories relies not only on quantitative data and statistical analysis, but also qualitative data and analysis, and instinct.

Card sorting programs use algorithms to standardise categories and to produce conclusions, and these conclusions do not take into account any qualitative data, such as the comments and thinking a participant may impart during a manual, face-to-face card sort.

Such qualitative information is just as valuable as the statistical analysis, if not more. Not gaining that insight significantly reduces the quality of the card sorting as an activity, perhaps by as much as 50 per cent.

Faster is not always better for the participant

While an online card sort may be completed by an individual in less than an hour, it will typically be one of the most unrewarding, abstract hours they will ever have spent.

For a recruited participant, there is vast dissonance between the project and this activity. Additionally, because the participant is acting on their own, there are no interactions or opportunities to ask questions (either of the project owner or other colleagues).

This means that in an online card sorting session, the participant’s main aims will be to do a decent enough job, but to also finish the activity as soon as possible, often consciously sacrificing quality in the process; “That will do” is an often uttered phrase among online card sorting participants and it further devalues the data being collected.

Faster doesn’t usually mean better and here, too, the limited interaction further reduces the value of the exercise. This has to be considered.

Online card sorts are typically faster than physical sorts

Faster is not always better for the tester

For the tester, online card sorting is infinitely faster than a manual process. After all, the tester doesn’t have to host or conduct the session, whereas with manual card sorting the tester would typically facilitate, and usually a colleague would help with note taking and observation too. And there’s the task of creating of the cards themselves.

In reality, engaging and managing a group of remote participants isn’t a quick task either. Recruiting, explaining and following up post testing all take time, and it’s an individual task, whereas managing a group talks to many people at once. Again, this has to be kept in mind.

A screenshot from Optimal Workshop’s online card sorting tool. Online tools work best when part of a broader research project that incorporates offline sorts and user testing.

When to use online sorting tools

Considering the above points, many questions come to mind:

  • When should I use online card sorting?
  • Should I only use manual sorts?
  • Should I dismiss card sorting completely?

Despite the above cons, online card sorting remains a very useful tool, but it is vital to know its limitations. This means to apply it where its strengths can be maximised, and failings mitigated.

Online card sorting should therefore only be used as a supplementary activity to more substantial testing, such as manual (offline) card sorts.

Use online card sorts to increase your research reach

Optimising online sorts

What’s key to a successful online card sort? Considering the context that typical participants will find themselves in — by themselves, generally survey fatigued, desk-based, with only the introduction and on-screen instructions to guide them — follow these guidelines:

Use online card sorting to increase your research reach

Offline card sorting can be used to validate hunches and inclinations gained through paper based, facilitated sessions. Facilitated, moderated sessions will usually give you the insight required to solve a problem through qualitative observation, while online, unfacilitated, unmoderated card sorting can provide the data required to convince stakeholders through quantitative analysis.

Use 50 cards or less in online sorting

Any more, and the on-screen experience begins to fall over. Too many cards means more chance of being overwhelmed, more navigating the screen up and down, and mentally sifting through too much information without suitable support. (A manual sort can easily contain 100 or more cards.)

Target the focus of the online sort

With 50 cards, it’s unlikely a whole site will be able to fit in a card sort. As such, limit the card content to specific areas of a site and ask users to sort that section.

Make it a short task

While an hour was mentioned earlier in the article, realistically the target should be a far lower completion time of 5–10 minutes. Data provided by Optimal Workshop suggests that the majority of successful sorts are completed in around 7–10 minutes.

Limit online sorts to 50 cards or less

Summary

Online card sorting is a great way of enhancing and supplementing the research that goes into developing new navigation for a intranet or website.

Working alongside other techniques, including offline card sorting, it works best as targeted sessions that can be completed in less than ten minutes.

Conducted well, online card sorting enables higher participation and completion levels, meaning better representation, wider project reach and visibility, and substantial quantitative data.

(Thank you to Andrew Mayfield, CEO at Optimal Workshop, for providing images and input into this article.)

Alex Manchester
Alex Manchester
Alex Manchester is an alumni of Step Two Designs. He specialises in intranet and enterprise social network research strategy and user experience design. Alex works with a wide range of public and private sector organisations and has over eight years experience in this field.