August 2011

Card sorting: online versus offline

Written by , published August 29th, 2011

Categorised under: articles, intranets, usability & information architecture

Card sorting is a common technique within user-centred design (UCD) methodologies. It’s also an activity that can polarise opinions; people either believe in it, or think it’s a waste of time.

There are many articles and blog posts out there that measure the value of this UCD technique. This article focuses on a more specific aspect of the technique, which comes out of experience conducting a number of projects using card sorting — the use of offline and online card sorting methods.

What is card sorting?

Card sorting is an activity which asks participants to sort a selection of content — web pages, menu options etc — into categories that make sense to the individual or group.

Conducting this activity provides researchers with a user’s perspective into the groupings, and will typically inform information architecture (IA) designs.

It’s a simple technique, and perhaps the most widely used during intranet and website redesigns. Run with multiple groups of staff, it can provide valuable insights.

For a full rundown on card sorting, read the ‘Definitive guide to card sorting‘ on Boxes and Arrows.

Card sorting is also described in the context of a full intranet redesign in the book Designing intranets: creating sites that work.

Online card sorting sits alongside other options

How useful is card sorting?

Treated as one input among several in a UCD process, card sorting is valuable in sparking ideas, understanding nuances in nomenclature, and identifying confusing, contentious and unnecessary architectural components and structures. Card sorting will also invariably produce some surprises.

Yet, on its own, and even at its most informative, card sorting will not produce a finished, perfect blueprint for an IA, particularly in the intranet space. Other research and testing phases are just as valuable, if not more so.

For the best results, card sorting should only be carried out as part of a UCD process that includes other activities such as needs analysis and contextual enquiry research, and further IA and usability tests using drafted structures, prototypes and so on.

Offline, manual card sorting (above) is typically a group activity. It provides a very different experience from online, automated card sorting, which is typically an individual activity. As such, the two should be treated and applied very differently.

Online versus offline

There are two main ways of conducting card sorting: offline card sorting workshops where participants sort physical index cards on a table (usually as a group), and online programs used on a computer (usually one participant involved at a time). There is plenty of debate on the pros and cons of each approach.

Often this discussion is one of competition; one technique is seen as better than the other. Or, the two are seen as equally useful, and online card sorting is chosen because it’s deemed easier, faster, and with broader reach capabilities.

In practice, card sorting of either type should not be used without other supporting research techniques. The following sections provide guidelines on when and how to use each technique.

Pros of online card sorting

There are a range of online card sorting tools, including Optimal Sort, Websort and Usabilitest. These are popular, well-used tools, and for good reason. They offer a number of benefits:

Accessible

Anyone with a web connection can participate in a card sort. This is arguably online card sorting’s killer feature. It allows project teams to involve participants who may be geographically too far away to be involved in person.

Fast for the tester

All the tester needs to do to set up an online card sort is export the content data as a list, paste it into the card sorting software and email a link to participants. This is much faster than manually preparing physical index cards.

Fast for the participant

An online card sort of around 100 cards can be completed in under an hour. This is also, seemingly, a compelling aspect for many project teams eager to conduct a decent amount of testing in a short space of time. A manual card sort with a group will typically take 1–2 hours.

Analysis provided

With all data included in one program, an automatic analysis of card grouping correlation is provided. Data is also made available to download in numerous formats.

Cons of online card sorting

The benefits of online card sorting seem clear, yet there are drawbacks and counter points, described below.

The analysis provided is limited

IA development is an iterative process that requires an in-depth understanding of the content involved. Creating effective, intuitive headings and subcategories relies not only on quantitative data and statistical analysis, but also qualitative data and analysis, and instinct.

Card sorting programs use algorithms to standardise categories and to produce conclusions, and these conclusions do not take into account any qualitative data, such as the comments and thinking a participant may impart during a manual, face-to-face card sort.

Such qualitative information is just as valuable as the statistical analysis, if not more. Not gaining that insight significantly reduces the quality of the card sorting as an activity, perhaps by as much as 50 per cent.

Faster is not always better for the participant

While an online card sort may be completed by an individual in less than an hour, it will typically be one of the most unrewarding, abstract hours they will ever have spent.

For a recruited participant, there is vast dissonance between the project and this activity. Additionally, because the participant is acting on their own, there are no interactions or opportunities to ask questions (either of the project owner or other colleagues).

This means that in an online card sorting session, the participant’s main aims will be to do a decent enough job, but to also finish the activity as soon as possible, often consciously sacrificing quality in the process; “That will do” is an often uttered phrase among online card sorting participants and it further devalues the data being collected.

Faster doesn’t usually mean better and here, too, the limited interaction further reduces the value of the exercise. This has to be considered.

Online card sorts are typically faster than physical sorts

Faster is not always better for the tester

For the tester, online card sorting is infinitely faster than a manual process. After all, the tester doesn’t have to host or conduct the session, whereas with manual card sorting the tester would typically facilitate, and usually a colleague would help with note taking and observation too. And there’s the task of creating of the cards themselves.

In reality, engaging and managing a group of remote participants isn’t a quick task either. Recruiting, explaining and following up post testing all take time, and it’s an individual task, whereas managing a group talks to many people at once. Again, this has to be kept in mind.

A screenshot from Optimal Workshop’s online card sorting tool. Online tools work best when part of a broader research project that incorporates offline sorts and user testing.

When to use online sorting tools

Considering the above points, many questions come to mind:

  • When should I use online card sorting?
  • Should I only use manual sorts?
  • Should I dismiss card sorting completely?

Despite the above cons, online card sorting remains a very useful tool, but it is vital to know its limitations. This means to apply it where its strengths can be maximised, and failings mitigated.

Online card sorting should therefore only be used as a supplementary activity to more substantial testing, such as manual (offline) card sorts.

Use online card sorts to increase your research reach

Optimising online sorts

What’s key to a successful online card sort? Considering the context that typical participants will find themselves in — by themselves, generally survey fatigued, desk-based, with only the introduction and on-screen instructions to guide them — follow these guidelines:

Use online card sorting to increase your research reach

Offline card sorting can be used to validate hunches and inclinations gained through paper based, facilitated sessions. Facilitated, moderated sessions will usually give you the insight required to solve a problem through qualitative observation, while online, unfacilitated, unmoderated card sorting can provide the data required to convince stakeholders through quantitative analysis.

Use 50 cards or less in online sorting

Any more, and the on-screen experience begins to fall over. Too many cards means more chance of being overwhelmed, more navigating the screen up and down, and mentally sifting through too much information without suitable support. (A manual sort can easily contain 100 or more cards.)

Target the focus of the online sort

With 50 cards, it’s unlikely a whole site will be able to fit in a card sort. As such, limit the card content to specific areas of a site and ask users to sort that section.

Make it a short task

While an hour was mentioned earlier in the article, realistically the target should be a far lower completion time of 5–10 minutes. Data provided by Optimal Workshop suggests that the majority of successful sorts are completed in around 7–10 minutes.

Limit online sorts to 50 cards or less

Summary

Online card sorting is a great way of enhancing and supplementing the research that goes into developing new navigation for a intranet or website.

Working alongside other techniques, including offline card sorting, it works best as targeted sessions that can be completed in less than ten minutes.

Conducted well, online card sorting enables higher participation and completion levels, meaning better representation, wider project reach and visibility, and substantial quantitative data.

(Thank you to Andrew Mayfield, CEO at Optimal Workshop, for providing images and input into this article.)

Tags: , , , , , ,

Don't miss out!

Every month we publish two new articles, just as good as the one you've just read. Join our monthly newsletter, and we'll notify you when new articles are published, along with other Step Two updates.

(We won't use your details for any other purposes, or provide it to third parties.)



7 Comments:

  1. Alex, this is an excellent article. I agree with every point and recommendation you make here. I would only add that your sage advice is just as applicable to other kinds of web sites as it is to intranets. Well done!

  2. Alex commented on August 30th, 2011

    Hi Jacqui,

    Thanks for the feedback, great to hear and I agree the points are applicable beyond the intranet.

  3. Fred Beecher commented on August 30th, 2011

    Hi Alex. I think it’s great to look at the pros and cons of these two types of sorts, but I think one distinction hasn’t been made clear, the distinction between moderated and unmoderated sorts.

    Where I work, the most common type of card sorting we do is *moderated* online sorting, and for us it offers the best of both worlds. We get all the qualitative goodness of a physical sort, reduce the abstraction of unmoderated online sorts, and the analysis benefits of an online sort. Your article doesn’t state it outright, but it seems you’re focused on *unmoderated* online sorts. We do use those too, but only to extend the reach of our research as you recommend in the article.

  4. Alex commented on August 30th, 2011

    Hi Fred, you’re right, I have not focused on that distinction here. It’s a good point but I will need convincing by you that moderated online sorts are worth doing! I actually began writing this article after a session of moderated online sorts that I was particularly unhappy with. I think the points above apply in terms of guidelines for moderated or unmoderated online sorts as the computer screen really is not conducive to large-scale, long sessions, even on bigger screens.

    A question for you: do you run them as one-on-one sessions or multiple users? I wonder how the latter would work but can foresee some problems.

  5. We always run them one-on-one. Mostly we work on business-critical Web applications, so we’re often sorting tasks rather than informational content (e.g., “Update Address”). This typically keeps the number of cards down, so maybe that’s why we seem to have had more success with moderated online sorts.

    I can see how this would be more unwieldy with a larger sort, like for an intranet. A method that might help with that might be to conduct a delphi sort, in which a knowledgeable user or stakeholder takes a first stab at organizing the content and then card sort participants “fix” the organization until it seems right to them. When very few cards move around, that’s when you know the sort is done. It’s pretty slick because it validates the sort at the same time you’re conducting it.

  6. Alex commented on August 31st, 2011

    The focus on tasks… as much as the Delphi method is valid, i’m inclined to use tree testing as a way to test the suitability of a structure. Asking users to ‘fix’ seems a little arbitrary?

  7. I was probably condensing the language too much there. : ) Yes, tree testing & reverse card sorts are best for validating an existing structure. But the delphi method is for *creating* the structure, not validating it. The idea is that each user evaluates and improves upon the structure created by the previous user. Once users stop moving cards around (or only move a few of the same cards around consistently), you know with a reasonable amount of certainty that you’ve got a structure that will work. I’d still reverse card sort whatever came out of the post-sort analysis, though.

7ads6x98y