Interpret and standardize the categories in OptimalSort

This article describes:

  • what the categories table displays for open, hybrid, and closed card sorts 
  • why standardizing categories in open and hybrid card sorts is a good idea
  • how to standardize categories accurately
  • how standardizing categories affects your results.

What the categories table displays for open, hybrid, and closed card sorts

In an open card sort, the table displays all categories created by your participants, and the cards placed in each category. The two columns on the right (agreement and number of participants) become relevant when you standardize categories, which we talk about below. 

For example, we can see here that one participant created the category 'Resort Destinations' containing 11 cards.

In a closed card sort, the table lists your pre-set categories, the number of unique cards placed in the category, the cards themselves, and the number of participants who placed each card in that category. 

For example, we can see that 42 unique cards were placed in the 'Not important at all' category. And in particular, the card 'has a dj' was put in this category by 37 participants.

And in a hybrid card sort (in which you have pre-set categories and allow participants to create categories of their own) the table displays what you see in open and closed sorts: your categories and the participants' categories, the unique cards placed in each category, the number of participants, and the agreement between participants.

Why standardizing categories in open and hybrid card sorts is a good idea 

Standardizing categories means merging similiar categories together to turn them into one category.

When you allow participants to come up with their own category names, there's a good chance you'll see the similar labels that have variations in wording, spelling, capitalizations, and so on. Furthermore, when you take a closer look at the cards, you can often deduce that different participants mean exactly the same thing by their category labels.

It's a good idea to standardize categories before exploring your results in depth. It prevents cross-over and reduces the complexity of the analysis, particularly if you have a large number of completed card sorts (like in a recent study we ran, which had 195 completed card sorts and over 1400 suggested categories!).

How to standardize categories accurately

Before you eye up a bunch of similar-looking labels and standardize straight away, it's important to look at the similarities between the categories in more depth.

Here's a simple process to follow.

First, look for similar words and phrases

The categories table is initially arranged alphabetically, so you'll be able to see  similar categories next to each other by just glancing through the table. You can also use search to display only categories that contain key words.

In this example, we've searched for the key word 'America', and have found that 7 participants included that word in a category label. This could make it a great candidate for standardization. 

Second, establish if participants mean the same thing by their labels

We might be inclined to select all 7 categories containing the word 'America', and hit 'Standardize' with a sense of achievement and delight. But first, we need to make sure the distinct categories we're about to merge are similar enough to become one.

Two options for approaching this include looking more closely at the category labels and the cards in each category, and checking the agreement score after we've created a standardized category. The first is more subjective, and involves your own insights and decisions, and the second give you a hard figure. Both approaches complement the other.

Take a closer look at the category labels and the cards in each category

The table displays by default the first three cards in each category by alphabet, so this might be enough for you to spot similarities and differences. If not, you can opt to view all the cards in each category.

In our example, we can see below that five of our 'America' categories contain at least two of the same cards as the others. This gives a good indication that the participants had similar intentions when creating those groupings, and we could consider merging them into one category. 

However, when we look again at the category labels, we see that one is actually 'America (North & South)', and that it includes countries also in South America. Another category only contains 'Mexico', and another only 'Istanbul'. These groupings suggest a different intention than the others, so we could probably exclude them.

Therefore, we could create a standardized category called 'America' containing the four categories with similar cards.

 

Check the Agreement score of your standardized categories

The agreement score tells you the agreement level between included participants on the cards that belong in each category. Beside the agreement score, you can see the number of participants included in that score.

A perfect agreement score is 1.0, which is what you'll see before standardizing (as each category has only been created by one person).

Once you standardize a category, check the agreement score to get an objective assessment of how similar the groupings are. Agreement scores from .8 and over means that 80% of participants in the new category agree with the grouping. This is a good standard.

So, what agreement score do we get for our 'America' category? Only 0.51.

When you see an agreement score this low, participants are probably thinking about the categories in different ways, or may have placed cards in categories that don't seem to make sense. At this point, we could reassess our merged categories.

However, when we look at the cards, we can still see something useful. The cards are displayed in order of occurrence, and we can see that the top four cards have perfect 1.0 agreement among participants (included in this category 4/4 times). Furthermore, the next three cards have an agreement of .75, which can also give us confidence that grouping these cards together is a good idea. 

How standardizing categories affects the results you see

Your standardized categories will be displayed in the Standardization grid, so you can clearly how often each card appears in the category you've created.