Interpret the OptimalSort Dendrograms for open and hybrid card sorts

This article tells you how to interpret the two different dendrograms in your open and hybrid card sorting results, and describes the main differences between the two.

It also explains why you won't see dendrograms in your closed card sort results, and the results you'll get instead.

The two dendrograms in your results that are generated by different algorithms:

  • The Actual Agreement Method (AAM), which is most useful when you have 30 or more completed card sorts.
  • The Best Merge Method (BMM), which is most useful when you have fewer completed card sorts.

The AAM dendrogram shows the % of participants who agree with the grouping 

The Actual Agreement Method depicts only factual relationships, and provides the most useful data if over 30 participants have completed your survey. The scores tell you accurately that 'X% of participants agree with this exact grouping.' 

The vertical blue lines represent agreements between participants, and the vertical gray line shows you the percentage of participants that agree the highlighted cards should be grouped together. You can move the gray line along the axis.  

Example: Open card sort for a travel website

Participants sorted cards labelled with place names from around the world, and their completed card sorts were used to help design a travel website.

When we move the vertical line and hover the mouse over the cluster in the example below, we can see that 70% of participants agreed that the highlighted cards should be grouped together. We also shown three category labels created by participants who grouped those cards together (they're just suggestions, and they get more accurate the higher the agreement).

This data gives us ideas for how we could group and label that particular content on our website, and potentially all the content on our website. Though we might have planned to arrange our travel website by geographic location, we could instead arrange it by what people are interested in, with one category like this: 

Historical Interest — London, Athens, Paris, Rome, Italy, Milan

The further we take the vertical line towards the left, the higher the agreement between participants, and the more reliable the suggested category labels become. In this example, 82% of participants agree that the three Italian cities should be grouped together. 

The BMM dendrogram shows the % participants who agree with parts of the grouping

The Best Merge Method dendrogram makes the most of a smaller number of completed card sorts. The AAM dendrogram shows only 'factual' relationships. In contrast, the BMM dendrogram makes assumptions about larger clusters based on pair relationships, and tells you that 'X% of participants agree with parts of this grouping.'

BMM's ability to compromise and extrapolate helps you squeeze the most out of small or incomplete responses.

In this example, we can see that 75% of participants agree with parts of this highlighted grouping: that is, have all grouped at least two of these cards together:

The main differences between AAM and BMM

If 10 participants sorted their cards {A,B},{C} and another 10 participants {A},{B,C}... 

...the AAM algorithm will tell you that {A,B,C} is not a very good category because nobody created that exact combination of cards.

...the BMM algorithm will tell you that {A,B,C} is quite a good category because every participant found it partially acceptable. 

Actual Agreement Method algorithm (detail)

The AAM algorithm counts each instance of a complete category from every participant. Each category with a non-zero score (a "real category") inherits the base score (i.e. Before inheritance) of all superset categories. The category with the highest score is taken, and all conflicting categories are eliminated.

Best Merge Method algorithm (detail)

The BMM algorithm breaks each instance of a category from every participant down into their base pairs. The pair with the highest score is locked in. This repeats, and where the pair being locked in intersects with an existing locked category, it is agglomerated with this category. All subsets of this new category are eliminated.

Why dendrograms are not as useful for closed card sorts

Dendrograms explore the way participants group things according to their own logic, and so work best when participants face no constraints while doing so. In a closed card sort, you define the categories that participants must sort cards into, and this introduces a bias that is visible in a dendrogram. Furthermore, dendrograms help you generate ideas for grouping and labelling content, and when you're running a closed card sort you already have your categories set.

As a small illustration, let's say you have 50 cards labelled with food items, and you run an open card sort to generate ideas for how to arrange the items on your website. Some participants might group the items by type (fruit, vegetables, dairy), some by price, some by color, some by recipe, and so on. Dendrograms will reduce the complexity of this data by showing the cross-overs between the different categorisation, and help you generate ideas for different ways to group and label your content.

However, if you run a closed card sort and ask participants to sort the items into the categories Fruit, Vegetables, Dairy, and so on, high rates of agreement in the results will be because your category labels encouraged participants to think in a fixed way, rather than because participants all used the same logic.

Instead of dendrograms, you'll get more useful insights from the following two visualizations:

  • The Results Matrix displays the number of times each card was sorted into your pre-set categories. 
  • The Popular Placements Matrix displays the percentage of participants who sorted cards into particular categories, and ranks them from most popular to least popular.