Treejack scores are provided for three aspects of participant responses: success, directness, and speed. Successes, failures, and skips are also broken down into direct and indirect, and each Task is given an overall score. This article explains what each of these things mean.

The Success score refers to the percentage of participants who selected a correct destination, regardless of whether or not they had to jump around the tree a few times before doing so.

A success score of around 80% or more is considered a good score for a task.

The success score is also broken down into Direct and Indirect.

Direct success: Number of participants who went directly to the correct destination, without going back up the tree at any point. A direct success score suggests the participants had a measure of confidence when going to and selecting an answer — though we cannot be certain of what they were actually thinking.

Indirect success: Number of participants who got the destination correct, but moved back up through the tree at least once before doing so. This is useful information to see: if you see a high success score, it will pay to look at the number of indirect successes before celebrating.

Skips: A direct skip is one where the participant chooses 'skip this task' without clicking anywhere within the tree first. An indirect skip is one where they explored the tree before skipping the task.

The Directness score is the percentage of participants who did not backtrack at all when selecting a destination, whether it was correct or incorrect. It attempts to measure how certain participants were in selecting their destination, though it's important to not assume too much about what your participants were thinking.

If a participant considers that they have gone the wrong way at any point and backtracks to go down a different branch, they will be excluded from the Directness score.

The time taken score is a box plot of the recorded task completion times. We exclude participants who took longer than 100 seconds to complete the task because we've found from our data that people who take longer than 2 minutes to complete a single task have almost certainly left it idle. We remove these outliers to give a better indication of the actual median completion time.

The overall score in the colored box on the right is a weighted average of the squares of success and directness. It favors success over directness at a ratio of 3:1, scaled to be a value out of 10.

Here's an example of how we calculate the overall score:

Success: 57%

Directness: 67%

Success score: 57^2 / 1000 = 3.249

Directness score: 67^2 / 1000 = 4.489

Weighted combined score: 3.249 x 3/4 + 4.489 x 1/4 = 2.437 + 1.122 = 3.559 (rounded up to 4).

If you included post-task questions in your study, you’ll find the results for your questions at the bottom of each Task.

You can also view a pietree for each of your tasks. Simply click the ‘View the Pietree’ button to watch your pietree grow before your eyes. Read more about how to interpret your pietree.