Treejack scores are provided for three aspects of participant responses: success, directness, and speed. Successes, failures, and skips are also broken down into direct and indirect, and each Task is given an overall score. This article explains what each of these things mean.
Success Score: How many participants chose a correct answer
The Success score refers to the percentage of participants who selected a correct answer, regardless of whether or not they had to jump around the tree a few times before doing so.
A success score of around 80% or more is considered a good score for a task.
The success score is also broken down into Direct and Indirect.
Direct success: Number of participants who went directly to the correct answer, without going back up the tree at any point. A direct success score suggests the participants had a measure of confidence when going to and selecting an answer — though we cannot be certain of what they were actually thinking.
Indirect success: Number of participants who got the answer correct, but moved back up through the tree at least once before doing so. This is useful information to see: if you see a high success score, it will pay to look at the number of indirect successes before celebrating.
Skips: A direct skip is one where the participant chooses 'skip this task' without clicking anywhere within the tree first. An indirect skip is one where they explored the tree before skipping the task.
Direct / Indirect fail: Similar to a direct or indirect successful result — this score is showing you the same information in regards to the participant's confidence when selecting an answer, however they have not selected a 'correct' node within your tree.
Directness score: How many participants went directly to an answer (correct or incorrect) without going back
The Directness score is the percentage of participants who did not backtrack at all when selecting an answer, whether their answer was correct or incorrect. It attempts to measure how certain participants were in selecting their answer, though it's important to not assume too much about what your participants were thinking.
If a participant considers that they have gone the wrong way at any point and backtracks to go down a different branch, they will be excluded from the Directness score.
Time taken score
The time taken score is a box plot of the recorded task completion times. We exclude participants who took longer than 100 seconds to complete the task because we've found from our data that people who take longer than two minutes to complete a single task have almost certainly left it idle. We remove these outliers to give a better indication of the actual median completion time.
The overall score
The overall score in the colored box on the left is a weighted average of the squares of success and directness. It favors success over directness at a ratio of 3:1, scaled to be a value out of 10.
Here's an example of how we calculate the overall score:
Success score: 57^2 / 1000 = 3.249
Directness score: 67^2 / 1000 = 4.489
Weighted combined score: 3.249 x 3/4 + 4.489 x 1/4 = 2.437 + 1.122 = 3.559 (rounded up to 4).
Post-task questions and pietrees
If you included post-task questions in your study, you’ll find the results for your questions at the bottom of each Task.
You can also view a pietree for each of your tasks. Simply hit the ‘View the Pietree’ button to watch your pietree grow before your eyes. Click here for more information on how to interpret your pietree.