How and why to rate your observations

Reframer is currently in beta, which means this content is too! We'll do our best to update it promptly as improvements are made. Thanks for your patience and feedback as we work on making Reframer awesome. Notice something that needs work? Please email support@optimalworkshop.com.

Reframer enables you to rate your observations on a 1–5 significance scale. You will rate your observations as you capture them, and the frequency of each rating will be displayed on your Results Overview screen, giving you an instant insight into your data.

Rating is intended to be a super-fast, intuitive assessment of an observation you've just captured. Therefore, before you start a project, and certainly before you start a session, it's important for everyone working on the project to agree on what is meant by 'significance' in that context.

Yes, that's right. The meaning you give the rating is up to you. We've given you some useful suggestions below, but really, the possibilities are endless.  

How to rate your observations quickly

To rate your observations quickly as you write them, use the tab and arrow keys:

Type > Tab > Arrow⇅ > Tab

You can enter numbers manually if you find this quicker, but once you get the hang of the process above you'll probably stick with it.

How to get the most out of rating observations

Before you start a project, it's important for you and your team to decide exactly what the significance rating will measure. What constitutes 'significance' for you will depend on things like your own research practice, the nature of your project, what your objectives are, and the big-picture insights you'd like to get. 

You could use the Project Notes section to record your rating key, so that everyone — both the researchers and the interested stakeholders — is clear about what the ratings mean.

Here's an example of how you can make significance ratings work for you.

Example: Rate observations by usability implications

You might decide that your significance rating stands for the seriousness of the usability problem, so that 1=No usability implications and 5=Serious usability implications. You could then add your key to the Project Notes so that everyone involved in the project is up with the play:

Reframer starts tallying your ratings from the first observation of the first session, so you, and anyone else, can see results straight away on the Results Overview page. You can get to this page by clicking the 'Results' button in the top right of your Project and Session pages.

You'll see the significance rating tally on the right, next to your most frequently occurring tags and the themes created so far:

 

In this example, because we understand the ratings to represent usability implications, we can instantly see that most observations have at least moderate usability implications, and that 17 observations indicate serious usability implications.

Showing this chart to stakeholders will communicate instantly that there's work to do on the product. You can also use these ratings as a benchmark, so that after you've run your post-design research sessions, you can show how much the usability implications have reduced — thanks to your hard work, of course.

Other ideas for what 'significance' could represent

As we've mentioned, the meaning you give significance ratings will depend on many things, and be unique to your project. Here's some more ideas to get you thinking.

1=Participant found this easy and 5=Participant found this difficult

1=Participant happy and 5=Participant frustrated

1=No pain point and 5=Significant pain point

1=No relevance to business goals and 5=Very relevant to business goals

 

Ask Us a Question

If you couldn't find the information your were looking for within our knowledge base, email your question to support@optimalworkshop.com.