DR Determining Research Method for the FAQ Supporting My Projects ‐ General Comprehension Usability Testing - hackforla/tdm-calculator GitHub Wiki

This is a record in the Decision Records on Solutions Adopted.

Issue

Issue #2456

Problem Statement

Determining the best Research Method for the FAQ Supporting My Projects ‐ General Comprehension Usability Testing

Potential Solution

The two possible usability testings that the UXR team determined as possible options for this test were Cloze and Highlighter Tests. We decided to utilize Highlighter Testing for this RP because it is more user-friendly within the context of a dogfooding approach. The Cloze Test requires users to have prior knowledge of the product to yield valuable feedback, while the Highlighter test does not require any prior knowledge. Below are the initial resources that informed this decision:

Resources for Content Testing/ Comprehension Usability Testing

Highlight Tests literary findings that influenced decision

Highlight testing: Best for getting direct content feedback

  • Source: Maze Guides, CHAPTER 2 - 8 Content testing methods you need to know: Step-by-step for evaluating UX content
  • A highlight test is particularly useful for pinpointing strengths and weaknesses in your content—whether that’s in terms of branding, customer confidence, comprehension, or any other aspect. It’s a versatile method that you can use to measure anything from comprehension to content effectiveness. To do a highlight test, you’ll need a sample of your text and a red and green highlighter. Next, choose what aspect of your content you’d like to test, such as tone or clarity. After giving participants your chosen text, ask them to highlight content that aligns with their expectations regarding the specific aspect you’re testing in green, and content that doesn’t in red.

Highlighter tests

  • Source: 3 Effective Methods for Content Tests (Beyond Usability Testing), Nikki Anderson-Stanier & Thumy Phan in People Nerds 2021
  • Finally, one of my favorite ways to test content is the highlighter method. Here, you ask users to indicate what’s clear and unclear by highlighting text. This test can help determine precisely what needs to be rewritten. It can also reduce the amount of text on a page to focus on what is essential to the user. The highlighter test is excellent for determining value proposition and what information is necessary to help users achieve their goals. Running a highlighter test is relatively simple:
Image

Highlighter method

  • Source: How to Do Content Testing? 5 Key Content Testing Methods That Help Improve Conversions, by Marek Strba, 2025 This method will help you find out if the copywriting content you created conveys the right tone of voice to the customer.
  • Determine your priorities in whether you want to sound friendly, professional, trustworthy, etc. Then put your content in a Google Document and ask the participants to highlight with green all words and sentences that made them feel that way. Any parts that they felt the opposite about or the ones they were confused about can be highlighted in red and yellow respectively. After the study, you’ll have colorful text which will perfectly represent how you’ve managed to convey the right tone of voice with your content. Eliminate or rewrite the phrases highlighted in red and try the test again to see if it helped.
Cloze Tests iterary findings that influenced decision

Cloze test: Best for measuring content comprehension

  • Source: Maze Guides, CHAPTER 2 - 8 Content testing methods you need to know: Step-by-step for evaluating UX content
  • Cloze testing is a content testing method in which participants fill in missing words from a passage. This method helps you evaluate how understandable your content is. You can design a cloze test by taking a 125- to 250-word piece of text and removing one every few words, typically every fifth or sixth. You present this text to a test participant and see how many of the missing words they can guess correctly. An overall score of 60% or above is deemed comprehensible enough to meet user needs.
Image

Cloze tests

  • Source: 3 Effective Methods for Content Tests (Beyond Usability Testing), Nikki Anderson-Stanier & Thumy Phan in People Nerds 2021
  • A cloze test is a fancy way of describing what some of us might remember as "Mad Libs." In this test, you take a sample of text, remove specific words, and ask participants to fill in what they believe the missing words are. Through this test, participants must rely on the context and their knowledge of your product. Cloze tests are great for determining how appropriate and understandable text is for your audience. These tests are beneficial when dealing with highly complex topics, such as legal or healthcare information. However, you can still use a cloze test to assess the understandability of any website. If you're interested in running a cloze test, follow these steps: Take a bit of text from your website, about 250 words. Use text you aren't sure about, or people have had problems with in the past. For example, if customer support gets many calls on how your product works, test the text currently on your website about that subject. Once you choose the text, take out every fifth word and replace it with a blank space. You will, ideally, have around 25 blanks in your text and no more than 50. Mad Libs can be cognitively exhausting! Ask participants to fill in the blank spaces with the word they believe should be used. To score the test, count the number of correct answers and divide that by the total blank words. Then, turn this number into a percentage. For example, if I had 25 total blanks and a participant got 15 right, they would get a score of 60%.
Image

Cloze test

A Cloze test helps you measure reading comprehension of your content and find out how well users understand its context. To run such a test, you’ll need a piece of text, for example from your website’s homepage.

You then remove every 6th word from it and ask testers to fill out the gaps. If they get more than 60% of the words right, your text is comprehensive enough. See the below example of a Cloze test by NN/group (Facebook privacy policy used as basis):

Image
Testing content with users literary findings that influenced decision

How to Test Content with Users

  • Source: Kate Moran, 2021 for the Nielsen Norman Group

Use Moderated Instead of Unmoderated Studies

  • In remote unmoderated studies, participants work on their own, with no facilitator present. Even though this variation of usability testing is cheaper, we recommend that you do not use it for content studies. When trying to discover how people research a topic, compare offerings, and make decisions, the best approach is to conduct a moderated study, where a facilitator is present (physically or remotely). Facilitators can ensure that participants process the content naturalistically instead of approaching the task superficially. Content studies tend to have long stretches of time when the user is simply scanning page after page—in silence. When left alone (such as in a remote unmoderated test), participants may feel awkward and wonder whether they’re helpful. Without proper feedback and reassurance, participants may rush through the test and approach the task in superficial manner. This behavior is often reinforced by the shorter session times common in unmoderated testing (typically 20–30 minutes). Additionally, having a facilitator enables specific, personalized follow-up and clarification questions, such as “I noticed you hesitated on this paragraph, can you tell me what you were thinking?” The table below contains more examples of valuable follow-up questions for content studies.
Image

Feasibility Determination

Highlighter testing is the most feasible solution as it allows the UXR team to utilize dogfooding which is the better approach for testing with novice users and requires less time needed for recruitment and less resources utilized.

⚠️ **GitHub.com Fallback** ⚠️