View  Edit  Attributes  History  Attach  Print  Search
Site / Usability Study

Assignment - Usability Study (15%)


Purpose

This assignment will give you first hand experience at understanding how to evaluate an existing user interface. You will learn how to interact with study participants, develop usability study tasks, understand what problems users face with the interface, and understand what successes they have. You will also learn how to document your findings.

Overview

You have been asked by a company to evaluate the user interface for one of the systems listed below. You will perform a usability study with four participants and utilize the think-aloud and constructive interaction techniques. You will create a usability report detailing the findings of your study and recommendations for system redesign.

Focus your usability study on the main features of the site. Include the frequent tasks people need to do plus the infrequent yet important tasks. You should avoid going beyond this; do not test the entire user interface as this can easily make the assignment too broad in scope.

You will be evaluating the system Facebook Live for this assignment. The assignment must be done in pairs. You can create a 'usability study account' for the system if you do not want to use your own account. You should use the interface and become familiar with it.

Steps

1. Plan the Study: Plan out the usability study by creating:

  • Pre-Test Questionnaire: include 8-10 questions that ask the user about their general demographics, computer skills, how often they have used the system in question, and what they think of the system in question
  • Post-Test Questionnaire: include questions that ask the user how they felt about the usability of the system, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
  • Representative Tasks: create a set of realistic tasks that the users will do during the study. Ideally you would generate these from randomly sampling users about their tasks, or performing some design ethnography or interviews prior to the study. It is fine to generate some tasks based on your own experiences with the software though (at least for the assignment). So try it out, play with it, and then create some realistic user tasks. Make sure these vary from common tasks to infrequent yet very important ones. Create 7-9 different tasks. Choose tasks that contain a range of interactions - you will be graded on how well you come up with appropriate tasks.

Some tasks should be done in a lab setting, while others should be done in the outdoors where one may be more likely to use the system. You will need to translate what you know from conducting usability studies in lab settings into conducting them outside of the lab.

2. Pilot Study: Try out your usability study with someone before you actually do it with real participants. This is important so you are sure your tasks and questionnaires make sense, you can run the study in less than one hour, and that users won't become extremely frustrated.

3. Find Participants: Find family members or friends who are willing to be participants in your usability study. Try to choose a range of users (e.g., novices, experts, young, old). Your study participants cannot be students at SFU. This is required because I really want you to explore the system from the perspective of someone who is not overly familiar with computers.

4. Perform the Study: You need to run the study with 4 PEOPLE IN TOTAL (2 Think-aloud separately, 2 Constructive Interaction together).

  • Perform the usability study by first having the participant complete the pre-test questionnaire. Have two participants perform the tasks using the think aloud protocol. Have the remaining two participants perform the tasks using the constructive interaction protocol. Ensure that one participant will act as the novice and the other as the coach. It works best if the novice is actually a novice and the coach is an experienced user. Otherwise the coach will just do everything while the novice watches.

Throughout the study, try to get an understanding of what the user's conceptual model is of the system. It is a good idea to ask users about their conceptual model when they first see the interface.

While the experimenter should not help the subject with the task, there are a few exceptions to this rule. If a subject has problems getting started, record the problems and give them a hint to get going. This is OK, because if they can't get started, they will not be able to do the tasks! If a subject cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task. Or, give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, record all problems.

Collect data using one or more of the observational methods discussed in class (e.g., paper notes, audio, video).

After completing the tasks, have the participant complete the post-test questionnaire.

5. Interview and Transcription: Conduct a short retrospective interview to understand where the participant feels they had the most problems with the interface. You will need to generate a set of interview questions and ask them in a semi-structured manner. The interview needs to be audio recorded and then you must transcribe the interview. This means you need to type it out verbatim, including a note of which person is speaking (the interviewer or the participant). You can use a service like Temi as a transcription aid, but you should correct mistakes that Temi makes.

6. Analysis: Analyze your findings to see what usability problems are common amongst users. You may want to use affinity diagramming.

7. Report: Create a report that details your findings. This should include detailing the core problems people had with the interface and suggestions for redesign (see below for more detail).

Deliverables

You will provide a detailed report - maximum 5 pages in length, single spaced (include appendices outside of this page limit). It should include the following sections:

  • Introduction: describe the situation you are studying and why
  • Methodology: describe your study methods
    • explain that you performed a usability study and describe your study steps
    • describe your participants' demographics briefly
  • Observations: describe your observations and where appropriate use data to articulate what you saw (e.g., participant quotes, interview data, questionnaire data, screen captures)
  • Interpretations: describe the strengths and weaknesses of the system. This should be more than a simple checklist. Try to generalize them when all possible (e.g., if each screen is difficult to navigate through, then rather than describing each screen as having a problem, describe this as a general navigation issue with the user interface).
  • Suggested Improvements: describe five key improvements that could be made to improve the interface. Don't provide nit-picky improvements (e.g., I would change the button color to blue). Try to provide high level improvements (e.g., The buttons are not easy to see so I would improve the color contrast in the interface in several ways.)
  • Conclusion: summarize briefly and conclude elegantly
  • Appendix 1: include all raw data (e.g., handwritten notes regardless of how messy)
  • Appendix 2: include interview transcriptions

Be sure to complete the team member evaluations here. Leave the team # blank in the form. This is due at the same time as the assignment.