View  Edit  Attributes  History  Attach  Print  Search
Site / Controlled Study Assignment

Assignment - Controlled Study (20%) - Individual



Purpose

This assignment is a hands-on exercise on quantitative evaluation. Its immediate purpose is to give you experience conducting a controlled experiment, performing a simple statistical analysis, interpreting the results, and considering its implications to design decisions. Its other purpose is to provide you with enough knowledge of the experimental process to help you understand and appreciate the interaction design evaluation and HCI literature that uses this methodology.

Overview

You have been asked by a company to evaluate two user interfaces for interacting with a Beam telepresence robot for driving navigating. One is an XBox or Playstation controller and one is a keyboard and mouse. You will perform a controlled study and create a study report detailing the findings of your study and recommendations for system design. You will complete the assignment individually.

Steps

1. Plan the Study: Plan out the controlled study by creating:

  • Hypotheses: You need to decide on hypotheses and null hypotheses. These need to include the keyboard layout as the independent variable. You will have two dependent variables: speed and accuracy.
  • Pre-Test Questionnaire: include questions that ask the user about their general demographics, computer skills, how often they have used user interfaces like those in question, and what they think of the user interfaces in question.
  • Post-Test Questionnaire: include questions that ask the user how they felt about the user interfaces, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
  • Representative Tasks: Participants will use both interfaces to drive a Beam telepresence robot. You need to design a series of tasks for them to do using the system. You will present these one week before the assignment is due, then, in class, we will provide feedback on a correct experimental setup. You will use the corrected setup, rather than your own task design.

2. Perform the Study: you will perform the study with 6 people. Normally, you'd run around 20 people in a study like this but we will reduce that number because it is a class assignment. Prior to having your participant perform the study tasks, you should first give them some time to play with the system so that they become familiar with it.

3. Analysis: Analyze your findings using known statistical methods that are taught in class.

4. Report: Create a report that details your findings.

Deliverables

You will provide a detailed report (maximum 2 pages in length, single spaced + appendices). It should include the sections below. You should also refer to the Marking Guide to see how we will grade each section.

  • Introduction: describe the situation you are studying and why
  • Methodology:
    • participants: describe your participants' demographics briefly
    • hypotheses: describe your hypothesis and rationale for them
    • method: explain that you performed a controlled study and describe your study steps
    • environment: describe where you conducted the study and how you controlled the environment
    • data collection: explain what data was collected and how
    • validity: describe any potential concerns with validity
  • Results: describe your data and statistical tests. Put all graphs in the appendix.

For your statistical testing, report:

  • What data we are running the t-test on
  • The result (p-value)
  • Whether you can reject the null hypothesis and what this means

For example (M is mean, SD is standard deviation):

“An independent-samples t-test was conducted to compare memory for words in sugar and no sugar conditions. There was a significant difference in the scores for sugar (M=4.2, SD=1.3) and no sugar (M=2.2, SD=0.84) conditions; p = 0.02. Thus, we can reject the null hypothesis at a 95% confidence interval (p < 0.05). These results suggest that sugar really does have an effect on memory for words. Specifically, our results suggest that when humans consume sugar, their memory for words increases.” (Source)

  • Discussion and Conclusions: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?
  • Appendix 1: include all raw data from your study. You need to Include your Excel spreadsheet twice. This might include printing out multiple data sheets/tabs within Excel depending on how you structure your spreadsheet.
  1. Show the Excel spreadsheet with the data output (e.g., the mean, standard deviation, p values)
  2. Show the Excel spreadsheet with formulas visible (to show formulas, see the instructions on this page)
  • Appendix 2: include any graphs that you feel are valuable. include images of your two test screens (e.g., the two keyboard layouts as shown on your phone)

Bring a printed copy of your report to class on the due date in the course calendar and hand it in at the beginning of class. Failure to hand it in at the start of class means it will be considered late.