View  Edit  Attributes  History  Attach  Print  Search
Site / Controlled Study Assignment

Assignment - Controlled Study (20%) - Individual



Purpose

This assignment is a hands-on exercise on quantitative evaluation. Its immediate purpose is to give you experience conducting a controlled experiment, performing a simple statistical analysis, interpreting the results, and considering its implications to design decisions. Its other purpose is to provide you with enough knowledge of the experimental process to help you understand and appreciate the interaction design evaluation and HCI literature that uses this methodology.

Overview

You have been asked by a company to evaluate two user interfaces for interacting with a Beam telepresence robot for driving navigating. One is an XBox or Playstation controller and one is a keyboard and mouse. You will perform a controlled study and create a study report detailing the findings of your study and recommendations for system design. You will complete the assignment individually.

Steps

1. Plan the Study: Plan out the controlled study by creating:

  • Hypotheses and Variables: You need to decide on hypotheses and null hypotheses, as well as independent, dependent, and control variables.
  • Pre-Test Questionnaire: include questions that ask the user about their general demographics, computer skills, how often they have used user interfaces like those in question, and what they think of the user interfaces in question.
  • Post-Test Questionnaire: include questions that ask the user how they felt about the user interfaces, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
  • Representative Tasks: Participants will use both interfaces to drive a Beam telepresence robot. You need to design a series of tasks for them to do using the system. Try to construct tasks such that they will take about 10-20 minutes per participant. You will present these one week before the assignment is due through informal discussions in class.

Then, in class, we will provide feedback on a correct experimental setup. You will use the corrected setup, rather than your own task design to gather data.

2. Perform the Study: Normally, you'd run around 20 people in a study like this but we will reduce that number because it is a class assignment. During class, we will collect data from 12 students in the class. You will then use this data as your participant data.

3. Analysis: Analyze your findings using known statistical methods that are taught in class.

4. Report: Create a report that details your findings.

Deliverables

You will provide a detailed report. It should include the sections below. You should also refer to the Marking Guide to see how we will grade each section.

The following sections are due at the FIRST HAND-IN DEADLINE where you will be describing the ideal study you would run but not actually running it (maximum 2 pages, single spaced + appendices). Write everything in past tense as though you have already run the study.

  • Introduction: describe the situation you are studying and why
  • Methodology:
    • participants: describe your ideal participants' demographics briefly
    • hypotheses: describe your desired hypotheses and rationale for them
    • method: explain that you performed a controlled study and describe your planned study steps including the tasks. use figures to illustrate the task.
    • environment: describe where you would conduct the study and how you would control the environment
    • data collection: explain how you would collect your data
    • validity: describe any potential concerns with validity; talk about how you ensure validity
    • questionnaires: place your questionnaires in an appendix
  • Appendix 1: include your pre and post-test questionnaires

Bring a printed copy of your report to class on the due date in the course calendar and hand it in at the beginning of class. Failure to hand it in at the start of class means it will be considered late.

The following sections are due at the SECOND HAND-IN DEADLINE (maximum 2 pages single spaced + appendices). You will have results based on the study plan we come up with in class and the data given in class. You do not have to rewrite the first hand-in as part of the second deliverable.

  • Results: describe your data and statistical tests. Include any graphs to show your descriptive statistics.

For your statistical testing, report:

  • What data we are running the t-test on
  • The result (p-value)
  • Whether you can reject the null hypothesis and what this means

For example (M is mean, SD is standard deviation):

“An independent-samples t-test was conducted to compare memory for words in sugar and no sugar conditions. There was a significant difference in the scores for sugar (M=4.2, SD=1.3) and no sugar (M=2.2, SD=0.84) conditions; p = 0.02. Thus, we can reject the null hypothesis at a 95% confidence interval (p < 0.05). These results suggest that sugar really does have an effect on memory for words. Specifically, our results suggest that when humans consume sugar, their memory for words increases.” (Source)

  • Discussion and Conclusions: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?
  • Appendix 1: include all raw data from your study. You need to Include your Excel spreadsheet twice. This might include printing out multiple data sheets/tabs within Excel depending on how you structure your spreadsheet.
  1. Show the Excel spreadsheet with the data output (e.g., the mean, standard deviation, p values)
  2. Show the Excel spreadsheet with formulas visible (to show formulas, see the instructions on this page)

Bring a printed copy of your report to class on the due date in the course calendar and hand it in at the beginning of class. Failure to hand it in at the start of class means it will be considered late.