View  Edit  Attributes  History  Attach  Print  Search
Site / Controlled Study Assignment

Assignment - Controlled Study (15%) - Individual



Purpose

This assignment is a hands-on exercise on quantitative evaluation. Its immediate purpose is to give you experience conducting a controlled experiment, performing a simple statistical analysis, interpreting the results, and considering its implications to design decisions. Its other purpose is to provide you with enough knowledge of the experimental process to help you understand and appreciate the interaction design evaluation and HCI literature that uses this methodology.

Overview

You have been asked by a company to evaluate two user interfaces. You will perform a controlled study and create a study report detailing the findings of your study and recommendations for system design. You will complete the assignment individually.

You will be evaluating keyboard input on a smartphone device. Imagine that you have been asked by a company to decide whether it is better to continue with their current keyboard input technique or migrate back to T9 input, which was common before smartphones arrived in the marketplace.

  • The T9 keyboard input can be found at this web page. When you run your study, you will load up this web page on a smartphone and participants will use the keyboard emulator to type in phrases as part of the study tasks.
  • For the regular keyboard, you will need to create a text input setup on your smartphone that is as simple as you can get it. e.g., choose an application that just supports text input and nothing else.

Steps

1. Plan the Study: Plan out the controlled study by creating:

  • Hypotheses: You need to decide on hypotheses and null hypotheses. These need to include the keyboard layout as the independent variable. You will have two dependent variables: speed and accuracy.
  • Pre-Test Questionnaire: include questions that ask the user about their general demographics, computer skills, how often they have used user interfaces like those in question, and what they think of the user interfaces in question.
  • Post-Test Questionnaire: include questions that ask the user how they felt about the user interfaces, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
  • Representative Tasks: Participants will use both the T9 keyboard layout and the regular keyboard on a smartphone to type the following sentences. You should collect data for your dependant variables (speed and accuracy) by timing the user when they complete each sentence and also counting how many letters they misspelled at the end of each trial. You need to counter balance the ordering of the keyboards - half of the people should use the T9 layout first and half of the people should use the regular keyboard first.

The test sentences are:

  • the quick brown fox jumps over the lazy yellow dog
  • Samarth Singhal is my teaching assistant for this class
  • controlled experiments are one way to evaluate a computer system design

OR

  • the quick brown fox jump over the lazy yellow dog
  • come back next week and we will play some more
  • controlled experiments are one way to evaluate a computer system design

2. Perform the Study: you will perform the study with 6 people. Normally, you'd run around 20 people in a study like this but we will reduce that number because it is a class assignment. Prior to having your participant perform the study tasks, you should first give them some time to play with the T9 keyboard so that they become familiar with it.

3. Analysis: Analyze your findings using known statistical methods that are taught in class.

4. Report: Create a report that details your findings.

Deliverables

You will provide a detailed report (maximum 2 pages in length, single spaced + appendices). It should include the sections below. You should also refer to the Marking Guide to see how we will grade each section.

  • Introduction: describe the situation you are studying and why
  • Methodology:
    • participants: describe your participants' demographics briefly
    • hypotheses: describe your hypothesis and rationale for them
    • method: explain that you performed a controlled study and describe your study steps
    • environment: describe where you conducted the study and how you controlled the environment
    • data collection: explain what data was collected and how
    • validity: describe any potential concerns with validity
  • Results: describe your data and statistical tests. Put all graphs in the appendix.

For your statistical testing, report:

  • What data we are running the t-test on
  • The result (p-value)
  • Whether you can reject the null hypothesis and what this means

For example (M is mean, SD is standard deviation):

“An independent-samples t-test was conducted to compare memory for words in sugar and no sugar conditions. There was a significant difference in the scores for sugar (M=4.2, SD=1.3) and no sugar (M=2.2, SD=0.84) conditions; p = 0.02. Thus, we can reject the null hypothesis at a 95% confidence interval (p < 0.05). These results suggest that sugar really does have an effect on memory for words. Specifically, our results suggest that when humans consume sugar, their memory for words increases.” (Source)

  • Discussion and Conclusions: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?
  • Appendix 1: include all raw data from your study. You need to Include your Excel spreadsheet twice. This might include printing out multiple data sheets/tabs within Excel depending on how you structure your spreadsheet.
  1. Show the Excel spreadsheet with the data output (e.g., the mean, standard deviation, p values)
  2. Show the Excel spreadsheet with formulas visible (to show formulas, see the instructions on this page)
  • Appendix 2: include any graphs that you feel are valuable. include images of your two test screens (e.g., the two keyboard layouts as shown on your phone)

Bring a printed copy of your report to class on the due date in the course calendar and hand it in at the beginning of class. Failure to hand it in at the start of class means it will be considered late.