This assignment is a hands-on exercise on quantitative evaluation. Its immediate purpose is to give you experience conducting a controlled experiment, performing a simple statistical analysis, interpreting the results, and considering its implications to design decisions. Its other purpose is to provide you with enough knowledge of the experimental process to help you understand and appreciate the interaction design evaluation and HCI literature that uses this methodology.
Imagine you have been asked by a company to evaluate their product where they are interested in knowing if Interface Style A is better than Interface Style B. You will plan and perform a controlled study and create a study report detailing the findings of your study and recommendations for system design.
You will complete the assignment individually.
Steps
1. Plan the Study:
- Decide on a Design to Evaluate: Choose a design that you can evaluate and compare to a different design. For example, imagine driving a telepresence robot using one of two techniques: a mouse and keyboard vs. a gaming controller.
- Hypotheses and Variables: You need to decide on hypotheses and null hypotheses, as well as independent, dependent, and control variables.
- Pre-Test Questionnaire: include questions that ask the user about their general demographics, computer skills, how often they have used user interfaces like those in question, and what they think of the user interfaces in question.
- Post-Test Questionnaire: include questions that ask the user how they felt about the user interfaces, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
- Representative Tasks: Participants will use both interfaces . You need to design a series of tasks for them to do using the system. Try to construct tasks such that they will take about 10-20 minutes per participant.
2. Perform the Study: Normally, you’d run around 20 people in a study like this but we will reduce that number because it is a class assignment. You must run the study with at least 5 people. In total, this shouldn’t take more than a couple of hours to do if you plan your tasks appropriately.
3. Analysis: Analyze your findings using known statistical methods that are taught in class.
4. Report: Create a report that details your findings.
Deliverables
You will provide a detailed report that is 2 pages, single spaced (not including appendices and figures). It should include the following sections:
1. Introduction: describe the situation you are studying and why
2. Study Methods:
- participants: describe your participants’ demographics briefly
- hypotheses: describe your hypotheses and rationale for them
- method: explain that you performed a controlled study and describe your study steps including the tasks. use figures to illustrate the tasks.
- environment: describe where you conducted the study and how you controlled the environment
- data collection: explain how you collected your data
- validity: describe any potential concerns with validity; talk about how you ensured validity
- questionnaires: place your completed questionnaires in an appendix
For your statistical testing, report:
-
- What data we are running the t-test on
- The result (p-value)
- Whether you can reject the null hypothesis and what this means
For example (M is mean, SD is standard deviation):
“An independent-samples t-test was conducted to compare memory for words in sugar and no sugar conditions. There was a significant difference in the scores for sugar (M=4.2, SD=1.3) and no sugar (M=2.2, SD=0.84) conditions; p = 0.02. Thus, we can reject the null hypothesis at a 95% confidence interval (p < 0.05). These results suggest that sugar really does have an effect on memory for words. Specifically, our results suggest that when humans consume sugar, their memory for words increases.” (Source)
4. Discussion and Conclusions: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?
5. Appendix 1: include all raw data from your study. You need to Include your Excel spreadsheet twice. This might include printing out multiple data sheets/tabs within Excel depending on how you structure your spreadsheet:
– Show the Excel spreadsheet with formulas visible (to show formulas, see the instructions on this page)
6. Appendix 2: include your pre and post-test questionnaires
Upload the assignment on Canvas on the due date in the course calendar before the beginning of class. Failure to upload on time will be considered late.