Sept 15Introduction

  • Introduction of the course, assignments, and textbooks
  • Explanations of why we evaluate and the goal of evaluation frameworks
  • Analytic vs. participative evaluation methods
  • Objective vs. subjective evaluation methods
  • Formative vs. summative methods
  • Quantitative vs. qualitative methods
  • Research/study questions

Studio Lab Activities
Review Materials
Sept 22Controlled Experiments

  • Scientific Method
  • Hypothesis testing
  • Independent & dependent variables
  • Ethics of controlled experiments

  • Doing Psychology Experiments, Chapter 2: How to Do Experiments
  • Doing Psychology Experiments, Chapter 7: How to Decide Which Variables to Manipulate and Measure
Studio Lab Activities

Exercise Sheets

Assignment 1: Ethics (5%) Due
at the start of class.
Sept 29Reliability and Validity

  • Reliability, consistency, repeatability
  • Confounding variables
  • Face validity, construct validity, internal validity, external validity

  • Doing Psychology Experiments, Chapter 8: How to Decide on a Between-Subjects versus Within-Subject Design

Studio Lab Activities
Oct 6Statistics for Controlled Experiments

  • Descriptive statistics: mean, standard deviation, median, range
  • Basic Inferential tests: t-test
  • Graphing basics

  • Doing Psychology Experiments, Chapter 12: How to Interpret Experimental Results
  • Doing Psychology Experiments, Chapter 13: How to Report Experimental Results

Studio Lab Activities
Oct 13Evaluating Emotion and Affect

  • Measuring affect and emotion
  • Theories of emotion
  • Quantifying qualitative codes
  • Cued Debrief Recall

  • Bentley, T., Johnston, L., and Baggo, K. Evaluation using cued-recall debrief to elicit information about a user's affective experiences, Proc. OzCHI, ACM Press (2005). (PDF)

Studio Lab Activities
Assignment 2: Controlled Study (15%) Due
at the start of class.
Oct 20Usability Evaluations of Collaborative Systems

  • Collocated collaboration systems
  • Distributed collaboration systems
  • Selecting participants as pairs or small groups
  • Planning tasks for collocated and distributed collaborative systems
  • Analyzing group data

  • None

Studio Lab Activities
Oct 27Field Deployments

  • Planning and timeframes’
  • Pilot testing
  • Managing home visits
  • Minimum viable prototypes
  • Group vs. individual interviews
  • Data validity
  • Ethics for field evaluations

  • Judge and Neustaedter, Studying and Designing Technology for Domestic Life, Morgan Kaufmann (2015). Chapters 9, 10

Studio Lab Activities

  • Practice for exam
Assignment 3: Affective Evaluation (15%) Due
at the start of class.
Nov 3Midterm Exam
Choosing Evaluation Strategies

  • Usability evaluation considered harmful (some of the time)
  • Applicability of usability evaluation for various scenarios

  • Greenberg S., and Buxton, B. Usability Evaluation Considered Harmful (Some of the time), Proceedings of the ACM Conference on Computer Human Interaction, ACM Press (2008) (PDF)
  • Watch this video of the paper's presentation: the video

Nov 10Design Evaluation Presentations

  • Come to class with an updated plan for the study you want to run
Come to class with a company and product you want to evaluate. Bring a document of up to one page that lists:

  • Your team members and email addresses
  • The name of the organization that you plan to work with
  • The name and email address of a contact person at the organization
  • Details of how you have talked with them
  • An initial idea of the type of evaluation you could conduct and on what
Nov 17Design Evaluation Presentations

  • Come to class with an updated plan for the study you want to run
In this class we meet with every group and work on the study plan together
Nov 24Initial Report of Results

  • Come to class and talk about the initial results you are getting from participants
You should send us a complete set of draft study materials in a single PDF. This means that you should create an initial plan that includes details of:

  • Participants
  • Methods to be used, including all questionnaires
  • Data collection: what you will collect and why
  • Data analysis: how you will analyze your data
Dec 1 Feedback on the project

  • Come to class and talk about the final results
Dec 8Final Report Presentation

  • Final presentation of your project
Assignment 4: Project (40%) Due
by 11:59pm, Dec 9. Share a single PDF containing all materials into your group's Slack channel.