View  Edit  Attributes  History  Attach  Print  Search
Site / Field Study

Assignment - Affective Evaluation (15%)

Resources

The Game

You and your team are going to use a field study evaluate four people’s subjective emotional and affective experiences to video game play. Your goal is to find out how the participants felt during and after game play. You will use this information to suggest improvements to the design of two aspects of the interface (see redesign goals below). You will run your study with 4 participants who will all play the SAME game. You must focus on one of these games using any console that they appear on:

  • NHL 14, 15, 16 or any sports game (such as EA FIFA 2016)
  • UFC 1
  • Borderlands II (or any other role-playing strategy game)
  • Left 4 Dead II
  • Mass Effect 2 (or any other first person shooter)
  • Brutal Legend (or any other firstperson, open world game)
  • Assassin's Creed (or any other quest-based adventure fantasy game)

This list of games was picked because of their varied genres as well as their availability on a number of different consoles/systems.

The assignment must be done in a team of three.

The Location

Ideally you will run this as a field study. This means that you should evaluate the players in any environment where you might naturally find them playing these kinds of games. For example, you could host a video game party with several friends and then conduct the evaluation in someone’s living or party room. If this is not possible, you can run the study as a lab style study in any space where you get set up the equipment (e.g., bookable room at SFU). In the latter case you should be explicit in your report that you ran a laboratory style study not a field study.

The Redesign Goals

The goals for this assignment are to explore player’s emotional and affective responses to playing video games in order to offer suggestions about what works and what needs redesign. You should report on key features to keep and key features to redesign with redesign suggestions.

The Evaluation Methods

You will be evaluating each of four participant’s affective experiences using two methods:

1. A simple post-play questionnaire called the Intrinsic Motivation Inventory (IMI) which measures the overall emotional experience of enjoyment of the game or activity.

2. The cued-recall debrief method is a first person, video-based method of reviewing and talking about the participant’s play session with the participant after the play session in order to elicit information about the short term affective states the player experienced during the game or activity. See paper above and notes on affect and emotion below. You must read the Bentley paper in order to understand how to use this method.

Study Protocol

Be familiar with this protocol before you start. A session will look like this:

1. Play: One participant will play a game for about 15 minutes. Their experience will be video taped in first person – the video should capture what the participant is seeing. That is you will need a camera directly behind them or mounted on their head to capture what they see while they play.

2. IMI Questionnaire: After they are done playing, the participant is to fill out the IMI questionnaire using the enjoyment, perceived competence, perceived choice and pressure/tension subscales.

3. Cued Debrief Recall: Immediately after the questionnaire part, is the review or debriefing part. One team evaluator and the participant will sit together and review the video of the play session and “debrief” or talk about what the participant was feeling during the play session. The participant will talk about this as they watch the video of the session. You may need to use prompts (like in think aloud) to encourage them to talk about what they were feeling. The debrief part should also be video or audio taped. This means you will need to find a quiet place where you can replay your video and discuss the session. Note that you need one camera to play the video (or alternative play back display) and a second camera to video tape the interview (focusing on audio).

When you are done the sessions you will have four videos of participants playing (shot in first person) and four videos of participants talking about their play (eight videos total). You are not done yet. You will still need to analyze all your data.

Data Analysis

IMI Enjoyment questionnaire: Score the questionnaire as given in the instructions below. You will need to use some simple descriptive statistics to do this (mean and standard deviation). Refer to web site for clarification

Notes: The Intrinsic Motivation Inventory (IMI) is a multidimensional measurement device intended to assess participants’ subjective experience related to a target activity in laboratory (or contrived field) experiments. The questionnaire assesses participants’ interest/enjoyment, perceived competence, perceived choice and felt pressure and tension while performing a given activity, thus yielding four subscale scores for our purposes.

The interest/enjoyment subscale is considered the self-report measure of intrinsic motivation. Although the overall questionnaire is called the Intrinsic Motivation Inventory, it is only the one subscale that assesses intrinsic motivation, per se. As a result, the interest/enjoyment subscale often has more items on it that do the other subscales.

The perceived choice and perceived competence concepts are theorized to be positive predictors of both self-report and behavioral measures of intrinsic motivation.

Pressure/tension is theorized to be a negative predictor of intrinsic motivation.

Affective Content of Cued Recall Debriefing Session: After all four of the play-debrief sessions are finished, you will analyze the four debrief video/audios (i.e., not the video of playing). To do this you need two evaluators for each of the four sessions.

Two evaluators will each review each debriefing session to listen for affect comments. They will record these comments on a data sheet.

Have two evaluators individually (i.e., separately) review each participant’s debriefing session. Focus on what the participant said (i.e., audio only). Your goal is to write down (on the data sheet) any comments the participants makes which contain affective words or contain affective expressions. If these words or phrases are related to controller mapping or screen layout design you should add a note about this. Count (add up) the total number of positive, neutral and negative comments. Classify into categories for comments about overall experience, controller mapping design, screen layout design and other game or interface features.

Inter-rater Reliability on Cued Recall Debrief Affect Comment Coding: For each session/participant, compare the results of the two evaluators/raters. Did you find the same kinds of affective statements? Did you find the same number of for each kind? Did you find the same overall number of positive, neutral and negative affect comments?

You also need to calculate the inter-rater reliability value for each session. Use the excel example file posted with this assignment to do so.

The inter-rater reliability or R value indicates how similar the results from the two evaluators are. For each session your R should be better than .90. If it is not, have the two evaluators work together to reach a better consensus.

You will have four inter-rater reliability values, one for each session.

Deliverables

You will provide a detailed report (maximum 5 pages in length, single spaced + appendices). It should include the following sections:

1. Introduction: describe the situation you are studying and why

2. Methodology: describe your study methods

  • participants: describe your participants' demographics briefly
  • method: explain that you performed a field study and describe your study steps
  • data collection: explain what data was collected and how

3. Results: describe your data and the generalized results

  • What does your analysis of the IMI/Enjoyment questionnaire data tell you about

each of the player’s experiences with game play.

  • What does the affective content analysis of participant’s cued recall debriefing

session tell you about each of the player’s experiences with game play?

  • For each participant, were there any parallels or inconsistencies between the

questionnaire and the cued recall debrief results?

4. Discussion and Implications: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?

5. Conclusion: conclude your report by summarizing the study and its findings.

6. Appendices: include all raw data from your study.

You will also create a video summary of the highlights, findings and redesign suggestions of your evaluation of user’s affective experiences. It is to be a maximum of 3 minutes in length.

Include the following information:

  • A narrated summary of your key findings using segments of your play sessions AND

debriefing sessions to illustrate key points and support your findings.

  • A narrated overview of your suggestions for redesign.

Bring a printed copy of your report to class on the due date in the course calendar and hand it in at the beginning of class. You need to include a URL to your summary video very clearly in your report. Email the URL to the video to both your instructor and TA. Videos can be posted on Vimeo, You Tube?, or your own personal site.

Failure to hand it in at the start of class means it will be considered late.