You and your team are going to evaluate a video game’s user interface for emotional and affective experiences. Your goal is to find out how participants felt during and after game play. You will use this information to suggest improvements to the design of the game.
The assignment must be done individually; however, there is one portion where you must pair up with a classmate.
The Location
Ideally you will run this as a field study. This means that you should evaluate the players in any environment where you might naturally find them playing these kinds of games. For example, you could host a video game party with several friends and then conduct the evaluation in someone’s living or party room. Due to COVID-19 restrictions, you can run the study as a lab style study in any space where you get to set up the equipment and would be safe in (e.g., at home with your close social bubble).
The Evaluation Methods
You will be evaluating each of participant’s affective experiences using two methods:
1. A simple post-play questionnaire called the Intrinsic Motivation Inventory (IMI) which measures the overall emotional experience of enjoyment of the game or activity.
2. The cued-recall debrief method is a first person, video-based method of reviewing and talking about the participant’s play session with the participant after the play session in order to elicit information about the short term affective states the player experienced during the game or activity.
Study Protocol
A study session should look like this for each participant. You will run the study with just one participant.
1. Play: One participant will play a game for about 15 minutes. Their experience will be video recorded in first person – the video should capture what the participant is seeing. That is you will need a camera directly behind them or mounted on their head to capture what they see while they play.
2. IMI Questionnaire: After they are done playing, the participant is to fill out the IMI questionnaire using the enjoyment, perceived competence, perceived choice and pressure/tension subscales.
3. Cued Debrief Recall: Immediately after the questionnaire part is the review or debriefing part. One team evaluator and the participant will sit together and review the video of the play session and “debrief” or talk about what the participant was feeling during the play session. The participant will talk about this as they watch the video of the session. You may need to use prompts (like in think aloud) to encourage them to talk about what they were feeling. The debrief part should also be video or audio recorded. This means you will need to find a quiet place where you can replay your video and discuss the session. Note that you need one camera to play the video (or alternative play back display) and a second camera to video record the interview (focusing on audio).
When you are done the session you will have a video of the participant playing (shot in first person) and a video of the participant talking about their play (shot in third person).
Data Analysis
1. IMI Enjoyment Questionnaire
The Intrinsic Motivation Inventory (IMI) is a multidimensional measurement device intended to assess participants’ subjective experience related to a target activity in laboratory (or contrived field) experiments. The questionnaire assesses participants’ interest/enjoyment, perceived competence, perceived choice and felt pressure and tension while performing a given activity, thus yielding four subscale scores for our purposes.
The interest/enjoyment subscale is considered the self-report measure of intrinsic motivation. Although the overall questionnaire is called the Intrinsic Motivation Inventory, it is only the one subscale that assesses intrinsic motivation, per se. As a result, the interest/enjoyment subscale often has more items on it that do the other subscales. The perceived choice and perceived competence concepts are theorized to be positive predictors of both self-report and behavioral measures of intrinsic motivation. Pressure/tension is theorized to be a negative predictor of intrinsic motivation. You will score the IMI questionnaire using the procedure described in class.
2. Affective Content of Cued Recall Debriefing Session
After the play-debrief sessions are finished, you will analyze the debrief video/audios (i.e., not the video of playing). To do this you need two evaluators – thus, you need to pair up with someone else from the class for this part of the assignment.
Two evaluators will each review each debriefing session to listen for affect comments. They will record these comments on a data sheet. Have two evaluators individually (i.e., separately) review each participant’s debriefing session. Focus on what the participant said (i.e., audio only). Your goal is to write down (on the data sheet) any comments the participants makes which contain affective words or contain affective expressions. Count (add up) the total number of positive, neutral and negative comments.
3. Inter-rater Reliability on Cued Recall Debrief Affect Comment Coding
For each session/participant, compare the results of the two evaluators/raters. Did you find the same kinds of affective statements? Did you find the same number of for each kind? Did you find the same overall number of positive, neutral and negative affect comments?
You also need to calculate the inter-rater reliability value for each session. The inter-rater reliability or R value indicates how similar the results from the two evaluators are. For each session your R should be better than .90. If it is not, have the two evaluators work together to reach a better consensus.
Deliverables
You will provide a detailed report (maximum 2 pages in length, single spaced + appendices). It should include the following sections:
1. Introduction: describe the situation you are studying and why
2. Methodology: describe your study methods
- participants: describe your participants’ demographics briefly
- method: explain that you performed a field study and describe your study steps
- data collection: explain what data was collected and how
3. Results: describe your data and the generalized results
- What does your analysis of the IMI/Enjoyment questionnaire data tell you about each of the player’s experiences with game play?
- What does the affective content analysis of participant’s cued recall debriefing session tell you about each of the player’s experiences with game play?
- For each participant, were there any parallels or inconsistencies between the questionnaire and the cued recall debrief results?
4. Discussion and Implications: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?
5. Conclusion: conclude your report by summarizing the study and its findings.
6. Appendices: include all raw data from your study.
upload your assignment on Canvas before class time on the due date in the course calendar. Failure to upload on time means it will be considered late.