biofeedback, video games, video game, emotional state, Digital Games Research Association Conference, the game, Timo Saari, modifications, commercial integration, first person shooter games, Conference, Deighan, Jen Allanson, Stephen Fairclough, Research, Sarah Deighan, Video game development, incorporation, emotional arousal, authors, emotional response, horror games
The Integration of Biofeedback into video games
by Sarah Deighan HCC 729 Professor Kuber March 22, 2012
Deighan 1 The Integration of Biofeedback into Video Games INTRODUCTION Video games have come a long way since Pong. Recent blockbuster hits Halo and Dragon Age are excellent examples of innovation in the gaming industry. Video game development
has even helped drive improvement in hardware and software for personal computers
and boosted the distribution of new interface technology. The Nintendo Wii is one illustration of video games exponentially increasing the speed with which a new technology
was adopted. The more active interaction method was adopted much more rapidly than it might have been without such an intense advertising push. The integration of biofeedback into commercial video games However, there are challenges that would need to be overcome. BIOMETRIC DATA Biofeedback comes in a variety of forms including, but not limited to, heart rate
, Galvanic Skin Response (GSR), Electrocardiogram (ECG), Electromyography (EMG), Pupillography, Electrooculography, Spirometry, and Electroencephalography. In Psychophysiological Recording, the authors cover the recording and basic analysis of these types of biofeedback. The theme throughout the analysis sections is the recognition of patterns. The level of a person's concentration is based on the frequency of the different patterns of waves. It is deviation from the norm that tells the most about a person's reaction. For example, surprise can be indicated through brainwaves by the amplitude of a segment of the brainwave pattern called P300 (91). For most physiological data, an increase in the frequency of the patterns or the amplitude of the waves means an increase in emotional arousal
. This can be used to identify an increase in stress, frustration, or excitement. It would depend upon the particular biofeedback being collected, but
Deighan 2 this theme is most readily apparent in GSR, heart rate, EEG, and EMG. They are some of the simplest forms of biofeedback to gather and read for this reason (Stern, Ray, and Quigley). The biofeedback produced by the human body creates a veritable wealth of possible inputs for technological systems. There are serious issues at the present with the incorporation of biofeedback into commercial video games. In Physiological Computing, the authors cover each type of biofeedback collection and analysis of the patterns in reasonable detail. However, the also admit that some variations of pattern, like irregular rhythms of a heart, may not be indicative at all of emotional arousal. Instead, it may be indicative of health problem
s, diet, hormones, exercise level (181) or they may be caused by stray data, like artifacts in EEG data
(88). During their overview of the biofeedback relating to pupils, they provided a list of 23 stimuli that could lead to changes in pupil size (128). While the authors did provide a great deal of information about how to collect and analyze various types of biological data, they did not provide any suggestions on how to combine the data together in order to better triangulate a participant's emotional response
level. Generally people attempting to gather a type of biofeedback are using it as an indicator of a particular emotional response. Although the practical information that was provided in each chapter could be very practically useful, it doesn't go very deep into any theoretical issues, like attempting to get to the heart of the participant's emotional state. PARADIGMS OF BIOFEEDBACK INCORPORATION No matter how the biofeedback is used within a game, it is often referred to in generalities. While many authors lay out the various models of incorporating biofeedback into a game, very few bother to perform a general categorization. In "Affective Videogames and Modes of
Deighan 3 Affective Gaming," the authors lay out a categorization of biofeedback video games into three categories, "Assist Me," "Challenge Me," and "Emote Me." "Assist Me" is a paradigm in which the system monitors the user's frustration level and, should the user become too frustrated, begin helping them. The "Challenge Me" paradigm is where the system monitors the user for level of emotional response, linking the overall level of emotion or agitation with how much the user is being challenged. The computer increases the level of difficulty whenever the user is too relaxed in order to keep the game challenging. "Emote Me" is a paradigm in which the user is monitored by the system for a particular emotional response and alters the game in order to evoke a desired level of emotional response in the user (Gilleade, Dix, and Allanson). The names of the categories, named as though they were demands from an impetuous user, do not do the categorization system justice. This is a quite clear delineation of affective gaming paradigms. The authors did a thorough job in exploring samples of experimental video games employing biofeedback in an affective capacity. The paper was written in 2005 and there have been a relatively large number of fairly mainstream employments of biofeedback in commercial games since then. Additionally, while the categories are clearcut, even the authors admit that some games will fit multiple categories (6). The article itself does precisely what needed to be done; it carefully illustrated the difference between using biofeedback as a simple input device and affective gaming, or the implicit use of biofeedback to fine tune normal game processes. One drawback to the article is that they do not expound upon when affective gaming would be most effective.
Deighan 4 REALITIES OF BIOFEEDBACK INCORPORATION For many years, researchers have integrated biofeedback into video games in the form of the primary game controller. Games like BrainBall (Reuderink) , MindFlex, or the Star Wars Jedi MindForce simply measure the number of waves produced and use that as a direct input. It is possible for people to learn to gain a basic amount of control over the brain wave
s being produced (Pineda et al.), it may not enhance the experience. It may simply be a novel type of controller. One of the first examples of a biofeedback incorporation experiment involving first person shooter games was "Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game Using Biofeedback". The experiment was important because of the type of game and the modifications to the game were enhancements rather than the use of biofeedback as a controller. They employed two types of biometric data in creating in-game modifications, making the assessment of user emotional state more reliable. Many popular games are firstperson shooters, so they have an important place in the commercial video game market. The use of a first person shooter implies a correlation with other first person shooter games. Dekker and Champion even modified an existing commercial video game, which kept the gaming graphics at a commercially competitive level. Dekker and Champion's overall methods were sound but they made several mistakes. However, one of the valuable aspects of the article is the full, open and honest account of their experiment, including shortcomings. One possible problem with Dekker and Champion's experiment was the number of modifications that they chose to make based on the biofeedback signals. If a player was excited, the game would increase the character's speed until it triggered a "bullet time effect" (552) which let the character be substantially faster than its environment. On the other hand it would
Deighan 5 also trigger a "stealth mode effect" (552) if the player was calm that allowed the character to become invisible to enemies and increased the damage of the weapon. If a player became too excited, the screen would turn red and shake (552). However, there was also a modification to allow the game to notice when users were too calm and frighten them by creating new enemies. There were also "shader effects"(553) that reflected the user's emotional state. There were far too many alterations to the game based on the data to draw conclusions about the efficacy or any general bias about the individual effects. The changes probably made the level seem too different for the participants to fully understand what was going on. The data is little aside from users self reported findings on whether they noticed or appreciated the changes. Dekker and Champion collected a lot of data but were unable to use the majority of it because of issues with experiment implementation or choices to ignore relevant demographics like age and gender. Participants were all prescreened for video game playing, but they were not balanced by occupation, "gender, race, age or gaming experience" (554). Although the participant group might have been representative of the demographics of anyone who has played computer games, different types of games have different audiences. The genre of horror games is a fairly specific category of game that, while popular, is not universally popular. The horror genre may have affected results dramatically. The authors themselves noted that whether participants liked the genre of horror had a correlation to their levels of immersion and their reactions to several modifications (557). In this case, it appears that they actually had more people who liked the horror genre than not. Ideally, they would have chosen to include either participants who like horror or a balanced number between likes and dislikes, or likes, dislikes and a neutral category. Although the participant pool was unbalanced, the data supports a simple hypothesis about the ability of biofeedback data to reflect a user's emotional state. Although they
Deighan 6 could have created much richer results, they did confirm that biofeedback could be collected and used as a reasonably affective game input. In contrast, "The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games" also used two types of biological data into a first-person shooter game. However, they chose to test the different biofeedback signals against each other and have the modifications to the game be offsetting. The biofeedback data might make one aspect of the game easier, but another one more difficult to compensate. It reflected the player's state, but did not give the player a clear advantage. The theoretical mechanic was much simpler, while the player was excited, the character was quicker, but shook more. While the player was calmer, he was slower but steadier. This simple mechanic was a much cleaner one to measure and evaluate. Additionally, Kuikkaniemi et al. used the biofeedback data and quantified assessments of player performance to assess each aspect of the modifications throughout the game. This statistical assessment provides more practical information for researchers and video game developers
. For example, that some types of implicit biofeedback produce no effects and that other types may produce a better score, but do not alter the emotional responses that gamers experience (867). These discoveries will dramatically improve the ability of video game producers to effectively choose the type of biofeedback and the method of incorporation. Kuikkaiemi et al. even found problems with using electrodermal activity (EDA); participants attempting to influence it did much more poorly than those participants asked to modify their breathing (867). This has significant implications for any game creator attempting to craft a balanced video game because the type of biofeedback, like EDA or respiration, will have a dramatic impact on the experience of the gamer. Video game manufacturers carefully craft the experience of playing a game and must account for all of the drains on Human resources
Deighan 7 attention, very carefully. By illustrating gamers do poorly when trying to manipulate EDA, the authors have identified a "challenge" factor that could be added to a game or avoided by the manufacturer. Nacke et al. focused on identifying player preferences for different types of biofeedback. Although the type of game was different, an arcade shooter instead of first person shooter, they also pursued a considerably important aspect of commercial game construction, player preference. Nacke et al. found that players who enjoyed having biofeedback control
lers also enjoyed having multiple biofeedback controller
s because they felt it was more immersive (108). They also found that sensors should match the users mental models of associations, like flexing a leg in order to speed up the player's character (108). They noted that inputs that were harder to control were not perceived as working correctly (109). "Many participants noted that direct control was their preferred way of controlling the game mechanics as they felt it provided direct feedback to them and made the game more responsive" (110). The user's mental models and perceptions of the biofeedback controller affect their opinion of the game. If a biofeedback type is difficult to control, it could make a production video game seem as though it is not working properly. Therefore, a commercial video game producer would probably stay away from indirect biofeedback as a direct controller for a player character. Using Dekker and Champion's article as a rough basis for moving forward, researchers have been developing findings that will allow commercial video games to incorporate biofeedback. Some researchers, like Kuikkaiemi et al., have been leaning towards using known popular genres in order to ensure relevance. Although Nacke et al. did not utilize contemporary graphics, they did use an older and proven style of video game which also produced a reasonably representative gaming environment.
Deighan 8 FINANCES AND HARDWARE Money hampered some aspects of Dekker and Champion's experiment. The set-up utilized a Wild Divine Lightstone sensor which was "chosen due to a low cost..., unobtrusiveness..., and a SDK (software development
Kit...)" (Dekker and Champion 552). Unfortunately, while it was the best that they could do, it caused problems later on. Although the device was low profile, it covered three fingertips of one of the participant's hands. Originally, these sensors were worn on the participant's keyboard hand, but it hampered their interactions so greatly that the sensors were switched to the mouse hand. However, when worn on the mouse hand, the sensors blocked any tactile feedback from the mouse. Other aspects of the experiment, like lighting requirements for recording, also interfered with the users' experience. Since the user's emotional state would be affected by emotions that could be caused by these conditional factors, it might not have been an accurate representation of how well the game responded to biofeedback thresholds, which was a factor that affected gameplay. The financial issue is an important factor in the possible success of commercial video game integration. Any company that chooses to incorporate biofeedback into a video game will need to identify an existing recording device or build a device that will allow users to be unimpeded while still collecting reliable data. Biofeedback recording tools are getting less expensive, but few are cheap enough for students and unfunded research project
s to purchase. The cheap recording devices have drawbacks, like the Lightstone device. Since every person that owned a video game with biofeedback would need to purchase a specific recording device, these hardware limitations have significant implications for any commercial video game.
Deighan 9 ETHICS OF BIOFEEDBACK Biofeedback, in its many forms and applications, cannot just be interpreted to understand context but could, on occasion be misused. As mentioned before, some patterns indicate unusual physiology or possible medical issues. Biofeedback data can be incredibly personal, identifiable, and could even be used against the participant by an organization like an insurance company
. When the data being collected is so sensitive, I would expect a large chapter in a book like Psychophysiological Recording to be dedicated to the proper care and handling that the data and participants should receive. Instead, the authors dedicated a meager six pages to guidelines written using terminology that must be unraveled. It is incredibly disappointing that such an important chapter would be left in such a state. In the next edition, the safety and ethics section should be given greater consideration. Most researchers do not consider the ramifications of future experiments on game players' lives. For example, Dekker and Champion claimed that males' biofeedback showed much larger physiological responses to game stimuli and females' signals were much more steady, even though their presented data did not even include gender. These unverifiable findings might lead someone to conclude that women with a high variation of biofeedback are somehow abnormal, or that men are more at risk of a heart attack during video games. While researchers are less likely to make unsubstantiated claims like the above, few experiments bother to dedicate a paragraph to a discussion about how Future research
should be treat participants' data or possible safeguards against the harvesting of data from a video game. By laying down ethical guidelines in the foundational experiments' reports, researchers could set a precedent that would protect users of future systems.
Deighan 10 CONCLUSION Researchers are taking significant steps forward in the incorporation of biofeedback into video games. The earlier experiments and papers have helped to define the genres and prove that biofeedback is a viable input for video games. Some of the early examples of experimentation are weaker in their goals and their implementation but those who followed in their wake have tightened up their experiments. More Current Research
has more defined goals, more quantitative data, and more practicality. Research, like that in "The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games" and "Biofeedback Game Design," is providing a foundation of knowledge that will allow commercial manufacturers to incorporate biofeedback effectively into their products. As this becomes more and more possible, reserachers need to consider the consequences of that eventuality. They need to include some guidance on the ethics of biofeedback's inclusion within sections of their papers dedicated to future research. There is one aspect of biofeedback that is not ready for commercial integration, the recording hardware. Dekker and Champion were the only researchers to use cheap, general use sensors and their participants' gameplay experience suffered because of the sensors. Time and resources would need to be dedicated to developing cheap, effective, and unobtrusive biofeedback sensors for integration into commercial games to become a viable option.
Deighan 11 Works Cited Allanson, Jennifer, and Stephen Fairclough. "A Research Agenda
for Physiological Computing." Interacting with Computers 16.5 (2004): 857-78. Print. Dekker, Andrew, and Erik Champion. "Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game Using Biofeedback." Situated Play, Proceedings of the DiGRA 2007 Conference. Digital Games Research Association Conference. 550-58. Gilleade, Kiel Mark, Alan Dix, and Jen Allanson. "Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me." Proceedings of DiGRA 2005 Conference: Changing Views Worlds in Play. Digital Games Research Association Conference. 2005. 1-7. Kuikkaniemi, Kai, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, and Niklas Ravaja. "The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games." CHI 2010: Brains and Brawn. Proc. of ACM Computer-Human Interaction, Atlanta, GA. ACM, 2010. 85968. Leeb, Robert, Claudia Keinrath, Doron Friedman, Christoph Guger, Christa Neuper, Maia Garau, Angus Antley, Anthony Steed, Mel Slater, and Gert Pfurtscheller. "Walking from Thoughts: Not the Muscles Are Crucial, but the Brain Waves!"Presence 2005. 25-32. Nacke, Lennart E., Michael Kalyn, Calvin Lough, and Regan Mandryk. "Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction." CHI 2011. Proc. of ACM Computer-Human Interaction, Vancouver, BC, Canada
. 2011. 103-12. Pineda, J. A., D. S. Silverman, A. Vankov, and J. Hestenes. "Learning to Control Brain Rhythms: Making A Brain-Computer Interface Possible." Neural Systems and Rehabilitation Engineering IEEE Transactions
11.2 (June 2003): 181-84. Rani, Pramila, Nilanjan Sarkar, and Changchun Liu. "Maintaining Optimal Challenge in Computer Games through Real-Time Physiological Feedback." The 11th Human-Computer Interaction
International. Las Vegas
, NV. 2005. 1-7. Ravaja, Niklas, Timo Saari, Jari Laarni, Kari Kallinen, Mikko Salminen, Jussi Holopainen, and Aki Jдrvinen. "The Psychophysiology of Video Gaming: Phasic Emotional Responses to Game
Deighan 12 Events." Proceedings of the DiGRA 2005 Conference: Changing Views - Worlds in Play. Digital Games Research Association Conference. 2005. 1-13. Reuderink, B. (2008). Games and Brain-computer Interfaces: the State of the Art. Rovers, A. F., L. M.G Feijs, G. J.M. Van Boxtel, and P. J.M Cluitmans. "Flanker Shooting Game; Model-Based Design of Biofeedback Games." InterNational Conference
on Designing Pleasurable Products and Interfaces, DPPI09. Compiegne University of Technology
, Compiegne, France. 2009. Stern, Robert M., William J. Ray, and Christopher M. Davis. Psychophysiological Recording. 2nd ed. New York: Oxford UP, 1980.