Publications

Thesis

The Effects of Emotions and Their Regulation on Decision-making Performance in Affective Serious Games
Blekinge Institute of Technology, Karlskrona (Sweden), on 26 April 2019., supervisor: Veronica Sundstedt
abstract download


Journal publications


Jerčić P., Sundstedt V., (2019). PRACTICING EMOTION–REGULATION THROUGH BIOFEEDBACK ON THE DECISION–MAKING PERFORMANCE IN THE CONTEXT OF SERIOUS GAMES: A SYSTEMATIC REVIEW, Entertainment Computing, Elsevier.
abstract download

Jerčić P., Sennersten C., Lindley C. (2018). MODELING COGNITIVE LOAD AND PHYSIOLOGICAL AROUSAL THROUGH PUPIL DIAMETER AND HEART RATE, Multimedia Tools and Applications.
abstract download

Jerčić P., Wen W., Hagelbäck J., Sundstedt V., THE EFFECT OF EMOTIONS AND SOCIAL BEHAVIOR ON PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AUTONOMOUS ROBOTS, International Journal of Social Robotics, pp. 1-15.
abstract download


Conference publications


Jerčić, P., Hagelbäck, J., Lindley, C. (2018). PHYSIOLOGICAL AFFECT AND PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AN AUTONOMOUS ROBOT. In: Clua E., Roque L., Lugmayr A., Tuomi P. (eds) Entertainment Computing – ICEC 2018. Lecture Notes in Computer Science, vol 11112. Springer, Cham
abstract download

Jerčić, P., Astor, P. J., Adam, M., Hilborn, O., Schaff, K., Lindley, C. A., Sennersten, C., et al. (2012). A SERIOUS GAME USING PHYSIOLOGICAL INTERFACES FOR EMOTION REGULATION TRAINING IN THE CONTEXT OF FINANCIAL DECISION MAKING. ECIS 2012 Proceedings. AIS Electronic Library (AISeL).
abstract download

Jerčić P., Sennersten C., Lindley C., THE EFFECT OF COGNITIVE LOAD ON PHYSIOLOGICAL AROUSAL IN A DECISION-MAKING SERIOUS GAME, in IEEE Virtual Worlds and Games for Serious Applications (VS-Games), 2017 9th International Conference on, 2017, pp. 153-156.
abstract download

Adam M. T. P., Astor P. J., Jerčić P., Schaaff K. (2013). INTEGRATING BIOSIGNALS INTO INFORMATION SYSTEMS: A NEUROIS TOOL FOR IMPROVING EMOTION REGULATION. The Journal of Management Information Systems.
abstract download

Jerčić, P., Cederholm H. (2010). THE FUTURE OF BRAIN-COMPUTER INTERFACE FOR GAMES AND INTERACTION DESIGN. Biosplay workshop at Fun and Games Conference 2010.
abstract download

Sohaib A.T., Qureshi S., Hagelbäck J., Hilborn O., Jerčić P. (2013). EVALUATING CLASSIFIERS FOR EMOTION RECOGNITION USING EEG. 15:th International Conference on Human-Computer Interaction.
abstract download

M. Horvat, M. Dobrinić, M. Novosel and P. Jerčić (2018) Assessing emotional responses induced in virtual reality using a consumer EEG headset: A preliminary report, 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, 2018, pp. 1006-1010.
abstract download


Other publications


Peffer G., Cederholm H., Clough G., Jerčić P. (2010). EVALUATING GAMES DESIGNED TO IMPROVE FINANCIAL CAPABILITY. ECEL 2010 9th European Conference on e-Learning.
abstract download

Hagelbäck J., Hilborn O., Jerčić P., Johansson S. J., Lindley C. A., Svensson J, Wen W. (2013). PSYCHOPHYSIOLOGICAL INTERACTION AND EMPATHIC COGNITION FOR HUMAN-ROBOT COOPERATIVE WORK (PSYINTEC). Gearing Up and Accelerating Cross-Fertilization between Academic and Industrial Robotics Research in Europe, Springer Tracts in Advanced Robotics 94.
abstract download

Hilborn O., Eriksson J., Jerčić P., Lindley C., Petersson J., Sennersten C., A STUDY OF THE EFFECT OF EMOTION REGULATION IN A SHOOTING GAME USING BIO-FEEDBACK FROM ELECTROENCEPHALOGRAPHY FOR TRAINING IN EMOTION REGULATION, A Serious Game for Training in Emotion Regulation, p. 79.

Publication details


Jerčić, P. (2019). The Effects of Emotions and Their Regulation on Decision-making Performance in Affective Serious Games (Doctoral dissertation, Blekinge Institute of Technology).
download

Abstract

Emotions are thought to be one of the key factors that critically influence human decision-making. Emotion-regulation can help to mitigate emotion-related decision biases and eventually lead to a better decision performance. Serious games emerged as a new angle introducing technological methods to practicing emotion-regulation, where meaningful biofeedback information communicates player's affective states to a series of informed gameplay choices. These findings motivate the notion that in the decision context of serious games, one would benefit from awareness and regulation of such emerging emotions.

This thesis explores the design and evaluation methods for creating serious games where emotion-regulation can be practiced using physiological biofeedback measures. Furthermore, it investigates emotions and the effect of emotion-regulation on decision performance in serious games. Using the psychophysiological methods in the design of such games, emotions and their underlying neural mechanism have been explored.

The results showed the benefits of practicing emotion-regulation in serious games, where decision-making performance was increased for the individuals who down-regulated high levels of arousal while having an experience of positive valence. Moreover, it increased also for the individuals who received the necessary biofeedback information. The results also suggested that emotion-regulation strategies (i.e., cognitive reappraisal) are highly dependent on the serious game context. Therefore, the reappraisal strategy was shown to benefit the decision-making tasks investigated in this thesis. The results further suggested that using psychophysiological methods in emotionally arousing serious games, the interplay between sympathetic and parasympathetic pathways could be mapped through the underlying emotions which activate those two pathways. Following this conjecture, the results identified the optimal arousal level for increased performance of an individual on a decision-making task, by carefully balancing the activation of those two pathways. The investigations also validated these findings in the collaborative serious game context, where the robot collaborators were found to elicit diverse affect in their human partners, influencing performance on a decision-making task. Furthermore, the evidence suggested that arousal is equally or more important than valence for the decision-making performance, but once optimal arousal has been reached, a further increase in performance may be achieved by regulating valence. Furthermore, the results showed that serious games designed in this thesis elicited high physiological arousal and positive valence. This makes them suitable as research platforms for the investigation of how these emotions influence the activation of sympathetic and parasympathetic pathways and influence performance on a decision-making task.

Taking these findings into consideration, the serious games designed in this thesis allowed for the training of cognitive reappraisal emotion-regulation strategy on the decision-making tasks. This thesis suggests that using evaluated design and development methods, it is possible to design and develop serious games that provide a helpful environment where individuals could practice emotion-regulation through raising awareness of emotions, and subsequently improve their decision-making performance.


Jerčić P., Sundstedt V., (2019). PRACTICING EMOTION–REGULATION THROUGH BIOFEEDBACK ON THE DECISION–MAKING PERFORMANCE IN THE CONTEXT OF SERIOUS GAMES: A SYSTEMATIC REVIEW, *Entertainment Computing, Elsevier.*
download

Abstract

Evidence shows that emotions critically influence human decision-making. Therefore, emotion-regulation using biofeedback has been extensively investigated. Nevertheless, serious games have emerged as a valuable tool for such investigations set in the decision-making context. This review sets out to investigate the scientific evidence regarding the effects of practicing emotion-regulation through biofeedback on the decision-making performance in the context of serious games. A systematic search of five electronic databases (Scopus, Web of Science, IEEE, PubMed Central, Science Direct), followed by the author and snowballing investigation, was conducted from a publication's year of inception to October 2018. The search identified 16 randomized controlled experiment/quasi-experiment studies that quantitatively assessed the performance on decision-making tasks in serious games, involving students, military, and brain-injured participants. It was found that the participants who raised awareness of emotions and increased the skill of emotion-regulation were able to successfully regulate their arousal, which resulted in better decision performance, reaction time, and attention scores on the decision-making tasks. It is suggested that serious games provide an effective platform validated through the evaluative and playtesting studies, that supports the acquisition of the emotion-regulation skill through the direct (visual) and indirect (gameplay) biofeedback presentation on decision-making tasks.


Jerčić, P., Astor, P. J., Adam, M., Hilborn, O., Schaff, K., Lindley, C. A., Sennersten, C., et al. (2012). A SERIOUS GAME USING PHYSIOLOGICAL INTERFACES FOR EMOTION REGULATION TRAINING IN THE CONTEXT OF FINANCIAL DECISION MAKING. *ECIS 2012 Proceedings. AIS Electronic Library (AISeL).*
download

Abstract

Research on financial decision-making shows that traders and investors with high emotion regulation capabilities perform better in trading. But how can the others learn to regulate their emotions? 'Learning by doing' sounds like a straightforward approach. But how can one perform ?learning by doing? when there is no feedback? This problem particularly applies to learning emotion regulation, because learners can get practically no feedback on their level of emotion regulation. Our research aims at providing a learning environment that can help decision-makers to improve their emotion regulation. The approach is based on a serious game with real-time biofeedback. The game is settled in a financial context and the decision scenario is directly linked to the individual biofeedback of the learner?s heart rate data. More specifically, depending on the learner?s ability to regulate emotions, the decision scenario of the game continuously adjusts and thereby becomes more (or less) difficult. The learner wears an electrocardiogram sensor that transfers the data via Bluetooth to the game. The game itself is evaluated at several levels.


Jerčić, P., Hagelbäck, J., Lindley, C. (2018). PHYSIOLOGICAL AFFECT AND PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AN AUTONOMOUS ROBOT. *In: Clua E., Roque L., Lugmayr A., Tuomi P. (eds) Entertainment Computing – ICEC 2018. Lecture Notes in Computer Science, vol 11112. Springer, Cham*
download

Abstract

This paper sets out to examine how elicited physiological affect influences the performance of human participants collaborating with the robot partners on a shared serious game task; furthermore, to investigate physiological affect underlying such human-robot proximate collaboration. The participants collaboratively played a turn-taking version of a serious game Tower of Hanoi, where physiological affect was investigated in a valence-arousal space. The arousal was inferred from the galvanic skin response data, while the valence was inferred from the electrocardiography data. It was found that the robot collaborators elicited a higher physiological affect in regard to both arousal and valence, in contrast to their human collaborator counterparts. Furthermore, a comparable performance between all collaborators was found on the serious game task.


Jerčić P., Sennersten C., Lindley C., THE EFFECT OF COGNITIVE LOAD ON PHYSIOLOGICAL AROUSAL IN A DECISION-MAKING SERIOUS GAME, *in IEEE Virtual Worlds and Games for Serious Applications (VS-Games), 2017 9th International Conference on, 2017, pp. 153-156.*
download

Abstract

The aim of this paper is to investigate how a substantial cognitive load overshadows the physiological arousal effect, in an attempt to study cognitive abilities of participants engaged on decision-making tasks in serious games. Participants were engaged in a dynamic serious game environment displaying online biofeedback based on the physiological measurements of arousal. The pupil diameter was analyzed in relation to the heart rate during a challenging decision-making task. It was found that the moment when a substantial cognitive load overshadows the physiological arousal effect is observable on the pupil diameter in relation to the heart rate.


Jerčić P., Wen W., Hagelbäck J., Sundstedt V., THE EFFECT OF EMOTIONS AND SOCIAL BEHAVIOR ON PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AUTONOMOUS ROBOTS, *International Journal of Social Robotics, pp. 1-15.*
download

Abstract

The aim of this paper is to investigate performance in a collaborative human–robot interaction on a shared serious game task. Furthermore, the effect of elicited emotions and perceived social behavior categories on players’ performance will be investigated. The participants collaboratively played a turn-taking version of the Tower of Hanoi serious game, together with the human and robot collaborators. The elicited emotions were analyzed in regards to the arousal and valence variables, computed from the Geneva Emotion Wheel questionnaire. Moreover, the perceived social behavior categories were obtained from analyzing and grouping replies to the Interactive Experiences and Trust and Respect questionnaires. It was found that the results did not show a statistically significant difference in participants’ performance between the human or robot collaborators. Moreover, all of the collaborators elicited similar emotions, where the human collaborator was perceived as more credible and socially present than the robot one. It is suggested that using robot collaborators might be as efficient as using human ones, in the context of serious game collaborative tasks.


Jerčić P., Sennersten C., Lindley C. (2018). MODELING COGNITIVE LOAD AND PHYSIOLOGICAL AROUSAL THROUGH PUPIL DIAMETER AND HEART RATE, *Multimedia Tools and Applications.*
download

Abstract

This study investigates individuals’ cognitive load processing abilities while engaged on a decision-making task in serious games, to explore how a substantial cognitive load dominates over the physiological arousal effect on pupil diameter. A serious game was presented to the participants, which displayed the on–line biofeedback based on physiological measurements of arousal. In such dynamic decision-making environment, the pupil diameter was analyzed in relation to the heart rate, to evaluate if the former could be a useful measure of cognitive abilities of individuals. As pupil might reflect both cognitive activity and physiological arousal, the pupillary response will show an arousal effect only when the cognitive demands of the situation are minimal. Evidence shows that in a situation where a substantial level of cognitive activity is required, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. It is suggested that it might be possible to design serious games tailored to the cognitive abilities of an individual player, using the proposed physiological measurements to observe the moment when such dominance occurs.


Adam M. T. P., Astor P. J., Jerčić P., Schaaff K. (2013). INTEGRATING BIOSIGNALS INTO INFORMATION SYSTEMS: A NEUROIS TOOL FOR IMPROVING EMOTION REGULATION. *The Journal of Management Information Systems.*
download

Abstract

Traders and investors are aware that emotional processes can have material consequences on their financial decision performance. However, typical learning approaches for debiasing fail to overcome emotionally driven financial dispositions, mostly because of subjects' limited capacity for self-monitoring. Our research aims at improving decision makers' performance by (1) boosting their awareness to their emotional state and (2) improving their skills for effective emotion regulation. To that end, we designed and implemented a serious game-based NeuroIS tool that continuously displays the player's individual emotional state, via biofeedback, and adapts the difficulty of the decision environment to this emotional state. The design artifact was then evaluated in two laboratory experiments. Taken together, our study demonstrates how information systems design science research can contribute to improving financial decision making by integrating physiological data into information technology artifacts. Moreover, we provide specific design guidelines for how biofeedback can be integrated into information systems.


Jerčić, P., Cederholm H. (2010). THE FUTURE OF BRAIN-COMPUTER INTERFACE FOR GAMES AND INTERACTION DESIGN. *Biosplay workshop at Fun and Games Conference 2010.*
download

Abstract

In this paper we discuss the potential application areas and uses of modern Brain-Computer Interface technology such as the EPOC. We divide the discussing into two subgroups namely, Game design and Interaction design and hence discuss the future of these research areas in regards to such technology.


Sohaib A.T., Qureshi S., Hagelbäck J., Hilborn O., Jerčić P. (2013). EVALUATING CLASSIFIERS FOR EMOTION RECOGNITION USING EEG. *15:th International Conference on Human-Computer Interaction.*
download

Abstract

There are several ways of recording psychophysiology data from humans, for example Galvanic Skin Response (GSR), Electromyography (EMG), Electrocardiogram (ECG) and Electroencephalography (EEG). In this paper we focus on emotion detection using EEG. Various machine learning techniques can be used on the recorded EEG data to classify emotional states. K-Nearest Neighbor (KNN), Bayesian Network (BN), Artificial Neural Network (ANN) and Support Vector Machine (SVM) are some machine learning techniques that previously have been used to classify EEG data in various experiments. Five different machine learning techniques were evaluated in this paper, classifying EEG data associated with specific affective/emotional states. The emotions were elicited in the subjects using pictures from the International Affective Picture System (IAPS) database. The raw EEG data were processed to remove artifacts and a number of features were selected as input to the classifiers. The results showed that it is difficult to train a classifier to be accurate over large datasets (15 subjects) but KNN and SVM with the proposed features were reasonably accurate over smaller datasets (5 subjects) identifying the emotional states with an accuracy up to 77.78%.


M. Horvat, M. Dobrinić, M. Novosel and P. Jerčić (2018) Assessing emotional responses induced in virtual reality using a consumer EEG headset: A preliminary report, *41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, 2018, pp. 1006-1010.*
download

Abstract

We report on a pilot study involving emotion elicitation in virtual reality (VR) and assessment of emotional responses with a consumer-grade EEG device. The stimulation used HTC Vive VR system showing pictures from NAPS database within a specifically designed virtual environment. The stimulation consisted of two distinct sequences with 10 pictures of happiness and 10 pictures of fear. Each picture was contained in a separate virtual room that the participants traveled through along a preset path. The estimation employed EMOTIV EPOC+ 14-channel EEG headset and a custom-developed application. The software wirelessly received EEG signals from alpha, beta low, beta high, gamma and theta bands, time-stamped them and dynamically stored in a relational database for subsequent analysis. Our preliminary results show that statistically significant correlations between valence and arousal ratings of pictures and EEG bands are present but highly personalized. Simultaneous correct placement of VR and EEG headsets is demanding and precise localization of electrodes is difficult. In fact, if emotion estimation is not strictly necessary we recommend using devices with fewer electrodes. Nevertheless, we found the EEG to be effective. By acknowledging its limitations, and using the headset in the correct context, experiments involving emotions may be significantly amended.


Peffer G., Cederholm H., Clough G., Jerčić P. (2010). EVALUATING GAMES DESIGNED TO IMPROVE FINANCIAL CAPABILITY. *ECEL 2010 9th European Conference on e-Learning.*
download

Abstract

A multi-level approach to evaluation of serious games for financial capability is presented in this poster. The approach has been implemented as a toolkit in the context of xDelia, a collaborative project on game-based learning with a focus on emotions in financial decision making. The toolkit has been developed as part of a larger design an evaluation framework for the project. Four facets for financial c apability games are targeted by the evaluation: game design, financial capability, behaviour c hange, and learning with technology. The development of this toolkit is work in progress. An evaluation exercise is planned with existing financial capability games, where we want to assess the toolkit and refine its design to make it more effective for evaluators to use.


Hagelbäck J., Hilborn O., Jerčić P., Johansson S. J., Lindley C. A., Svensson J, Wen W. (2013). PSYCHOPHYSIOLOGICAL INTERACTION AND EMPATHIC COGNITION FOR HUMAN-ROBOT COOPERATIVE WORK (PSYINTEC). *Gearing Up and Accelerating Cross-Fertilization between Academic and Industrial Robotics Research in Europe, Springer Tracts in Advanced Robotics 94.*
download

Abstract

The aim of the PsyIntEC project is to explore affective and cognitive modeling of humans in human-robot interaction (Hri) as a basis for behavioral adaptation. To achieve this we have explored human affective perception of relevant modalities in human-human and human-robot interaction on a collaborative problem-solving task using psychophysiological measurements. The experiments conducted have given us valuable insight into the communicational and affective queues interplaying in such interactions from the human perspective. The results indicate that there is an increase in both positive and negative emotions when interacting with robots compared to interacting with another human or solving the task alone, but detailed analysis on shorter time segments is required for the results from all sensors to be conclusive and significant.