Publications

Thesis

LICENTIATE THESIS - DESIGN AND EVALUATION OF AFFECTIVE SERIOUS GAMES FOR EMOTION REGULATION TRAINING (at Blekinge Institute of Technology, Karlskrona (Sweden), on 24 April 2013., supervisor: Stefan Johansson)
abstract download


Journal publications


Jerčić P., Sennersten C., Lindley C. (2018). MODELING COGNITIVE LOAD AND PHYSIOLOGICAL AROUSAL THROUGH PUPIL DIAMETER AND HEART RATE, Multimedia Tools and Applications.
abstract download

Jerčić P., Wen W., Hagelbäck J., Sundstedt V., THE EFFECT OF EMOTIONS AND SOCIAL BEHAVIOR ON PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AUTONOMOUS ROBOTS, International Journal of Social Robotics, pp. 1-15.
abstract download


Conference publications


Jerčić, P., Hagelbäck, J., Lindley, C. (2018). PHYSIOLOGICAL AFFECT AND PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AN AUTONOMOUS ROBOT. In: Clua E., Roque L., Lugmayr A., Tuomi P. (eds) Entertainment Computing – ICEC 2018. Lecture Notes in Computer Science, vol 11112. Springer, Cham
abstract download

Jerčić, P., Astor, P. J., Adam, M., Hilborn, O., Schaff, K., Lindley, C. A., Sennersten, C., et al. (2012). A SERIOUS GAME USING PHYSIOLOGICAL INTERFACES FOR EMOTION REGULATION TRAINING IN THE CONTEXT OF FINANCIAL DECISION MAKING. ECIS 2012 Proceedings. AIS Electronic Library (AISeL).
abstract download

Jerčić P., Sennersten C., Lindley C., THE EFFECT OF COGNITIVE LOAD ON PHYSIOLOGICAL AROUSAL IN A DECISION-MAKING SERIOUS GAME, in IEEE Virtual Worlds and Games for Serious Applications (VS-Games), 2017 9th International Conference on, 2017, pp. 153-156.
abstract download

Adam M. T. P., Astor P. J., Jerčić P., Schaaff K. (2013). INTEGRATING BIOSIGNALS INTO INFORMATION SYSTEMS: A NEUROIS TOOL FOR IMPROVING EMOTION REGULATION. The Journal of Management Information Systems.
abstract download

Jerčić, P., Cederholm H. (2010). THE FUTURE OF BRAIN-COMPUTER INTERFACE FOR GAMES AND INTERACTION DESIGN. Biosplay workshop at Fun and Games Conference 2010.
abstract download

Sohaib A.T., Qureshi S., Hagelbäck J., Hilborn O., Jerčić P. (2013). EVALUATING CLASSIFIERS FOR EMOTION RECOGNITION USING EEG. 15:th International Conference on Human-Computer Interaction.
abstract download

M. Horvat, M. Dobrinić, M. Novosel and P. Jerčić (2018) Assessing emotional responses induced in virtual reality using a consumer EEG headset: A preliminary report, 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, 2018, pp. 1006-1010.
abstract download


Other publications


Peffer G., Cederholm H., Clough G., Jerčić P. (2010). EVALUATING GAMES DESIGNED TO IMPROVE FINANCIAL CAPABILITY. ECEL 2010 9th European Conference on e-Learning.
abstract download

Hagelbäck J., Hilborn O., Jerčić P., Johansson S. J., Lindley C. A., Svensson J, Wen W. (2013). PSYCHOPHYSIOLOGICAL INTERACTION AND EMPATHIC COGNITION FOR HUMAN-ROBOT COOPERATIVE WORK (PSYINTEC). Gearing Up and Accelerating Cross-Fertilization between Academic and Industrial Robotics Research in Europe, Springer Tracts in Advanced Robotics 94.
abstract download

Hilborn O., Eriksson J., Jerčić P., Lindley C., Petersson J., Sennersten C., A STUDY OF THE EFFECT OF EMOTION REGULATION IN A SHOOTING GAME USING BIO-FEEDBACK FROM ELECTROENCEPHALOGRAPHY FOR TRAINING IN EMOTION REGULATION, A Serious Game for Training in Emotion Regulation, p. 79.

Publication details


Jerčić, P. (2013). Design and Evaluation of Affective Serious Games for Emotion Regulation Training (Licentiate dissertation, Blekinge Institute of Technology).
download

Abstract

Emotions are thought to be a key factor that critically influences human decision-making. Emotion regulation can help to mitigate emotion related decision biases and eventually lead to a better decision performance. Serious games emerged as a new angle introducing technological methods to learning emotion regulation, where meaningful biofeedback information displays player's emotional state. This thesis investigates emotions and the effect of emotion regulation on decision performance. Furthermore, it explores design and evaluation methods for creating serious games where emotion regulation can be learned and practiced. The scope of this thesis was limited to serious games for emotion regulation training using psychophysiological methods to communicate user's affective information. Using the psychophysiological methods, emotions and their underlying neural mechanism have been explored. Through design and evaluation of serious games using those methods, effects of emotion regulation have been investigated where decision performance has been measured and analyzed. The proposed metrics for designing and evaluating such affective serious games have been exhaustively evaluated. The research methods used in this thesis were based on both quantitative and qualitative aspects, with true experiment and evaluation research, respectively. Serious games approach to emotion regulation was investigated. The results suggested that two different emotion regulation strategies, suppression and cognitive reappraisal, are optimal for different decision tasks contexts. With careful design methods, valid serious games for training those different strategies could be produced. Moreover, using psychophysiological methods, underlying emotion neural mechanism could be mapped to provide optimal level of arousal for a certain task. The results suggest that it is possible to design and develop serious game applications that provide helpful learning environment where decision makers could practice emotion regulation and subsequently improve their decision making.


Jerčić, P., Astor, P. J., Adam, M., Hilborn, O., Schaff, K., Lindley, C. A., Sennersten, C., et al. (2012). A SERIOUS GAME USING PHYSIOLOGICAL INTERFACES FOR EMOTION REGULATION TRAINING IN THE CONTEXT OF FINANCIAL DECISION MAKING. *ECIS 2012 Proceedings. AIS Electronic Library (AISeL).*
download

Abstract

Research on financial decision-making shows that traders and investors with high emotion regulation capabilities perform better in trading. But how can the others learn to regulate their emotions? 'Learning by doing' sounds like a straightforward approach. But how can one perform ?learning by doing? when there is no feedback? This problem particularly applies to learning emotion regulation, because learners can get practically no feedback on their level of emotion regulation. Our research aims at providing a learning environment that can help decision-makers to improve their emotion regulation. The approach is based on a serious game with real-time biofeedback. The game is settled in a financial context and the decision scenario is directly linked to the individual biofeedback of the learner?s heart rate data. More specifically, depending on the learner?s ability to regulate emotions, the decision scenario of the game continuously adjusts and thereby becomes more (or less) difficult. The learner wears an electrocardiogram sensor that transfers the data via Bluetooth to the game. The game itself is evaluated at several levels.


Jerčić, P., Hagelbäck, J., Lindley, C. (2018). PHYSIOLOGICAL AFFECT AND PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AN AUTONOMOUS ROBOT. *In: Clua E., Roque L., Lugmayr A., Tuomi P. (eds) Entertainment Computing – ICEC 2018. Lecture Notes in Computer Science, vol 11112. Springer, Cham*
download

Abstract

This paper sets out to examine how elicited physiological affect influences the performance of human participants collaborating with the robot partners on a shared serious game task; furthermore, to investigate physiological affect underlying such human-robot proximate collaboration. The participants collaboratively played a turn-taking version of a serious game Tower of Hanoi, where physiological affect was investigated in a valence-arousal space. The arousal was inferred from the galvanic skin response data, while the valence was inferred from the electrocardiography data. It was found that the robot collaborators elicited a higher physiological affect in regard to both arousal and valence, in contrast to their human collaborator counterparts. Furthermore, a comparable performance between all collaborators was found on the serious game task.


Jerčić P., Sennersten C., Lindley C., THE EFFECT OF COGNITIVE LOAD ON PHYSIOLOGICAL AROUSAL IN A DECISION-MAKING SERIOUS GAME, *in IEEE Virtual Worlds and Games for Serious Applications (VS-Games), 2017 9th International Conference on, 2017, pp. 153-156.*
download

Abstract

The aim of this paper is to investigate how a substantial cognitive load overshadows the physiological arousal effect, in an attempt to study cognitive abilities of participants engaged on decision-making tasks in serious games. Participants were engaged in a dynamic serious game environment displaying online biofeedback based on the physiological measurements of arousal. The pupil diameter was analyzed in relation to the heart rate during a challenging decision-making task. It was found that the moment when a substantial cognitive load overshadows the physiological arousal effect is observable on the pupil diameter in relation to the heart rate.


Jerčić P., Wen W., Hagelbäck J., Sundstedt V., THE EFFECT OF EMOTIONS AND SOCIAL BEHAVIOR ON PERFORMANCE IN A COLLABORATIVE SERIOUS GAME BETWEEN HUMANS AND AUTONOMOUS ROBOTS, *International Journal of Social Robotics, pp. 1-15.*
download

Abstract

The aim of this paper is to investigate performance in a collaborative human–robot interaction on a shared serious game task. Furthermore, the effect of elicited emotions and perceived social behavior categories on players’ performance will be investigated. The participants collaboratively played a turn-taking version of the Tower of Hanoi serious game, together with the human and robot collaborators. The elicited emotions were analyzed in regards to the arousal and valence variables, computed from the Geneva Emotion Wheel questionnaire. Moreover, the perceived social behavior categories were obtained from analyzing and grouping replies to the Interactive Experiences and Trust and Respect questionnaires. It was found that the results did not show a statistically significant difference in participants’ performance between the human or robot collaborators. Moreover, all of the collaborators elicited similar emotions, where the human collaborator was perceived as more credible and socially present than the robot one. It is suggested that using robot collaborators might be as efficient as using human ones, in the context of serious game collaborative tasks.


Jerčić P., Sennersten C., Lindley C. (2018). MODELING COGNITIVE LOAD AND PHYSIOLOGICAL AROUSAL THROUGH PUPIL DIAMETER AND HEART RATE, *Multimedia Tools and Applications.*
download

Abstract

This study investigates individuals’ cognitive load processing abilities while engaged on a decision-making task in serious games, to explore how a substantial cognitive load dominates over the physiological arousal effect on pupil diameter. A serious game was presented to the participants, which displayed the on–line biofeedback based on physiological measurements of arousal. In such dynamic decision-making environment, the pupil diameter was analyzed in relation to the heart rate, to evaluate if the former could be a useful measure of cognitive abilities of individuals. As pupil might reflect both cognitive activity and physiological arousal, the pupillary response will show an arousal effect only when the cognitive demands of the situation are minimal. Evidence shows that in a situation where a substantial level of cognitive activity is required, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. It is suggested that it might be possible to design serious games tailored to the cognitive abilities of an individual player, using the proposed physiological measurements to observe the moment when such dominance occurs.


Adam M. T. P., Astor P. J., Jerčić P., Schaaff K. (2013). INTEGRATING BIOSIGNALS INTO INFORMATION SYSTEMS: A NEUROIS TOOL FOR IMPROVING EMOTION REGULATION. *The Journal of Management Information Systems.*
download

Abstract

Traders and investors are aware that emotional processes can have material consequences on their financial decision performance. However, typical learning approaches for debiasing fail to overcome emotionally driven financial dispositions, mostly because of subjects' limited capacity for self-monitoring. Our research aims at improving decision makers' performance by (1) boosting their awareness to their emotional state and (2) improving their skills for effective emotion regulation. To that end, we designed and implemented a serious game-based NeuroIS tool that continuously displays the player's individual emotional state, via biofeedback, and adapts the difficulty of the decision environment to this emotional state. The design artifact was then evaluated in two laboratory experiments. Taken together, our study demonstrates how information systems design science research can contribute to improving financial decision making by integrating physiological data into information technology artifacts. Moreover, we provide specific design guidelines for how biofeedback can be integrated into information systems.


Jerčić, P., Cederholm H. (2010). THE FUTURE OF BRAIN-COMPUTER INTERFACE FOR GAMES AND INTERACTION DESIGN. *Biosplay workshop at Fun and Games Conference 2010.*
download

Abstract

In this paper we discuss the potential application areas and uses of modern Brain-Computer Interface technology such as the EPOC. We divide the discussing into two subgroups namely, Game design and Interaction design and hence discuss the future of these research areas in regards to such technology.


Sohaib A.T., Qureshi S., Hagelbäck J., Hilborn O., Jerčić P. (2013). EVALUATING CLASSIFIERS FOR EMOTION RECOGNITION USING EEG. *15:th International Conference on Human-Computer Interaction.*
download

Abstract

There are several ways of recording psychophysiology data from humans, for example Galvanic Skin Response (GSR), Electromyography (EMG), Electrocardiogram (ECG) and Electroencephalography (EEG). In this paper we focus on emotion detection using EEG. Various machine learning techniques can be used on the recorded EEG data to classify emotional states. K-Nearest Neighbor (KNN), Bayesian Network (BN), Artificial Neural Network (ANN) and Support Vector Machine (SVM) are some machine learning techniques that previously have been used to classify EEG data in various experiments. Five different machine learning techniques were evaluated in this paper, classifying EEG data associated with specific affective/emotional states. The emotions were elicited in the subjects using pictures from the International Affective Picture System (IAPS) database. The raw EEG data were processed to remove artifacts and a number of features were selected as input to the classifiers. The results showed that it is difficult to train a classifier to be accurate over large datasets (15 subjects) but KNN and SVM with the proposed features were reasonably accurate over smaller datasets (5 subjects) identifying the emotional states with an accuracy up to 77.78%.


M. Horvat, M. Dobrinić, M. Novosel and P. Jerčić (2018) Assessing emotional responses induced in virtual reality using a consumer EEG headset: A preliminary report, *41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, 2018, pp. 1006-1010.*
download

Abstract

We report on a pilot study involving emotion elicitation in virtual reality (VR) and assessment of emotional responses with a consumer-grade EEG device. The stimulation used HTC Vive VR system showing pictures from NAPS database within a specifically designed virtual environment. The stimulation consisted of two distinct sequences with 10 pictures of happiness and 10 pictures of fear. Each picture was contained in a separate virtual room that the participants traveled through along a preset path. The estimation employed EMOTIV EPOC+ 14-channel EEG headset and a custom-developed application. The software wirelessly received EEG signals from alpha, beta low, beta high, gamma and theta bands, time-stamped them and dynamically stored in a relational database for subsequent analysis. Our preliminary results show that statistically significant correlations between valence and arousal ratings of pictures and EEG bands are present but highly personalized. Simultaneous correct placement of VR and EEG headsets is demanding and precise localization of electrodes is difficult. In fact, if emotion estimation is not strictly necessary we recommend using devices with fewer electrodes. Nevertheless, we found the EEG to be effective. By acknowledging its limitations, and using the headset in the correct context, experiments involving emotions may be significantly amended.


Peffer G., Cederholm H., Clough G., Jerčić P. (2010). EVALUATING GAMES DESIGNED TO IMPROVE FINANCIAL CAPABILITY. *ECEL 2010 9th European Conference on e-Learning.*
download

Abstract

A multi-level approach to evaluation of serious games for financial capability is presented in this poster. The approach has been implemented as a toolkit in the context of xDelia, a collaborative project on game-based learning with a focus on emotions in financial decision making. The toolkit has been developed as part of a larger design an evaluation framework for the project. Four facets for financial c apability games are targeted by the evaluation: game design, financial capability, behaviour c hange, and learning with technology. The development of this toolkit is work in progress. An evaluation exercise is planned with existing financial capability games, where we want to assess the toolkit and refine its design to make it more effective for evaluators to use.


Hagelbäck J., Hilborn O., Jerčić P., Johansson S. J., Lindley C. A., Svensson J, Wen W. (2013). PSYCHOPHYSIOLOGICAL INTERACTION AND EMPATHIC COGNITION FOR HUMAN-ROBOT COOPERATIVE WORK (PSYINTEC). *Gearing Up and Accelerating Cross-Fertilization between Academic and Industrial Robotics Research in Europe, Springer Tracts in Advanced Robotics 94.*
download

Abstract

The aim of the PsyIntEC project is to explore affective and cognitive modeling of humans in human-robot interaction (Hri) as a basis for behavioral adaptation. To achieve this we have explored human affective perception of relevant modalities in human-human and human-robot interaction on a collaborative problem-solving task using psychophysiological measurements. The experiments conducted have given us valuable insight into the communicational and affective queues interplaying in such interactions from the human perspective. The results indicate that there is an increase in both positive and negative emotions when interacting with robots compared to interacting with another human or solving the task alone, but detailed analysis on shorter time segments is required for the results from all sensors to be conclusive and significant.