Behavioral Biometrics in Assisted Living: A Methodology for Emotion Recognition

  • S. Xefteris Distributed Knowledge and Media Systems Group, Department of Electrical and Computer Engineering, National Technical University of Athens, Greece
  • N. Doulamis Photogrammetry and Computer Vision Lab, Dept. of Rural and Survey Engineering, National Technical University of Athens, Greece
  • V. Andronikou Distributed Knowledge and Media Systems Group, Department of Electrical and Computer Engineering, National Technical University of Athens, Greece
  • T. Varvarigou Distributed Knowledge and Media Systems Group, Department of Electrical and Computer Engineering, National Technical University of Athens, Greece
  • G. Cambourakis Division of Communication, Electronic and Information Engineering, Department of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece
Keywords: ambient assisted living, dementia, emotion, image processing, telemedicine


Behavioral biometrics aim at providing algorithms for the automatic recognition of individual behavioral traits, stemming from a person’s actions, attitude, expressions and conduct. In the field of ambient assisted living, behavioral biometrics find an important niche. Individuals suffering from the early stages of neurodegenerative diseases (MCI, Alzheimer’s, dementia) need supervision in their daily activities. In this context, an unobtrusive system to monitor subjects and alert formal and informal carers providing information on both physical and emotional status is of great importance and positively affects multiple stakeholders. The primary aim of this paper is to describe a methodology for recognizing the emotional status of a subject using facial expressions and to identify its uses, in conjunction with pre-existing risk-assessment methodologies, for its integration into the context of a smart monitoring system for subjects suffering from neurodegenerative diseases. Paul Ekman’s research provided the background on the universality of facial expressions as indicators of underlying emotions. The methodology then makes use of computational geometry, image processing and graph theory algorithms for the detection of regions of interest and then a neural network is used for the final classification. Findings are coupled with previous published work for risk assessment and alert generation in the context of an ambient assisted living environment based on Service oriented architecture principles, aimed at remote web-based estimation of the cognitive and physical status of MCI and dementia patients.


Download data is not yet available.


Eurostat, “Population structure and ageing,” 2015, available at:


A. L. Busse, G. Gil, J. M. Santarém, W. J. Filho, “Physical activity and cognition in the elderly: A review”, Dement. Neuropsychol., Vol. 3, No. 3, pp. 204–208, 2009

A. Di Carlo, M. Baldereschi, L. Amaducci, S. Maggi, F. Grigoletto, G. Scarlato, D. Inzitari, “Cognitive impairment without dementia in older people: prevalence, vascular risk factors, impact on disability. the italian longitudinal study on aging”, J. Am. Geriatr. Soc., Vol. 48, No. 7, pp. 775–782, 2000

D. Laurin, R. Verreault, J. Lindsay, K. MacPherson, K. Rockwood, “Physical activity and risk of cognitive impairment and dementia in elderly persons”, Arch. Neurol., Vol. 58, No. 3, pp. 498–504, 2001

A. Bradford, M. E. Kunik, P. Schulz, S. P. Williams, H. Singh, “Missed and delayed diagnosis of dementia in primary care: prevalence and contributing factors”, Alzheimer Dis. Assoc. Disord., Vol. 23, No. 4, pp. 306–314, 2009

T. Kitwood, Dementia Reconsidered: the Person Comes First, Open University Press, 1999

M. Haritou, Y. Glickman, A. Androulidakis, S. Xefteris, A. Anastasiou, A. Baboshin, S. Cuno, and D. Koutsouris, “A technology platform for a novel home care delivery service to patients with dementia”, J. Med. Imaging Heal. Informatics, Vol. 2, No. 1, pp. 49–55, 2012

M. Torkamani, L. McDonald, I. S. Aguayo, C. Kanios, M.-N. Katsanou, L. Madeley, P. D. Limousin, A. J. Lees, M. Haritou, M. Jahanshahi, “A randomized controlled pilot study to evaluate a technology platform for the assisted living of people with dementia and their carers”, J. Alzheimers Dis., Vol. 41, No. 2, pp. 515–523, 2014

P. Rashidi, A. Mihailidis, “A survey on ambient-assisted living tools for older adults”, IEEE J. Biomed. Heal. Informatics, Vol. 17, No. 3, pp. 579–590, 2013

C. Frantzidis, C. Bratsas, C. L. Papadelis, E. Konstantinidis, C. Pappas, P. D. Bamidis, “Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli”, Inf. Technol. Biomed. IEEE Trans., Vol. 14, No. 3, pp. 589–597, 2010

F. Bellotti, B. Kapralos, K. Lee, P. Moreno-Ger, R. Berta, “Assessment in and of serious games: an overview,” Adv. Human-Computer Interact., Vol. 2013, pp. 1–11, 2013

E. I. Konstantinidis, A. S. Billis, C. Mouzakidis, V. Zilidou, P. E. Antoniou, P. D. Bamidis, “Design, implementation and wide pilot deployment of FitForAll: an easy to use exergaming platform improving physical fitness and life quality of senior citizens”, IEEE J. Biomed. Heal. Informatics, Vol. 20, No. 1, pp. 189 - 200, 2014

A. S. Billis, E. I. Papageorgiou, C. Frantzidis, E. I. Konstantinidis, P. D. Bamidis, “Towards a hierarchically-structured decision support tool for improving seniors’ independent living”, 6th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’13), pp. 1–4, Rhodes, Greece, May 29-31, 2013

A. Artikis, P. D. Bamidis, A. Billis, C. Bratsas, C. Frantzidis, V. Karkaletsis, M. Klados, E. Konstantinidis, S. Konstantopoulos, D. Kosmopoulos, H. Papadopoulos, S. Perantonis, S. Petridis, C. S. Spyropoulos, “Supporting tele-health and AI-based clinical decision making with sensor data fusion and semantic interpretation: The USEFIL case study”, International Workshop on Artificial Intelligence and NetMedicine, Montpellier, France, August 27, 2012

S. Xefteris, M. Haritou, K. Tserpes, A. Serretti, J. R. Llopart, R. Calati, T. Varvarigou, “Analysis of requirements and specifications for a monitoring system to support the self-management of dementia patients at home”, 3rd International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’10), Samos, Greece, June 23-25, 2010

S. Xefteris, A. Baboshin, K. Tserpes, A. Androulidakis, Y. Glickman, T. Varvarigou, M. Haritou, F. D’Andria, “Enabling risk assessment and analysis by event detection in dementia patients using a reconfigurable rule set”, 4th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’11), Crete, Greece, May 25-27, 2011

P. D. Bamidis, P. Fissler, S. G. Papageorgiou, V. Zilidou, E. I. Konstantinidis, A. S. Billis, E. Romanopoulou, M. Karagianni, I. Bearatis, A. Tsapanou, G. Tsilikopoulou, E. Grigoriadou, A. Ladas, A. Kyrillidou, A. Tsolaki, C. Frantzidis, E. Sidiropoulos, A. Siountas, S. Matsi, J. Papatriantafyllou, E. Margioti, A. Nika, W. Schlee, T. Elbert, M. Tsolaki, A. B. Vivas, I.-T. Kolassa, “Gains in cognition through combined cognitive and physical training: the role of training dosage and severity of neurocognitive disorder”, Front. Aging Neurosci., Vol. 7, Art. No. 152, pp. 1-15, 2015.

C. Styliadis, P. Kartsidis, E. Paraskevopoulos, A. A. Ioannides, P. D. Bamidis, “Neuroplastic effects of combined computerized physical and cognitive training in elderly individuals at risk for dementia : an eLORETA controlled study on resting states”, Neural Plast., Vol. 2015, Article ID 172192, 2015

A. S. Billis, N. Katzouris, A. Artikis, P. D. Bamidis, “Clinical decision support for active and healthy ageing: an intelligent monitoring approach of daily living activities”, 17th Portuguese Conference on Artificial Intelligence (EPIA 2015), Coimbra, Portugal, September 8-11, 2015

D. N. Monekosso, P. Remagnino, “Behavior analysis for assisted living”, IEEE Trans. Autom. Sci. Eng., Vol. 7, No. 4, pp. 879–886, 2010

R. A. Stucki, P. Urwyler, L. Rampa, R. Müri, U. P. Mosimann, T. Nef, “A web-based non-intrusive ambient system to measure and classify activities of daily living”, J. Med. Internet Res., Vol. 16, No. 7, Article ID e175, 2014

A. S. Billis, E. I. Papageorgiou, C. A. Frantzidis, M. S. Tsatali, A. C. Tsolaki, P. D. Bamidis, “A decision-support framework for promoting independent living and ageing well”, IEEE J. Biomed. Heal. Informatics, Vol. 19, No. 1, pp. 199–209, 2015

A. K. Jain, A. Ross, S. Prabhakar, “An introduction to biometric identification”, IEEE Trans. circuits Syst. Video Technol. Spec. issue image-and video-based biometrics, Vol. 14, No. 1, pp. 4-20, 2004

E. M. Tapia, S. S. Intille, K. Larson, Activity recognition in the home using simple and ubiquitous sensors. Springer, 2004

M. Mozer, “Lessons from an adaptive house”, in Smart Environments: Technologies, Protocols, and Applications, Hoboken, NJ, pp. 273–294, 2004

A. Steinhage, C. Lauterbach, “Monitoring movement behavior by means of a large area proximity sensor array in the floor”, 2nd Workshop on Behaviour Monitoring and Interpretation (BMI'08), Kaiserslautern, Germany, September 23, 2008

M. Muhlenbrock, O. Brdiczka, D. Snowdon, J. -L. Meunier, “Learning to detect user activity and availability from a variety of sensor data”, 2nd IEEE International Conference on Pervasive Computing and Communications (PerCom’04), pp. 13-22, Florida, USA, March 14-17, 2004

C. Darwin, P. Ekman, P. Prodger, The expression of the emotions in man and animals. Oxford University Press, USA, 1998

B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,” Pattern Recognit., Vol. 36, No. 1, pp. 259–275, 2003

S. Xefteris, A. Baboshin, K. Tserpes, A. Androulidakis, Y. Glickman, T. Varvarigou, M. Haritou, F. D’Andria, “Enabling risk assessment and analysis by event detection in dementia patients using a reconfigurable rule set”, 4th International Conference on PErvasive Technologies Related to Assistive Environments, (PETRA’11), Crete, Greece, May 25-27, 2011

P. Ekman, W. V Friesen, Unmasking the face: A guide to recognizing emotions from facial clues. Ishk, 2003

P. Ekman, R. J. Davidson, The nature of emotion: Fundamental questions. Oxford University Press, 1994

C. Darwin, The Expression of the Emotions in Man and Animals. 1872.

P. Ekman, W. V Friesen, P. Ellsworth, Emotion in the human face: Guidelines for research and an integration of findings, Oxford Pergamon Press, 1972

P. Ekman, W. V Friesen, M. O’Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-Bitti, and others, “Universals and cultural differences in the judgments of facial expressions of emotion”, J. Pers. Soc. Psychol., Vol. 53, No. 4, pp. 712-717, 1987

P. Ekman, “An argument for basic emotions”, Cogn. Emot., Vol. 6, No. 3–4, pp. 169–200, 1992

P. Ekman, Darwin and Facial Expression: A century of Research. New York: Academic Press, 1973

C. Izzard, The Face of Emotion. New York: Appleton-Century-Crofts, 1971.

P. Ekman, W. V. Friesen, P. Ellsworth, Emotion in the human face.Guidelines for research and an integration of findings, Pergamon Press, USA, 2013

H. Hwang, D. Matsumoto, “Evidence for the Universality of Facial Expressions of Emotion”, in Understanding Facial Expressions in Communication, Springer India, pp. 41–56, 2015,

I. E. Eibelsfeldt, Ethology, the Biology of Behavior . 1970

P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression”, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101, 2010

T. Kanade, J. F. Cohn, Y. Tian, “Comprehensive database for facial expression analysis”, 4th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53, 2000

V. Andronikou, S. Xefteris, T. Varvarigou, “A novel, algorithm metadata-aware architecture for biometric systems”, IEEE Workshop on Biometric Measurements and Systems for Security and Medical Applications (BIOMS), pp. 1–6, 2012

M. Jones, P. Viola, “Fast multi-view face detection”, Mitsubishi Electr. Res. Lab TR-20003-96, Vol. 3, 2003, available at

A. Okabe, B. Boots, K. Sugihara, S. N. Chiu, Spatial tessellations: concepts and applications of Voronoi diagrams, Vol. 501. John Wiley & Sons, 2009

F. P. Preparata, M. Shamos, Computational geometry: an introduction. Springer Science & Business Media, 2012.

B. Aronov, M. Dulieu, F. Hurtado, “Witness (Delaunay) graphs”, Comput. Geom., Vol. 44, No. 6–7, pp. 329–344, 2011

J. Gan, Y. Tao, “DBSCAN Revisited: Mis-Claim, Un-Fixability, and Approximation”, ACM SIGMOD International Conference on Management of Data, pp. 519–530, 2015

R. Xu, D. Wunsch, and others, “Survey of clustering algorithms”, IEEE Transactions on Neural Networks, , Vol. 16, No. 3, pp. 645–678, 2005

I. Kunttu, L. Lepistö, A. Visa, “Efficient Fourier shape descriptor for industrial defect images using wavelets”, Opt. Eng., Vol. 44, No. 8, p. 80503-1 - 80503-3, 2005

F. J. van Rensburg, J. Treurnicht, C. J. Fourie, “The Use of Fourier Descriptors for Object Recognition in Robotic Assembly”, 5th CIRP ISME Intelligent Computation in Manufacturing Engineering (CIRP ISME), Italy, July 25-28, 2006

S.-W. Lin, S.-Y. Chou, S.-C. Chen, “Irregular shapes classification by back-propagation neural networks”, Int. J. Adv. Manuf. Technol., Vol. 34, No. 11–12, pp. 1164–1172, 2007

A. D. Doulamis, N. D. Doulamis, S. D. Kollias, “An adaptable neural-network model for recursive nonlinear traffic prediction and modeling of MPEG video sources”, IEEE Transactions on Neural Networks, Vol. 14, No. 1, pp. 150–166, 2003

P. Delivery, “Home Precautions For A Family Member With Dementia”, Adolesc. Heal., Vol. 56, No. 5, pp. 502–507, 2015

E. H. Rickenbach, K. L. Condeelis, W. E. Haley, “Daily Stressors and Emotional Reactivity in Individuals With Mild Cognitive Impairment and Cognitively Healthy Controls”, Psychol. Aging, Vol. 30, No. 2, pp. 420-431, 2015

S. Mussele, K. Bekelaar, N. Le Bastard, Y. Vermeiren, J. Saerens, N. Somers, P. Mariën, J. Goeman, P. P. De Deyn, S. Engelborghs, “Prevalence and associated behavioral symptoms of depression in mild cognitive impairment and dementia due to Alzheimer’s disease”, Int. J. Geriatr. Psychiatry, Vol. 28, No. 9, pp. 947–958, 2013

E. Richard, C. Reitz, L. H. Honig, N. Schupf, M. X. Tang, J. J. Manly, R. Mayeux, D. Devanand, J. A. Luchsinger, “Late-life depression, mild cognitive impairment, and dementia”, JAMA Neurol., Vol. 70, No. 3, pp. 383–389, 2013

D. Cristinacce, T. F. Cootes, “Feature detection and tracking with constrained local models”, BMVC, Vol. 3, pp. 929-938, Edinburgh, September 4-7, 2006

T. Baltrušaitis, P. Robinson, L. -P. Morency, “3D constrained local model for rigid and non-rigid facial tracking”, 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2610–2617, 2012

Y. Bengio, I. J. Goodfellow, A. Courville, Deep Learning, MIT Press, 2016 (in preparation)

H. Zhang, X. Xiao, O. Hasegawa, “A Load-Balancing Self-Organizing Incremental Neural Network”, IEEE Trans. Neural Networks Learn. Syst., Vol. 25, No. 6, pp. 1096–1105, 2014

H.-I. Suk, S.-W. Lee, D. Shen, “Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis”, Neuroimage, Vol. 101, pp. 569–582, 2014


Abstract Views: 348
PDF Downloads: 163

Metrics Information
Bookmark and Share