Behavioral Biometrics in Assisted Living: A Methodology for Emotion Recognition
Abstract
Behavioral biometrics aim at providing algorithms for the automatic recognition of individual behavioral traits, stemming from a person’s actions, attitude, expressions and conduct. In the field of ambient assisted living, behavioral biometrics find an important niche. Individuals suffering from the early stages of neurodegenerative diseases (MCI, Alzheimer’s, dementia) need supervision in their daily activities. In this context, an unobtrusive system to monitor subjects and alert formal and informal carers providing information on both physical and emotional status is of great importance and positively affects multiple stakeholders. The primary aim of this paper is to describe a methodology for recognizing the emotional status of a subject using facial expressions and to identify its uses, in conjunction with pre-existing risk-assessment methodologies, for its integration into the context of a smart monitoring system for subjects suffering from neurodegenerative diseases. Paul Ekman’s research provided the background on the universality of facial expressions as indicators of underlying emotions. The methodology then makes use of computational geometry, image processing and graph theory algorithms for the detection of regions of interest and then a neural network is used for the final classification. Findings are coupled with previous published work for risk assessment and alert generation in the context of an ambient assisted living environment based on Service oriented architecture principles, aimed at remote web-based estimation of the cognitive and physical status of MCI and dementia patients.
Keywords:
ambient assisted living, dementia, emotion, image processing, telemedicineDownloads
References
Eurostat, “Population structure and ageing,” 2015, available at: http://ec.europa.eu/eurostat/statistics-explained/index.php/Population_
structure_and_ageing
A. L. Busse, G. Gil, J. M. Santarém, W. J. Filho, “Physical activity and cognition in the elderly: A review”, Dement. Neuropsychol., Vol. 3, No. 3, pp. 204–208, 2009 DOI: https://doi.org/10.1590/S1980-57642009DN30300005
A. Di Carlo, M. Baldereschi, L. Amaducci, S. Maggi, F. Grigoletto, G. Scarlato, D. Inzitari, “Cognitive impairment without dementia in older people: prevalence, vascular risk factors, impact on disability. the italian longitudinal study on aging”, J. Am. Geriatr. Soc., Vol. 48, No. 7, pp. 775–782, 2000 DOI: https://doi.org/10.1111/j.1532-5415.2000.tb04752.x
D. Laurin, R. Verreault, J. Lindsay, K. MacPherson, K. Rockwood, “Physical activity and risk of cognitive impairment and dementia in elderly persons”, Arch. Neurol., Vol. 58, No. 3, pp. 498–504, 2001 DOI: https://doi.org/10.1001/archneur.58.3.498
A. Bradford, M. E. Kunik, P. Schulz, S. P. Williams, H. Singh, “Missed and delayed diagnosis of dementia in primary care: prevalence and contributing factors”, Alzheimer Dis. Assoc. Disord., Vol. 23, No. 4, pp. 306–314, 2009 DOI: https://doi.org/10.1097/WAD.0b013e3181a6bebc
T. Kitwood, Dementia Reconsidered: the Person Comes First, Open University Press, 1999
M. Haritou, Y. Glickman, A. Androulidakis, S. Xefteris, A. Anastasiou, A. Baboshin, S. Cuno, and D. Koutsouris, “A technology platform for a novel home care delivery service to patients with dementia”, J. Med. Imaging Heal. Informatics, Vol. 2, No. 1, pp. 49–55, 2012 DOI: https://doi.org/10.1166/jmihi.2012.1060
M. Torkamani, L. McDonald, I. S. Aguayo, C. Kanios, M.-N. Katsanou, L. Madeley, P. D. Limousin, A. J. Lees, M. Haritou, M. Jahanshahi, “A randomized controlled pilot study to evaluate a technology platform for the assisted living of people with dementia and their carers”, J. Alzheimers Dis., Vol. 41, No. 2, pp. 515–523, 2014 DOI: https://doi.org/10.3233/JAD-132156
P. Rashidi, A. Mihailidis, “A survey on ambient-assisted living tools for older adults”, IEEE J. Biomed. Heal. Informatics, Vol. 17, No. 3, pp. 579–590, 2013 DOI: https://doi.org/10.1109/JBHI.2012.2234129
C. Frantzidis, C. Bratsas, C. L. Papadelis, E. Konstantinidis, C. Pappas, P. D. Bamidis, “Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli”, Inf. Technol. Biomed. IEEE Trans., Vol. 14, No. 3, pp. 589–597, 2010 DOI: https://doi.org/10.1109/TITB.2010.2041553
F. Bellotti, B. Kapralos, K. Lee, P. Moreno-Ger, R. Berta, “Assessment in and of serious games: an overview,” Adv. Human-Computer Interact., Vol. 2013, pp. 1–11, 2013 DOI: https://doi.org/10.1155/2013/136864
E. I. Konstantinidis, A. S. Billis, C. Mouzakidis, V. Zilidou, P. E. Antoniou, P. D. Bamidis, “Design, implementation and wide pilot deployment of FitForAll: an easy to use exergaming platform improving physical fitness and life quality of senior citizens”, IEEE J. Biomed. Heal. Informatics, Vol. 20, No. 1, pp. 189 - 200, 2014 DOI: https://doi.org/10.1109/JBHI.2014.2378814
A. S. Billis, E. I. Papageorgiou, C. Frantzidis, E. I. Konstantinidis, P. D. Bamidis, “Towards a hierarchically-structured decision support tool for improving seniors’ independent living”, 6th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’13), pp. 1–4, Rhodes, Greece, May 29-31, 2013 DOI: https://doi.org/10.1145/2504335.2504362
A. Artikis, P. D. Bamidis, A. Billis, C. Bratsas, C. Frantzidis, V. Karkaletsis, M. Klados, E. Konstantinidis, S. Konstantopoulos, D. Kosmopoulos, H. Papadopoulos, S. Perantonis, S. Petridis, C. S. Spyropoulos, “Supporting tele-health and AI-based clinical decision making with sensor data fusion and semantic interpretation: The USEFIL case study”, International Workshop on Artificial Intelligence and NetMedicine, Montpellier, France, August 27, 2012
S. Xefteris, M. Haritou, K. Tserpes, A. Serretti, J. R. Llopart, R. Calati, T. Varvarigou, “Analysis of requirements and specifications for a monitoring system to support the self-management of dementia patients at home”, 3rd International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’10), Samos, Greece, June 23-25, 2010 DOI: https://doi.org/10.1145/1839294.1839356
S. Xefteris, A. Baboshin, K. Tserpes, A. Androulidakis, Y. Glickman, T. Varvarigou, M. Haritou, F. D’Andria, “Enabling risk assessment and analysis by event detection in dementia patients using a reconfigurable rule set”, 4th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’11), Crete, Greece, May 25-27, 2011
P. D. Bamidis, P. Fissler, S. G. Papageorgiou, V. Zilidou, E. I. Konstantinidis, A. S. Billis, E. Romanopoulou, M. Karagianni, I. Bearatis, A. Tsapanou, G. Tsilikopoulou, E. Grigoriadou, A. Ladas, A. Kyrillidou, A. Tsolaki, C. Frantzidis, E. Sidiropoulos, A. Siountas, S. Matsi, J. Papatriantafyllou, E. Margioti, A. Nika, W. Schlee, T. Elbert, M. Tsolaki, A. B. Vivas, I.-T. Kolassa, “Gains in cognition through combined cognitive and physical training: the role of training dosage and severity of neurocognitive disorder”, Front. Aging Neurosci., Vol. 7, Art. No. 152, pp. 1-15, 2015. DOI: https://doi.org/10.3389/fnagi.2015.00152
C. Styliadis, P. Kartsidis, E. Paraskevopoulos, A. A. Ioannides, P. D. Bamidis, “Neuroplastic effects of combined computerized physical and cognitive training in elderly individuals at risk for dementia : an eLORETA controlled study on resting states”, Neural Plast., Vol. 2015, Article ID 172192, 2015 DOI: https://doi.org/10.1155/2015/172192
A. S. Billis, N. Katzouris, A. Artikis, P. D. Bamidis, “Clinical decision support for active and healthy ageing: an intelligent monitoring approach of daily living activities”, 17th Portuguese Conference on Artificial Intelligence (EPIA 2015), Coimbra, Portugal, September 8-11, 2015 DOI: https://doi.org/10.1007/978-3-319-23485-4_14
D. N. Monekosso, P. Remagnino, “Behavior analysis for assisted living”, IEEE Trans. Autom. Sci. Eng., Vol. 7, No. 4, pp. 879–886, 2010 DOI: https://doi.org/10.1109/TASE.2010.2049840
R. A. Stucki, P. Urwyler, L. Rampa, R. Müri, U. P. Mosimann, T. Nef, “A web-based non-intrusive ambient system to measure and classify activities of daily living”, J. Med. Internet Res., Vol. 16, No. 7, Article ID e175, 2014 DOI: https://doi.org/10.2196/jmir.3465
A. S. Billis, E. I. Papageorgiou, C. A. Frantzidis, M. S. Tsatali, A. C. Tsolaki, P. D. Bamidis, “A decision-support framework for promoting independent living and ageing well”, IEEE J. Biomed. Heal. Informatics, Vol. 19, No. 1, pp. 199–209, 2015 DOI: https://doi.org/10.1109/JBHI.2014.2336757
A. K. Jain, A. Ross, S. Prabhakar, “An introduction to biometric identification”, IEEE Trans. circuits Syst. Video Technol. Spec. issue image-and video-based biometrics, Vol. 14, No. 1, pp. 4-20, 2004 DOI: https://doi.org/10.1109/TCSVT.2003.818349
E. M. Tapia, S. S. Intille, K. Larson, Activity recognition in the home using simple and ubiquitous sensors. Springer, 2004 DOI: https://doi.org/10.1007/978-3-540-24646-6_10
M. Mozer, “Lessons from an adaptive house”, in Smart Environments: Technologies, Protocols, and Applications, Hoboken, NJ, pp. 273–294, 2004 DOI: https://doi.org/10.1002/047168659X.ch12
A. Steinhage, C. Lauterbach, “Monitoring movement behavior by means of a large area proximity sensor array in the floor”, 2nd Workshop on Behaviour Monitoring and Interpretation (BMI'08), Kaiserslautern, Germany, September 23, 2008
M. Muhlenbrock, O. Brdiczka, D. Snowdon, J. -L. Meunier, “Learning to detect user activity and availability from a variety of sensor data”, 2nd IEEE International Conference on Pervasive Computing and Communications (PerCom’04), pp. 13-22, Florida, USA, March 14-17, 2004
C. Darwin, P. Ekman, P. Prodger, The expression of the emotions in man and animals. Oxford University Press, USA, 1998
B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,” Pattern Recognit., Vol. 36, No. 1, pp. 259–275, 2003 DOI: https://doi.org/10.1016/S0031-3203(02)00052-3
S. Xefteris, A. Baboshin, K. Tserpes, A. Androulidakis, Y. Glickman, T. Varvarigou, M. Haritou, F. D’Andria, “Enabling risk assessment and analysis by event detection in dementia patients using a reconfigurable rule set”, 4th International Conference on PErvasive Technologies Related to Assistive Environments, (PETRA’11), Crete, Greece, May 25-27, 2011 DOI: https://doi.org/10.1145/2141622.2141678
P. Ekman, W. V Friesen, Unmasking the face: A guide to recognizing emotions from facial clues. Ishk, 2003
P. Ekman, R. J. Davidson, The nature of emotion: Fundamental questions. Oxford University Press, 1994
C. Darwin, The Expression of the Emotions in Man and Animals. 1872. DOI: https://doi.org/10.1037/10001-000
P. Ekman, W. V Friesen, P. Ellsworth, Emotion in the human face: Guidelines for research and an integration of findings, Oxford Pergamon Press, 1972
P. Ekman, W. V Friesen, M. O’Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-Bitti, and others, “Universals and cultural differences in the judgments of facial expressions of emotion”, J. Pers. Soc. Psychol., Vol. 53, No. 4, pp. 712-717, 1987 DOI: https://doi.org/10.1037/0022-3514.53.4.712
P. Ekman, “An argument for basic emotions”, Cogn. Emot., Vol. 6, No. 3–4, pp. 169–200, 1992 DOI: https://doi.org/10.1080/02699939208411068
P. Ekman, Darwin and Facial Expression: A century of Research. New York: Academic Press, 1973
C. Izzard, The Face of Emotion. New York: Appleton-Century-Crofts, 1971.
P. Ekman, W. V. Friesen, P. Ellsworth, Emotion in the human face.Guidelines for research and an integration of findings, Pergamon Press, USA, 2013
H. Hwang, D. Matsumoto, “Evidence for the Universality of Facial Expressions of Emotion”, in Understanding Facial Expressions in Communication, Springer India, pp. 41–56, 2015, DOI: https://doi.org/10.1007/978-81-322-1934-7_3
I. E. Eibelsfeldt, Ethology, the Biology of Behavior . 1970
P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression”, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101, 2010 DOI: https://doi.org/10.1109/CVPRW.2010.5543262
T. Kanade, J. F. Cohn, Y. Tian, “Comprehensive database for facial expression analysis”, 4th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53, 2000
V. Andronikou, S. Xefteris, T. Varvarigou, “A novel, algorithm metadata-aware architecture for biometric systems”, IEEE Workshop on Biometric Measurements and Systems for Security and Medical Applications (BIOMS), pp. 1–6, 2012 DOI: https://doi.org/10.1109/BIOMS.2012.6345771
M. Jones, P. Viola, “Fast multi-view face detection”, Mitsubishi Electr. Res. Lab TR-20003-96, Vol. 3, 2003, available at http://www.merl.com/publications/docs/TR2003-96.pdf
A. Okabe, B. Boots, K. Sugihara, S. N. Chiu, Spatial tessellations: concepts and applications of Voronoi diagrams, Vol. 501. John Wiley & Sons, 2009
F. P. Preparata, M. Shamos, Computational geometry: an introduction. Springer Science & Business Media, 2012.
B. Aronov, M. Dulieu, F. Hurtado, “Witness (Delaunay) graphs”, Comput. Geom., Vol. 44, No. 6–7, pp. 329–344, 2011 DOI: https://doi.org/10.1016/j.comgeo.2011.01.001
J. Gan, Y. Tao, “DBSCAN Revisited: Mis-Claim, Un-Fixability, and Approximation”, ACM SIGMOD International Conference on Management of Data, pp. 519–530, 2015 DOI: https://doi.org/10.1145/2723372.2737792
R. Xu, D. Wunsch, and others, “Survey of clustering algorithms”, IEEE Transactions on Neural Networks, , Vol. 16, No. 3, pp. 645–678, 2005 DOI: https://doi.org/10.1109/TNN.2005.845141
I. Kunttu, L. Lepistö, A. Visa, “Efficient Fourier shape descriptor for industrial defect images using wavelets”, Opt. Eng., Vol. 44, No. 8, p. 80503-1 - 80503-3, 2005 DOI: https://doi.org/10.1117/1.1993687
F. J. van Rensburg, J. Treurnicht, C. J. Fourie, “The Use of Fourier Descriptors for Object Recognition in Robotic Assembly”, 5th CIRP ISME Intelligent Computation in Manufacturing Engineering (CIRP ISME), Italy, July 25-28, 2006
S.-W. Lin, S.-Y. Chou, S.-C. Chen, “Irregular shapes classification by back-propagation neural networks”, Int. J. Adv. Manuf. Technol., Vol. 34, No. 11–12, pp. 1164–1172, 2007 DOI: https://doi.org/10.1007/s00170-006-0667-3
A. D. Doulamis, N. D. Doulamis, S. D. Kollias, “An adaptable neural-network model for recursive nonlinear traffic prediction and modeling of MPEG video sources”, IEEE Transactions on Neural Networks, Vol. 14, No. 1, pp. 150–166, 2003 DOI: https://doi.org/10.1109/TNN.2002.806645
P. Delivery, “Home Precautions For A Family Member With Dementia”, Adolesc. Heal., Vol. 56, No. 5, pp. 502–507, 2015
E. H. Rickenbach, K. L. Condeelis, W. E. Haley, “Daily Stressors and Emotional Reactivity in Individuals With Mild Cognitive Impairment and Cognitively Healthy Controls”, Psychol. Aging, Vol. 30, No. 2, pp. 420-431, 2015 DOI: https://doi.org/10.1037/a0038973
S. Mussele, K. Bekelaar, N. Le Bastard, Y. Vermeiren, J. Saerens, N. Somers, P. Mariën, J. Goeman, P. P. De Deyn, S. Engelborghs, “Prevalence and associated behavioral symptoms of depression in mild cognitive impairment and dementia due to Alzheimer’s disease”, Int. J. Geriatr. Psychiatry, Vol. 28, No. 9, pp. 947–958, 2013 DOI: https://doi.org/10.1002/gps.3909
E. Richard, C. Reitz, L. H. Honig, N. Schupf, M. X. Tang, J. J. Manly, R. Mayeux, D. Devanand, J. A. Luchsinger, “Late-life depression, mild cognitive impairment, and dementia”, JAMA Neurol., Vol. 70, No. 3, pp. 383–389, 2013 DOI: https://doi.org/10.1001/jamaneurol.2013.603
D. Cristinacce, T. F. Cootes, “Feature detection and tracking with constrained local models”, BMVC, Vol. 3, pp. 929-938, Edinburgh, September 4-7, 2006
T. Baltrušaitis, P. Robinson, L. -P. Morency, “3D constrained local model for rigid and non-rigid facial tracking”, 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2610–2617, 2012 DOI: https://doi.org/10.1109/CVPR.2012.6247980
Y. Bengio, I. J. Goodfellow, A. Courville, Deep Learning, MIT Press, 2016 (in preparation)
H. Zhang, X. Xiao, O. Hasegawa, “A Load-Balancing Self-Organizing Incremental Neural Network”, IEEE Trans. Neural Networks Learn. Syst., Vol. 25, No. 6, pp. 1096–1105, 2014 DOI: https://doi.org/10.1109/TNNLS.2013.2287884
H.-I. Suk, S.-W. Lee, D. Shen, “Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis”, Neuroimage, Vol. 101, pp. 569–582, 2014 DOI: https://doi.org/10.1016/j.neuroimage.2014.06.077
Downloads
How to Cite
License
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
