Engineering:Nadine Social Robot

From HandWiki
Short description: Social Humanoid Robot
Nadine
Nadine Robot.jpg
Year of creation2013

Nadine is a gynoid humanoid social robot that is modelled on Professor Nadia Magnenat Thalmann.[1] The robot has a strong human-likeness with a natural-looking skin and hair and realistic hands. Nadine is a socially intelligent robot which returns a greeting, makes eye contact, and can remember all the conversations had with it. It is able to answer questions autonomously in several languages, simulate emotions both in gestures and facially, depending on the content of the interaction with the user.[2][3][4] Nadine can recognise persons it has previously seen, and engage in flowing conversation.[5][6][7] Nadine has been programmed with a "personality", in that its demeanour can change according to what is said to it.[8] Nadine has a total of 27 degrees of freedom for facial expressions and upper body movements. With persons it has previously encountered, it remembers facts and events related to each person.[9][10] It can assist people with special needs by reading stories, showing images, put on Skype sessions, send emails, and communicate with other members of the family.[11][12][13][14] It can play the role of a receptionist in an office or be dedicated to be a personal coach.[15][16]

Nadine interacted with more than 100,000 visitors at the ArtScience Museum in Singapore during the exhibition, "HUMAN+: The Future of our Species", that was held from May to October 2017.[17][18][19] Nadine has worked as a customer service agent in AIA Insurance Company in Singapore.[20][21][22] This is the first time in the world that a humanoid robot is used as a customer service agent.

History

Nadine is a next-generation humanoid robot that is a successor from Eva,[23] a humanoid robot head manufactured by Hanson Robotics in 2008. Eva's software platform was developed at MIRALab,[24] University of Geneva. Eva's head shows very realistic moods and emotions[25] and short term memory.[26][27] Eva has also performed in a play in the Roten Fabrik Theatre at Zurich.[28]

Nadine has been created in 2013 by Kokoro, Japan and has been modelled after Professor Nadia Magnenat Thalmann. Nadine has a head and full body with a natural appearance. Nadine software platform which has been developed at the Institute for Media Innovation in Singapore's Nanyang Technological University is able to show emotions, speak naturally, understand some gestures, and remember and retrieve facts during dialogue sessions.[29][30] Nadine also interacts with arm movements. Ongoing research provides the social robot with two articulated hands and natural grasping.[31][32][33][34] Nadine is also linked to all kinds of databases such as its personal dataset, Wikipedia, weather channels, and many others.

Platform

Nadine (social robot) is built with a classic perception – processing/decision – interaction layer framework. The design of Nadine platform with objectives of maintaining human-like natural behavior even in complex situation, be generic to handle any kind of data and place of operation, multi-lingual support etc.

Nadine's functionalities are based on her understanding of environment and perception of users/people in front of her. Nadine's perception layer is focused on this task. Nadine uses a 3D depth cameras, webcam and microphone to pick up vision and audio inputs from her environment and users. Perception layer is composed of independent sub-modules that operate on different input streams of the above-mentioned devices to recognize faces,[35] emotions,[13] gestures,[36][37] user location, intention, behavior etc. and other environmental attributes such as object recognition,[38][39] location etc.

The processing layer functions as Nadine's brain that uses the perception outputs to gauge the situation and decide on how to act according to it. The main component of this layer is a behavior tree planner, Nadine's central processing unit allows to process all perceived inputs. Based on the inputs received from perception layer, the behavior tree planner updates the other sub-modules of processing layer, which include processing dialog between user and Nadine, affective system and memories of her interaction. To process dialog, generic chatbots[40][41] have been built to handle different situations and questions. An online search based on Google Assistant is also integrated to answer questions outside the trained corpus. Based on the user's speech, emotion and Nadine's current emotion, Nadine can exhibit different human motion to user.[13] Nadine's memory model[42] also allows her to remember specific facts about the user and context of current conversation in order to provide appropriate responses. Upon understanding the user interaction and environment context, an appropriate verbal or non-verbal response is decided. For this purpose, Nadine's processing layer maps each perception layer stimuli to an activation and threshold. Based on the processing of each stimulus by each sub-module, the activation levels are varied. When thresholds are reached, each winning action is passed on to interaction layer to show the various responses in Nadine.

The interaction layer or Nadine controller is responsible for executing each of the responses received from processing layer to show it in Nadine's face or gesture. For example, based on user location modify Nadine's head to maintain eye gaze with user. Apart from this, the interaction layer is also responsible for controlling her motors to show different gestures and facial expressions. For verbal responses, it includes a speech synthesizer and lip synchronization module. Based on the verbal response, corresponding phonemes and visemes are generated. The speech synthesizer also takes into account the tone of dialog (to show various emotions) while generating speech. The lip synchronization converts the visemes into corresponding facial motor position to move Nadine's lips according to her speech. Currently, Nadine can support six languages including English, German, French, Chinese, Hindi and Japanese.

Events

Nadine has participated in live demos on stage and engaged with people from all walks of life. Proclaimed as one of the world's most realistic humanoid robot,[43] Nadine made her first public appearance as a key highlight at the “Human+: The Future of Our Species” exhibition held in Singapore's ArtScience Museum.[44]

She has interacted with many people from corporate companies across various industries such as Dentsu Aegis Network (DAN), Credit Suisse[45] and Deutsche Bank.[46]

Nadine also interacted with Prime Minister of India, His Excellency Narendra Modi during his historic visit to NTU Singapore, on 1 June 2018, which was one of the innovations he took special interest in.[47][48]

Nadine has worked as a customer service agent at AIA Singapore.[20][21][22] She has been trained to handle questions that are usually asked to AIA customer service agents. She also encourages AIA customers to sign up with AIA e-care registration portal. Customer service interactions were used to train a machine-learning based conversational dialog engine. A client-server architecture was also set up between our platform and AIA portal to allow fast and secure communication.[49]

In late 2020 and until April 2021, Nadine has spent 6 months at Bright Hill Evergreen[50] Home in Singapore to assist elderly in playing Bingo and interacting with them. With the ethical committee agreement of NTU, a thorough study has been done for the first time on the interaction of Nadine social robot with light dementia patients.[51] The film of this interaction can be seen on Nadine YouTube Channel.[52]

References

  1. ↑ Gieriet, Marc (2016-10-09). "Nadia und Nadine". SRF Schweizer Radio und Fernsehen. https://www.srf.ch/sendungen/reporter/nadia-und-nadine. 
  2. ↑ "Media coverage on Nadine exhibition". http://www.8days.sg/seeanddo/thingstodo/chat-with-a-female-robot-who-s-made-to-look-just-like-her-8882876. 
  3. ↑ Mulligan, Gabriella (2017-03-17). "Is robotics a solution to the growing needs of the elderly?". BBC. https://www.bbc.com/news/business-39255244. 
  4. ↑ Lim, Paige (2016-03-07). "Now you're talking: human-like robot may one day care for dementia patients". Reuters. https://www.reuters.com/article/singapore-humanoid-idUSKCN0W9120. 
  5. ↑ Lee, Jocelyn (2017-05-27). "Chat With A Female Robot (Who's Made To Look Just Like Her Creator)". 8 Days. https://www.8days.sg/seeanddo/thingstodo/chat-with-a-female-robot-who-s-made-to-look-just-like-her-8882876. 
  6. ↑ J Ren, X Jiang and J Yuan, Quantized Fuzzy LBP for Face Recognition, 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2015, Brisbane, Australia, 2015
  7. ↑ "NTU scientists unveil social and telepresence robots". Nanyang Technological University. 2015-12-29. http://media.ntu.edu.sg/NewsReleases/Pages/newsdetail.aspx?news=fde9bfb6-ee3f-45f0-8c7b-f08bc1a9a179. 
  8. ↑ Kochanowicz J, A. H. Tan and D. Thalmann, Modeling human-like non-rationality for social agents, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 11-20, Geneva, Switzerland, May 23–25, 2016
  9. ↑ J. Zhang J, N. Magnenat Thalmann and J. Zheng, Combining Memory and Emotion With Dialog on Social Companion: A Review, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 1-9, Geneva, Switzerland, May 23–25, 2016
  10. ↑ Berger, Sarah (2015-12-31). "Humanlike, Social Robot 'Nadine' Can Feel Emotions And Has A Good Memory, Scientists Claim". International Business Times. https://www.ibtimes.com/humanlike-social-robot-nadine-can-feel-emotions-has-good-memory-scientists-claim-2245600. 
  11. ↑ A. Beck, Z. Zhang and N. Magnenat Thalmann, Motion Control for Social Behaviors, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 237-256, 2015
  12. ↑ Z.P. Bian, J. Hou, L.P. Chau and N. Magnenat Thalmann, Fall Detection Based on Body Part Tracking Using a Depth Camera, IEEE Journal of Biomedical and Health Informatics, Vol. 19, No. 2, Pp. 430-439, 2015
  13. ↑ 13.0 13.1 13.2 J. Zhang, J. Zheng and N. Magnenat Thalmann, PCMD: personality‐characterized mood dynamics model toward personalized virtual characters, Computer Animation and Virtual Worlds, Vol. 26, Issue 3-4, Pp. 237-245, 2015
  14. ↑ J. Zhang, J. Zheng and N. Magnenat Thalmann, Modeling Personality, Mood, and Emotions, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 211-236, 2015
  15. ↑ Y. Xiao, Z. Zhang, A. Beck, J. Yuan and D. Thalmann, Human-Robot Interaction by Understanding Upper Body Gestures, MIT Press Journals - Presence: Teleoperators and Virtual Environments, Vol. 23, No. 2, Pp. 133-154, 2014
  16. ↑ Z. Yumak, J. Ren, N. Magnenat Thalmann, and J. Yuan, Modelling Multi-Party Interactions among Virtual Characters, Robots, and Humans, MIT Press Journals - Presence: Teleoperators and Virtual Environments, Vol. 23, No. 2, Pp. 172-190, 2014
  17. ↑ Martin, Mayo (2017-05-19). "Singapore's receptionist robot makes her public debut at ArtScience Museum's futuristic show". Channel NewsAsia. https://www.channelnewsasia.com/news/lifestyle/singapore-robot-nadine-public-debut-art-science-museum-8857788. 
  18. ↑ Wong, Cara (2017-05-19). "Conversation with a humanoid robot". The Straits Times. https://www.straitstimes.com/lifestyle/entertainment/conversation-with-a-humanoid-robot. 
  19. ↑ "Chat With A Female Robot (Who's Made To Look Just Like Her Creator)". https://www.8days.sg/seeanddo/thingstodo/chat-with-a-female-robot-who-s-made-to-look-just-like-her-8882876. 
  20. ↑ 20.0 20.1 "AIA To Pioneer Adoption Of Artificial Intelligence In Singapore's Life Insurance Sector". AIA Singapore. 2018-10-12. https://www.aia.com.sg/en/about-aia/media-centre/press-releases/2018/aia-to-pioneer-adoption-of-artificial-intelligence-in-singapores-life-insurance-sector.html. 
  21. ↑ 21.0 21.1 Benjamin, Ang (2018-10-22). "Singapore: AIA transforms customer service with insurance industry's first humanoids". Asia Insurance Review. http://www3.asiainsurancereview.com/News/View-NewsLetter-Article/id/44576/Type/agent/Singapore-AIA-transforms-customer-service-with-insurance-industry-s-first-humanoids#. 
  22. ↑ 22.0 22.1 Gabriel, Olano (2018-10-19). "AIA Singapore welcomes two new robotic hires". Insurance Business Asia. https://www.insurancebusinessmag.com/asia/news/technology/aia-singapore-welcomes-two-new-robotic-hires-114145.aspx. 
  23. ↑ Social Robotics
  24. ↑ MIRALab
  25. ↑ C. Cig, Z. Kasap, A. Egges and N. Magnenat Thalmann, Realistic Emotional Gaze and Head Behavior Generation Based on Arousal and Dominance Factors, The 3rd International Conference on Motion in Games 2010, Springer, 2010
  26. ↑ Z. Kasap and N. Magnenat Thalmann, Building long-term relationships with virtual and robotic characters: the role of remembering, The Visual Computer (IF: 1.073), vol. 28, no. 1, pp. 87-97, January 2012
  27. ↑ Z. Kasap, M. Ben Moussa, P. Chaudhuri and N. Magnenat Thalmann, Making Them Remember—Emotional Virtual Characters with Memory, IEEE Computer Graphics and Applications (IF: 1.116), vol. 29, no. 2, pp. 20-29, March 2009
  28. ↑ The robot EVA playing in the Roten Fabrik theatre in Zurich
  29. ↑ ColdFusion, The Most Realistic Robots! (2018)
  30. ↑ J. Zhang J, N. Magnenat Thalmann and J. Zheng, Combining Memory and Emotion With Dialog on Social Companion: A Review, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 1-9, Geneva, Switzerland, May 23–25, 2016
  31. ↑ L. Tian, N. Magnenat Thalmann, D. Thalmann, J. Zheng, A methodology to model and simulate customized human robotic realistic hand, Proceedings of the 35th Computer Graphics International (CGI 2018), ACM, Bintan, Indonesia, June 11–14, 2018
  32. ↑ L. Tian, N. Magnenat Thalmann, D. Thalmann, J. Zheng, The Making of a 3D-Printed, Cable-Driven, Single-Model, Lightweight Humanoid Robotic Hand, Frontiers in Robotics and AI, DOI: 10.3389/frobt.2017.00065, pp. 65, December 04, 2017
  33. ↑ N. Magnenat Thalmann, L. Tian and F. Yao, Nadine: A Social Robot that Can Localize Objects and Grasp Them in a Human Way, Frontiers in Electronic Technologies, Springer, pp. 1-23, 2017
  34. ↑ H. Liang, J. Yuan, D. Thalmann and N. Magnenat Thalmann, AR in Hand: Egocentric Palm Pose Tracking and Gesture Recognition for Augmented Reality Applications, ACM Multimedia Conference 2015 (ACMMM 2015), Brisbane, Australia, 2015
  35. ↑ J. Ren, X. Jiang, and J. Yuan, Face and Facial Expressions Recognition and Analysis, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 3-29, 2015
  36. ↑ L. Ge, H. Liang, J. Yuan and D. Thalmann, Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 24 June 2016
  37. ↑ Q. Ke, M. Bennamoun, S. An, F. Sohel and F. Boussaid, A New Representation of Skeleton Sequences for 3D Action Recognition, CVPR 2017
  38. ↑ Z. Fang, J. Yuan and N. Magnenat Thalmann, Understanding Human-Object Interaction in RGB-D videos for Human Robot Interaction, Proceedings of the 35th Computer Graphics International (CGI 2018), ACM, Indonesia, June 11–14, 2018
  39. ↑ J. Redmon and A. Farhadi, YOLO9000: Better, Faster, Stronger, arXiv preprint, 2017
  40. ↑ R.S Wallace, The Anatomy of A.L.I.C.E., In Parsing the Turing Test (pp. 181-210). Springer, Dordrecht, 2009
  41. ↑ Chatterbot
  42. ↑ J. Zhang, J. Zheng and N. Magnenat Thalmann, MCAEM: Mixed-Correlation-Analysis based Episodic Memory for Companion-User Interactions, The Visual Computer, DOI: 10.1007/s00371-018-1537-3, Vol 34, Issue 6-8, pp. 1129-1141, May 10, 2018
  43. ↑ Aloysius, Low (2017-05-19). "'Human+' exhibit explores the future of humanity". CNET. https://www.cnet.com/pictures/human-plus-exhibition-artscience-museum-singapore/12/. 
  44. ↑ "Human+: The Future of Our Species". http://www.rosettemedia.com/2017/05/the-cyborgs-have-arrived/. 
  45. ↑ Credit Suisse Global Megatrends Conference 2018
  46. ↑ 9th Annual deAccess Asia 2018 Conference Highlights
  47. ↑ "NTU marks historic visit by Indian Prime Minister". http://media.ntu.edu.sg/news/Pages/NTU-marks-historic-visit-by-Indian-Prime-Minister.aspx. 
  48. ↑ PM Modi Interacts With Robot In Nanyang Tech University In Singapore
  49. ↑ NTU Singapore’s social robot Nadine starts work as a receptionist
  50. ↑ Bright Hill Evergreen
  51. ↑ Gauri Tulsulkar, Nadia Magnenat Thalmann et al., Can a humanoid social robot stimulate the interactivity of cognitively impaired elderly? A thorough study based on computer vision methods, The Visual Computer, vol. 37, pp. 3019-3038, July 2021
  52. ↑ Nadine Social Robot @ Brighthill Evergreen Home



Retrieved from "https://handwiki.org/wiki/index.php?title=Engineering:Nadine_Social_Robot&oldid=3235445"