Premise

Classifying daily emotions in children using an empathic voice assistant chatbot

Synopsis

In order to classify the child’s emotions, we first used Landbat.io to clarify and formulate questions, and then by implementing the web-based chat-bot with IBM Assistant on the Heroku server, I gradually added a voice assistant and tone analyzer to it, and add real-time conversation into the dataset. I will also mention the implementation method and the challenges/limitations in terms of the IBM, and in the end, we try to make chatbot a more user-friendly environment for children.

Background

Many of the adult psychiatric crises we struggle with are caused by not treating the early signs of our moods in the first few years of our childhood. Imagine that storytelling can be a good solution when you ask children to talk freely about their feelings and moods and develop their emotional intelligence. While parents play an important role in teaching children emotional intelligence, there have been many changes in emotional and communication technologies to help people understand their emotions. Based on Ekman and Friesen’s (1971)’s list of six basic emotions are happiness, anger, disgust, fear, sadness, surprise. Given that full knowledge of the child’s feelings and emotions is an extensive topic and requires multifaceted studies, in this article, we focus on anger and aggression in children.With this introduction, it can be said that aggression, as shown in the picture, appears in virtual and real-life and in different ways. Therefore, it is imperative to identify the source of the root of aggression in children:

 

 

In this research, we enable a conversation agent to enable children to recognize and express their feelings. We describe a chatbot that uses a communication interface to talk and listen to children as they share daily emotional events they have experienced, then converts existing communication data to the dataset of child emotions. We classify the types of emotions that children choose to share with the chatbot, the date, ID, and types of chatbot conversation, the challenges during the conversation, and the children’s perception of the chatbot. From our findings, and the child’s experience, this chatbot is used to examine the child’s mood swings and, if necessary, use it for early treatment of the child. As the first and one of the most influential factors on a child’s social-emotional development, families play a fundamental role in influencing the propensity for aggressive behavior. Families can have protective and risk-inducing influences on their children’s likelihood of engaging in interpersonal violence and suicidal behavior as they grow older. The creation of environments that boost the ability of families to protect their children from violence will be a key activity. Increasing parents’ knowledge and awareness about raising children in difficult situations is a good way to improve children’s behavior (Krahé, 2013).

 

Conversation Design Process

What is the problem?

The idea for this chatbot came from the fact that “The idea for this chatbot came from the fact that “aggressiveness in children often have to be monitored frequently by trained specialists or therapist through clinical interviews to avoid side effects in adulthood by early detection in childhood. With AI chatbots, parents can collect their children’s behavior data to find a pattern for better reactions. ”

It does not seem that artificial intelligence can replace doctors in the short term, but computer artificial intelligence models can be at least as accurate as doctors. Also If the program detects a worrying change, it can notify the patient’s physician for review (Graham, S., Depp, C., Lee, E.E. et al., 2019).

Accordingly, by creating a seemingly simple chatbot, parents can answer a number of questions daily about their child’s status and behaviors in relation to aggression, and based on that, a behavioral pattern can be created based on these answers. Among various other tasks, they will be asked about their emotional state, as well as some of the child’s motor skills, feeding pattern, and clinical condition.
By creating an artificial intelligence system that evaluates speech samples, it compares current data with previous samples by parents (and even larger populations) and then assesses the child’s mental state.

 

 

Exploring technologies! Chatbots

Conversational agents, also known as AI chatbots, use dialog systems to enable natural language interactions with users via speech, text, or both (Laranjo et al., 2018, p. 1253). In terms of concept, AI chatbots differ from embodied virtual conversational agents or avatars, focusing on synthesizing multimodal signals (such as images, videos, and sounds) to simulate human face-to-face communication. This paper aimed to improve the AI chatbot’s classification feature of natural language conversation for improving mental health so that humans and the chatbot could exchange information more freely. Conversational capacity can range from constrained (i.e., users can only respond by selecting predefined conversational lines) to unconstrained (i.e., users can respond by selecting predefined conversational lines).

As of 2019, Amazon Alexa had over 100,000 programs, and Facebook Messenger had over 300,000 active chatbots, many of which were for health care and wellbeing. The World Health Organization, for example, launched a chatbot on Facebook Messenger in April 2020 to tackle misinformation and provide instant and accurate information about COVID-19 (Miner et al., 2020, p. 3).

First Conversational Design with Landbot.io

To explore technology, I started my prototype by developing more knowledge about chatbots and how to make my first chatbot. This was my first step because I never had created or worked with a chatbot before. To turn literature into chatbots, I found ‘Landbot.io.’ This was an easy tool to create an initial chatbot to create a basic chatbot quickly. I decided to create a simple chatbot and perform a quick test. I wanted to investigate whether a chatbot actually has a different result than a survey of children regarding sharing feelings, especially aggressive behavior.

[aesop_video src=”youtube” id=”Uk1Tma80HrY” align=”center” disable_for_mobile=”off” loop=”on” controls=”on” mute=”off” autoplay=”off” viewstart=”off” viewend=”off” show_subtitles=”off” revealfx=”off” overlay_revealfx=”off”]

User Testing:

After making this basic chatbot, I tried to test two versions of the bot on three mothers, which had different results. One bot contained explanations of the questions, and the other was more simple and quick.

First Feedback (User Engagement): Given that this bot takes daily information from parents, and that may be hard to collect every day, it seems, it would be better to collect data from a child.
Second (Ease of Use): If this bot is supposed to collect all of the children’s behavioral information over a long time, it is important to start with simple questions and open questions. Parents or kids do not have much time during the day to type the information to this bot.

The conclusion from the first bot: The above three items are significant for users in the long term, and to raise user engagement, it is essential to have a better user experience (for example adding voice to chatbot) , which is impossible with Landbat. In general, Landbat has the following weaknesses.

Landbot weaknesses

• Landbot is mostly used for lead capture and qualification, customer service and frequently asked questions (FAQs), onboarding automation, product recommendations, human resource management and recruiting, survey data and quizzes, booking, and reservations.
• Chatbots must use advanced natural language processing (NLP) to comprehend your visitor’s urgency and sentiment fully; otherwise, the results will be frustrating and drive users away.
• Chatbots should provide advanced consulting via connecting to counseling service teams with rich data through notifications or integrations to an online knowledge base, while in Landbot, even it’s hard to connect to Wikipedia.

Second Round, IBM Watson! Integrating Bot with AI

The second iteration started with the chatbot with IBM Watson, I review several video tutorials on youtube and thought this would be an improvement to use IBM Watson. The ability to recognize more advanced speech as well as the possibility of better conversation with chatbots and of course advanced flows will be very helpful. Of course, I also tried solutions like Microsoft PowerBI, which for this purpose seemed very simple to me. Be it. The possibility of development was one of the most important factors in choosing Watson.

 What Improved?

[aesop_video src=”youtube” id=”hvd1Hd6iGaM” align=”center” disable_for_mobile=”off” loop=”on” controls=”on” mute=”off” autoplay=”off” viewstart=”off” viewend=”off” show_subtitles=”off” revealfx=”off” overlay_revealfx=”off”]

In the second iteration, I tried not to ask for personal information (at least in the initial conversations) .so that I could gain more trust from the user. I also tried to focus more on the sequence of related conversations. With this format, the user will feel more empathy and engagement. Another point is that the user will be able to type open-end conversations and see fewer multiple-choice questions. This is possible thanks to the two features of Entities and Intends, which allow the system to recognize the signs of each word from within the user dialogs and create new conversation loops. Given these features, in the next block and for the final project, I can consider for bot development is creating different conversation scenarios.

Third Iteration; Integration with Voice

For the third round, and to improve the user flow, I decided to start and communicate directly with the child instead of taking the data from the parents. Based on this, considering that text communication with a child (especially young ones) seems a bit difficult, I considered using the two APIs in IBM TTS (Text to speech ) and STT ( Speech to text).
In order for all APIs to be called at one point, they must be running on a server, which I first wanted to execute my code in IBM Studio and Notebook, but after several times I experienced a lot of errors.

So I was looking for another option that could provide me with the resources I need to run Python code and be free. Heroku is a cloud platform as a service that supports several programming languages including Python, JavaScript, and HTML code

I spent several weeks trying to reach an ideal format and learning the libraries and tutorials I needed.

For those who want to work with IBM APIs, I suggest watching this series of videos:

One of these tutorials was a link on the IBM site that taught you how to run a web-based voice chatbot outside of IBM.

The process I did to build a web chatbot was as follows:

 

 

I needed an ENV creator to implement the code in the web version. Python-dotenv reads key-value pairs from a .env file and can set them as environment variables. It helps in the development of applications following the 12-factor principles. In my project, all API keys are defined in ENV.

Adding Voice To chatbot

For the voice assistant, I routed various unpredictable situations where the sound, or the content of the message, was not clear:

my chatbots so far have been created in such a way that of course, I need to add emotion and interactive improvements to the application.

Due to the fact that the quality of the recorded audio was very high by default, and on the other hand Heroku server was very slow in the free version, I used a reducing codec to send Voices faster to the IBM server. RecordRTC is a WebRTC JavaScript library for audio/video as well as screen activity recording. It supports Chrome, Firefox, Opera, Android, and Microsoft Edge. Platforms: Linux, Mac, and Windows. Also, I used Flac js library to Compress and convert Raw Wave to speed up the transferring voices.

Limitation

Unfortunately, the free version of IBM-Text to Speech was limited and was deactivated after three days, and I had to create several different accounts to test during this run. For this reason, I will not publish the chatbot link publicly, but I will send it to the lecturer.

 

Tone Analyzer

Next, I needed to add Tone-analyzer to the project. For this purpose, I used Watson Tone Analyzer V3. This API records text, emotion, and emotion intensity. At this point, I needed to save the original texts after they became a feeling, so I also saved the date and time along with the original text in the dataset for further review.

Saving Data into Dataset

Given that we had to update the database instantly after each call, simple ways to store data in the database were not very routine. That’s why I used the psycopg library, which works relatively live with the database. Based on the description on their website: Psycopg is the most popular Python programming language database adapter. Its main features are the flawless implementation of Python DB API 2.0 specifications and thread safety (so that multiple strings can experience the same connection). This program is designed for fully multidisciplinary applications that create and destroy many cursors and create a large number of “INSERT” or “UPDATE” simultaneously in the database.

Finally, the data is stored in a CSV file so that it can be used for future review:

Finally, in order to make the look of the chatbot more usable for children and also for them to be able to use it more easily, I made the chatbot environment a bit fancy and used friendly fonts.

Eventually, the final version of the video became a demo, but due to the server’s slowness, there is a delay of a few seconds in sending and receiving sound from the IBM server to the Heroku server, which will probably be reduced as the sound is streamed and the server is the same. In the final version, I did not work much on the flow of conversations, but it is natural that with a suitable scenario, the conversations can be continued up to several stages so that more real feelings can be expressed.

[aesop_video src=”youtube” id=”A1An4K6U8XQ” align=”center” disable_for_mobile=”on” loop=”on” controls=”on” mute=”off” autoplay=”off” viewstart=”off” viewend=”off” show_subtitles=”off” revealfx=”off” overlay_revealfx=”off”]

CONCLUSION AND NEXT STEPS

The main thing I have learned with creating these chatbots is how to create a more advanced chatbot by dragging and dropping and a basic understanding of programming knowledge. Second, I learned how developed a chatbot with a voice assistant to have better interactions. Of course, I think the IBM Tone analyzer worked perfectly and I really surprised with the result. It motivated me to use this feature more for analyzing text. However, I’m keen on exploring advanced sentiment analyzes, by using other NLP language processing technologies.

Next Steps

For the next step, I’ll analyze my dataset, which I collected from my user while asking the user open-ended questions. I think it’s possible to do more analysis from these sentences. it can detect a child’s personality type, And respond to the user based on the user type. ​Given that I have researched fake tweets in my machine learning course, I am familiar with some effective text analysis features. With my research, I concluded that I could use some of the following features on behavior data. Each of these features and how they impact aggression in children can help classify it

 

Possible Features	Hypothesis/tool/resource behind each feature
Part-of-speech frequency	Normalized frequencies of each tag: adjectives, verbs, nouns, and adverbs (four features).
Negation	Normalized frequencies of negative words like no, not, didn’t, can’t. The idea behind this is to detect non-direct aggressiveness.
Capitalized	The idea behind this is that aggressive replies tend to emphasize the target they mention. It can indicate feelings or speaking volume.
Punctuation marks	! or ? or any c combination of both can emphasize the offensiveness of texts.
Sentiment	Use of NRC-Sentiment-Emotion-Lexicons4 to trace the polarity in text.
Emotions	Frequency of emotions from specific categories: anger, fear, surprise, sadness, and disgust. The idea behind this is to check the categories related to aggressiveness.
Readability Feature	Measure the difficulty of comprehension that persons encounter when reading a text. The Flesch Reading Ease measure inspires it.
Swear words	The intuition behind this is that the texts containing insults are often aggressive.
Lexical analysis with python library empath	An empath is a tool for analyzing text across lexical categories. By default, it has 194 lexical categories, and each category is considered a feature.

 

LITERATURE

Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social psychology, 17(2), 124.

Krug, E. G., Mercy, J. A., Dahlberg, L. L., & Zwi, A. B. (2002). The world report on violence and health. The Lancet, 360(9339), 1083–1088. https://doi.org/10.1016/s0140-6736(02)11133-0

Sharma, M. K., & Marimuthu, P. (2014). Prevalence and Psychosocial Factors of Aggression Among Youth. Indian Journal of Psychological Medicine, 36(1), 48–53. https://doi.org/10.4103/0253-7176.127249

Nærde, A., Ogden, T., Janson, H., & Zachrisson, H. D. (2014). Normative development of physical aggression from 8 to 26 months. Developmental Psychology, 50(6), 1710–1720. https://doi.org/10.1037/a0036324

Graham, S., Depp, C., Lee, E.E. et al. Artificial Intelligence for Mental Health and Mental Illnesses: an Overview. Curr Psychiatry Rep 21, 116 (2019). https://doi.org/10.1007/s11920-019-1094-0

Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., & Coiera, E. (2018). Conversational agents in healthcare: a systematic review. Journal of the American Medical Informatics Association, 25(9), 1248–1258. https://doi.org/10.1093/jamia/ocy072

Krahé, B. (2020). The social psychology of aggression. Routledge.Leff, S. S., & Waasdorp, T. E. (2013). Effect of Aggression and Bullying on Children and Adolescents: Implications for Prevention and Intervention. Current Psychiatry Reports, 15(3), 15(3):343. https://doi.org/10.1007/s11920-012-0343-2

Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020). Chatbots in the fight against the COVID-19 pandemic. npj Digital Medicine, 3(1), 3. https://doi.org/10.1038/s41746-020-0280-0