Premise

Creating a responsive chatbot against loneliness among students.

Synopsis

The lockdown has continued to isolate students from their old social life. The mental health state of many is dwindling significantly compared to pre-corona times. When we are unable to meet people, what do we do?
With that as a leading question, a chatbot was created to fight loneliness among younger people. What started out as a narrow chatbot with clickable elements turned into a responsive chatbot, able to interpret user utterances. Future recommendations include the use of sentiment analysis based on Azure and more training based on the user utterances.
Chatbots without enough learning data are rather narrow. Therefore, it is essential to keep training the chatbot with the utterances that users generate. Additionally, if the chatbot is able to interpret the sentiment of the user, it could respond accordingly. Thus, increasing the responsiveness of the chatbot drastically.

The Problem

In the current pandemic the worries about the mental wellbeing of young adults is an almost daily topic. However, nothing seems to be done about their isolated state. Social anxiety is part of the problem for people during the pandemic (Panchal et al, 2021).
Especially among the student demographic, the consequences of a lack of social life is felt (Krendl, 2021; Lippke, Fischer & Ratz, 2021; Padmanabhanunni, 2021). As a student myself I am feeling the consequences as well. With no social outlet other than my housemates, my social life has come to a stop.

The Approach

The initial idea for the prototype was a chatbot created with IBM Watson. The user would be able to have natural conversations with the chatbot in order to feel less lonely. The chatbot would be less chatbot and more of a listening ear for socially isolated individuals.
The first conversation flow was meant to divide the conversation in three distinct emotional states, bad, neutral and positive. The chatbot would then be able to discuss the reason for the emotional state with the individual. However, it quickly became apparent that IBM Watson did not fit the goal of the chatbot.

It was then decided to create the chatbot through the Bot Framework Composer. The composer offers more options and has pre-trained models, which is convenient for the chatbot. Additionally, the Composer allows for an open conversation design.
Later in the process, both Azure Cognitive Services and Tensorflow were researched to implement sentiment analysis in the chatbot. Thus, improving the ability of the chatbot to interpret user utterances. 

1st Iteration User Testing

General feedback was that the chatbot did not feel like a “chatbot against loneliness”, but rather as a social hub. Additionally, the testers noted the static feeling of the conversation, since the options are clickable and not typable. The primary goal of the second iteration is to create a less restrictive chatbot. A chatbot that can understand the user and where the user is able to actually type messages instead of static clicking.

The Design Process Second Iteration

The second iteration uses the Bot Framework Composer, which is a visual development environment developed by Microsoft. It allows the developer to create intricate bots with various goals. It is important to note down exactly what we want to accomplish and what the bot should be able to do. Some rough guidelines for the design process:

• What utterances are common in human conversations?
• How do we fight loneliness? Should we create a bot that feels like it is listening to the person? Or should the bot talk in a more proactive manner, maybe even give suggestions to the person to battle their own loneliness.

Two main concepts that should be identified straight away are Intent and Utterance. Intent is identified as the intention that a user has when saying/typing something. The utterance is the entity of text/speech that the user writes/says. The bot extracts the intent from the utterance by analysing the text and searching for certain trigger words. The intent(s) drive the conversation forward, as it provides the bot with the knowledge of what the user is attempting to achieve. 

The primary goal for the second iteration in Composer was to ask the user for their name and let the bot greet them with their name. It was quite difficult to make it work as some limitations are in place. 

When the bot asked “What is your name?”, the expected utterance that acted as training data were along the lines of “My name is X” Or “I am X”. The Composer works with LUIS, a language understanding module. The convenient thing about LUIS is that it is trained on a larger scale and can thus easily work with a smaller amount of expected utterances. However, it was difficult to extract the name of the user, as a response along the lines of “My name is X” would see the Bot thinking that the name of the user was My Name Is X, instead of just X. Documentation on the possibilities of the Composer are rather constrictive and after much trying, the bot is now able to extract the name from an utterance.

Another hiccup was with the age. Similar problems arose as with the age, however the solution that worked with the extraction of the name, did not work with the extraction of the age of the user. 

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/ChatBotGif.gif” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

2nd Iteration User Testing

The user testing with the second iteration was more positive than the first iteration. The users enjoyed the way in which they communicated with the chatbot. It also felt familiar to normally chat with the chatbot, in contrast with the clickable elements from the first iteration.
Increased adaptiveness and responsiveness of the chatbot was a feedback point of both user testings. As such, the final iteration is an attempt on the integration of sentiment analysis into the chatbot. 

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/JoesTest.jpeg” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Final Iteration

The final iteration of the prototype was aimed at improving the responsiveness of the chatbot. For this reason, research was done into both Tensorflow and Azure Cognitive Services. Both cloud services provide excellent sentiment analysis.
However, Azure Cognitive Services proves to be the most beneficial for the prototype. The reason for this is that Tensorflow can only interpret the sentiment of a given paragraph. Additionally, the Tensorflow model needs training data from the creator of the model. As the research was not as broad scale, it would be more efficient to have access to a model with pre-trained models.
Azure Cognitive Service allows for better extraction of sentiment analysis through an API call. The API call is rather sophisticated as it also provides opinion mining. Which means that the model is able to see what the subject is of an opinion. E.g. from an utterance such as, “The pizza was great.” Azure is able to extract that the user’s opinion was regarding the “pizza” and that the opinion was “great”, which would return a positive sentiment. 

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/SentimentGif.gif” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Reflection

The project was fun to create. IBM Watson was easy to use and understand. However, it was rather restrictive in its free version capabilities. Luckily, the Bot Framework Composer offered many more possibilities. 
Additionally, it was difficult to find solutions to certain problems. Websites such as Stackoverflow are normally of high value for developers. A similar website for Composer does not exist. Which was frustrating, as the solution for the extraction of the user’s age was not found sadly. 

Another problem was the connection between the Azure Cognitive Services sentiment analysis API and the Composer. The API call needs a Python environment to properly send the request and receive the response. However, the Composer does not provide the adaptability for such a call, as only simple API calls are possible through the Composer. Rasa would be an interesting point of research as well. However, Rasa does need a lot of training data in comparison with Azure.

Lastly, more research has to be done on the extraction of sentiment from the chat logs. The Azure API call splits paragraphs up at the “.” mark. However, the question has to be asked, what part of the paragraph is most important for the bot to respond to? Is a positive sentiment more important than a negative sentiment?

Recommendation/Conclusion

Future recommendations for improved iterations are to integrate sentiment analysis from Azure if possible. The chatbot will be more adaptive and is able to both ask and interpret open ended questions. While the chatbot is able to acknowledge the user after asking for their name, it is not able to ask how the day of the user was. When fighting loneliness, it is important to create a chatbot that is able to ask “how was your day?”. Additionally, through sentiment analysis and perhaps even opinion mining, the chatbot is able to adapt to a user’s mood and ask questions accordingly. 

Furthermore, the chat logs of future iterations should be used to extract utterances of users that are not yet recognized by the model. E.g. if people often ask “how was your day?” to the chatbot, the designer should provide such an intent for the conversation flow. Thus, improving the adaptiveness of the chatbot. 

References

Hudson, J., Ungar, R., Albright, L., Tkatch, R., Schaeffer, J., & Wicker, E. R. (2020). Robotic Pet Use Among Community-Dwelling Older Adults. The Journals of Gerontology: Series B, 75(9), 2018-2028.

Krendl, A. C. (2021). Changes in stress predict worse mental health outcomes for college students than does loneliness; evidence from the COVID-19 pandemic. Journal of American College Health, 1-4.

Lippke, S., Fischer, M. A., & Ratz, T. (2021). Physical activity, loneliness, and meaning of friendship in young individuals–a mixed-methods investigation prior to and during the COVID-19 pandemic with three cross-sectional studies. Frontiers in Psychology, 12, 146

Padmanabhanunni, A., & Pretorius, T. B. (2021). The unbearable loneliness of COVID-19: COVID-19-related correlates of loneliness in South Africa in young adults. Psychiatry research, 296, 113658.

Panchal, N., Kamal, R., Orgera, K., Cox, C., Garfield, R., Hamel, L., & Chidambaram, P. (2020). The implications of COVID-19 for mental health and substance use. Kaiser family foundation.

Vasileiou, K., Barnett, J., Barreto, M., Vines, J., Atkinson, M., Long, K., … & Wilson, M. (2019). Coping with loneliness at University: A qualitative interview study with students in the UK. Mental Health & Prevention, 13, 21-30.

Premise

Exploring techniques that can ease calorie tracking and optimize the quality of nutritional data.

Synopsis

During other courses I already did some research into dietary applications and their pros and cons. The three most common ways to collect data are manual input, data collection by sensors and data collection by cameras. Manual input is a threshold for users to maintain their data collection, automated technologies such as smartwatches and barcode scanning can help users with this task. During this research I want to discover which technologies are already used and which technologies have the potential to take dietary data collection to a next level.

Background

Obesity numbers have tripled since 1975. 1,9 billion adults and 340 million children and adolescents were overweight in 2016 (World Health Organization, 2020). Obesity is a serious problem; it is associated with mental health problems and reduced quality of life. This includes diabetes, heart disease, stroke and several types of cancer (Adult Obesity, 2020). Behavior usually plays an important role in obesity (Jebb, 2004). Treatment for obesity is therefore often focused on food intake and physical activity. Maintaining a behavioral change is often difficult. The adherence of a diet to maintain weight loss is crucial. New technologies that can help users adhere a diet can make a real change (Thomas et al., 2014). There are applications that help users with calorie tracking but users have to give input themselves which is a threshold to maintain calorie tracking. Therefore, I want to discover technologies that can help with calorie tracking.

Experiments

Arduino

 This course started with a few Arduino workshops which enthused me to try to work with Arduino. At first, I did some research into Arduino compatible sensors. I inspired my research on the HEALBE GoBe3 device which I already did research on during another course. This device uses a bioelectrical impedance sensor to measure calorie intake. It turns out that bioelectrical impedance sensors are quite expensive and that I had to think of another way to measure calorie intake with Arduino. I found this project based on a game of Sures Kumar in which they can measure foods by their unique resistance value. I managed to make it work and measure the resistance which I could link to different fruits. However, the results were not very promising. The resistance values of the fruits would differ from time-to-time due to temperature differences and ripeness. Secondly, there was an overlap of values by different fruits which means that the system could not detect the right fruit. This means that this method is not suitable for precise food detection. Therefore I, switched to a method I already did some experimenting with.

[aesop_video src=”self” hosted=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/Schermopname-2021-06-13-om-12.20.45.mp4″ width=”content” align=”center” disable_for_mobile=”on” loop=”on” controls=”on” mute=”off” autoplay=”off” viewstart=”off” viewend=”off” show_subtitles=”off” revealfx=”off” overlay_revealfx=”off”]

IBM Watson

 I already did some experimenting with IBM Watson in the past. Therefore, I knew that image recognition is a technology that can recognize foods. During my experiment with IBM Watson, I managed to use the IBM food model. This model could detect images of fruit very well with a confidence score of 90 percent. The disadvantage of using a pre-made model is that you do not have the possibility to change or complement the model with your own data. Therefore, I tried to make my own model that could detect different apple types.The model was trained with 20 apples of each type and 20 negative images of fruits that were not an apple. After the training, the model showed different confidence scores per type of apple. I checked the training images with the lowest confidence scores and noticed why they did not achieve the same score as the rest. The images that scored the lowest were not present at the same amount as the images that scored a high score. This made me realize that it is important to be very cautious choosing which image is suitable for the training data. Another thing I realized was the limitation of a system such as IBM Watson. Training customization and free implementation options were limited. The challenge is to dive into image recognition technologies that could be implemented into python.

OpenCV

The first technology that caught my attention was OpenCV. They provide computer vision libraries and tools to execute real time computer vision. I completed a tutorial in which the computer could recognize a stop sign in a picture. This method used a single image and a pre-trained Haar classifier. During my search for a tutorial to make a custom Haar classifier I found a tutorial that could make realtime object detections. This would mean that users only have to open their camera and immediately see the object detection. This caught my attention and challenged me to follow a new tutorial which would teach me to create a real time object detection model. This was pretty easy; the model used the coco dataset. This dataset is known as one of the best image datasets available. It has around 200.000 labeled images and 80 different categories. I was surprised how easy it was to install this model and detect objects in real time. It could detect 80 different objects including persons, cellphones, bananas and apples. Although this dataset worked really well it was not suitable for my goal. It could not detect any fruits other than bananas and apples. A solution for this problem was the creation of my own coco dataset that would be focused on fruits. Unfortunately, due to my limited programming knowledge and regular errors I could not manage to make this work. Therefore, I tried to find a tool that could help me to make my own custom dataset.

[aesop_video src=”self” hosted=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/Schermopname-2021-06-13-om-12.23.56.mp4″ align=”center” disable_for_mobile=”on” loop=”on” controls=”on” mute=”off” autoplay=”off” viewstart=”off” viewend=”off” show_subtitles=”off” revealfx=”off” overlay_revealfx=”off”]

Roboflow and Yolov5

First iteration

After some research I found a tutorial that showed me how to make a custom object detection model for multiple objects. This tutorial showed me how to use the YOLOv5 (You Only Look Once) environment to train the model and Roboflow to annotate the training images. Roboflow is a program that makes it possible to annotate every image and export the annotations into the filetype that you need. In this case I exported my file to a YOLOv5 PyTorch file. The annotation process was a lot of work because I had to draw bounding boxes on every picture. At first, I used a dataset from Kaggle. This was a dataset of 90.000 images consisting of 130 different fruits or vegetables. I selected 600 images for 6 different fruits. The images were already cut out but had to be annotated still. The results after training these images were not very good. It could not recognize the images very well. The images below show that the model created wrong bounding boxes and predictions.

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/wrong-results.jpeg” panorama=”on” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Second iteration

Therefore, I created a new dataset that I collected myself. This time I used a mix between images of fruit with a blank background and images of fruits with a mixed background. This time the results of the training were much better. I used YOLOv5 because this model is easier to implement than YOLOv4 and the results are similar (Nelson, 2021). The image below shows the score of different elements. The most elements show a flattening curve at the top which shows there were enough training iterations.  The mAP@0,5 (mean average precision) scored above the 0,5 which is commonly seen as a success (Yohanandan, 2020). Although the training of my model was a success, I still could not manage to connect the model to a camera. This is a disappointment and a challenge in future projects.

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/06/Good-graph.png” panorama=”on” imgwidth=”content” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

The images above show a wrong and a good prediction. Although the prediction of the lemon is wrong, it is interesting to see that it is recognized for a banana. This is understandable because there are training images of bananas that look similar.

Future Research

Image recognition is a technology that has an enormous potential for food recognition. There are a lot of datasets and models available that are perfectly capable of recognizing fruits. Which would be a very useful addition to dietary applications when linked to a database with the caloric values. The next step would be a model that could recognize sizes or portions of food. This step would improve the quality of dietary intake data.

Conclusion

 My search for technologies that could support dietary apps was successful. I enjoyed working with Arduino and came across many amazing projects. However, I learned more during my dive into the different image recognition techniques. I discovered how many different models there are and how they work. My lack of programming knowledge was sometimes frustrating because I could not finish some projects I started with. Nevertheless, the search for solutions or different methods gave me new insights that I otherwise would not have acquired. It would be nice if I could implement some of the techniques, I learned during my graduation project.

References

Nelson, J. (2021, 4 maart). Responding to the Controversy about YOLOv5. Roboflow Blog. https://blog.roboflow.com/yolov4-versus-yolov5/

Yohanandan, S. (2020, 24 juni). mAP (mean Average Precision) might confuse you! – Towards Data Science. Medium. https://towardsdatascience.com/map-mean-average-precision-might-confuse-you-5956f1bfa9e2