PREMISE

Detecting the emotional state of the employees with the computer vision technique and applying music therapy to increase productivity and efficiency of the employees.

SYNOPSIS

During my term as an industrial engineer, the biggest problem of companies was that the productivity and efficiency of their employees were not stable. This is due to the fact that the emotional status of the employees varies on different days or within the same day. Music was one of the methods used to change the emotional states of me and other employees in such situations. Employees apply music therapy to themselves, both consciously and unconsciously. In this study, in order to increase the awareness of the employees’ emotional status, a song recommendation music therapy will be applied that will make the employees aware of their emotions and change their current emotion status, with the emotion status detection model to be developed with computer vision.

PREFACE

Music is an ubiquitous piece of art that has an important place in people’s lives and can be found at any time of the day in order to change their thoughts, emotions and distract their attention. While people are listening to music, their minds are imprinting the music into their minds with their current emotions without being aware of it, and whenever they listen to that song, the mind invokes these emotions without the person being aware of it (Jäncke, 2008). Because of this feature of the human mind, music therapies have been started to be applied as well as psychotherapies. While psychotherapies focus on changing people’s perspectives on their problems, music therapies are a type of therapy used to improve people’s emotional states (Mao, 2022). The emotional state is one of the factors that affect people’s productivity and efficiency. For example, while rap music makes people feel more positive, depressive music makes people feel melancholy.

Employees cannot work as productively or efficiently as they would like when they are upset, stressed, or frustrated. In other words, employees who are not happy while working have difficulty showing their performance (Mao, 2022). In addition, since there is no department in the companies regarding the psychological or emotional states of the employees, the employees try to cope with this problem on their own (Zhou, 2018). To give an example, when the famous scientist Albert Einstein could not find enough inspiration to write, he changed his current emotion by playing the violin and returned to writing again (Mao, 2022). The aim of this project is to determine the unhappy emotions of the employees during the day and to recommend music in order to turn their current emotional status into happiness with music therapy.

PROTOTYPING

PREPARATIONS

In order to understand and apply how Computer Vision technology works in practice, I did research on medium, google scholar, and various sources. My technical background and level of knowledge made it easier for me to obtain information on how this technology can be applied. As a result of the research I have done, I learned that CV technology is a method that is formed by the combination of deep learning and machine learning algorithms and is used in fields such as object detection, motion tracking, action recognition, human emotion recognition, and that it can be created both with and without coding. Since I preferred the non-coding method in this study, I used the Lobe application, and I watched videos related to this subject on YouTube in order to understand how this application works. In order to train the Emotional status recognition CV model, I first started by scraping the photos on Google images. I gathered the image data collected from hereunder 2 different labels happy and unhappy. The gender and emotion distribution of the image data is shown in Graph -1. However, many factors such as the light level in the photographs, the accessories on the people, gestures, and mimics affect the learning of the system while applying emotions. In order to prevent erroneous learning of the system, I included grayscale and color photos, photos taken both outdoors and indoors, and photos with and without accessories in both emotion data sets. Thus, I have ensured that the system understands emotional states as accurately as possible. It is important for the system to understand the emotional states correctly in order to correctly recommend the music to be recommended. I scrapped the songs to be used in music therapy and the features of the songs from the Spotify API using python and collected them in a csv file. Figure 1 shows how song features are gathered from Spotify API.

Iteration -1

Although the accuracy value of the model trained using the Lobe is 92%, the predictions made during the test phase do not fully reflect reality. The system cannot define an Asian person as happy during the testing phase. In my opinion, the reason for this is that his eyes get smaller when he smiles and his face looks like a sad emotion is formed. In order to solve this problem, images of Asian people’s emotions were included in the model and retrained. Also, the system cannot fully understand the unhappy emotion of people whose facial features are not obvious. In order to solve this problem, the data set has been retrained by adding data with this face feature to the system. The new accuracy value created at the end of these processes is 96%.

Before Improvement                                             After Improvement           

Iteration -2

In order to be able to change the emotional state of people, 2 playlists were created to understand if the recommended song is more effective than the songs they have listened to or not listened to before. One of the playlists consists of songs that the users may have heard before, as it is taken from the Spotify popular songs playlist, while the other consists of newly released songs. Before the test, the participants were offered songs from both lists, and the emotional status of the participants was determined by the CV model. If the song from the popular song’s playlist has not been listened to by the user, then he/she would be asked to say a song they like, and the user test continued with this song. Following the user testing, a survey was given to the users, and they said that the songs they listened to before were more effective in changing their emotional status. The survey includes questions such as felt emotional state, the emotional state according to Lobe, listened to popular and newly released songs if there was a change in their emotional state afterward, and which song helped them. I made face to face usability test with 10 people and, the distribution of the survey result is shown in Graph 2.

Reflections

The process of understanding CV technology and developing the prototype can be summarized as quite challenging but also fun and exploratory. The reason I chose to study this technology, which I had not experienced before, was to push my limits. Learning, understanding, and applying a new subject has been a very developmental gain for me. Being able to do this in such a short time is proof that I have proven myself. Searching for many resources, reading articles, and benefiting from educational content was like sailing to new horizons. Exploring new technologies, Spotify’s python libraries, examining API documents, and understanding how the system works increased my desire to learn and my passion for this subject. Overcoming the difficulties, I faced with limited resources in limited time and producing solutions improved my problem-solving ability. Gaining new skills with this experience increased my self-confidence.

CONCLUSION

The emotional status changes experienced by the employees reduce their productivity and motivation from time to time. It has been revealed as a result of various studies that music has an effect on the emotional states of people. Based on this, I developed an application that will follow the employees at random times of the day and make them happy with music therapy in an unhappy emotional state. This application aims to get people out of their unhappy moods by suggesting songs with a happy mood among the songs they have listened to before. For the prototype, it was ensured that music suggestions were made according to the relevant mood on the data set, which was trained by using images of people with various facial expressions and mimics.

References

Alake, R. (2021, December 16). A Beginner’s Guide To Computer Vision – Towards Data Science. Medium. Retrieved March 28, 2022, from https://towardsdatascience.com/a-beginners-guide-to-computer-vision-dca81b0e94b4

Jäncke, L. (2008). Music, memory, and emotion. Journal of biology, 7(6), 1-5.

Lobe. (2020, October 26). YouTube. Retrieved March 25, 2022, from https://www.youtube.com/c/Lobe_ai/featured

Mao, N. (2022). The role of music therapy in the emotional regulation and psychological stress relief of employees in the workplace. Journal of Healthcare Engineering, 2022.

Rao, M. A. (2021, December 28). Realtime Face Emotion Recognition using transfer learning in TensorFlow. Medium. Retrieved March 30, 2022, from https://medium.com/analytics-vidhya/realtime-face-emotion-recognition-using-transfer-learning-in-tensorflow-3add4f4f3ff3

Web API | Spotify for Developers. (n.d.). Spotify. Retrieved January 3, 2022, from https://developer.spotify.com/documentation/web-api/

X. Juan, “The practice path and significance of group music therapy in the psychological treatment of orphans and disabled children,” Contemporary music, no. 12, pp. 37–39, 2021.

X. Zhou, “Skillful use of music therapy to relieve the learning pressure of high school students,” Psychological Monthly, no. 10, pp. 15-16, 2018.