Premise

Understanding food consumption routines of consumers while doing grocery shopping.

Synopsis

In recent years I have come to realize more and more the seriousness of the climate crisis and I also believe that, as a consumer, I can contribute to solving this crisis by adjusting my food consumption and diet. And I am not the only one, research shows that a large part of the inhabitants of The Netherlands are aware that they can play a role in solving sustainability problems and are willing to consume more sustainably (PBL, 2020). Nevertheless, it appears in practice that the shift to a more sustainable diet is difficult to achieve. I also recognize this in my own behavior, I have barely adjusted my consuming behavior and diet in recent years for the benefit of the climate. By experimenting with different technologies I tried to build a tool that helps me to get insights into what sociocultural routines facilitate or make it more difficult for consumers to choose more sustainable and plant-based meals in their daily lives.

Changing food consumption

Changing our food consumption can help reduce the ecological impact of the food production. In particular, eating patterns in which the consumption of animal proteins are reduced make a relatively large contribution to reduce the ecological impact of the food production. This is because animal products have a greater impact on the environment than plant products (Nijdam 2011; Tilman & Clark 2014).

In addition to consumers, other parties (such as the government, producers, restaurants and grocery stores) also play an important role in influencing our consuming behavior. Yet the transition to a more sustainable lifestyle can only take place if it is supported by consumers. My wish is that we as consumers are actually going to make more plant-based and therefore sustainable choices. The approach to sustainability in our consumption can be tackled for many different situations. For now, I focus on our food consumption at home.

Research

To better understand why people aren’t making sustainable choices I did some literature research. Different researches show that consumption behavior is not (only) influenced by concerns about climate related problems but more by the physical and social environment in which consumption takes place.

We are stuck inside our routines

Consumers often revert to previously learned values and preferences while performing a routine, without thoroughly considering their motives and actions each time (Warde 2016). Examples of these routines are for example grocery shopping or cooking. In 2020, the Netherlands Environmental Assessment Agency (Planbureau van de Leefomgeving) published a study that describes consumption behavior as undertaking (or performing) collective, socially shared and culturally anchored “social practices” (PBL, 2020). According to the research, sociocultural routines consist of different, interrelated elements:

• Meanings: the reasons why people perform certain routines the way they do (including motivations and the significance of certain products and their properties in the routine.) 
• Skills: the knowledge and skills required to perform a particular routine properly.
• Materials: physical aspects →  food environment, food products and cooking materials

Interventions

The optimal strategy for addressing the impact of our food consumption is to cultivate significant changes in the dietary and consumption behaviour of consumers (PBL, 2020). Nowadays many people are connected to the internet by various technologies. Therefore, digital interventions offer an ideal opportunity to engender necessary changes in dietary behaviour (Chen et al., 2020). A digital behaviour change intervention (DBCI) is defined as ‘ . . . a product or service that uses computer technology to promote behaviour change’ (Michie et al., 2017). DBCIs are responsive to dynamic contextual factors that could influence the user’s behavior and tailor aspects of the intervention based on these contextual factors.

During creating this scheme (below) I realized that I have to focus on finding the moment when we decide what we are eating (for dinner, lunch, breakfast, snacks) in different situations. In order to visualize when these moments are occurring, I want to map the routines of consumers.

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/03/Scannable_document_op_10_mrt._2021_11_23_01.png” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Experiments

In order to get more insight into consumers’ food consumption routines, I want to know when and where the user does his shopping and what they are buying. To make this insight possible, I developed an application that tracks the location and asks the user to record their consumption by taking pictures of their receipts. The ultimate aim is to combine this data and make it visual in a dashboard. While developing, I experimented with and combined different technologies. 

Step 1: Sharing location.

To detect where a user is, I am using Geolocation API.

Geolocation API

The Geolocation API allows the user to provide their location to web applications if they so desire. For privacy reasons, the user is asked for permission to report location information (Geolocation API – Web APIs | MDN, 2020).

Step 2: Detects when someone is inside a supermarket.

Google Stores Api

To detect whether someone is visiting a supermarket, I intended to use the Google Places Api. Unfortunately you have to pay to use the API. To still be able to use the technology, I faked the supermarket locations without the Google Api.

Step 3: Notifying  the user and remembers to upload receipt

Botpress

I wanted to use a chatbot in combination with the Geolocation API to notify the user to upload their receipt after they visited a grocery store. Botpress wrote a promising article on location-based chatbots. Botpress is an open-source all-in-one bot creation platform that provides all the tools you need to build, debug and deploy AI-based conversational assistants (botpress, 2021). While working with botpress I tried to implement the Geolocation API in the bot. But this unfortunately didn’t work. There were no solutions on the internet, and neither in the Botpress documentation. Therefore I decided to switch to developing a chat app with Socket.io. 

Sockets

Socket.IO is a JavaScript library for realtime web applications. It enables real time communication between web clients and servers. Thus, with socket.io you can create a chat app and send notifications. It has two parts: a client-side library that runs in the browser, and a server-side library for Node.js. 

Websites built with Node.js and socket.io can be deployed as Progressive web apps. A PWA is a website built using web technologies but acts and feels like an app.

The browser uses the geolocation to keep track of when and to what coordinates the user moves. This information will be sent to the server. The server checks whether the user is in a supermarket. If so, after leaving the supermarket, the server sends a message to the user with a reminder to upload a receipt.

The code I wrote for detecting the user’s location can be found here on Github.

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/03/steps-sterre-scaled.jpg” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” caption=”Notifying the user.” captionposition=”center” revealfx=”off” overlay_revealfx=”off”]

 

Step 4: uploading receipt.

Nanonets

To store what the user is consuming I am using an OCR system. OCR stands for: Optical Character Recognition. It’s a system that transforms a two-dimensional image of text, that could contain machine printed or handwritten text from its image representation into machine-readable text (Zelic, 2021). There is a lot of good software or APIs out there, like: Wave Accounting, Verify and Taggun Unfortunately you have to pay for a lot or leave your credit card details. However Tesseract is an open source text recognition engine. It can be used directly, or (for programmers) using an API to extract printed text from images.

Working with Tesseract needs a lot of preprocessing and takes a few days to a couple of weeks for training from scratch. However, The Nanonets OCR API allows you to build models a lot faster. You do not need to pre-process the images or worry about matching templates or build rule based engines to increase the accuracy of the OCR model. I uploaded example receipts into Nanonets, annotated it and trained the model. After training the model, I am able to upload the receipt and it will get all the products on the receipt. Unfortunately, I was only able to train my dataset twice before I had to pay for it as well. Nevertheless, this tool has given me an insight into the technology and possibilities of scanning recipes. This is why the results of the coupons I enter are not accurate (see picture below).

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/03/Schermafbeelding-2021-03-29-om-16.31.25.png” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Tesseract

Since I could only train my dataset twice with the help of Nanonets and to make another iteration, I decided to still develop a more accurate selection of products from receipts with Tesseract. With the help of Tesseract and the digital coupons from Lidl I have been able to save the products from the coupon in a CSV file, along with the location, date and time of purchase. The disadvantage is that for now only Lidl coupons can be used for this, since the code is completely customized for the Lidl receipt structure. Receipts from other supermarkets will likely need a custom code.

To keep the accuracy of the tesseract output from dropping, the image was first preprocessed. Subsequently, all text of the receipt is read by a Python package from Tesseract and placed in a .txt file. The date, time, location and product data was selected and cleaned with the help of Python and exported to a .csv file (see image below).

The code I wrote for collecting, cleaning and merging the data can be found here on Github.

[aesop_image img=”https://cmdstudio.nl/mddd/wp-content/uploads/sites/58/2021/03/Schermafbeelding-2021-05-15-om-17.47.43.png” panorama=”off” align=”center” lightbox=”on” captionsrc=”custom” captionposition=”left” revealfx=”off” overlay_revealfx=”off”]

Next steps

As soon as the data can be collected in the right way, the different types of products still have to be distinguished from each other. The next steps would be exporting the products list and combining it with the location data and visualizing this into a dashboard. For visualizing the data I could use a tool like PowerBi.

Conclusion

Before I started experimenting with the technologies, I did a relatively large amount of research. I had a hard time already trying to experiment without having a good understanding of the problem and I clear idea on what I wanted to work on. It took me a while to switch from my design thinking routine to the critical thinking and creating mindset. It however triggered my mind in a different way, critical making is something I would like to get better at. 

During this process I noticed that there are already very good technologies that I could have applied to this project. Unfortunately, you have to pay for many of these software or APIs. While trying out what works best, I notice that it takes a lot of time to really set this project up properly yourself, if you don’t want to pay for certain services. Because I haven’t been able to test and use this project with real users yet, I did not gain insights into food consumption routines.

References

botpress. (2021, January 20). botpress/botpress. GitHub. https://github.com/botpress/botpress

Chen, Y., Perez-Cueto, F. J. A., Giboreau, A., Mavridis, I., & Hartwell, H. (2020). The Promotion of Eating Behaviour Change through Digital Interventions. International Journal of Environmental Research and Public Health, 17(20), 7488. https://doi.org/10.3390/ijerph17207488

Geolocation API – Web APIs | MDN. (2020, September 15). MDN. https://developer.mozilla.org/en-US/docs/Web/API/Geolocation_API

Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017). Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. Journal of Medical Internet Research, 19(6). https://doi.org/10.2196/jmir.7126

Warde, A. (2014). After taste: Culture, consumption and theories of practice. Journal of Consumer Culture, 14(3), 279–303. https://doi.org/10.1177/1469540514547828

PBL Planbureau voor de Leefomgeving. (2019). Dagelijkse Kost: Hoe overheden, bedrijven en consumenten kunnen bijdragen aan een duurzaam voedselsysteem (No. 2638). https://www.pbl.nl/sites/default/files/downloads/PBL-2019-dagelijkse-kost-bijdragen-aan-duurzaam-voedselsysteem-2638.pdf

Zelic, F. (2021, March 9). [Tutorial] OCR in Python with Tesseract, OpenCV and Pytesseract. AI & Machine Learning Blog. https://nanonets.com/blog/ocr-with-tesseract/