Ubiquitous Computing | Tan Zhou

Voice Interaction Design: Food Journaling via Voice Assistant

Thu, 07 Mar 2019 06:32:08 +0000

Overview

My Role: UX Researcher and Designer

Methods: Literature review, Interviews, Competitive analysis, Journey mapping, Heuristic evaluation

Data Sources: Interview transcripts, Literature

Deliverables: Research report, Dialogic flow, Sample conversations

Tools: Google Assistant, Amazon Alexa, Google Sheet

Background/Context: Studies following diabetic patients and weight watchers found food journaling to be an effective means of managing one’s diet. Although automating the journaling process using smart devices could increase adherence by decreasing the effort and mental burden required, it could also lead to a decrease in users reflecting on collected data. After searching the field, I failed to find a well-designed voice-based interface that supports food tracking.

Project Overview: The overarching goal was to answer the question “When people track foods they eat daily (food journaling) via a voice assistant, how can we design the dialogic flow to facilitate users’ reflections on their eating habits?”.

Table of Contents

Objective

Understand users needs and pain points in food journaling
Design the flow of the conversation and its underlying logic to facilitate voice-based food journaling
Adapt Nielsen’s heuristics to evaluate voice-based interface

Opportunity and Process

Opportunity

The study of automated food journaling show that such automation could lower the burden of tracking and increase adherence (Beenish et al., 2016), but it could lead to a decrease in users reflection on collected data(Choe et al., 2014). Also, the majority of these studies leverages only graphical interfaces, building on the legacy of hand-and-finger input devices.These approaches are limited by the required input since users might not always be able to log their entries.

Voice assistants like Amazon Alexa, Apple Siri, and Google assistant are getting more and more ubiquitous. Their support of hands-free interaction makes voice-based food journaling an ideal use case for voice assistants.

Voice Assistants are more accessible than ever. Photo: Stratablue.com

Process

In order to define the users’ requirements, I conducted a competitive analysis of existing tools that support voice-based food journaling. I also interviewed two participants who have previously used their mobile phones for food journaling to understand users’ needs, behaviors, and motivations of the user during the journaling process.

Using the information collected from the field review and interviews, I wrote a series of sample dialogues to capture the “sound-and-feel” of the interaction under different scenarios. These sample dialogues convey the flow that the user will experience and allow me to experiment with different design strategies, such as how to promote the discoverability of new features or how to confirm a user’s request.

At the usability test stage, a friend who is unfamiliar with the project was asked to role-play the simple dialogues with me. This helped me curate the conversation, defining the flow and the underlying logic that represents the complete food journaling experience. I also conducted system evaluations with a set of adapted heuristics to expose usability issues.

Strategy

Exploratory Interviews

My initial goal of exploratory interviews was to understand users’ needs in the journaling process. Two interviews were conducted with informants who had experience journaling the food they ate.

Heuristic Evaluation

1. Awareness of system status

The Nielsen heuristics emphasize the visibility of system status. Although visibility does not apply to voice-based interfaces, user awareness and feedback are still important. The system needs to inform users about what it is doing in a timely and appropriate fashion.

2. Error prevention

As with any system, it is best to prevent errors from occurring or handle them in a way that is less intrusive to user experience. This becomes even more important in a system with no visual interface since people cannot “unsay” what they have previously said. This system addresses this issue by providing users their options for the next step explicitly in the conversation.

3. Flexibility and efficiency

Novice users become experts after they become familiar with a system, and experts benefit from efficiency. As a result, some instructions, which would be helpful to novice users, may become redundant to experts. The logic flow supporting setting up customized shortcuts for frequently used terms to speed up the process.

4. Accessibility

Voice based interaction is great for people who are unable to use a graphic interface. It doesn’t require input from hands which offers more accessibility since users could interact with the system while carrying groceries, cooking a meal, or driving a car.

5. Ambiguity

People don’t communicate in syntax the way computers do. They sometimes use metaphors or slang. They sometimes forget words or pause when speaking. Voice technology should accommodate the users’ communication style and needs. When the user pauses, the system just repeats the previous response with a much more detailed instruction of what users can do until they make a selection.

6. Discoverability

The invisibility of a voice-based interface makes it difficult for users to explore new ways of interacting with the system. In this logic flow, new actions will be introduced in the form of quick tips at the end of each conversation (except those involving specific customized shortcuts). During the reflection stage when a user views their journals on a display, the system suggests new interactions that are more efficient that would increase the accuracy of the users’ food journaling.

7. Multimodal Reflection

In the five-stage personal informatics model proposed by Li et al.(2010), the reflection stage may involve looking at lists of collected personal information or exploring or interacting with information visualizations. For a voice-based system, exploration is inherently difficult for exploration and visualization is impossible. To address this, the logic flow allows for short-term reflection by repeating and confirming the list of input items during every conversion. The system also expands to another modality for long-term reflection. The design flow falls back to a visual interface after collecting information. A user can access their journaling data on their smartphone and on the website where auxiliary graphical interfaces would be available. However, this part is not accomplished during this project.

Outcomes

Dialogic flow for food journaling via voice assistant
Sample dialogue for voice-based interactions

The logic flow of conversation shows four stages of the voice-based food journaling system. In the beginning, users start with the voice assistant by saying specific trigger words. In the collection phase, novices and experts can take different approaches. The reflection stage suggests a multimodal approach – short-term reflection is embedded in the conversation while long-term reflection involves reports and visualizations viewed on another display. At the final stage, users can take actions after they reflect on their food journal. They can correct something that was incorrectly recorded via speech, set a shortcut for more efficient interaction, or explore new ways to interact with the voice assistant.

Sample Conversations

A novice user adding Food. (Inputs from the user is highlighted)

- Hi My Food Journal,

- Hi! Welcome to My Food Journal. I can help you add new food to the journal or check previous journals. Which would you like?

- Add new food.

- Okay, go ahead and say the names.

- Ummm…

- You can add food by saying its name and serving size. Go ahead.

- One in-n-out double-double, French Fries, and Chocolate shake.

- Got it. Would you like to add more?

- No, that’s it.

- Sure. I got one in-n-out double-double, a medium French Fries, and a medium chocolate shake. Is that correct?

- Yup.

- Alright. Quick tip: You can always add more food by saying “Hi My Food Journal, quick add.”

In the sample conversation, note that the system always provides an affirmation when it gets any information from the users. At the end of each conversation, new actions are introduced in the form of quick tips to increase discoverability.

Quick add and customized shortcuts (Inputs from the user is highlighted)

- My Food Journal, quick add.

- (A response tone playing indicating the system is listening)

- “McDonald Lunch Combo”

- (A confirmative tone playing indicating the system have received your information)

In this example, a user has set up a shortcut named “McDonald Lunch Combo” which includes a list of food they usually get from McDonald. This way the user could skip the instructive steps and complete the journaling in a much more efficient fashion.

Conclusion/Reflection

The focus of this project was on the bottom-up process of conversation design. After gathering insights from competitive analysis, interviews, sample conversations, and adapted heuristic evaluations, I delivered a dialogic flow for food journaling via voice assistant and sample conversations. The next step would be to expand the dialogs based on the flow and implement the system using a real-world voice assistant platform.

Quantitative Analysis: Personal Data Sticker

Mon, 01 Jan 0001 00:00:00 +0000

Overview

My Role: User Researcher working under a leading researcher

Methods: Survey, R, Factorial design, Factorial Analysis, and Literature review

Data Sources: Literature review and resulting survey

Deliverables: Research report describing users’ perceptions of social media stickers containing personal informatics

Tools: R Studio, JSON, Windows Powershell, Photoshop, and Google Doc

Background/Context: When people set health and behavior goals such as training for a half-marathon, going to bed earlier, or losing weight, they often use social media to share their progress with friends and family (Munson, 2012). Epstein et al. (2015) identified that many people share personal informatics data to receive advisory information, emotional support, motivation, or accountability from their audience. However, prior research has shown that sharing this data on social media generally doesn’t not result in the users’ desired outcomes.(Epstein, 2019).

Project Overview: I explored design principles for incorporating users’ step counts, one of the most commonly-tracked and shared pieces of personal informatics, into stickers in preparation for creating a similar feature for use in Snapchat posts or Instagram Stories.

Table of Contents

Objective

When users send or receive Snapchat stickers containing their personal step count data, what factors influence the users’ perceptions of those stickers? And how do these factors actually impact the users’ perceptions?

Opportunity and Process

Opportunity

In Snapchat or Instagram stories, people are able to use stickers to customize their posts for informative, aesthetic, or entertainment purposes. Available stickers are organized in a variety of categories depending on their content, format, style, etc. Currently, these stickers include methods for sharing some types of data, but the data is rarely about the user themself and instead typically pertains to the user’s location or surroundings. (Habib et al., 2019). This gives me an opportunity to design a series of stickers to support more types of user-generated content focused on users’ reasons for sharing.

Process

Through a review of research on multimedia and online advertising, I identified a series of predictors that may influence Snapchat users’ perceptions of stickers. I also determined the evaluation metrics for measuring these user perceptions. My team then designed a series of stickers for Snapchat that contain personal data about step counts. Finally, I conducted an online survey to determine how those stickers’ designs influenced users’ perceptions.

Samples of data-driven stickers incorporating step counts

Strategy

Experiment Design

I used a factorial study design to evaluate varying levels of context, presentation, and style’s influence on participant perception and preference. After they consented to participate in the study, participants were asked to identify one person who they frequently Snap with to imagine as their conversation partner. They then gave feedback on six (6) randomly generated posts featuring data-driven stickers, answering some demographic questions upon completion.

The post generated varied on three dimensions:

Presentation styles

Badge style annotates objects with the specific tracked value, for example a shoe or ribbon with “5,793 steps” written on it.
Embellished style presents common objects as charts, picking one dimension to be the axis and shading the object partway according to the tracked value.
Analogy style re-expresses tracked values as better-known quantities through comparisons.

Relevance levels

Domain-relevant Domain-relevant designs use objects or comparisons specifically related to steps, such as snicker, track field.
Domain-irrelevant Domain-irrelevant designs use well-known objects and comparisons that are not commonly associated with steps, such as a star or a speedometer.

Background styles

Background of the post is a photo of specific scenario
Background is from public CSS patterns with abstract shapes

	Domain-relevant	Domain-irrelevant
Badge
Embellished
Analogy

User Perception measurements

Participants answered questions from four widely-used scales from online marketing and advertising literature, modified to this context. The validated scale measures:

Entertainment Value: how entertaining the shared content is

Attitude: attitude toward the content

Intention to use: How inclined a user find to use this feature

Information Value: how informative a user find receiving the content

Privacy Considerations: how invasive the a user find sharing the content

Participants answered each question on a 7-item Likert scale with endpoints “Strongly Disagree” and “Strongly Agree”.’

Survey Sampling and Participant Screening

I used convenience sampling to gather information on adult Snapchat users’ opinions of the stickers. I sent out recruitment emails to a student research subject list, and distributed flyers with a survey link in university classrooms and meetings.

Only participants who were at least 18 years of age and who, on average, sent or viewed at least one post on Snapchat per week were chosen to participate in the study

Outcomes

The regression analysis showed that:

Users’ perceptions of stickers they’ve received correlated more significantly with our independent variables, compared with sharers’ perceptions.
And that influence that factors have on the two sides – Sharer and Recipient – are not received equally.
When the relevance of stickers’ presentations changes from irrelevant to relevant, the ratings increase in both sharers and recipients’ intention to use, recipients’ level of entertaining, as well as informative.
Context is significant to ratings of Attitude and Entertaining from recipients.
Stickers design with “Analogy” is perceived to be more exciting to receive than “Plaintext”.

Conclusion/Reflection

In the project, I explore design principles for incorporating self-tracked step counts in data-driven stickers as a first step towards integrating these data into Snapchat posts or Instagram Stories. I examine the effect of a sticker’s presentation style, domain-relevance, and background through surveys. We uncover the importance of domain-relevant backgrounds and stickers, identify the situational value of stickers styled as analogies, embellished, and badges, and demonstrate that data-driven stickers can make content more informative and entertaining.

Some limitations of the study: I only received valid answers from 19 participants. This means that our regression analysis results are highly troubled by sampling error. Thus, it is unlikely to draw statistically valid conclusion from such a small sample size. Another flaw in the analysis is that no interaction terms were included in the model, due to that our data sample were simply not powerful enough to discern any differences.

However, considering the pilot nature of the project, the process I took to reach these conclusions provided meaningful learning experience. Indeed, the learnings generated from this pilot directly influence the framing of research questions and analysis approach when the pilot was later developed into a larger, more comprehensive study with 506 total participants. The findings of the large-scaled study were published in Proceedings of the ACM on Human-Computer Interaction in October 2020.