Usability Evaluation | Tan Zhou

Voice Interaction Design: Food Journaling via Voice Assistant

Thu, 07 Mar 2019 06:32:08 +0000

Overview

My Role: UX Researcher and Designer

Methods: Literature review, Interviews, Competitive analysis, Journey mapping, Heuristic evaluation

Data Sources: Interview transcripts, Literature

Deliverables: Research report, Dialogic flow, Sample conversations

Tools: Google Assistant, Amazon Alexa, Google Sheet

Background/Context: Studies following diabetic patients and weight watchers found food journaling to be an effective means of managing one’s diet. Although automating the journaling process using smart devices could increase adherence by decreasing the effort and mental burden required, it could also lead to a decrease in users reflecting on collected data. After searching the field, I failed to find a well-designed voice-based interface that supports food tracking.

Project Overview: The overarching goal was to answer the question “When people track foods they eat daily (food journaling) via a voice assistant, how can we design the dialogic flow to facilitate users’ reflections on their eating habits?”.

Table of Contents

Objective

Understand users needs and pain points in food journaling
Design the flow of the conversation and its underlying logic to facilitate voice-based food journaling
Adapt Nielsen’s heuristics to evaluate voice-based interface

Opportunity and Process

Opportunity

The study of automated food journaling show that such automation could lower the burden of tracking and increase adherence (Beenish et al., 2016), but it could lead to a decrease in users reflection on collected data(Choe et al., 2014). Also, the majority of these studies leverages only graphical interfaces, building on the legacy of hand-and-finger input devices.These approaches are limited by the required input since users might not always be able to log their entries.

Voice assistants like Amazon Alexa, Apple Siri, and Google assistant are getting more and more ubiquitous. Their support of hands-free interaction makes voice-based food journaling an ideal use case for voice assistants.

Voice Assistants are more accessible than ever. Photo: Stratablue.com

Process

In order to define the users’ requirements, I conducted a competitive analysis of existing tools that support voice-based food journaling. I also interviewed two participants who have previously used their mobile phones for food journaling to understand users’ needs, behaviors, and motivations of the user during the journaling process.

Using the information collected from the field review and interviews, I wrote a series of sample dialogues to capture the “sound-and-feel” of the interaction under different scenarios. These sample dialogues convey the flow that the user will experience and allow me to experiment with different design strategies, such as how to promote the discoverability of new features or how to confirm a user’s request.

At the usability test stage, a friend who is unfamiliar with the project was asked to role-play the simple dialogues with me. This helped me curate the conversation, defining the flow and the underlying logic that represents the complete food journaling experience. I also conducted system evaluations with a set of adapted heuristics to expose usability issues.

Strategy

Exploratory Interviews

My initial goal of exploratory interviews was to understand users’ needs in the journaling process. Two interviews were conducted with informants who had experience journaling the food they ate.

Heuristic Evaluation

1. Awareness of system status

The Nielsen heuristics emphasize the visibility of system status. Although visibility does not apply to voice-based interfaces, user awareness and feedback are still important. The system needs to inform users about what it is doing in a timely and appropriate fashion.

2. Error prevention

As with any system, it is best to prevent errors from occurring or handle them in a way that is less intrusive to user experience. This becomes even more important in a system with no visual interface since people cannot “unsay” what they have previously said. This system addresses this issue by providing users their options for the next step explicitly in the conversation.

3. Flexibility and efficiency

Novice users become experts after they become familiar with a system, and experts benefit from efficiency. As a result, some instructions, which would be helpful to novice users, may become redundant to experts. The logic flow supporting setting up customized shortcuts for frequently used terms to speed up the process.

4. Accessibility

Voice based interaction is great for people who are unable to use a graphic interface. It doesn’t require input from hands which offers more accessibility since users could interact with the system while carrying groceries, cooking a meal, or driving a car.

5. Ambiguity

People don’t communicate in syntax the way computers do. They sometimes use metaphors or slang. They sometimes forget words or pause when speaking. Voice technology should accommodate the users’ communication style and needs. When the user pauses, the system just repeats the previous response with a much more detailed instruction of what users can do until they make a selection.

6. Discoverability

The invisibility of a voice-based interface makes it difficult for users to explore new ways of interacting with the system. In this logic flow, new actions will be introduced in the form of quick tips at the end of each conversation (except those involving specific customized shortcuts). During the reflection stage when a user views their journals on a display, the system suggests new interactions that are more efficient that would increase the accuracy of the users’ food journaling.

7. Multimodal Reflection

In the five-stage personal informatics model proposed by Li et al.(2010), the reflection stage may involve looking at lists of collected personal information or exploring or interacting with information visualizations. For a voice-based system, exploration is inherently difficult for exploration and visualization is impossible. To address this, the logic flow allows for short-term reflection by repeating and confirming the list of input items during every conversion. The system also expands to another modality for long-term reflection. The design flow falls back to a visual interface after collecting information. A user can access their journaling data on their smartphone and on the website where auxiliary graphical interfaces would be available. However, this part is not accomplished during this project.

Outcomes

Dialogic flow for food journaling via voice assistant
Sample dialogue for voice-based interactions

The logic flow of conversation shows four stages of the voice-based food journaling system. In the beginning, users start with the voice assistant by saying specific trigger words. In the collection phase, novices and experts can take different approaches. The reflection stage suggests a multimodal approach – short-term reflection is embedded in the conversation while long-term reflection involves reports and visualizations viewed on another display. At the final stage, users can take actions after they reflect on their food journal. They can correct something that was incorrectly recorded via speech, set a shortcut for more efficient interaction, or explore new ways to interact with the voice assistant.

Sample Conversations

A novice user adding Food. (Inputs from the user is highlighted)

- Hi My Food Journal,

- Hi! Welcome to My Food Journal. I can help you add new food to the journal or check previous journals. Which would you like?

- Add new food.

- Okay, go ahead and say the names.

- Ummm…

- You can add food by saying its name and serving size. Go ahead.

- One in-n-out double-double, French Fries, and Chocolate shake.

- Got it. Would you like to add more?

- No, that’s it.

- Sure. I got one in-n-out double-double, a medium French Fries, and a medium chocolate shake. Is that correct?

- Yup.

- Alright. Quick tip: You can always add more food by saying “Hi My Food Journal, quick add.”

In the sample conversation, note that the system always provides an affirmation when it gets any information from the users. At the end of each conversation, new actions are introduced in the form of quick tips to increase discoverability.

Quick add and customized shortcuts (Inputs from the user is highlighted)

- My Food Journal, quick add.

- (A response tone playing indicating the system is listening)

- “McDonald Lunch Combo”

- (A confirmative tone playing indicating the system have received your information)

In this example, a user has set up a shortcut named “McDonald Lunch Combo” which includes a list of food they usually get from McDonald. This way the user could skip the instructive steps and complete the journaling in a much more efficient fashion.

Conclusion/Reflection

The focus of this project was on the bottom-up process of conversation design. After gathering insights from competitive analysis, interviews, sample conversations, and adapted heuristic evaluations, I delivered a dialogic flow for food journaling via voice assistant and sample conversations. The next step would be to expand the dialogs based on the flow and implement the system using a real-world voice assistant platform.

Usability Study: League of Legends Chatbot

Wed, 07 Mar 2018 01:00:00 +0000

Overview

My Role: Lead UX Researcher in a team of 4 researchers and 2 developers

Methods: Interviews, Focus groups, Surveys, Thematics analysis, Observations, Moderated usability test, and Descriptive statistical analysis

Data Sources: Interview transcripts, Observation notes, Survey result, and In-game chat logs

Deliverables: Research report and Chatbot prototype

Tools: AutoHotKeys, League of Legends spectator mode, Google Docs, Google Slides, Google Forms, Excel, OBS Studio, and Slack

Background/Context: In this project, the client requested our team study the usefulness of a chatbot for improving group collaboration. We grounded our research in the context of League of Legends games, where team collaboration is the key theme of the gameplay experience. In an LoL game, the temporarily assembled team encounters various challenges in communicating and coordinating, e.g. not understanding other player’s intentions, requests for help going unanswered. These challenges, if not resolved, could be detrimental to the team in League of Legends’ fast-paced gameplay.

Project Overview: The overarching question is to understand “How can a chatbot help facilitate collaborations among a temporarily assembled team in League of Legends?”.

Client: Dakuo Wang, IBM Research

Table of Contents

Objective

Understand what obstacles impede collaborations among a temporarily assembled team in League of Legends
Understand how a chatbot might help players collaborate with strangers on their team
Evaluate the usability of the chatbot

Opportunity and Process

Opportunity

Many League of Legend players said that they had a better chance of winning when they coordinated with their team. However, the current game doesn’t have any built-in mechanism that proactively encourages players to communicate with their teammates. A chatbot can provide additional useful information for the team members and engage them to better communicate with each other.

Process

Several UX research approaches were used to design the chatbot.

We completed interviews and focus groups to understand problems players experience in team collaboration. We conducted surveys to determine the chatbot functions. We built a chatbot prototype with learnings from the previous two steps. We carried out a Wizard of Oz experiment to verify the usability of the functions

Strategy

Interview and Focus Groups

We conducted 9 interviews with players to learn more about their attitude toward and experience with team collaborations during their most recent gameplay. We encouraged participants to describe frustrating situations where their teammate didn’t respond timely to their calls for help, where they had difficulties with the built-in chatbot - the main channel of in-game communication. We also asked about positive moments when they felt connected to the team, and when they successfully operate tactics. These interviews allowed us to better understand the current state of in-game collaborations and their points of breakdown.

We then held 2 focus groups, in which we specifically asked participants to list the things they asked their teammates to do, the channels through which they made their requests and the responses they received from their teammates. The focus groups’ back and forth conversations allowed us to generate a rather extensive list of ideas for potential chatbot functions.

Survey

After thematic analysis of the interviews and focus groups, we synthesized a few broad genres of chatbot functions to improve upon. We designed a survey to gather quantitative data on the qualitative findings so we could then prioritize the list of topics.

Prototype

We then developed prototype functions around the areas deemed most important. We developed a list of functions showing what the chatbot would say in response to a variety of scenarios.

Wizard of Oz Test

Due to the difficulty of inserting a chatbot function into an already mature and complex game system, we have decided to use a Wizard of Oz study in which a research team member would play as the chatbot in testing sessions.

A fundamental requirement of this Wizard of Oz study is that the participants cannot know that an actual person is behind the curtain. Instead, they must believe they are interacting with a fully automated chatbot.

Prototype Chatbot Introduction Function

Prototype Chatbot Encourage & Response Function

Although this was a challenge, we leveraged AutoHotKey - a scripting software that allows the test operator to send pre-scripted messages with keyboard shortcuts - to simulate the speed, accuracy, and efficiency of a real-world chatbot system.

User Testing Process

Recruit participants via emails with details about the study and compensation
Participants play two rounds of the game, the first without a chatbot facilitating and the second with our Wizard of Oz chatbot
Survey participants about their experience after each round of game
Complete additional survey about the chatbot’s performance after final round

Outcomes

Objective 1: Understand what obstacles impede in-game collaborations among a temporarily assembled team in a League of Legends game

After analyzing interview transcripts, the most highly mentioned codes were:

“Play as a team”: Players don’t feel like they were playing as a team and want to enhance team collaboration.
“Extra information”: Players are not getting enough from the game’s built-in mechanism and want more information
“Communication”: Players find it difficult to communicate with teammates.
“Encouragement”: Players expect the chatbot to send encouragement when they played poorly, and praise when they played well
“Chatbox”: Players feel the current chat box in the game is difficult to use

Objective 2: Understand how a chatbot might help facilitate collaboration among a team of strangers

Medians were calculated for all Likert Scale measurements shown below

From the quantitative results, we determined that a chatbot could:

facilitate communication by providing contextual information to “ping” signals
remind players their progress
aid in coordination at the beginning of the game for inexperienced players
amplify players’ “call for help” when they are under attack
encourage players to respond to calls for help

After pseudo-pilot test data, we defined the functions of our chatbot. We planned to increase players’ sense of being a team by adding functions as “introduction and reminder before game” and mediating “When somebody insults you/team”. We decided to provide extra information as “Tower/Inhibitor/Base under attack” and “more information beyond pin”. The full function list here.

Objective 3: Evaluate the usability of chatbot

The above chart showed players generally saw improvements in every measured category, with players’ self-measurement and teammate-measurement of teamwork showing the greatest improvements.

The chatbot also appeared to facilitate a greater sense of coordination among players, make it easier for them to get help from teammates, increase their perceptions of speedy responses from teammates, and improve their awareness of teammates’ expectations.

Players appreciated the chatbot to some extent on every measured scale. Participants strongly indicated that they enjoyed the chatbot’s personality, appreciated the chatbot asking others to help them, felt that the chatbot made them feel like part of the team, and believed the chatbot helped them coordinate. Since these were the primary functions that the chatbot was meant to fulfill, we can say that it was, in these trials at least, a success.

Key Takeaways

The Positives😀:

Did the chatbot help players collaborate better?
Yes. The chatbot alleviated the burden of both direct and indirect collaboration.
Was the chatbot intrusive to their gaming experience?
No. The chatbot reminded players to help/coordinate with their teammates, but it was easy for players to ignore messages if they wanted to.
Was the Wizard of Oz Study successful?
Yes. One player asked me how we made the bot work so seamlessly.

-Did players like our chatbot?

Yes. Players especially appreciated the chatbot’s personality, functions, and encouragement.

The negatives🙁:

One player briefly mistook the chatbot for another player.
One player was distracted by the chat box and did not pay enough attention to the gameplay.
One player stated that the chatbot would work really well for entry-level players but wasn’t as helpful for more experienced players.

Conclusion

In this project, we explored the use of a chatbot in League of Legends and its influence on in-game team collaboration. At the initial stage of the project, we studied players’ issues and needs using interviews, focus groups, and surveys. We then built a prototype based on our research results and verified it through two rounds of Wizard of Oz user tests. Finally, we evaluated the chatbot’s performance using surveys and interviews following players’ interaction with the chatbot during gameplay.

Players found the chatbot useful and said it improved their collaborations with teammates while maintaining the fairness of the game. We used players’ self-reported perceptions of teamwork as our measurement. However, similar studies on team collaboration may want to also include more objective measurements for cross validation.

Quantitative Analysis: Personal Data Sticker

Mon, 01 Jan 0001 00:00:00 +0000

Overview

My Role: User Researcher working under a leading researcher

Methods: Survey, R, Factorial design, Factorial Analysis, and Literature review

Data Sources: Literature review and resulting survey

Deliverables: Research report describing users’ perceptions of social media stickers containing personal informatics

Tools: R Studio, JSON, Windows Powershell, Photoshop, and Google Doc

Background/Context: When people set health and behavior goals such as training for a half-marathon, going to bed earlier, or losing weight, they often use social media to share their progress with friends and family (Munson, 2012). Epstein et al. (2015) identified that many people share personal informatics data to receive advisory information, emotional support, motivation, or accountability from their audience. However, prior research has shown that sharing this data on social media generally doesn’t not result in the users’ desired outcomes.(Epstein, 2019).

Project Overview: I explored design principles for incorporating users’ step counts, one of the most commonly-tracked and shared pieces of personal informatics, into stickers in preparation for creating a similar feature for use in Snapchat posts or Instagram Stories.

Table of Contents

Objective

When users send or receive Snapchat stickers containing their personal step count data, what factors influence the users’ perceptions of those stickers? And how do these factors actually impact the users’ perceptions?

Opportunity and Process

Opportunity

In Snapchat or Instagram stories, people are able to use stickers to customize their posts for informative, aesthetic, or entertainment purposes. Available stickers are organized in a variety of categories depending on their content, format, style, etc. Currently, these stickers include methods for sharing some types of data, but the data is rarely about the user themself and instead typically pertains to the user’s location or surroundings. (Habib et al., 2019). This gives me an opportunity to design a series of stickers to support more types of user-generated content focused on users’ reasons for sharing.

Process

Through a review of research on multimedia and online advertising, I identified a series of predictors that may influence Snapchat users’ perceptions of stickers. I also determined the evaluation metrics for measuring these user perceptions. My team then designed a series of stickers for Snapchat that contain personal data about step counts. Finally, I conducted an online survey to determine how those stickers’ designs influenced users’ perceptions.

Samples of data-driven stickers incorporating step counts

Strategy

Experiment Design

I used a factorial study design to evaluate varying levels of context, presentation, and style’s influence on participant perception and preference. After they consented to participate in the study, participants were asked to identify one person who they frequently Snap with to imagine as their conversation partner. They then gave feedback on six (6) randomly generated posts featuring data-driven stickers, answering some demographic questions upon completion.

The post generated varied on three dimensions:

Presentation styles

Badge style annotates objects with the specific tracked value, for example a shoe or ribbon with “5,793 steps” written on it.
Embellished style presents common objects as charts, picking one dimension to be the axis and shading the object partway according to the tracked value.
Analogy style re-expresses tracked values as better-known quantities through comparisons.

Relevance levels

Domain-relevant Domain-relevant designs use objects or comparisons specifically related to steps, such as snicker, track field.
Domain-irrelevant Domain-irrelevant designs use well-known objects and comparisons that are not commonly associated with steps, such as a star or a speedometer.

Background styles

Background of the post is a photo of specific scenario
Background is from public CSS patterns with abstract shapes

	Domain-relevant	Domain-irrelevant
Badge
Embellished
Analogy

User Perception measurements

Participants answered questions from four widely-used scales from online marketing and advertising literature, modified to this context. The validated scale measures:

Entertainment Value: how entertaining the shared content is

Attitude: attitude toward the content

Intention to use: How inclined a user find to use this feature

Information Value: how informative a user find receiving the content

Privacy Considerations: how invasive the a user find sharing the content

Participants answered each question on a 7-item Likert scale with endpoints “Strongly Disagree” and “Strongly Agree”.’

Survey Sampling and Participant Screening

I used convenience sampling to gather information on adult Snapchat users’ opinions of the stickers. I sent out recruitment emails to a student research subject list, and distributed flyers with a survey link in university classrooms and meetings.

Only participants who were at least 18 years of age and who, on average, sent or viewed at least one post on Snapchat per week were chosen to participate in the study

Outcomes

The regression analysis showed that:

Users’ perceptions of stickers they’ve received correlated more significantly with our independent variables, compared with sharers’ perceptions.
And that influence that factors have on the two sides – Sharer and Recipient – are not received equally.
When the relevance of stickers’ presentations changes from irrelevant to relevant, the ratings increase in both sharers and recipients’ intention to use, recipients’ level of entertaining, as well as informative.
Context is significant to ratings of Attitude and Entertaining from recipients.
Stickers design with “Analogy” is perceived to be more exciting to receive than “Plaintext”.

Conclusion/Reflection

In the project, I explore design principles for incorporating self-tracked step counts in data-driven stickers as a first step towards integrating these data into Snapchat posts or Instagram Stories. I examine the effect of a sticker’s presentation style, domain-relevance, and background through surveys. We uncover the importance of domain-relevant backgrounds and stickers, identify the situational value of stickers styled as analogies, embellished, and badges, and demonstrate that data-driven stickers can make content more informative and entertaining.

Some limitations of the study: I only received valid answers from 19 participants. This means that our regression analysis results are highly troubled by sampling error. Thus, it is unlikely to draw statistically valid conclusion from such a small sample size. Another flaw in the analysis is that no interaction terms were included in the model, due to that our data sample were simply not powerful enough to discern any differences.

However, considering the pilot nature of the project, the process I took to reach these conclusions provided meaningful learning experience. Indeed, the learnings generated from this pilot directly influence the framing of research questions and analysis approach when the pilot was later developed into a larger, more comprehensive study with 506 total participants. The findings of the large-scaled study were published in Proceedings of the ACM on Human-Computer Interaction in October 2020.