Projects | Tan Zhou

From Messy Dataset to “At-a-Glance” Visualizations of Competitive Landscape

Sat, 01 Nov 2025 05:26:35 +0000

Overview

As AI and automation accelerated across the industry, my stakeholders need to understand the competitive space of AI, automation, and technology in title/settlement platforms. The challenge wasn’t collecting information—it was making complex, uneven competitive data understandable and actionable for decision-makers.

This case study focuses on how I translated a large competitive dataset into a clear visualization system. It intentionally avoids sharing competitive “insights” or conclusions about specific companies.

The Business Problem

Leadership needed decision support for product strategy questions, like:

Where are competitors investing in automation across the transaction workflow?
What types of solutions exist (end-to-end platforms vs. narrow tools)?
Which parts of the ecosystem are truly comparable to our context?

To answer these, stakeholders needed a landscape they could trust and interpret quickly—without reading a long report.

The Research Challenge

This was not a clean comparison set. The competitive space had three structural issues:

1. “Apples-to-oranges” offerings

Some products are broad workflow platforms. Others specialize in one slice (e.g., document automation, search, closing coordination, post-close). Comparing them on a single axis would oversimplify and mislead.

2. “AI” claims were inconsistent

Many vendors used similar language (“AI-powered,” “automation,” “intelligent workflow”), but the underlying capability varied widely. The dataset needed a way to separate marketing terms from meaningful maturity indicators.

3. Too much information to be usable

Raw competitive research often becomes a dense spreadsheet that only the researcher can navigate. Stakeholders needed clarity at a glance, with enough structure to support follow-up questions.

My Role

I led the work end-to-end across:

Research framing (what decisions the landscape needed to support)
Data modeling and taxonomy creation (how we normalized inconsistent inputs)
Classification logic and decision rules
Information design and visualization system
Stakeholder alignment through iterative readouts and refinement

What Success Looked Like

We defined success as a set of outputs that were:

Strategic: tied to product decisions, not just market description
Trustworthy: classification logic visible and repeatable
Scannable: usable in seconds, not minutes
Multi-dimensional without being messy: complexity represented through a system, not a single overloaded chart
Reusable: designed as an artifact we could update as the market changed

Process: From Research Needs to Visualization System

Step 1: Translate stakeholder questions into “decision views”

Before making any visual, I reframed stakeholder needs into explicit questions the landscape must answer:

Orientation question: “Where does each solution fit in the workflow?”
Capability question: “How advanced is automation/AI—and how broadly does it apply?”
Context question: “Which solutions are actually relevant to our domain focus?”
Ecosystem question: “What’s plug-and-play vs. what changes switching costs and integration realities?”

This step prevented a common failure mode: building one beautiful chart that answers none of the real decisions.

Step 2: Build a classification model to normalize messy data

To compare uneven offerings, I created a shared taxonomy—essentially a “data contract” for the landscape.

What we standardized (examples)

Primary workflow focus: where the product anchors its value (even if it touches multiple steps)
Workflow breadth: narrow point solution → broad end-to-end platform
Automation mechanism: rules-based automation vs. AI-driven vs. hybrid
Integration posture: standalone tool → integrated suite → ecosystem
Domain relevance: relevance based on transaction complexity and operational needs (rather than vendor labels)

The key: explicit decision rules

I documented rules for edge cases, such as:

Platforms spanning multiple workflow stages
Suites that bundle unrelated modules
Tools that market “AI” but primarily deliver rules-based automation
Products that appear comparable but serve fundamentally different transaction contexts

This turned subjective categorization into something stakeholders could understand, challenge, and trust.

Step 3: Choose a “backbone” view for orientation

I started with Workflow Stage because it matches how most stakeholders naturally reason about real estate closing: as a lifecycle with handoffs and dependencies.

Why this came first

It gives immediate context to non-experts: “Where in the process does this help?”
It avoids premature ranking or “winners/losers”
It makes later views easier to interpret by grounding them in a shared mental model

Design principle: Always orient before differentiating.

Step 4: Avoid the single 2×2 trap—use complementary, orthogonal views

A single chart can’t responsibly represent a market where:

some products are broad platforms,
some are specialized,
and “AI” is not consistently defined.

So I designed a system of four views, each answering a different strategic question with minimal cognitive load.

The Solution: A Four-View Competitive Landscape System

1. Workflow Stage Landscape

Question answered: “Where does each solution primarily contribute within the closing workflow?”

**Why it works:**It helps teams understand the ecosystem without needing domain expertise. It also prevents false comparisons by showing that many solutions aren’t trying to solve the same problem.

How it’s designed for clarity:

Grouped by workflow stages with short “expectations” per stage (what buyers typically look for there)
A dedicated representation for cross-lifecycle platforms so multi-stage tools don’t distort stage-specific comparisons

2. AI & Automation Maturity × Workflow Breadth

Question answered: “How mature is automation/AI—and how broadly does it apply across the workflow?”

**Why it works:**This separates two things stakeholders often conflate:

maturity of automation capability
how much of the workflow the product claims to cover

How it’s designed for responsible interpretation:

“Maturity” is grounded in observable capability indicators rather than marketing terms
“Breadth” is framed as workflow ownership, not simply feature count

Design principle: Keep axes orthogonal so the chart stays truthful.

3. Commercial vs. Residential Relevance

Question answered: “Which solutions are most comparable to our operational context?”

**Why it works:**Transaction types differ in complexity, documentation, risk, and workflow variability. Without this lens, stakeholders may draw incorrect strategic conclusions from superficially similar tools.

How it’s designed:

A simple segmentation that scopes interpretation rather than ranking vendors
Helps stakeholders quickly identify “directly relevant” vs. “adjacent signals” in the market

4. Ecosystem Integration Landscape

Question answered: “What’s a tool we can plug in vs. an ecosystem that changes interoperability and switching costs?”

**Why it works:**Integration posture shapes adoption dynamics: procurement, implementation effort, dependency risk, and long-term flexibility.

How it’s designed:

Clear categories that highlight whether a solution is:
- a standalone product,
- part of an integrated suite,
- or operating as an ecosystem strategy

Design principle: Strategy isn’t only about features—it’s about constraints.

Making It Usable: Storytelling and Stakeholder Alignment

Progressive disclosure (how the readout was structured)

I presented the work like a product narrative:

Start with workflow stage to establish orientation
Move to maturity × breadth to discuss capability patterns
Add relevance to prevent misinterpretation
End with ecosystem integration to connect to strategic leverage and constraints

This sequence reduced debate and increased clarity: stakeholders could follow the logic rather than getting stuck on definitions.

Built-in “how to read” guidance

Each view includes lightweight framing—axis definitions, category labels, and reading cues—so the landscape can stand alone without the researcher in the room.

Outcomes

This work created an artifact stakeholders could actually use:

A shared vocabulary for discussing a fragmented market
A trustworthy classification model that made comparisons feel grounded
A decision-ready visualization system that supported strategy discussions without requiring deep domain knowledge
A framework designed to be maintained and updated, not a one-time research dump

(Deliberately omitted here: any market-specific conclusions or vendor evaluations.)

What This Demonstrates

This project is a snapshot of the kind of UX work that sits at the intersection of:

research strategy (defining what must be true to make a decision),
analytics and synthesis (normalizing messy inputs),
information design (reducing cognitive load),
and stakeholder alignment (building shared understanding through clear frameworks).

Key Takeaways I’d Reuse

Start with the decisions, not the data.
Normalize first; visualize second.
Use multiple simple views instead of one complex chart.
Make classification rules explicit so the work earns trust.
Design for scanning—then support deeper follow-up.

Modernizing document workflows in a complex transaction platform

Sun, 01 Jun 2025 05:26:35 +0000

Problem

Business & product challenge

Before: Document coordination lived in email threads and attachments—forcing manual tracking, follow-ups, and low confidence in ‘what’s latest’.

In complex, high-stakes transactions, “documents” aren’t a feature—they’re the operating system. Internal teams and external clients must request, collect, verify, and reference dozens of items across multiple parties, deadlines, and handoffs.

The legacy reality looked like this:

Requirements defined through contracts + back-and-forth Q&A
Documents arriving in scattered email threads and attachments
Manual tracking (“what’s missing, who owes what, what changed?”)
Version confusion and rework (duplicate uploads, wrong file shared, unclear latest)

This is a product problem (lack of shared visibility and trusted status), and a business problem (time and risk). Industry benchmarks show why this matters: “interaction workers” spend ~28% of time on email and ~19% searching/gathering information, and improving collaboration/searchability can create ~20–25% productivity uplift in the right conditions.

Users

Internal transaction teams (ops/service/processing): need a reliable source of truth to coordinate work, maintain confidentiality, and avoid errors.
External clients/partners: need clarity on what’s required, what’s outstanding, and confidence that the right version was received.

Why this was important

Document handling is repeated constantly. If the workflow is unclear, people default to email and personal workarounds—creating compounding cost (minutes lost per document × many users × many transactions) and compounding risk (wrong versions, missed requirements, delayed approvals).

Strategy

Research goal

To define a modern document workflow that:

makes requirements visible,
makes progress trackable,
makes document status trustworthy, and
scales to high volumes without forcing users back into email or local folders.

Approach: a program of research, not “one study”

I ran this as a multi-phase research arc where each phase answered the next logical question:

Discovery (workflow reality): What actually happens today—and where does it break?
Concept shaping (new mental model): What structure reduces ambiguity (checklists, status, ownership, visibility)?
Validation (does it work for real users?): Can internal and external users understand it quickly, act confidently, and avoid errors?
Scale (second-order constraints): Once adoption grows, what breaks next (organization, findability, versioning, automation)?

Methods

Because the work spanned maturity stages, I matched method to decision:

Interviews / workflow mapping to surface real breakdowns and system constraints
Prototype concept testing to de-risk mental models (terminology, status, ownership, visibility)
Design validation to confirm comprehension and usability before rollout
Later-stage discovery focused on scale issues (high doc counts, search behavior, version control expectations)

My decision rationale

Why interviews first

At the start, the users didn’t have a UI problem—we had a coordination system problem. Interviews and workflow mapping were the fastest way to:

uncover the real “jobs to be done” (request → chase → receive → verify → organize → reuse)
expose hidden constraints (privacy boundaries, handoffs, audit needs)
identify why “email + attachments” persisted (it filled gaps the product didn’t cover) Decision logic: If we guessed at the workflow, we’d build a beautiful interface around the wrong system.

Why prototype testing next

Once I saw the breakdown was “tracking + trust,”, I needed to validate whether a checklist/status model could become the shared source of truth. Prototype testing was the right tool because it let us:

test comprehension of status/ownership (without expensive build)
test terminology and “professional tone” early (a known adoption lever)
measure whether users could correctly answer “what’s left?” in seconds Decision logic: We needed behavioral evidence that the model reduced ambiguity—not just opinions about it.

Why design validation (internal + external)

The workflow had two audiences with different risk profiles. Validation ensured:

internal users could move fast without creating errors
external users could act confidently without needing an explainer
status changes and version cues didn’t create false confidence or confusion Decision logic: In document workflows, clarity is safety—validation is risk management.

Why organization + versioning later

As the system matured, the next bottleneck wasn’t “can I upload?”—it was “can I find the right thing and trust it?” At scale, long document lists and multiple versions shift the problem from interaction design to information architecture and reliability.

Decision logic: Once the checklist model reduced “what’s missing,” the system’s limiting factor became “what’s correct and where is it?”—so research pivoted to structure, search behavior, and version control.

Key decisions

Checklist became the coordination layer that connects requests, uploads, ownerships, status, notifications, and version confidence.

Reframe documents from “file storage” to “workflow tracking”

Decision: Treat document handling as an end-to-end workflow (requirements → request → receipt → verification → history), not a repository. Why: Email persists because it supports coordination and status tracking—so the product had to do that job better.

Use a checklist model as the shared source of truth

Decision: Anchor the experience in a checklist/status structure that answers: what’s needed, what’s in progress, what’s done, what changed, who owns it. Why: This reduces ambiguity for both internal teams and external clients, and creates a consistent foundation for later features (notifications, organization, automation).

Make ownership, visibility, and status explicit (not implied)

Decision: Design for role-based visibility and unambiguous status transitions (with language that users trust). Why: In transaction workflows, unclear “who owns this” creates delays; unclear “status” creates rework and risk.

Standardize organization defaults before adding “more flexibility”

Decision: Provide sensible default structure (folders/tabs/categories, sorting and filtering patterns, and “pin/priority” behaviors) rather than relying on everyone inventing their own system. Why: Ad-hoc organization scales poorly and increases search time and error rates—especially across teams.

Invest in version confidence as a first-class requirement

Decision: Prioritize version history and clear draft/final cues (including stacking, timestamps, and traceability). Why: Version confusion is a trust-breaker; users can’t move fast if they fear sharing the wrong thing.

What changed

Roadmap & scope changes

The roadmap shifted from “improve upload” to “support workflow clarity” (tracking, status, ownership, visibility).
Document organization and versioning were treated as strategic enablers—not nice-to-haves—because they determine whether the system works at scale.

Design & UX changes

Checklist-based experience became the core navigation layer for document work (what’s outstanding, who owes what, what’s completed).
Status language and interaction patterns were refined through iterative testing to reduce misinterpretation.
Organization patterns were elevated: default structures, better sorting/filtering, and pathways to reduce scanning and “where did it go?” confusion.
Version confidence was explicitly designed (history, recency cues, clearer distinctions between draft/final).

Stakeholder alignment outcomes

Research artifacts created a shared mental model across product/design/ops/engineering—so decisions could be made faster and with less debate about what users “really do.”

How research translated into action: key inisghts were turned into concrete product decisions and measurable experience changes.

Impact

Industry research suggests that a large share of knowledge work is consumed by communication and information retrieval:

~28% of time is spent managing email (reading/writing/responding)
~19% of time is spent searching and gathering information
Making information more available and searchable can reduce information searching time by as much as ~35% in some contexts

A checklist-driven document system directly targets both buckets:

fewer emails needed to ask “what’s missing / did you get it?”
less time spent searching across inbox threads and attachments
fewer wrong-version loops and duplicate handling

Improving Order Status Communication for Insurance Products

Mon, 09 Jan 2023 05:26:35 +0000

My Role: UX Researcher

Research Type: Discovery, Generative, Primary research

Methods: User Interview (remote), Secondary Research, Cognitive Interview, Jobs to be Done

Deliverables: User journey map, Readout deck with research findings and recommendations

Tools: UserZoom, Microsoft Teams, Miro, Excel, PowerPoint

Research Motivation

After receiving anecdotal comments, the product team wanted to design a page to communicate with users the status of their insurance orders, but they were unsure of the benefits it would bring to the users. Additionally, they were not certain what specific status information would be helpful to users.

Process

I started by interviewing the Product team members to better understand the research request and define the problem statements. Some of the questions I asked include:

What information do you need to have to feel confident to start the project?

What do you know about our users’ preferences around this topic, and what are we not yet sure about?

Are there competitive examples of what we’re building that we should take a look at?

Do we understand how this information is being communicated to users currently? Through what channels?

Who are the difference user segments at play in your opinions?

To achieve this, I drafted research objectives and proposed a research plan to conduct 1:1 interviews with the targeting use segments. The plan involved recruiting 12 participants with different levels of product expertise and from different states, as the types of information in users' order updates for this insurnace product vary greatly by state. I then invited them to participate in remote interview sessions via UserZoom, and key stakeholders were encouraged to join and observe the sessions.

During the interviews, I collected data to understand foundational user needs, behaviors, pain points, and motivations by prompting users to tell stories around searhcing or requesting order status for their order and by following the “Job to be done”(JTBD) framework exploring the journey and the causes of user beheviors. After completing the analysis, I presented detailed findings and my recommendations to the product managers, designers, business analysts, and key executives working on this product.

Deliverable

15-minute post-session debrief: For interview sessions where stakeholders were present, I hosted a quick debrief call immediately after to discuss key takeaways.
30-minute readout: After completing my analysis, I presented detailed findings and my recommendations to the product managers, designers, business analysts, and key executives working on this product.

My deliverables included a journey map of inquiring order status: user actions, where the information came from, how it is provided, tools that are used, pain points, and needs, and potential areas of improvement, as well as a deck that identified the specific information that users find valuable with accompanying quotes and recommendations on next step product strategies.

Conclusion and Impact

The results of my research had a significant impact on the product team’s understanding of the user needs and provided foundational user knowledge that informed next-step product strategies and design decisions. The research empirically validated the assumed business value of order status updates and surfaced additional value adds for users, such as increasing user trust through greater transparency.

In conclusion, my research provided critical insights into the user needs and behaviors surrounding order status updates, which will help the product team make informed decision early on to design a page that meets users' specific needs and adds value to their overall experience.

Voice Interaction Design: Food Journaling via Voice Assistant

Thu, 07 Mar 2019 06:32:08 +0000

Overview

My Role: UX Researcher and Designer

Methods: Literature review, Interviews, Competitive analysis, Journey mapping, Heuristic evaluation

Data Sources: Interview transcripts, Literature

Deliverables: Research report, Dialogic flow, Sample conversations

Tools: Google Assistant, Amazon Alexa, Google Sheet

Background/Context: Studies following diabetic patients and weight watchers found food journaling to be an effective means of managing one’s diet. Although automating the journaling process using smart devices could increase adherence by decreasing the effort and mental burden required, it could also lead to a decrease in users reflecting on collected data. After searching the field, I failed to find a well-designed voice-based interface that supports food tracking.

Project Overview: The overarching goal was to answer the question “When people track foods they eat daily (food journaling) via a voice assistant, how can we design the dialogic flow to facilitate users’ reflections on their eating habits?”.

Table of Contents

Objective

Understand users needs and pain points in food journaling
Design the flow of the conversation and its underlying logic to facilitate voice-based food journaling
Adapt Nielsen’s heuristics to evaluate voice-based interface

Opportunity and Process

Opportunity

The study of automated food journaling show that such automation could lower the burden of tracking and increase adherence (Beenish et al., 2016), but it could lead to a decrease in users reflection on collected data(Choe et al., 2014). Also, the majority of these studies leverages only graphical interfaces, building on the legacy of hand-and-finger input devices.These approaches are limited by the required input since users might not always be able to log their entries.

Voice assistants like Amazon Alexa, Apple Siri, and Google assistant are getting more and more ubiquitous. Their support of hands-free interaction makes voice-based food journaling an ideal use case for voice assistants.

Voice Assistants are more accessible than ever. Photo: Stratablue.com

Process

In order to define the users’ requirements, I conducted a competitive analysis of existing tools that support voice-based food journaling. I also interviewed two participants who have previously used their mobile phones for food journaling to understand users’ needs, behaviors, and motivations of the user during the journaling process.

Using the information collected from the field review and interviews, I wrote a series of sample dialogues to capture the “sound-and-feel” of the interaction under different scenarios. These sample dialogues convey the flow that the user will experience and allow me to experiment with different design strategies, such as how to promote the discoverability of new features or how to confirm a user’s request.

At the usability test stage, a friend who is unfamiliar with the project was asked to role-play the simple dialogues with me. This helped me curate the conversation, defining the flow and the underlying logic that represents the complete food journaling experience. I also conducted system evaluations with a set of adapted heuristics to expose usability issues.

Strategy

Exploratory Interviews

My initial goal of exploratory interviews was to understand users’ needs in the journaling process. Two interviews were conducted with informants who had experience journaling the food they ate.

Heuristic Evaluation

1. Awareness of system status

The Nielsen heuristics emphasize the visibility of system status. Although visibility does not apply to voice-based interfaces, user awareness and feedback are still important. The system needs to inform users about what it is doing in a timely and appropriate fashion.

2. Error prevention

As with any system, it is best to prevent errors from occurring or handle them in a way that is less intrusive to user experience. This becomes even more important in a system with no visual interface since people cannot “unsay” what they have previously said. This system addresses this issue by providing users their options for the next step explicitly in the conversation.

3. Flexibility and efficiency

Novice users become experts after they become familiar with a system, and experts benefit from efficiency. As a result, some instructions, which would be helpful to novice users, may become redundant to experts. The logic flow supporting setting up customized shortcuts for frequently used terms to speed up the process.

4. Accessibility

Voice based interaction is great for people who are unable to use a graphic interface. It doesn’t require input from hands which offers more accessibility since users could interact with the system while carrying groceries, cooking a meal, or driving a car.

5. Ambiguity

People don’t communicate in syntax the way computers do. They sometimes use metaphors or slang. They sometimes forget words or pause when speaking. Voice technology should accommodate the users’ communication style and needs. When the user pauses, the system just repeats the previous response with a much more detailed instruction of what users can do until they make a selection.

6. Discoverability

The invisibility of a voice-based interface makes it difficult for users to explore new ways of interacting with the system. In this logic flow, new actions will be introduced in the form of quick tips at the end of each conversation (except those involving specific customized shortcuts). During the reflection stage when a user views their journals on a display, the system suggests new interactions that are more efficient that would increase the accuracy of the users’ food journaling.

7. Multimodal Reflection

In the five-stage personal informatics model proposed by Li et al.(2010), the reflection stage may involve looking at lists of collected personal information or exploring or interacting with information visualizations. For a voice-based system, exploration is inherently difficult for exploration and visualization is impossible. To address this, the logic flow allows for short-term reflection by repeating and confirming the list of input items during every conversion. The system also expands to another modality for long-term reflection. The design flow falls back to a visual interface after collecting information. A user can access their journaling data on their smartphone and on the website where auxiliary graphical interfaces would be available. However, this part is not accomplished during this project.

Outcomes

Dialogic flow for food journaling via voice assistant
Sample dialogue for voice-based interactions

The logic flow of conversation shows four stages of the voice-based food journaling system. In the beginning, users start with the voice assistant by saying specific trigger words. In the collection phase, novices and experts can take different approaches. The reflection stage suggests a multimodal approach – short-term reflection is embedded in the conversation while long-term reflection involves reports and visualizations viewed on another display. At the final stage, users can take actions after they reflect on their food journal. They can correct something that was incorrectly recorded via speech, set a shortcut for more efficient interaction, or explore new ways to interact with the voice assistant.

Sample Conversations

A novice user adding Food. (Inputs from the user is highlighted)

- Hi My Food Journal,

- Hi! Welcome to My Food Journal. I can help you add new food to the journal or check previous journals. Which would you like?

- Add new food.

- Okay, go ahead and say the names.

- Ummm…

- You can add food by saying its name and serving size. Go ahead.

- One in-n-out double-double, French Fries, and Chocolate shake.

- Got it. Would you like to add more?

- No, that’s it.

- Sure. I got one in-n-out double-double, a medium French Fries, and a medium chocolate shake. Is that correct?

- Yup.

- Alright. Quick tip: You can always add more food by saying “Hi My Food Journal, quick add.”

In the sample conversation, note that the system always provides an affirmation when it gets any information from the users. At the end of each conversation, new actions are introduced in the form of quick tips to increase discoverability.

Quick add and customized shortcuts (Inputs from the user is highlighted)

- My Food Journal, quick add.

- (A response tone playing indicating the system is listening)

- “McDonald Lunch Combo”

- (A confirmative tone playing indicating the system have received your information)

In this example, a user has set up a shortcut named “McDonald Lunch Combo” which includes a list of food they usually get from McDonald. This way the user could skip the instructive steps and complete the journaling in a much more efficient fashion.

Conclusion/Reflection

The focus of this project was on the bottom-up process of conversation design. After gathering insights from competitive analysis, interviews, sample conversations, and adapted heuristic evaluations, I delivered a dialogic flow for food journaling via voice assistant and sample conversations. The next step would be to expand the dialogs based on the flow and implement the system using a real-world voice assistant platform.

Usability Study: League of Legends Chatbot

Wed, 07 Mar 2018 01:00:00 +0000

Overview

My Role: Lead UX Researcher in a team of 4 researchers and 2 developers

Methods: Interviews, Focus groups, Surveys, Thematics analysis, Observations, Moderated usability test, and Descriptive statistical analysis

Data Sources: Interview transcripts, Observation notes, Survey result, and In-game chat logs

Deliverables: Research report and Chatbot prototype

Tools: AutoHotKeys, League of Legends spectator mode, Google Docs, Google Slides, Google Forms, Excel, OBS Studio, and Slack

Background/Context: In this project, the client requested our team study the usefulness of a chatbot for improving group collaboration. We grounded our research in the context of League of Legends games, where team collaboration is the key theme of the gameplay experience. In an LoL game, the temporarily assembled team encounters various challenges in communicating and coordinating, e.g. not understanding other player’s intentions, requests for help going unanswered. These challenges, if not resolved, could be detrimental to the team in League of Legends’ fast-paced gameplay.

Project Overview: The overarching question is to understand “How can a chatbot help facilitate collaborations among a temporarily assembled team in League of Legends?”.

Client: Dakuo Wang, IBM Research

Table of Contents

Objective

Understand what obstacles impede collaborations among a temporarily assembled team in League of Legends
Understand how a chatbot might help players collaborate with strangers on their team
Evaluate the usability of the chatbot

Opportunity and Process

Opportunity

Many League of Legend players said that they had a better chance of winning when they coordinated with their team. However, the current game doesn’t have any built-in mechanism that proactively encourages players to communicate with their teammates. A chatbot can provide additional useful information for the team members and engage them to better communicate with each other.

Process

Several UX research approaches were used to design the chatbot.

We completed interviews and focus groups to understand problems players experience in team collaboration. We conducted surveys to determine the chatbot functions. We built a chatbot prototype with learnings from the previous two steps. We carried out a Wizard of Oz experiment to verify the usability of the functions

Strategy

Interview and Focus Groups

We conducted 9 interviews with players to learn more about their attitude toward and experience with team collaborations during their most recent gameplay. We encouraged participants to describe frustrating situations where their teammate didn’t respond timely to their calls for help, where they had difficulties with the built-in chatbot - the main channel of in-game communication. We also asked about positive moments when they felt connected to the team, and when they successfully operate tactics. These interviews allowed us to better understand the current state of in-game collaborations and their points of breakdown.

We then held 2 focus groups, in which we specifically asked participants to list the things they asked their teammates to do, the channels through which they made their requests and the responses they received from their teammates. The focus groups’ back and forth conversations allowed us to generate a rather extensive list of ideas for potential chatbot functions.

Survey

After thematic analysis of the interviews and focus groups, we synthesized a few broad genres of chatbot functions to improve upon. We designed a survey to gather quantitative data on the qualitative findings so we could then prioritize the list of topics.

Prototype

We then developed prototype functions around the areas deemed most important. We developed a list of functions showing what the chatbot would say in response to a variety of scenarios.

Wizard of Oz Test

Due to the difficulty of inserting a chatbot function into an already mature and complex game system, we have decided to use a Wizard of Oz study in which a research team member would play as the chatbot in testing sessions.

A fundamental requirement of this Wizard of Oz study is that the participants cannot know that an actual person is behind the curtain. Instead, they must believe they are interacting with a fully automated chatbot.

Prototype Chatbot Introduction Function

Prototype Chatbot Encourage & Response Function

Although this was a challenge, we leveraged AutoHotKey - a scripting software that allows the test operator to send pre-scripted messages with keyboard shortcuts - to simulate the speed, accuracy, and efficiency of a real-world chatbot system.

User Testing Process

Recruit participants via emails with details about the study and compensation
Participants play two rounds of the game, the first without a chatbot facilitating and the second with our Wizard of Oz chatbot
Survey participants about their experience after each round of game
Complete additional survey about the chatbot’s performance after final round

Outcomes

Objective 1: Understand what obstacles impede in-game collaborations among a temporarily assembled team in a League of Legends game

After analyzing interview transcripts, the most highly mentioned codes were:

“Play as a team”: Players don’t feel like they were playing as a team and want to enhance team collaboration.
“Extra information”: Players are not getting enough from the game’s built-in mechanism and want more information
“Communication”: Players find it difficult to communicate with teammates.
“Encouragement”: Players expect the chatbot to send encouragement when they played poorly, and praise when they played well
“Chatbox”: Players feel the current chat box in the game is difficult to use

Objective 2: Understand how a chatbot might help facilitate collaboration among a team of strangers

Medians were calculated for all Likert Scale measurements shown below

From the quantitative results, we determined that a chatbot could:

facilitate communication by providing contextual information to “ping” signals
remind players their progress
aid in coordination at the beginning of the game for inexperienced players
amplify players’ “call for help” when they are under attack
encourage players to respond to calls for help

After pseudo-pilot test data, we defined the functions of our chatbot. We planned to increase players’ sense of being a team by adding functions as “introduction and reminder before game” and mediating “When somebody insults you/team”. We decided to provide extra information as “Tower/Inhibitor/Base under attack” and “more information beyond pin”. The full function list here.

Objective 3: Evaluate the usability of chatbot

The above chart showed players generally saw improvements in every measured category, with players’ self-measurement and teammate-measurement of teamwork showing the greatest improvements.

The chatbot also appeared to facilitate a greater sense of coordination among players, make it easier for them to get help from teammates, increase their perceptions of speedy responses from teammates, and improve their awareness of teammates’ expectations.

Players appreciated the chatbot to some extent on every measured scale. Participants strongly indicated that they enjoyed the chatbot’s personality, appreciated the chatbot asking others to help them, felt that the chatbot made them feel like part of the team, and believed the chatbot helped them coordinate. Since these were the primary functions that the chatbot was meant to fulfill, we can say that it was, in these trials at least, a success.

Key Takeaways

The Positives😀:

Did the chatbot help players collaborate better?
Yes. The chatbot alleviated the burden of both direct and indirect collaboration.
Was the chatbot intrusive to their gaming experience?
No. The chatbot reminded players to help/coordinate with their teammates, but it was easy for players to ignore messages if they wanted to.
Was the Wizard of Oz Study successful?
Yes. One player asked me how we made the bot work so seamlessly.

-Did players like our chatbot?

Yes. Players especially appreciated the chatbot’s personality, functions, and encouragement.

The negatives🙁:

One player briefly mistook the chatbot for another player.
One player was distracted by the chat box and did not pay enough attention to the gameplay.
One player stated that the chatbot would work really well for entry-level players but wasn’t as helpful for more experienced players.

Conclusion

In this project, we explored the use of a chatbot in League of Legends and its influence on in-game team collaboration. At the initial stage of the project, we studied players’ issues and needs using interviews, focus groups, and surveys. We then built a prototype based on our research results and verified it through two rounds of Wizard of Oz user tests. Finally, we evaluated the chatbot’s performance using surveys and interviews following players’ interaction with the chatbot during gameplay.

Players found the chatbot useful and said it improved their collaborations with teammates while maintaining the fairness of the game. We used players’ self-reported perceptions of teamwork as our measurement. However, similar studies on team collaboration may want to also include more objective measurements for cross validation.

Design Patterns: Help small non-profits build trust and engagement

Mon, 01 Jan 0001 00:00:00 +0000

Overview

My Role: UX Researcher in a team of five (5) researchers

Background/Context: Donors are often the core providers of a charity’s resource. Without donor support, some charities quickly struggle to function and are likely to enter a path of decline. Thus, it is important for charities to connect with their donors in order to gain their trust and increase their motivation.

Project Overview: I worked as part of a team to design best practices for facilitating online monetary donations to small charities and non-profits. Through interviews and analyses of established non-profit websites and their users’ behaviors and motivations, we developed design patterns allowing non-profit organizations to increase trust and engagement among donors.

Methods: Interviews, Competitive analysis, Design patterns, Literature review, and Desk research

Data Sources: Interview transcripts and Literature

Deliverables: Report describing research and suggested design patterns

Tools: Google Docs, Google Slides, and Google Forms

Table of Contents

Objective

What stops a motivated user from making donations to small charities online?
How can we convert a user’s motivation to donate to small charities online?
How can help small charities design their online presence to establish trust and engagement?

Opportunity and Process

Opportunity

Most HCI research related to non-profits focuses on facilitating transparency in order to build trust among stakeholders (Marshall et al., 2016, 2018). However, there is still little understanding of how a non-profit’s website and its features impact donors’ motivation and engagement.

Process

We analyzed charity websites to understand the design processes they follow to convert user motivation into calls for action. We interviewed nine (9) donors to understand their perspectives on these websites. We focused our research on establishing online trust and engagement through better design.

We delved into the Persuasive Design literature and created design patterns that enhance the experience for millennials visiting small charitable websites. Effective design patterns translate research findings into executable solutions and assist web designers and developers. They contain:

A concise and memorable title
Statement[s] describing the problem or challenge
Description and discussion on solving the problem
Solutions that are likely to succeed
Solutions that are typical yet undesirable

We identified impediments to charitable donations and recommended four design patterns that can help convert users’ motivation into action.

Strategy

Competitive Analysis

We analyzed websites of well-establish charities to explore different donation models, including WWF, UNICEF, Facebook Fundraisers, GoFundMe, etc.. We examined their user bases, information architectures, messaging approaches, calls to action, emotional designs, accreditations, and social influences. The competitive analysis provided us with key insights and informed our development of interview questions.

Interviews

We conducted interviews with nine (9) individual participants who either were active online donors or had donated through the above websites. The interviews highlighted the aspects that either motivated or discouraged donors as they navigated through the charity website. After our interviews, we inferred a relatively comprehensive list of statements taking the form “a person donates if he or she…”. We synthesized these statements into more general themes relevant to donors’ motivations.

Persuasive Design

We followed Fogg’s behavior model for persuasion to guide our designs. The behavior model highlights three factors relevant to accomplishing target behavior - motivation, ability to perform, and trigger.

There is some debate about the ethics of designing charity websites with the sole intention of encouraging people to donate their money. However, our goal in creating design patterns was not simply to persuade people to donate:

We wanted to leverage persuasive design for potential donors that already have medium levels of motivation but are missing ability and triggers to act on it.
We wanted to understand what discourages motivated people from performing actions related to donations in online platforms and how HCI can help address these aspects.

Outcome

Importance of trust

Trust is often critical when converting charitable motivations to actions
Monotonous, one-way interactions do not help donors help engaged or fulfilled

Suggested patterns

Our design patterns were concise and memorable with explicit descriptions of the relevant problems and common pitfalls that should be avoided.

The patterns that emerged were “Trust through reputation”, “Trust through transparency”, “Trust through social circle”, and “Commitment through Engagement”.

Delivered Design Patterns

Conclusion/Reflection

In this project, we conducted competitive analysis and interviews and analyzed user behavior/motivations pertaining to charities. We developed design patterns for non-profits seeking to increase trust and engagement among donors.

In the future, we hope to create additional design patterns incorporating other factors that are useful when designing a successful non-profit website (e.g. social media marketability, attractiveness to volunteers, etc.). These additional design patterns would eventually form a pattern language useful for converting different users’ motivations into actions furthering the non-profits’ overarching goals.

Quantitative Analysis: Personal Data Sticker

Mon, 01 Jan 0001 00:00:00 +0000

Overview

My Role: User Researcher working under a leading researcher

Methods: Survey, R, Factorial design, Factorial Analysis, and Literature review

Data Sources: Literature review and resulting survey

Deliverables: Research report describing users’ perceptions of social media stickers containing personal informatics

Tools: R Studio, JSON, Windows Powershell, Photoshop, and Google Doc

Background/Context: When people set health and behavior goals such as training for a half-marathon, going to bed earlier, or losing weight, they often use social media to share their progress with friends and family (Munson, 2012). Epstein et al. (2015) identified that many people share personal informatics data to receive advisory information, emotional support, motivation, or accountability from their audience. However, prior research has shown that sharing this data on social media generally doesn’t not result in the users’ desired outcomes.(Epstein, 2019).

Project Overview: I explored design principles for incorporating users’ step counts, one of the most commonly-tracked and shared pieces of personal informatics, into stickers in preparation for creating a similar feature for use in Snapchat posts or Instagram Stories.

Table of Contents

Objective

When users send or receive Snapchat stickers containing their personal step count data, what factors influence the users’ perceptions of those stickers? And how do these factors actually impact the users’ perceptions?

Opportunity and Process

Opportunity

In Snapchat or Instagram stories, people are able to use stickers to customize their posts for informative, aesthetic, or entertainment purposes. Available stickers are organized in a variety of categories depending on their content, format, style, etc. Currently, these stickers include methods for sharing some types of data, but the data is rarely about the user themself and instead typically pertains to the user’s location or surroundings. (Habib et al., 2019). This gives me an opportunity to design a series of stickers to support more types of user-generated content focused on users’ reasons for sharing.

Process

Through a review of research on multimedia and online advertising, I identified a series of predictors that may influence Snapchat users’ perceptions of stickers. I also determined the evaluation metrics for measuring these user perceptions. My team then designed a series of stickers for Snapchat that contain personal data about step counts. Finally, I conducted an online survey to determine how those stickers’ designs influenced users’ perceptions.

Samples of data-driven stickers incorporating step counts

Strategy

Experiment Design

I used a factorial study design to evaluate varying levels of context, presentation, and style’s influence on participant perception and preference. After they consented to participate in the study, participants were asked to identify one person who they frequently Snap with to imagine as their conversation partner. They then gave feedback on six (6) randomly generated posts featuring data-driven stickers, answering some demographic questions upon completion.

The post generated varied on three dimensions:

Presentation styles

Badge style annotates objects with the specific tracked value, for example a shoe or ribbon with “5,793 steps” written on it.
Embellished style presents common objects as charts, picking one dimension to be the axis and shading the object partway according to the tracked value.
Analogy style re-expresses tracked values as better-known quantities through comparisons.

Relevance levels

Domain-relevant Domain-relevant designs use objects or comparisons specifically related to steps, such as snicker, track field.
Domain-irrelevant Domain-irrelevant designs use well-known objects and comparisons that are not commonly associated with steps, such as a star or a speedometer.

Background styles

Background of the post is a photo of specific scenario
Background is from public CSS patterns with abstract shapes

	Domain-relevant	Domain-irrelevant
Badge
Embellished
Analogy

User Perception measurements

Participants answered questions from four widely-used scales from online marketing and advertising literature, modified to this context. The validated scale measures:

Entertainment Value: how entertaining the shared content is

Attitude: attitude toward the content

Intention to use: How inclined a user find to use this feature

Information Value: how informative a user find receiving the content

Privacy Considerations: how invasive the a user find sharing the content

Participants answered each question on a 7-item Likert scale with endpoints “Strongly Disagree” and “Strongly Agree”.’

Survey Sampling and Participant Screening

I used convenience sampling to gather information on adult Snapchat users’ opinions of the stickers. I sent out recruitment emails to a student research subject list, and distributed flyers with a survey link in university classrooms and meetings.

Only participants who were at least 18 years of age and who, on average, sent or viewed at least one post on Snapchat per week were chosen to participate in the study

Outcomes

The regression analysis showed that:

Users’ perceptions of stickers they’ve received correlated more significantly with our independent variables, compared with sharers’ perceptions.
And that influence that factors have on the two sides – Sharer and Recipient – are not received equally.
When the relevance of stickers’ presentations changes from irrelevant to relevant, the ratings increase in both sharers and recipients’ intention to use, recipients’ level of entertaining, as well as informative.
Context is significant to ratings of Attitude and Entertaining from recipients.
Stickers design with “Analogy” is perceived to be more exciting to receive than “Plaintext”.

Conclusion/Reflection

In the project, I explore design principles for incorporating self-tracked step counts in data-driven stickers as a first step towards integrating these data into Snapchat posts or Instagram Stories. I examine the effect of a sticker’s presentation style, domain-relevance, and background through surveys. We uncover the importance of domain-relevant backgrounds and stickers, identify the situational value of stickers styled as analogies, embellished, and badges, and demonstrate that data-driven stickers can make content more informative and entertaining.

Some limitations of the study: I only received valid answers from 19 participants. This means that our regression analysis results are highly troubled by sampling error. Thus, it is unlikely to draw statistically valid conclusion from such a small sample size. Another flaw in the analysis is that no interaction terms were included in the model, due to that our data sample were simply not powerful enough to discern any differences.

However, considering the pilot nature of the project, the process I took to reach these conclusions provided meaningful learning experience. Indeed, the learnings generated from this pilot directly influence the framing of research questions and analysis approach when the pilot was later developed into a larger, more comprehensive study with 506 total participants. The findings of the large-scaled study were published in Proceedings of the ACM on Human-Computer Interaction in October 2020.