Simon Data
2022-2023: Product design • User research • UX • UI • User testing
Simon Data is a customer data platform (CDP) that allows clients to unify customer identity, create audience segments, and set up marketing campaigns to target customers across multiple channels.
Simon Data website
Product images
• • •
Overview
Role: Product design lead
Team: 1 product manager, 5 engineers
Challenge: Reduce client reliance on Identity engineering team
Process: User interviews to scope out problem, leading to a survey redesign and new product feature, validated by user research
Outcome: 42% reduction in client tickets requiring engineering support
Some context
One of Simon Data’s functions as a Customer Data Platform is to keep and maintain all of a company’s customer profiles, which is referred to as Identity. Identity is intended to be the source of truth for a company’s customer data, meaning that it accurately and continuously amalgamates all known customer data over time, across all channels, on all devices. Essentially, as new data comes in, Identity figures out who that data belongs to.
In order for Identity to manage this new data correctly, it runs on a set of “Identity Rules” which dictate how data can exist and be shared across customer profiles. The best Identity Rules for a given company vary depending on the specificities of the company’s business model. This means that each company that onboards to Simon must decide their Identity Rules. This case study centers on the problems and solutions around helping company clients choose their best Identity Rules.
Image from Simon Data’s marketing materials indicating some of the data included in a customer profile.
• • •
A team in crisis
When I joined the Identity team, Simon had recently transitioned from its old Identity product, to its new one, referred to as ID2. ID2 was a lot more powerful than the first Identity product in allowing clients to set almost anything as their Identity Rules, which meant that, ideally, you’d have the most accurate data possible.
However, all of this choice was causing big problems. Since the change, the Identity team was spending all of their time dealing with client issues, to the point that they could no longer build any new product.
THE PROBLEM
“Even though I was hired as a frontend engineer, in 6 months, I haven’t touched the frontend yet. Most of my time has been spent putting out fires. Since Identity is so foundational to the product anytime we have an issue or bug, it’s kind of like all hands on deck.”
In short, this was a team in crisis. The goal was to figure out why engineering was bogged down with client problems and what we could do about it.
• • •
What’s really going on?
From the start, the engineers on the team collectively felt like they would be handed fewer client issues if there was some kind of tool or interface that could test Identity Rules and place more power in the hands of Client Solutions to help clients. They wanted to start building right away.
However, at this point, we only knew the problem from their perspective. Ultimately, this was a process that involved 3 groups of people: clients, Client Solutions, and engineers. Before building anything, I felt we needed to fully scope the problem and do user research on everyone involved.
RESEARCH
The research plan I wrote for the ID2 onboarding project.
• • •
Everyone is upset
I interviewed members from all 3 groups: I spoke to 3 Client Solutions members, 2 engineers on the Identity team, and 2 recently onboarded clients. One of the main takeaways was that it wasn’t just engineering that was frustrated since the changeover to ID2, it was everyone.
1. Clients were upset because…
They were having difficulty choosing their Identity Rules with so much choice available. This would delay the onboarding process, preventing them from using Simon’s platform, which they were currently paying for.
When they went to Client Solutions for help, they often couldn’t get the answers they needed. They might have to wait a week or more for engineering to get back to them.
If they ultimately chose the wrong set of Identity Rules, this would require engineering to flush all of their historical data and start the process over again, delaying onboarding even further.
“We’ve had to flush our data 3 times.”
2. Client Solutions was upset because…
They were struggling to help clients pick the best Identity Rules and answer their questions. They felt like they just didn’t understand the product well enough.
“ID2 is like a black box. When we get a client question we just don’t have enough information. We’re required to go to engineering for pretty much every question”
3. Engineering was upset because…
They were spending all of their time solving client issues rather than building product. Not only that, many of the issues they were dealing with didn’t require engineering support; it’s just that Client Solutions wasn’t able to help.
“More than half the time a client’s account is behaving correctly according to the rules they set. But since they don’t know that, we have to check it and explain it to them. It’s not an engineering problem”
• • •
Mapping it out
Based on what the interviewees told me, I mapped out this problematic process of resolving Identity Rule issues.
Map I created of the current process for resolving Identity Rule setting issues.
I identified two main areas for improvement to reduce reliance on engineering:
1. Setting incorrect Identity Rules at the beginning of the process
Getting the rules right from the beginning would make the need for flushing data and starting over less likely, reducing the need for engineering help.
2. Client Solutions handing off non-engineering problems to engineers
Though there will always be some client questions the require engineers, we can effectively reduce the ones that don’t require engineers. In interviews, engineers told me that in “more than half” of the problems coming to them, the Identity Rules were behaving correctly; they were just wrong for what the client expected. We should help Client Solutions understand the product well enough that they can determine whether or not the product is working correctly before asking engineers.
• • •
A reframe
Though the initial problem was presented as one to relieve engineering, that was only part of the story. The real problem was that client issues that were not being meaningfully addressed were cascading into engineering problems. Ultimately, what was presented as an engineering problem was a client problem. By better serving clients, we could reduce the burden on engineers.
Chart showing how client problems are currently sent directly to engineers. If we catch problems earlier, it becomes more like a funnel.
To reframe, the overall goal was really to find a way to measurably improve the Identity Rules setting process for users such that they no longer needed to rely on engineers.
• • •
What will solve these problems?
Now that we had two points of focus for process improvement, the question was, why are these problems happening? The research illuminated some answers for this as well.
1. Why are clients setting the wrong Identity Rules?
The general consensus from the Client Solutions members and clients I talked to was that the onboarding questionnaire was simply too complicated, leading to errors.
“They’re required to fill out this complicated questionnaire that they don’t fully understand and some of them just lose interest immediately because it’s too much to deal with.”
There were 2 main reasons the participants gave for this over-complication:
Difficult terminology
We’re asking them to translate their knowledge of their identity model into Simon’s terminology, which is confusing and unnecessary. Not all clients have the same knowledge of identity models and aren’t capable of doing this.
“It’s confusing because it’s like, why are you asking me all these questions? We just have an email address, why are you talking about a stable identifier?”
Irrelevant questions
We’re sending the same questionnaire to everyone, and clients who have a straightforward identity model are exposed to questions that don’t apply to them. They may answer these irrelevant questions out of confusion.
“Sometimes, especially if it’s multiple choice, they’ll just pick something, even if it doesn’t apply to them.”
2. Why can’t Client Solutions answer questions?
The only insight into a company’s Identity Rules once they were set was a table. All of the Client Solutions participants told me that this table was too complicated to actually be meaningful to them. It was too hard to look at the table and then reflect to determine why a client’s data may have behaved in an unexpected way. A couple Client Solutions participants mentioned the same thing as the engineers, that they thought having a tool to help test the Identity Rules would be helpful.
“Running a sample would be a much clearer way to see if things are working the way they’re supposed to rather than just looking at the rule table, which is really convoluted.”
The original Identity Rules table. It makes it difficult to diagnose problems relating to Identity Rule settings.
• • •
The low hanging fruit
Finding out that the complexity of the questionnaire was causing problems was exciting. This was not something the team was considering from the beginning, but it became clear that by just recreating the survey we could help clients set better rules. From the research, we knew that the two main points governing the redo were:
1. Difficult terminology: Rewriting the questions in a clear, simple way devoid of our terminology.
2. Irrelevant questions: Making sure that customers only see the questions they’re required to answer.
To only expose clients to questions that applied to them, we needed to put the survey into some sort of tool, as it currently existed as a word document. There was discussion of building the survey into our app, but starting with a tool like Typeform to prototype it first. I suggested we only use the Typeform and never build it.
The Typeform could do everything we needed and building a new product was just another thing our team would have to build (and maintain) when there were more important features to consider. In addition, the questions would be harder to change in anything we built (unless we also wanted to build a CMS) than a Typeform. With the Typeform, we could hand it off to Client Solutions to own and change the questions as necessary. I presented this argument to the team, and this was the direction we decided to go in.
To figure out the structure of the survey, I led workshops with my product manager and 2 Client Solutions members who had significant experience onboarding clients. We created a decision tree together to figure out the best order for the questions so clients were only asked what was necessary for their particular circumstances.
We then continued to refine the questions together. We removed Simon’s technical terminology from all of them and provided clear examples for to eliminate confusion. Below are some examples:
DESIGN + ITERATION
Before:
After:
• • •
The metrics
Since this was a low-lift project that didn’t require engineering, we didn’t feel the need to do extensive metrics. However, I did want to get a sense of if the survey was working better and reducing reliance on engineeers. In order to create a baseline, I looked at data from a two-month window for each of the last 6 companies that onboarded without the new survey. I then looked at the same data for last 6 companies that onboarded with the new survey. The data came from Slack conversations that I cross-referenced with Client Solutions. I asked the following questions:
How many of the clients had questions during the survey?
How many of the clients had to redo the survey?
How many of the clients set their rules wrong and required a data flush?
Before survey:
How many of the clients had questions during the survey?
6 out of 6
Every client that onboarded had questions about the survey.
How many of the clients had to redo the survey?
4 out of 6
Most of the clients had to redo the survey at least once.
How many of the clients set their rules wrong and required a data flush?
3 out of 6
One company flushed their data 3x!
After survey:
How many of the clients had questions during the survey?
4 out of 6
There was a reduction in clients who had questions.
How many of the clients had to redo the survey?
0 out of 6
None of the clients had to redo the survey after sending it in!
How many of the clients set their rules wrong and required a data flush?
1 out of 6
We reduced the number of clients needing data flushes significantly.
Once we determined that the survey was sufficiently reducing errors, we gave it to Client Solutions to own and iterate on as they deemed necessary.
• • •
Building a sandbox
Despite the survey making a significant dent in the number of companies setting incorrect Identity Rules, this didn’t fully rule out the need for better visibility into Identitiy Rules. When clients presented data issues, Client Solutions was still struggling to understand the product well enough to help them without engineering intervention. Since the idea of being able to test rules was brought up by both engineers and Client Solutions, that’s the direction we decided on.
I worked through design variations, sharing with the team and making changes based on feedback. My main goal with the design was to meet the very technical needs of the product with the highest degree of simplicity and clarity possible.
DESIGN + ITERATION
UI iterations I did of Profile Rules for the design.
The final version of the sandbox with my notes about the design choices are below:
• • •
Picking a use case
There were actually multiple possible users and use cases for the sandbox tool. For instance, should this be something that Client Solutions uses to help clients set the correct rules at the start of onboarding? Could clients eventually use this tool themselves?
But in terms of determining the sandbox’s viability in reducing reliance on engineers, I decided to look at the use case of Client Solutions’ going to engineering with non-engineering questions. From my research, engineers had told me that Client Solutions were coming to them with questions where the Identity Rules were actually functioning as set, but Client Solutions couldn’t determine that.
Could a sandbox allowing them to test Identity Rules reduce the number of non-engineering questions handed to engineering?
USER TESTING
• • •
The setup
I designed a study where I pulled real examples of Identity Rules questions from Slack. They came from Simon’s “Client Solutions helping Client Solutions” channel, whereby Client Solutions employees would post client questions and ask for help debugging difficult data problems. In all 3 examples, Client Solutions had to involve engineering. However, 2 cases didn’t require engineering.
I tested 3 members of the Client Solutions Team. None of the participants were involved in these real examples when they occurred (they didn’t post the original example or contribute in the channel). The following are the main questions I sought to answer:
1. Can they use the tool correctly?
Could they accurately translate the client question to a test in the sandbox? If it wasn’t accurate the first time, could they catch themselves?
2. Can they interpret the output correctly?
Could they determine if the data was behaving appropriately for the Identity Rules set or if this was an engineering problem?
3. Bonus: Can they explain what happened?
This wasn’t exactly necessary for reducing engineering reliance, but it would likely be beneficial to clients to get a clear explanation as to what was happening with their data.
• • •
The results
With 3 total participants and 3 examples per participant, there were 9 total tests.
1. Can they use the tool correctly?
Generally, yes! In 8 out of 9 tests, the participants were able to translate the example correctly into the sandbox. In 2 of those 8, they did it incorrectly and caught themselves part way through, but I still considered those successes.
2. Can they interpret the output correctly?
In 8 out of 9 tests, the participants correctly determined whether the Identity Rules were behaving correctly or it was an engineering bug. It seemed that as long as they entered the information correctly, they could interpret the output correctly.
3. Bonus: Could they explain what happened?
This was hard to quantify, but I gave the participants a “pass” on this one if the explanation made sense to me. In 6 out of 9 tests, I felt the explanation was correct and clear.
Anecdotally, the Client Solutions participants like the tool and seemed excited about using it in the future.
“This is a really easy way to talk about [Identity Rules]. Whereas before it had to be pretty high level.”
After both the survey and sandbox were in use for 3 months, I looked at the number of Jira tickets for client issues assigned to the Identity team. Compared to a 3 month period before the survey and sandbox, the client Jira tickets were reduced by 42%.
• • •
Reflections
Though I was happy with the results considering the problem I was presented with, I do think this process and the resulting feature we built reflect the consequences of not focusing enough on clients from the beginning, or an “engineering-led” design process. Engineering decided to offer every option available for Identity Rules even though most clients require a narrow set of Identity Rules. Had client needs been fully considered at the inception of ID2, I don’t think we would have ended up with a product with so many choices. In many ways, this project amounted to a “clean-up” job for a product that wasn’t user-focused.
Knowing what I know now, I would have probed more at the start of the project about what Identity Rules most companies actually need and pushed harder to only expose them to what’s necessary. It’s possible that we could have exposed most companies to far less options while the underlying infrastructure could still exist for edge cases.