Roadmap To Establishing A User Research Program

Hello, everybody. And thank you for joining me today for the session Roadmap to Establishing a User Research Program. My name is Yael Gutman, and I've introduced data driven decision making to three organizations, building up user research programs from the ground up. It's the tips and tricks that I've collected throughout my career that I want to share with you today, so that when you go on your own path, you can do so as smoothly as possible with the least bit of obstacles.

But first, a little bit about why this is so important to me, I'm obsessed with identifying opportunities and find the key for the solutions. But throughout my career, I've been in situations where I wasn't clear on how we got to the path we're on. There were three main contributors to this. First, receiving top down directives, and what solution to build with little to no discussion on what problem we're trying to solve. In other cases, where we did discuss what problems we're trying to solve, there was something based on organizational folklore or executive intuition, and not on empirical data. Another challenge was that some in the organizations believe they knew their users so well, there was no need to gauge the feedback from users. To combat these challenges, I started using data. And I found that when providing the same stakeholders and executives with data, they were more than happy to review the thinking and change strategy. And so, this became my strategy and how we started making more informed decision.

And this is backed of course by studies. A Mckinsey & company study from 2018 found a clear correlation between good design and superior financial performance. A Walker study from 2019, estimated that by 2020 customer experience will be the key contributor to brand loyalty, overtaking price and product quality. They also found that 86% of buyers are already willing to pay more for a better customer experience. Zappos and JetBlue are great examples of companies that put user experience at the top and benefit from it financially.

But although there's clear evidence to why companies should invest in this, there's still ways to go. Many companies are finding it difficult to adopt methodologies that will get them there. And McKenzie, the same McKenzie study found that 40% of companies do not talk to their customers during development. I use a Zoom study that focused on serving UX professionals found that the biggest challenge of UX professionals is getting UX research and testing to be part of the product development process. 49% of their professionals stated that executive buy-in is a challenge.

These are some of the challenges that I faced in my organizations that can give some color into the challenges. Limited resources. Some organizations believe that they don't have the budget or the time for user research. So, today we're going to talk about how you can start really small nibbled and cheap to really get their organization psyched and excited about user research, and its impact on the organization. Next, there are organizational structural issues that make it difficult for user research to be adopted. One we discussed before, when the organization already believes they know the users and there's no need for user research. And another obstacle is where user research is available in their organization, but it's only in specific areas done by specific teams. It’s not across the whole organization. Another area that's challenging is when departments that we should be collaborating with don't really understand the benefit of user research, and they may find it to be stifling their creativity or slowing down the development. And today, we'll talk about how you can turn those colleagues into ambassadors.

So, this is the roadmap we're going to review. The four steps to the roadmap, and we'll go through each of them. And storytelling is an element that grew out throughout the longevity of the program itself. So first order of business is to understand the baseline of what your users are doing and what they're thinking, and identifying the toolset in order to capture that data. So, in order to provide insights to your organization, you need two types of data sets. One is the quantitative data. What are the users actually doing? What are they clicking on? What pages they're on, on your website or on your app? The other type of data set is the qualitative data, which is why the users are doing what they're doing. And you need both of those in order to get the complete holistic view of what your opportunities are. And there are choices that are reasonably priced and three methodologies that will allow any organization, a low entry into the user research world.

So, for quantitative. When I joined my current organization, we had Google Analytics set, the free version. It wasn't implemented in a holistic way. And so, we had to first order of business was just to straighten that out, so we can have a baseline. Next, we did want to start with A/B testing. We tried the Google Analytics free A/B testing tool. We have a one-page application, so it wasn't flexible enough for us. But, definitely recommend that you guys check it out if that's something that would work with your platform. We ended up, after securing investment, integrating with Optimizely, which is the current tool we use for A/B testing, and we're very happy with it. CrazyEgg is a tool that I've used in the past. For heatmaps, it's limited in its capabilities, but for the beginning stages you can find it very helpful. We've since moved on to Clicktale, now Contentsquare, for heatmaps, as well as recordings.

For user research, in my current organization, we started with user interviews and user testing. And to ease organization into it, we did it with no additional costs to the organization. I planned and executed the interviews and the tests with a colleague. So, it was just within our regular roles, no additional cost. I'm very lucky. My current organization is a member organization, and so we can have easy access to members. We ask our membership team for the types of users we're interested in, and they provide us with participants. But regardless of what type of organization you are, you can start really small with user interviews and user testing, just to prove the validity of this methodology. And once you prove the methodology, you can then go and request more investment in the program.

You can also use Usertesting.com, which is very reasonably priced. It's an online tool where you can set up your actual test, and then observe the participants which Usertesting.com also provides. For surveys, I started again, very cheap, when I started in this organization with Survey monkey. Believe it, it was a thousand dollars a year and it served up purposes. It was limited. It wasn't as flexible as we needed, which is why eventually we moved to Qualtrics, so we could create more flexible ways of targeting our audience and prompting the survey. But Survey monkey was a very good tool for us.

Next, once you start establishing user research on your own project, as an individual contributor, it's time to share your insight. And you should share them as soon as you get them, so that your department, your management, and then eventually the organization get to hear about user research and its impact on KPIs and strategy. And when you storytell, it's really about telling the story behind the data. You should share the insights, package up the data so that it's meaningful to your audience and don't make them do mental gymnastics in order to understand the point. Just feed them the learnings that you've established and provide it in a digestible format, use plain English. Sometimes we forget that the terms that we use in our day to day is not shared with the whole organization. So, really use language that the organization, on a whole, understands. And lastly, you have to adjust your message to the audience. So when we present to the management and to the board, it's the bit-size nuggets, it's the most important learnings that we had. When we share it with the more broader team or with the working teams, we go more in depth and share the whole flow. So, it really depends on who you're talking to.

So, this is an example of a fictitious—a fictitious project of implementing chat. And this is something that we would share to the management. So, 67% of issues were resolved during chat. That was the goal of the project, and so that's what we would message back to them. And we're also providing cues in terms of the visuals. So, issues resolved are in green, which just visually means this is good. And for the 12% that's indicated here, which is members still has to do some action. That's in a more red tone indicating just by the color that that's something that we should look into.

When we would share information with a working team, we will provide additional information that wouldn't be of interest so much to executives. And so in this case, we would share the issue types and the issue of complexity. And this is because the working team could use this information in order to make decisions on the strategy and how to serve customer service.

After you've established user research on your own projects, it's time to expand and integrate it to the rest of the team. A challenge that I faced was that not everybody on my team was inclined to do user research. And in order to tackle that, we mandated it. So, we integrated it into the product development lifecycle as a must-zone in the project—within the project. But to make it easier on the team to manage user research, to plan it and to execute, I templatized everything I could think of. So, I templatize a discovery deck, I templatize a test plan, I templatize a test results of A/B test, I templatize a post-launch report. The idea of templatizing was to provide my colleagues with a template that would make it easy for them to focus on the content and not on the structure itself. Another thing that I did was that I leveraged potential ambassadors within my team. So, I identified specific staff that were more inclined to do user research, and then really focused on them. And I helped them plan user the research, monitor it, and then deliver the results of that. And these of course improve their rationalization of why they did a project, but also what it helped was to establish the importance of user research within the department itself.

Once you've established yourself as a ma—in your project, as an individual contributor, and then expanding it to the rest of the team, it's really important that we are able to provide an environment for creativity and for research ideas. When we started A/B testing, the team was so excited, we had so many ideas. But at some point after, the enthusiasm went down and we sort of went through all the low hanging fruits, we hit a dry spell. And it was a real challenge to come up with testing ideas. And so, to combat that, we did two things. One, we leveraged partners that we already had within our organization. So, we started sharing problem areas with those partners. And we listed—we solicited advice from them, from experience that they've had with other customers in a wide range of industries. And they provided us with solution ideas, as well as testing ideas. Another thing that we did was that we expanded how we collect ideas internally. So, we created an inclusive research tank, and we invited other teams to participate. In our case, we invited our Technical team, QA team, as well as Customer Service. And what we identified was that by including them in these sessions, we were able to get additional point of views that we didn't have before. And the breadth of ideas that we were hearing was much wider. And now the benefit of making this think tank inclusive was that these participants from other teams now became ambassadors, and went back to their teams and told their colleagues about the user research program.

But these think tanks were not a success from the get go. The first couple of sessions did not result in new ideas. And it was at the end of one of these sessions that a colleague from the Technical team asked me, “How do you actually come up with ideas?” It was so enlightening, product and design are tasked and expected to solve issues, where the tech team is usually charged with execution. And so it's not part of their daily routine to think of ideas. So, we changed the format of those sessions. And instead of asking the team members to come in with ideas, for testing, we asked them to come in with problem areas that they've identified within the user experience. And it was within that session that we discussed the problem area, the challenges, and organically came up with possible solutions and testing ideas. Another challenge that we faced was that not everybody was comfortable suggesting ideas. There was this sense that it was sort of a little bit dangerous, and what will happen if it doesn't succeed. So, in order to combat that, we started celebrating publicly failures, as they were once called, but we called them learnings. And so, when we have an A/B test that doesn't hit the KPI that we had expected, we shared that just as much as we share those that do. And we celebrate the fact that we didn't go onto implementing something, and getting technical resources to do something that would not have been as impactful as we had hoped. We also emphasize how good these learnings are, so we can know these are all building blocks towards additional tests. Another thing that I do personally is that I share when I thought, “Oh, I thought this test would do this, or would do that,” and how humbling it is to learn that it didn't. And the last thing is that we try to make these sessions fun. So, of course this was before COVID, when we did meet together, and we would do these contests and whoever would find, whoever would guesstimate, the—are closest to an A/B test result would receive a gift card. So, always good to make it as engaging as possible.

This is the last phase, where the research is really part of the organizational culture and indispensable to the organization. This is where we are now, and this is just a continuous phase where we want to stay. This is where the product teams that initially didn't understand why we were asking these questions, and may have said that this was stifling their creativity and putting a pause in development, these teams are now coming to us requesting data so that we can make more informed decisions. Through our continuous socialization of results within the organization, we've had other departments come to us and request that we help them with their research needs. So, apart from being very gratifying, this is also being an opportunity to further solidify user research as a very important component in the organization. And this is where management stats expecting user research. So, if they don't hear about user research, they ask for it, and they expect it to be part of the delivery.

So to summarize, just start, you can start small and you can start low budget. And you can start in a way that you're proving to the organization how impactful it can be. Once you prove this on your individual projects, as an individual contributor, it's time to spread the knowledge and integrate it within the product lifecycle itself so that all projects benefit from user research. And the further you extend it, the more important it is to facilitate ideation. Come up with ideas so the ideas are fresh, and you can continue to optimize and learn more. And the most important thing is to continue to share and socialize with your team, with management, with executives, that is the way that you get to a healthy, stable, and a long lasting program. Thank you so much for joining me today.