How To Maintain User Focus In A Metric-Driven Environment

How to maintain User Focus in a Metric Driven Environment
Speaker Name: Stijn Eversdijk, Product Designer, WeTransfer
So, hi everyone, I'm Stijn. I work as a product designer at WeTransfer.
This is one of the first times speaking at a conference like this for me. So please bear with me. They asked me to come to speak today and talk about how we work at WeTransfer, how we create products and how we struggle with that from time to time but the topic was fairly open. So I thought about it the last couple of weeks. And what kept coming up for me was metrics how we work with the metrics as a product development team, as product designers, as a company, as a whole, and how that impacts my day to day because I've seen colleagues get very frustrated by a lack of metrics or by the lack of vision or the complete opposite the metrics fixation or only having to rely on one person with a vision. I believe that there is a sensible middle ground. So I would like to talk to you about that today. So I work as a senior product designer, joined around three years ago at WeTransfer and before that, I used to work in a couple of small design studios here in Amsterdam and in Berlin.
And WeTransfer is mainly known as an [application] to send big files from A to B and we have been doing that since 2009. Our biggest audience is in the creative sector and we believe that creativity brings positive change. So over the course of the last year, we've begun building a suite of tools to cater to different parts of the creative process. So you have www.wetransfer.com, which is for sending those big files at the end of your process, we've got Collect, which is for saving content from different apps into one place. We've got Paste, which allows you to make simple presentations. We've WePresent, which is a platform where we write about interesting stuff that people do and we got Paper, which is an iOS app that allows you to draw and sketch.
So at WeTransfer, I got the chance to work on both established products with product market fit and a very constant revenue stream. Now for the last year or so, I've been working on Collect, which is a completely new app. It's out around a year now, and we are still looking for that product market fit and we're figuring things out while we go. So over the last three, four years with WeTransfer, we went through a lot of change and growth. During all that, I got the opportunity to experience a lot of different situations by being in this growing company and what I realised is that different contexts ask for different approaches in how you execute product design.
So it's March end of Q1 and Q1 is an interesting time because often OKRs get set for the rest of the year. Roadmaps get implemented, ambitious plans get kicked off and you as a designer or developer might feel a bit overwhelmed. Like you're only a small cog in the wheel but I can assure you that your input here can be super valuable. I assume we all strive to build products that deliver value to both the user and to the business and if you abstract something like making a product, what you can see is that you have to balance the intangible with more concrete, intangible parts. So you've got a vision, intuition, gut feel and what you want to make and how to do that and those intangible parts get supported, informed and often adjusted through a much more tangible site metrics, usage data, user research and it's a fine balance and pretty hard to get right.
And as a designer, you will find yourself right in the middle, you have to take these inputs and translate that into something, super tangible, a flow, a UI product, something that gets out the door and you affect that so directly that you really need to take a moment and weigh what approach you take at what time. So I'll tell you a bit about my experience at WeTransfer and show you how we sometimes completely miss that balance between the two and what the impact was on me and my colleagues and what we do to try to restore that balance. So when WeTransfer started, it was a typical example of a good idea, solving a relevant problem and finding a way to monetise that without impairing the user experience. So on www.wetransfer.com, we've combined these screen filling ads. We have to the point utility to send large files and we have an in-house design team that most of the wallpapers get designed by them and we allocate 30% of the ad inventory to highlight creative or good causes. So after a couple of years, WeTransfer became profitable but you could say that it was more of like a marketing or sales company than that you could consider it a tech or a product company.
So in 2015, WeTransfer received investment and was time for a step change. WeTransfer acquired a designs studio I used to work at. I decided to join as well and we really came in as the design team and we were tasked to do a big update to WeTransfer. WeTransfer 2.0 brings it up to date, brings it into now and is ready for the next steps. So you've probably heard a lot of advocating that design should have a seat at the table. Well, in 2016, we got a really big seat. As the design team, we were a very small group of like-minded designers, figuring it out all while we went, our data game was really weak. So we had nothing to leverage there and what we did was to create a hive mind by talking so frequently with each other day in, day out, we get so aligned on what we wanted to achieve and how we wanted to do that, that we could ask each other things like, "Does this design feel WeTransfer" and actually all understand what we meant. Every decision we made during that redesign was based on intuition, gut feel past experiences and a lot of discussions.
While we were making that 2.0 version, the company went on to grow and we were trying to become a more mature product. Leadership changed quite drastically and even got a new CEO and that all changed the way we worked. So from this one siloed design team, we shifted to more cross functional teams. That's nothing new, probably most of you working that way but it was really needed for us to grow and become more mature. So initially, at WeTransfer the direction was always set by one person having a vision or intuition or gut feel but when you grow, that becomes really hard to maintain and scale. So a need arose to know what the impact was of our work. So what is tangible and measurable: metrics.
Of course, we started to try and grow but it was a gradual transition from having one person with an Excel sheet in the corner to eventually adopting OKRs on every level of the company but what you started to notice was that metrics started to dominate every discussion that we had around the product. So you could give a perfect suggestion for like, "Hey, we should do X." Then the response you would get was obvious but how does that drive up the needle for this one goal that I have? The thing is that done wrong performance metrics promotes short term vision because it's a natural tendency to only put the effort towards that one goal that you're tasked to achieve and then you let go of the rest. While of course, a lot of things in a product are not just one silo. It's about looking at the broader perspective. So we got this completely wrong and many people felt frustrated and they felt like WeTransfer was turning into this big corporate, very slow moving thing where you could not exercise your craft anymore.
So in the meantime, I got to work on an exciting new project, which is called Collect. So www.wetransfer.com is mainly used on desktop and with mobile eating the world, we wanted to find a concept that would grow our presence on mobile as well. So with Collect, what we wanted to do was something different from www.wetransfer.com. Collect lets you store and bring together content from different apps in one place. That's what it does. So the thing is we released this app and we did not yet know what, whether this was what people actually wanted to use. So we were operating on a very small skill compared to www.wetransfer.com. So when we got an OKR assigned that was to fully focused on acquiring new users, we had to 10X our user base. We were all super new to this whole metric-driven way of doing products. So we all just accepted it and we were mainly happy that there was some guidance from leadership. So the thing is at WeTransfer acquisition is not that hard. We have a lot of organic traffic already. So when you advertise under the right channels, we grew our user base within a couple of months by five times.
But the thing is those monthly active users or acquiring users did not tell us anything about whether people actually liked the app and used the app in the way that we envisioned it. It only told us to focus on the top of the funnel. So it grew quite heavily but we now started to notice that very few people returned in the long term. So the idea didn't seem to click but our goal was to get people in and we delivered on that. So everything seems fine, right?
The thing is we, of course, have multiple products, also have multiple product teams and each team got assigned their own OKR. Our biggest acquisition channel was and still is the download page on www.wetransfer.com for mobile users. So to reach the goal, all we needed to do was to grow our presence there and a different team actually then owned that page. So to get anything done there, we needed to team up with them. The thing is they had their own OKR. So their goal was to have more people download files through this page. So us acquiring new users immediately negatively impacted their goal. So what happens when teams don't align? You're very likely to not be willing to work together, maybe in some cases even compete. Well, it was not a great situation to be in and this metric-driven way of doing product development actually led us to forget about the user and about the products that we were making. We were only weighing the efforts by the impact it had on the metric. So we realized that this was a bad situation. The things that we were chasing were only vanity metrics, which you could say are hollow numbers that work great for PRM marketing. We have X amount of people using this app but it will tell you very little when you're building a product. So we took a step back. Why are we doing this? What is the value of getting more people to sign up if you don't stick?
But what we also understood was that to be able to communicate with leadership, it needs to go both ways, right? We needed to set some sort of metric. So this time we did not want a vanity metric like that but something that will give us an ID while building the product that would inform us. So we make productivity tools that we transfer and what drives productivity that's habits. So what we did was we defined the habit moment together with our data scientist to see what was actually the moment that users saw value in Collect and that allowed us to shift our focus from acquisition to activation. So users that add different types of content over time really reshaped our conversations with other teams. So you can imagine how this makes a big difference when designing your presence on the download page, that I mentioned earlier, and going through the hassle of doing that was not easy but it allowed us to restore the balance between the matrix and the product's vision that we had and allowed us to focus on improving the proposition. So from there, we were able to actually take steps that are now showing traction and the metrics are not the driver but they support and align with our product's vision and can meaningfully indicate whether we are building the right thing.
I wanted to tell you this as an example, how easy it is to get sidetracked by a misbalance in how you do product. So a growing company inevitably comes with stakeholders, more departments, more designers, different ways of doing things, different ideas on what is the right thing but it can also make you feel like you're a part of a production line. Like you are not doing the creative work that you are hired to do in the first place. However, metrics are a super valuable source of information and if you have the skill, it's a competitive advantage that you need leverage. So you need to realize that it is important to use the right metrics at the right moment to support and inform your vision.
So when working with a metrics, it's important that they come with certain characteristics. Those are neither necessarily good nor bad but if you're not a trained data scientist but just a humble product designer like I am, it's good to know that metrics allows you to go wide. It's super easy to acquire insights from a big group of people. Metrics also measures things that are established and tangible innovation often is about doing things that are not yet established though. So metrics can help you point out what's an interesting behavior, unintended or unexpected but it will not tell you where to go necessarily also using metric. A metric loses the context that actually builds up that metrics. Think about feature X was requested 10,000 times but Y was that one feature requested. Then you lose that.
And metrics can give a false sense of certainty and that's dangerous when you're focusing on the wrong metrics as we did. A reason why I see performance metrics eagerly adopted is often this seeming sense of certainty but when it does not give this higher result often the answer is to just add more metrics to the problem but then you have to ask yourself whether you are measuring the right thing, do you have the right proxy for what you want to know? Will that help you find what you need to know? Just adding more data to it is also adding more noise. I think due to time, I might skip over this or just go really fast, one of the two. Rochelle King, she is VP of product design at Netflix and Spotify I believe she wrote a book about designing with data, which is super insightful. So check it out and she talks about different ways of thinking on how to use metric within your design process. So she talks about the ones that we already know metric driven design, where great for A/B experiment Metric informed where data is one of the deciding factors, but not necessarily the decider and she coins a third, which is metric-aware in which you are aware that there are many kinds of problem solving possible but you allow for various inputs into your design process.
It's a more strategic way of thinking. So when you're facing a design problem, start from the outside in and allow yourself to get as much diverse input before going towards that one goal.
Also, if you look at your product, you can probably see different stages for it and you can identify three stages and those also influence what kind of metric is important to focus on what is relevant for your product. So you can see that there's a pre-product market fit, like Collect where we are still finding the core value. You can see a growth stage where you try to acquire as many users as possible because you have that product market fit or you know, what works. You've got a value stage where you try to retain people and lower the churn and try to get as much value as possible. So for instance, we transfer has product market fit. That's great for more metric-driven decisions while collect has less established behavior. So different expectations from people and the vision of what we're building take a more prominent role there.
So I've not talked about product vision a whole lot but we might want to get into that another time just to wrap things up, building a great product is striking the right balance between the vision and the metrics and it's super easy to lose one of the two out of sight. When you're working with metrics, consider the stage your product is in, get to the right metrics to help inform your vision. Be aware that metrics come with their own characteristics, realise there are multiple ways of working with metrics and how you incorporate them in your workflow and leave those affinity metrics at the door and build and iterate on a vision that's shared broadly by the product team so that they can operate relatively autonomously. Thank you very much for your time.

Host
Thank you, Stein, for a nice talk, any questions? There is one together with the coffee, I guess coffee, and the microphone.

Question
I have energy for a question now. Nice talk, I have an example. So let's say you your main transaction page, like a search page or whatever and you have a registration pop up and that's where most of your users are coming from and then your data team or whatever notices that if we show this registration model every five seconds, then we're going to get more users and they show this. This is they are doing an experiment and apparently the people who see the registration model every five seconds after they see it 20 times, they say okay, I'm just going to register. So that increases your number. How do you combat that?

Answer
I think there, it's also a discussion on what you value as a product. So for WeTransfer design and user experience are very important. So I don't often get asked like Hey, can we make the login form flashing so that more people will sign up but of course, we've also had discussions about splash pages to sell up products or whatever. What we did there was mainly to, you probably can from a UX perspective, already build a case for like don't think that this is the right approach. What we then did was to actually counter that problem also with the metrics. So for instance we had a landing page that would sell you the plus tier where you can send bigger files and what we noticed was that there was a 10% drop off for first time users. So then was a very easy case to make on like it's important that people first experience the product and yes, of course, people will sign up if you put it in front of their nose because they just think you're a paid product or whatever but it's worth first having people experience the product in a certain way or first have them go through this or that before prompting them with that upsell. So it's probably the conversation that you need.

Question
So that makes sense as a developer or as a designer but if I don't know, your CEO comes to you like see these numbers, make it work, then you are going to have to give that just an explanation.

Answer
Yeah, definitely but that's what I think it also helps to come up with certain arguments that they are also leveraging. I am very fortunate to work at a company where they always have worked at a company where they value UX a lot and where there's also a dedication to trying to make something less about popups. That's why we have the full wallpapers as ads instead of banner ads. I mean, we could earn way more money but we've decided to not do that, and then that's also a differentiator in terms of who you are as a brand. So probably not the answer you're looking for.

Host
Just one question over there, I guess for the time constraints, this is the last one we're going to take but you can find Stein around right for the rest of the day.

Question
Thanks. Well saying first my compliments on your presentation, I found the no answer slides with great speaker content, really good combination, so especially the confusion is broad for not having shown anything on the screen. That's good. It really clicked for me at least. So my question was in, earlier at the start of your talk. You talked about creating that hive mind within the team that allowed you to ask questions. Does this feel like WeTransfer? Could you perhaps talk a little bit more about things you experienced were like critical success factors in establishing that right mind? What should we try and do to create that, just a few maybe concrete points?

Answer
So that's interesting because we really had that hive mind when we joined, WeTransfer and then we were split as a design team into all those different product development teams and what we started to notice is that people start becoming silos again. All of a sudden everyone is only focused on their product team and not necessarily like okay, so what is the whole UX of the product? People are going rogue and for the last year I've been investing a lot of time in, how can we bring the designers back together while working in those cross-functional teams? I think what's super key is to look at what can and rituals does your team has. So how do you communicate, how do you give feedback, promote feedback early and as often as possible and also look at how do people give feedback to each other?

Because we also notice that at some point people are getting annoyed. For other reasons of a sudden they also became very snarky in giving feedback which then turns into people protecting their work not sharing it and then you have this divide, instead of you wanting those people to really work together. So yeah be as transparent as possible because I think that's what happened thereby all the conversations that we had like hey should we do this? Should we do that? It was just a lot of conversation and I think even if you're not in the same team you need to try to facilitate that.

Host
Alright, thank you, Stijn.