Yield = Accuracy * Frequency

Anne Droge: The CEO banged his fists on the table and said, "Right, we'll be building two integrations before the end of the year". But suddenly, our roadmap was out of the window. Our team had to scramble for resources and we had no idea what we were signing up for. Our organization had just been dressed down by the CEO, our autonomy taken from us, and we had no idea why this just happened. We all of us here are hired to innovate. The company expects us to deliver. But how do you innovate when you don't know what problems you're meant to be solving. The unfortunate truth is that innovation is not a magic spark, a moment of genius. It's a repeatable grind that requires constant attention, diagnosing and communication. One after the other. We're going to tell you our story from when we joined Vend. And the lessons we learned along the way, as we tried to turn the ship around, regain trust, regain autonomy, and end up at a pace where we can be proud of where our colleagues say, wow, product engineering, design, we just can't keep up. My name is Anne. I'm the Director of Product at Vent. And at that time, I just joined the company.

Ludwig Wendzich: My name is Ludwig. I'm the Director of Product Design event. And at this moment, I was leading product at Vend.

Cara: I’m Cara VP of Engineering at Vend, and I joined shortly after this moment.

Ludwig Wendzich: You might wonder how did we end up here? Let me paint you a picture, you might recognize it. Imagine a product company. There's a number of product managers, they're all creating a backlog of things to do, and teams that churning out some of those things, you're shipping really fast. But the list of things to do that just continues to grow, and your customers aren't necessarily getting much happier. You're adding teams of engineers, and they're all autonomous. They're all in pods, or squads or something, and they've all got a PM, and they've all got a designer. And this is enabling you to increase the amount that you're delivering because you just keep having squads.

But the problem is that the different parts of your product, they're starting to look and work like very different products, the product is splintering. You're adding so many new things that the old things that are broken, they don't get fixed, the chase of the new of the shipping fast, it's become all-consuming. The ship is starting to come away at the seams. Now I'm sure you have never experienced anything like this before. And if you haven't great job, but I'm going to guess that at least a few of those symptoms are familiar to you.

So this was the ship that I joined. When I joined Vend about five years ago, we were moving fast, super fast. But the ship was starting to fall away at warp speed. Now, before I arrived at Vend, we had done a lot of work tweaking for frequency. And honestly, a huge part of the product world is just obsessed with improving frequency. In fact, one of the principles that we still hold your event is that we optimize for fixing mistakes quickly, rather than not making them in the first place. Being nimble is important. Yes, shipping is important. But when you overemphasise that shipping to learn, just fucking do it. Ship it!. You start asking questions like, "What are product managers even for? Why do we even have them?" Why shouldn't we just have these squads of engineers with designers?

So I joined Vend when we were at this juncture and asked these questions. We were shipping fast, and the rails had started coming off in terms of the quality of the product. You see embedded designers had no incentive to work together. And nobody was really incentivized to ask why or is this even working. Everyone was incentivized to come up with solutions to just get them out there. Ship, ship, ship.

Now, when I first started, I had this common refrain, if only I had been involved earlier, from literally everyone, engineers they wanted to be involved early in the design process, designers they want to be involved early in the decision making process about what we were going to make. So they could understand why. We were in this position where a designer would be asked to work on something and they'd have to go back and do a whole bunch of product work, you know, asking questions like, what problem are we solving with this solution? Is this the best way to solve this problem? Do we even have a problem to solve? What is the customer's context?

You see structure matters. And our teams were structured like so many product orgs talked about at the time. So we have these different squads. Yes and they were all fully autonomous. They all have the resources they needed to succeed. Each of those squads had a BA that had engineers, a designer, a PM, QA all of these folks wanting and by some definition, they were succeeding. Friction was low and we shipped a lot. Now, this is where I like to go on a particular rant about how we all have these negative associations with conflict. You know, we say conflict, and everyone immediately wants to know, how do I just make this conflict go away, we avoid conflict at all costs.

But conflict isn't necessarily bad. Resistance improves your thinking and outcomes. And we had just removed all conflict from the entire process. Designers, engineers, they reported directly to decision-makers, those people who would decide what we would make, and low and behold, we would make those things. Now, this sounds great in the short term. The problem is, is that we had one person with two incentives, long term, and a short term incentive and the short term incentive that always won out, our team leads were optimising for shipping stuff in the short term because that's where they were constantly feeling the pressure. And nobody was really thinking about the long game.

So we needed to split those responsibilities and introduce friction, we needed to introduce conflict. So over a few years, we moved the designers reporting line from that team lead, which was this hybrid product manager, engineering team lead role to reporting to a designer. We also split that team lead role into two, a lead engineer and a product manager. And we ensure that our practitioners all had functional reporting lines. Finally, we introduced the triangle, a designer, lead engineer and product manager would lead the software development lifecycle from start to finish together. Product managers, they needed to be responsible and accountable for the why, or the long term results. Engineering leads for the short term delivery, and designers for the overall consistency and usability of the application.

All of these people need to have skin in the game, they need to have something to stand for, they need to have a corner to pull into, they must all contribute to the different trade-offs that are made during the delivery process. And that's going to mean learning how to navigate conflict, they all have to be part of making the right trade-offs for the right reasons. So our first lesson was this, frequency is not your only lever. The theory is, is that if you increase shots on goal, the likelihood of striking gold will increase, and thus your yield or the value that you deliver to your customers should also increase. And sure you're going to ship a lot of things that aren't valuable don't really work as expected. But that's okay because you've optimized for shots on goal. So you have many more bowls of spaghetti that you can throw at the wall.

Now, this works in theory, it also works in practice, if you have seemingly endless capital, like Google and Facebook, but we don't have seemingly endless capital. And so just throwing stuff at the wall wasn't the best use of our resources, we needed to increase our focus and limit the area that we were trying to tackle. This meant that the bowls of spaghetti that we did throw were clustered around the goals we cared about. And we could achieve a high enough density to increase the yield where it mattered.

So with designers and product managers, given the voice and the conversation, we were able to look up at the long term direction of our product. We started doing user research and planning how different parts of our product would come together. We moved away from just fucking do it to jobs to be done. And we started to think about what does value actually mean to our customers? Like, why are they hiring us? We started to talk to our customers. And we learned that you know, we could learn stuff about our customers by talking to them, we didn't have to learn everything by shipping. So with the triangles, the why, accuracy became super important? And maybe a little too important because we also started to see lots of blue sky thinking and that often translated to long running projects, that pendulum had swung away from the ship, ship, ship to the more, more, more, accuracy.

Instead of increasing our overall yield, we were too focused on increasing the impact of each individual deliverable. And that often meant long periods of time with no delivery and thus no impact. We were stuck. So a clear example project that we had, we're going to call Project X. You don't need to know the details of this project. But you will probably recognize, it's a deathmatch project, we've all had. Anne is going to tell you more about it.

Anne Droge: We were stuck. It took us 550 days, 18 months to get value out of this Project X. We did really great user testing and really nailed accuracy. We needed to make a few foundational changes here and there to the page to solve the goal. But I think you've all been in that situation. The rebuild ended up taking so long that we ended up descoping. And we didn't descope based on minimal viability or our missteps that we needed to ship and learn to see whether we solved the problem. But we ran out of time we ran out of goodwill from the organization, it was getting too costly.

So I joined at the tail end of this project and joined as their PM, their PM number three. So speaking to the team, they had no appetite to roll out, they've done all the slog, and were still not done. The team was really demotivated and ashamed, although each and every single one of them did a really incredible job in their respective area. So while this was going on with the team, slightly fully demotivated and slightly ashamed, the company was actually promised this next big project, a vision of this next big chunk of value and piece of innovation that this team was gonna deliver on that was based on a burst of discovery, like many interviews cramped in a few weeks. And this was supposed to set out so for an entire job to be done. So here at Vend, we've identified three main jobs to be done for our customers. And they're the pillars of our product strategy.

So we really set ourselves up to fail, right? Because by trying to optimize for accuracy, we tried to optimize for accuracy across a whole job. And we actually ended up taking away our ability and also our autonomy, to learn and iterate, and on top of that robbed our customers of time with value. So as an organization that believes itself to be very agile, this looks dangerously like waterfall. So the trust of the organization felt like it was at an ultimate low. And why does this matter? So we've realized and we've experienced that first is really key to ensure autonomy for the teams, it's that secret that kind of helps you stay away from becoming a feature factory, or the CEO having to bring his hand on the table.

In my first few weeks speaking to the different teams, so support sales, the recurring sentiment was, what have you done for us lately? What are you doing there on the other end of the office? I don't understand what you're doing. If we only had this feature, if you could just go and build it, we could close all these deals and make some extra money. So this wasn't a sentiment only within the teams. As you heard in our introduction, the sentiment was shared by a CEO and also the executive this time. And their response to that was introducing a meeting called steering, it was a format that forced accountability on our part, a format that probably initially set out to exercise control, start influencing the roadmap, and get a sense of what we were actually doing. But it actually ended up being a blessing in disguise.

And let me tell you how. So there's this great article from Marianna Bellatti on all the best engineering advice that she saw from them non-technical people, where she explains why trust matters so much. So effective teams need trust. And trust degrades naturally over time. It's really observability. And not success, that is the key factor.

So organizations have to be constantly reminded why they've hired great people, why they've hired you. And you know, great people that know what they're doing. They notice in the beginning when you just jump on board. But over time, if the value of the employees and of the teams it's not observable, then it can't see it, then trust the grades and measures of control, such as starting to brief and features become more and more attractive to leadership. This is what we all as leaders and product engineering designers, PM's need to be constantly focused on, we need to constantly improve the actual observability and value of the work that the teams are delivering. It's about the value that we ship and remind the company why they hired us so that we can keep bureaucracy at bay by maintaining this high level of trust.

So just for you to double check, and you have lost the trust when you get a lot of what have you done, what are you doing, start doing this feature, or let's set up a meeting so we can prioritize by committee and extreme one is CEO having to come in and bang the table and tell you to build two integrations. But I hope you haven't experienced this. So steering actually really helped us as it provided a platform for us to regain that autonomy and create visibility. So you learned just from the start that you cannot discount accuracy, right? We don't have these unlimited balls of spaghetti to throw at the wall. And you should be shipping often and telling everybody about it as truistic rates and it's that visibility that observability of work that builds trust and This is really key to ensure that you have the autonomy. And we and I assume you believe that we need that autonomy, right?

We need it because we're the ones close. And we and our teams are the ones close to the data, the customer pain. So we need this to ensure that we solve the most important problems and drive outcomes for our customers. So product had to step up. Instead of solving that entire job. After research burst, we needed to bring back the ability to learn and iterate. Without a clear problem space, in definition, everything is on the table. So all the problems are connected, you're never done. And you'll probably end up trying to solve it all.

So we needed to be accurate. But the problem space and outcome needs to be defined small enough so that the team has a really clear target and goal and autonomy to then define and deliver the watch, ship and learn and come back, we need to refine the problem space. So how did we do this? So a few things we changed, we moved away from this burst discovery to continuous discovery. So currently, recent research is embedded in our daily practices. So we reach out to all of the customers who leave a survey with feedback in it. And we asked them if they want to share with us more about their experience. And we record and transcribe those interviews, and then highlight all the interesting insights and take the relevant people across the company.

So this really provides us with a constant stream of qualitative insights, that is also on top of that super accessible to anyone in the organization. Secondly, we write narratives about the problem space. So this is really to drive that clarity and transparency through writing. So that we're really clear on the problem to solve and what success looks like when we solve it. And we ensure buying on the outcome and the problem level. Because I think you all know that framing of on the feature level, everybody seems to have a different idea of what that feature will actually solve for them. We also scope on the functionality level. So based on the goal, we do this before we start working on it.

And notice again, once you start working on a project you get so emotionally invested. And that feeling of just give me one more day, just give me one more week, that's where you have high risk for scope creep. So we also learned that we needed to create visibility of our work, right? And speak the same language with the rest of the company. So we started our journey on understanding our retention metrics, we broke down our three main jobs to be done into habits, and habits are like the repeatable behaviours that the customer engages in that demonstrate their experiencing value.

So we correct our initial activation habit and have already seen massive value. This has helped us bridge our world that we live in the role of the jobs to be done and the customer problems to real measurable impact on our bottom line. So it now providers is a shared language across the organization, money really helpful.

And the communicate, like communicate, right? Communication about everything we ship as a product update in Slack. We ensure we create visibility at the steering meeting. So we demo, we also send out and release notes. And at the standard, we actually talk about what we've shipped. But we also make commitments, what can be expected in the next month. And then we have like I just mentioned feedback sessions with sales and support. So what have you learned so far? So in order to innovate, your goalpost needs to be accurate. Accuracy key, no spaghetti on the wall.

Ideas need to be able to survive that tension right in that conflict within the triangle. The goal needs to be small enough so that you can increase your chances of success and you know, have a clear outcome to iterate against. However, as you can see here, it's not just A, It's not just accuracy. What about F? What about frequency? In order to keep our autonomy in the room to innovate, to make sure that we solve the most important problem, the company needs to be reminded why they hired us that we're doing our job. It's about the observability of the work so that we can maintain the trust that we just learned degrades over time. So the frequency of shipping is super important to keep the trust. And this is where engineering takes the lead. And Cara will share with you some main lessons on how we brought frequency back into the equation.

Cara: So I started at Vend then six months after Anne and in my first week, I had two conversations that still stick out for me. The first was with our CEO, and I was expecting nothing more than a meeting grace. Project X was still very much on his mind. He said "Have you heard about Project X?" And of course by that point I had and he said we should try and figure out what happened there. Why is it taking so long? And the second conversation was with another leader. She said engineering used to have a lot of sweat at the end, it has changed now as other parts of the company have started to step up. It just seems like everything there is a drag. And looking at our metrics at the time, it really seemed like we were excelling.

We were deploying to production every 22 minutes, we had a culture which encouraged very small pull requests, which meant that our lead time was actually very quick. We had invested in tooling and processes which enabled us to recover from incidents very quickly. And actually, our change failure rate was really low. According to the DORA report we share the characteristics of high to elite DevOps performers. And similarly, our values with things like incrementalism, continuous improvement, and failing fast, we talked about favoring small slices of work over Big Bang releases, we had built a framework to roll out progressively so that we could learn and improve from failure. And in spite of this, as Anne described, the sentiment at the time was, what have you done for us lately?

So there was this dissonance between the beliefs we had about ourselves and the way the rest of the company saw us. So Ludwig talks about coming in at a time when we had successfully optimized for frequency. The focus then necessarily turned to addressing accuracy. And Anne talked about the cost of over correcting the accuracy, which took away frequency. Metrics were telling us only one part of the story, the pendulum swing was the other. We needed to address frequency in engineering delivery again. So I want to pause in the story for a bit to go back to Anne's point about autonomy. Our teams at the end are autonomous, they decide what they will work on and when and it is also up to them to decide the how, the technology, the trade offs, and so on.

But autonomy by itself is not enough. Autonomy needs to have a countervailing measure for it to work. We need to also have accountability. And at this point, at Vend, there was no process which encouraged accountability for our teams, there was just a little awareness of projects going off the rails, and in many more cases than Project X, projects will roll on almost unnoticed for months on end. And in the case of Project X, it was 18 months. And so we created an arena of accountability for each team, we introduced weekly check-ins. And a check-in is a meeting between the whole team including that triangle and product engineering and design leadership, split into two parts.

So the first part is the celebration of the work done, where the whole team is present to demo their work. And all workers work. So whether it's an investigation or performance analysis, or automated tests, or a new API endpoint, or something with a UI, it's all encouraged and celebrated. And the second part is with a smaller group, the team's triangle, and our triangle. And in the smaller group, we can be transparent about issues, we can help with blockers, and provide more context we needed. But importantly, it's an opportunity for the triangle to take a pulse check themselves, and reflect on progress.

Another way to look at it is that we have created an arena of visibility for each team. It was easy now to see how teams were doing. And as Anna said earlier, it is observability much more than success that builds trust. We also introduced the concept of a cycle, we start a cycle with a refined problem space that Anna described and we add to it an understanding of the value added by solving that core problem, the value to the product and the value to the customer. And this informs our investment or the amount of time we are willing to spend on solving that problem. This is the time budget for the solution. So with the parameters of the problem to solve and the time budget to solve it, the triangle worked together to define a solution. And this flips the traditional approach of problem to a design and estimate cycle between design and engineering. It shortens the time to implementation by pre constraining the solution space, and also by replacing the back and forth solution definition with high collaboration by the triangle.

The solution also needs to be clear on where its components are on the scale of must haves should could and bond tabs. And as the cycle progresses and dragons are discovered we remain within budget and on target by scoping out components from the won't have end of the scale. It's important to note also that a cycle is not a sprint. So where the over flow of one sprint is carried on to the next, a cycle is defined by its problem, which means that the cycle must deliver a solution to that problem, and there can be no overflow. We de-risk the cycle by following the principles we've always held. We have this thing at Vend to eat the frog. And it comes from a saying that is attributed to Mark Twain, which is if it's your job to eat a frog, it's best to do it first thing in the morning, we front-load the risky work to give us more time to react and adapt.

We integrate early and often so that any misalignment can be addressed. If we discover we've gone in the wrong direction, we cut our losses, learn from them and move on. So before these changes, we had ended up in a vicious cycle of focus on accuracy, which led to infrequent delivery, which encourage even more focus on accuracy. And our cycles now are typically six weeks or less. So we had only one shot and meeting months previously, we now have an opportunity every six weeks, we can be riskier because we have many more shots on goal, and the cost of missing one is minimized. Anne talked about forced accountability turning into a blessing for us. And I just talked about the introduction of check-ins and cycles to add an additional layer of accountability.

We've leveraged arenas of accountability to retain trust and stay aligned across the triangle, the team, the product engineering and design organization, and the wider business. So I wanted to come back to metrics, they didn't indicate a problem earlier because they focused on a very different part of the picture. And now almost 18 months on from introducing check-ins and cycles, we now deploy to production every seven minutes. This is down from the 22 minutes I mentioned at the start of this journey. And our tooling and processes have continued to enable us to keep our lead time, time to recovery, and change failure rates low. Improving our processes didn't have to be at the cost of engineering capability. But it pays to check.

So I want to be clear here, we're not pitching check-ins in cycles as practices for your teams to adopt. There is no one size fits all solution. The lesson here is about going deeper and honing in on the on the things that don't look right. If your metrics are telling you a different story from your stakeholders, dig into those issues, diagnose them and solve for them. Take this principle over the rituals which worked for us. So to recap, Lesson one, frequency without focus leads to spaghetti on your walls. Frequency is not your only lever, couple it with accuracy and you can achieve a high enough density to increase yield where it matters. Lesson two, trust degrades over time, increase visibility to retain trust and ensure autonomy. And lesson three, tune in to the performance signals for your organization, diagnose the issues and solve for them. This will be more impactful than just applying the rituals. Innovation is not a magic spot. Staying into the performance of your organization needs constant work and collaboration. Each discipline must step up and course correct. This is how we increase yield. And this is what enables us to innovate. This is the value of the triangle, the tension that keeps us moving. And this is the grind!