How To Experiment In A Zero Failure Environment

Hi there. It's a true privilege to be speaking to you here as part of UXDX. Today I'd like to tell you a story of how I learned how to do Lean experimental product development in an industry where failure can literally be a matter of life or death.

So I'll quickly introduce myself. My name is Tom Alterman, and I'm on the product team here at Asana. For those of you who don't know, Asana is a work management platform, with a mission to help humanity thrive by enabling the world's teams to work together effortlessly. Now, I've never worked with a company as mission-driven as Asana before. It also has an incredibly strong culture of collaborative product development and experimentation. And I'm not just saying that because they pay my paychecks and feed me delicious meals every day.

But most of my recent experience has actually been in the construction industry. I fell in love with construction back in 2015, when I joined a startup called PlanGrid. That our founders were the first to see how the form factor of the iPhone and the iPad could revolutionize how work got done on construction sites. They taught us to focus on building beautifully designed apps that foremen and superintendents would fall in love with. We had the highest-rated construction app in the App Store.

And at the time I left, we were being used in over 1.5 million projects across 100 countries. In 2018, we were acquired by Autodesk for $900 million. Now, while we always wanted to deeply understand our customer's needs, we had a challenge that will be familiar to many of you. And that's how to ship great products quickly when they have to work perfectly the first time. Much like getting ships to water, there's very little opportunity to revert.

Now, I'm sure many if not all of you have read this book. If not, I strongly recommend you do. It's really defined the last 10 years of how we build products and companies. In it, Eric Ries talks about how to get things to market fast using experimentation. With a key tenant being to fail fast, succeed faster.

You have also seen this quote from Reid Hoffman, if you're not embarrassed by the first version of your product, you've launched too late. And there's Facebook's famous, if not infamous mantra of move fast and break things. All of these ideas always really resonated with me. But another one also came to mind. And that was ambivalent building tools for these people. They don't want to hear about anything failing, ever. Their safety and their livelihoods depend on the tools that they have at their disposal, being 100%, reliable and dependable.

Now this has long been a challenging construction, any change can be incredibly risky. So people prefer to stick to tried and tested methods, even if they are incredibly inefficient. This is the world in which we as a relatively scrappy startup are trying to build new tools to help them get their work done faster, easier and safer. Now, you may not be working on construction, but I'm sure many of you are working on software that your customers rely on every day. And you know, they never want to feel like guinea pigs for your experiments. If that's the case, this talk is for you.

So I'm sure you're all paying incredibly close attention. But for those of you that have attention spans as bad as mine, I'm going to give you the quick TLDR upfront. So if you want to do experimentation in these environments, I recommend building a cross functional team that truly cares about the customer. Learning about that problem together, treating everything as an experiment, not just a product, but the process, and even the roles that you're going to have on the team and do the least amount of work needed to test your hypothesis.

Finally, when failure is not an option, use human power as the first functional MVP. So let me tell you a couple of stories about how I learned this using my own blood, sweat, and tears. Both of these stories revolve around an absolutely insane part of construction called Submittals. So what are Submittals? In the United States, you need approval from the architect for every product and method you are going to use to build a building. This means every brick, every door down to the actual nuts and bolts needs to get sent to the architect and approved before working continues. This means 1000s of documents that need to be collected, reviewed, and approved on time and correctly. And if that sounds crazy, it is because it absolutely is.

A two things to remember about this. Firstly, everyone truly hates it. It's hard, tedious and stressful for everyone involved. You're filling out spreadsheets for days, you're keeping track of millions of emails, the paperwork is endless. And in the best-case scenario, if one of these is missed or late, the work will be delayed, your company could lose 10s of 1000s of dollars and you could lose your job. That leads me on to the second important thing here is that these are incredibly important to get right. Because in the worst-case scenario, buildings fall down and people die. That's not an exaggeration.

This diagram shows a single bracket that was built incorrectly in a hotel in the 70s, which caused a walkway to collapse, killing 114 people. All because no one spotted this seemingly small mistake in one page for 100 pieces of metal document. It's an important process.

Our founders came to us and asked, Can we build something in PlanGrid that would fix this process, or at least make it not suck as badly? Obviously, this couldn't fail fast or at all. Being a scrappy startup, we had some pretty fun constraints, or be familiar to many of you. We had four engineers, actually two engineers, two others joined later, we have six months to get from an idea to something that we could sell to customers, or in the future, the business is dependent on this. So don't mess it up. All of this required us to rethink how we were going to build a product. So this is where we started to get experimental. The first step was learning about the problem together, we needed to truly understand the process to figure out how to tackle it.

We do this with interviews, site visits, and shadowing, everyone on the team spends time with users. And we really mean everyone. This photo here is also one of the visits. And it includes engineers, research, product design, QA, all of us for that. We really got to know them. What are their days like? What are the boring parts, all of the submittals processes takes place in spreadsheets and emails. I thought it was weird that we wanted to watch them do paperwork all day. But we learned so much from even the mundane bits of their day. As we went, we built up a picture of the process and pain points. And after each conversation, we'd list all the questions we still had, we'd vote on them as a team, then change your script for the next set of conversations. And we truly felt confident we understood enough of the process to propose a solution. And to reiterate, everyone was involved. And this was incredibly important for us because we felt everyone needed to be brought in and focus on these customer problems to really have that focus in the timeframes we had.

We knew there was one of our engineers who was incredibly talented and was more interested in systems work than working on this problem. So he left to go and do great work elsewhere in the company. It was more important for us that everyone was bought in, even if it meant being an engineer down for a while. So we understood the problem and built a great team that cared about the users. And this helped us flesh out a few key initial hypotheses. Now we need to test whether we were right or not. You're probably familiar with the standard iterative loop or product development. You come up with an idea, you build and launch it. And then you learn from it, and then iterate from that.

We had an idea, or we didn't have the time to build it. So we needed a way to shortcut directly to the learning of iterate faster. We needed experiments. We had all these hypotheses from our research. So we used a similar process before. We sat down as a team. We listed out all the ideas we had. And we ranked them by risk. What's the likelihood that the product will fail if we're wrong about this hypothesis? And then we thought about the cheapest way to test each of them. The mindset we had was to be lazy. Don't do any work that doesn't help us de-risk, the riskiest assumption. Sometimes the test was simple, like a survey. Some things you could test traditionally like clickable prototypes, concepts, tests, those kinds of things.

In the end, we were left with the hardest thing to test. Could we build something to replace spreadsheets and email? Everyone's process was a little different. Dominators and prototypes weren't really resonating. At some point, we realized that the only way we were going to learn more was by actually giving users a version of an idea to use with real projects with real data. This led us to the idea of a concierge MVP. We need to learn if the hypothesis was correct. We had real data, and we needed 100% reliability, no buildings falling down. And we need to do this quickly.

So how do we validate a fully functioning product without building the machinery behind it? For those of you who've seen the show Futurama, you know how the robot bender refers to humans. We create project meatbags, we would be the machinery, we will sub in humans for the technical parts of the product, and test it without writing any code.

Our plan was to slowly replace the people in the process with code over time. As we validate the ideas and build the product in parallel. We found five customers that represented our target audience. And we sent them a pitch saying, hey, we got this great new submittals tool, it's going to effortlessly handle your submittals process, track input and give you a report every week. And of course, we were the tool. Each of us took one of the user projects and did all of the data entry and all of the processing. User send us their existing spreadsheets and forward us all the email responses and files that came in.

This was V1 of our product, a pretty simple Google Sheet, mirroring the system that we wanted to build and populate it manually. We moved all their data into our template, we kept track of everything that came through email. And we looked at the files to figure out status. Now there's some basic automation here you can see colors and highlights. But fundamentally, it's a spreadsheet.

From there, we continue to automate things in the spreadsheet as we learn what the users need. A lot of conditional formatting, automatic dates, data validation, and the columns, really anything to reduce our workload here. But by and large, still, a spreadsheet, maintained by our team. The critical thing here is that for our use of this function like a product and a fairly magical one at that, they sent us data, it appeared, they didn't have to do any of the things, we knew what pain points in their existing process.

Then we set up a dashboard driven up by the spreadsheets. These were to provide insights and the next steps are critical hypotheses of our product, that are based in real-time driven by a bunch of queries. Really, excel scripts were the first code written for our product. And this was our first major breakthrough that showed us that we were really onto something valuable. As a lot of our users were genuinely shocked by what they were seeing. So much so that they thought that we were wrong. There was no way that there was stuff that was six months overdue, that was so critical. But when they investigated that they realised it was true, and they had potentially millions of cost overruns at risk, because these things had been missed.

Another thing we did was we had a Photoshop template that we will manually populate every week to send them a report that they could put their logo on, and share with architects, owners, and other stakeholders to keep everyone updated. And this was another really valuable element that we learned about.

These three things, the spreadsheet, the dashboard, and the report, were all the pillars of the solution that we wanted to build. Throughout this process, we met with the users every week, and we went to see how things were going. And we got the feedback. And because all of this was manual, we were able to take that feedback and immediately iterate on the format and the data, creating that tight loop we were looking for. Now, obviously, people are going to love you doing their job for them. But we need to turn this into a software product, not a concierge product. So we had a few constraints. We made sure we only did things in a standard way for all our customers. And we only did things that we knew that we could automate in code.

So how did this work out? The outcome of this was over 600 middles tracked across five projects for four months. And this included this half a billion-dollar highway interchange in California, which they estimated using our process would save them several million dollars in cost savings, this is way more than we thought we were doing. We learn so much from it. We were able to validate a product solution and quickly iterate it while real customers were using it.

This meant that before a single line of code was written we knew exactly what needs to be built and how it should work. We also built deep customer knowledge and empathy on the team, which meant everyone was focused on the details of the property. Now, that led us to something that we didn't even expect, during our customer discovery process, one of our engineers spotted an opportunity that set us on a completely new journey as well. So let me tell you my second story about that. Something that you need to know about construction is that people there like Ahmed, if you ask them, what do they hate about their job? Now think about it and say, nothing really, I love my job. They are wildly accommodating people. Ahmed is telling us this, having just walked us through one of the most terrible admin processes you've ever seen, that he has to endure every day.

So after a bit of experimentation, we finally landed on a question to help him and others reveal what the true pain points were. We'd ask him if you could clone yourself and convince that clone to do the part of the job you'd like the least? Or would you have that clone do? This opened up the floodgates and helped us see what we should really focus on to improve their lives.

And next time you're trying to figure out how to improve someone's productivity, and you feel they're being way too accommodating. I really recommend asking this question. Almost unanimously, they mentioned something we hadn't even thought about before. At the start of every new construction project, someone gets an incredibly crappy job of having to read a 3000 to 6000-page specification document and type out every single submittal required, creating what's called a submittal. This is incredibly tedious, but vitally important work.

On average, if you have a 10-hour day, it takes someone two weeks to complete. As a task so hated, it's often given to the person who annoyed the boss most on the previous project. Once we heard this, a couple of times, one of our engineers said, hey, this looks like a solvable machine learning problem, I don't think it'd be that hard to do. We had a challenge because we had no expertise on the team. And a crazy, tight deadline to meet as it was, this was out of our initial assumption of scope. It felt like a huge risk, but also a massive opportunity.

So we wondered, can we test whether customers will find this valuable? And luckily, we'd already worked out a process to answer that question. I sat down with engineers and wrote out a human-readable set of instructions for how to extract submittals from a specification, you set the constraint that these instructions had to be simple, and be able to be followed by any competent human with no prior knowledge of construction. They must also be instructions that can be easily replicated by a relatively simple machine learning algorithm.

We then reached out to a few customers and said, hey, we have this fantastic new tool that extracts all of your submittals for you. Just send us your specs, and we'll send you back a spreadsheet. Oh, it's still an early-stage product. So it might take a couple of days to get back to you. That we just hired some data entry contractors to do the work. This was really cheap. A couple of people at $20 an hour, that was simply picking up specs sent to an email address, and responding back with a spreadsheet of the results.

The response was incredible. We literally heard shrieks of joy from people, when they saw the results and realized they just got two weeks of their life back. And this was really a double whammy because not only did we validate the idea, but we also learned incredibly valuable insights on things like precision versus recall, it turns out that people were okay with us listing more than they needed. But they were not okay with us missing any important items.

We also got labeled training data as part of this process, all the stuff that was essential for us to be able to build and train the machine learning algorithms to all of these insights. And that data allowed us to build a product faster, and be way more confident that we had something the market really wanted. So what were the results? With only four engineers, we were able to learn, experiment, and launch a production-ready solution in six months. This was a paid add-on to our core solution that from day one accounted for 9% of the company's revenue, evidence that we've built something that our customers really need.

One of my favorite things about this was that our product was featured in a list of the 13 most interesting advances in construction technology that year, alongside this robot that builds walls. What I love about this is that this robot is an incredibly complicated piece of technology, solving a relatively simple problem for humans. Whereas we built some relatively simple technology to solve a very complex problem for humans. And initially, we actually use humans imitating machines to power it.

So that made us feel very smart. Now, this wasn't all sunshine and rainbows. So I wanted to share a few of the things that we messed up so that hopefully you can learn from them.

The first was we didn't have an exit strategy from our concierge MVPs. The idea was that we'd start tracking some middles manually, and then quickly replace ourselves with code as the engineers got up and running, maybe one month, two months max. Then as things do, they get delayed. But we have left hand cranking this process for almost four months. Now, we didn't want to leave our customers in the lurch as these were mission-critical processes that we've promised our customers with support.

So we were basically doing two jobs at the same time. It was particularly stressful for those supporting that half a billion-dollar highway interchange project. So if you're going to do this, think about the exit strategy. If things get delayed, or you decide not to pursue this product, how are you going to do right by yourselves and your customers? Another thing was we did a great job of involving the engineers in the early research. But once the concierge MVP started, we decided they need to be focused on writing code, and we'll share the knowledge with them later. At that point, we could already see the crack signed form in terms of that customer empathy. And that led to mistakes that cost us many weeks that could have been caught a lot sooner if the engineering team had had first-hand experience of managing a customer.

If we could do this, again, we would have had the engineers spend at least a couple of weeks actually being the human in the loop for those concierge MVPs as well as that would have saved us a huge amount of time in the long run.

Lastly, we only brought in marketing and sales at the very end of the process, which meant that it took much longer than it should have to understand the value propositions and therefore how to position and sell this in the market. Having at least the Product Marketing Manager involved in the discovery and experimentation phases with the rest of the team would have saved us a huge amount of coordination effort, and rewrites when it was time to develop and launch the go-to-market strategy.

So to recap, if you want to do experimentation in zero failure environments, build a cross-functional team that truly cares about the customer. Learn about the problem together. treat everything as an experiment. Do the least amount of work needed to test your hypothesis and use humans to power the first functional MVP? The last point I'll leave you with is that experiments come in many shapes and sizes. You can do this at any stage of development, be it a new idea, or something that's been in the market for a while. I encourage you to think about what it is that you're trying to learn next, and what's the least amount of work you need to do to learn it.

Thank you so much for your time. Please feel free to reach out to me with any questions, comments, or if you just like tips on how to implement this from challenges of your own, also I offer free coaching to people who are just starting out or looking to transition into product careers. that's of interest to you. Feel free to reach out to me anytime via email or on Twitter. We're all if you want to join me here at Asana. You'd be very welcome. I'd love to chat with you about that as well. Thank you so much and hopefully see you soon.