3 Takeaways From Duolingo's Product Culture

Okay, hi everyone, my name is Zan Gilani, it's close enough and I'm a product manager at Duolingo and actually, the topic for this talk is slightly different from what was presented. I'm going to be talking about three takeaways from Duolingo's product culture that hopefully you can apply to the projects or the products that you're working on. But before I do that, just a quick summary of what Duolingo is and what we're about.
How many people here, by the way, have heard of Duolingo before? Okay, and how many people have actually tried it? Okay, cool, very cool. So, this shouldn't take too long then. Duolingo is an app that teaches languages. We teach languages for free; we have 30 plus languages on the app right now. One of them is actually Irish and Irish is pretty special to Duolingo, especially in our hearts because there are around a hundred thousand native speakers of Irish, but on Duolingo, there are actually around 1 million active learners of Irish, and in fact last year, the Irish president gave an award to Duolingo for our contributions towards revitalizing the Irish language, there he is in the centre. Can I point? Yeah, there he is.
And actually in fact, since we did that, there's been a whole movement to get Scottish Gaelic onto our app as well, including this tweet from a Scottish member of parliament but we haven't gotten around to them unfortunately if anyone in the crowd is Scottish and speak Scottish Gaelic. We are very proud though, to help revitalize languages such as Irish, in fact, later this year, we're going to launch Hawaiian and Navajo courses as well, but that's actually a pretty small piece of our larger mission, which is to provide free language education to the entire world. Now, there are there over a billion people learning languages around the world and most of them satisfied the three, the following three criteria. First, they're learning English, the second is that they're not affluent, and the third is that they're actually learning English to get ahead in life, to make a better life for themselves.
So, our goal hopefully is to provide a free and high-quality way to help those people do so. And we've got a headstart, you know we have around 300 million users, we're actually the most downloaded educational app of all time so far. So, how did we get so far? Well, the first point is that we're accessible, anyone with an internet connection can use Duolingo for free. Now, in order to pursue this mission, we have to be a sustainable business. So rather than charge for content, inside the app, we have ads, we have a premium subscription called Duolingo plus, and we have in-app purchases as well. I'm going to speak a bit about these later on, but basically every lesson in every course you can learn on Duolingo for free.
The second is that we're effective. An independent study by the City University of New York actually found that 34 hours of Duolingo is roughly equivalent to one university semester or term of language instruction. And that's because we try to help people stay motivated, that's really the gist of it. Self-directed anything is pretty hard, you can think about the difficulty of forcing yourself to go to the gym every day, but self-directed education is even more so. And we tried to help by making language, learning a bit fun, a bit convenient, and hopefully, that keeps you motivated. The other thing is that we run hundreds of A/B tests, we're actually running 173 of them, I checked last night at this very moment and that helps us make Duolingo more fun and more smart just bit by bit incrementally.
But, where did the ideas of our test come from? And that's the crux of this talk, as, as I think has been a theme in this entire conference, every team is strapped for resources, whether it's time, it's money, its people. So working on the right ideas, working on the right experiments, that really is the name of the game. So, in this talk, I'm going to share three takeaways from our product culture that has helped us consistently come up with good ideas and I'm going to sprinkle in some parts of our experiment process and illustrate with, with our experiments as well, just to give you a bigger sense of how Duolingo operates.
So, the three points are, first, we borrow, particularly from outside our industry. The second is that we remix we're adapting these ideas that are borrowed and we're adapting them, particularly to our users and their specific context. And the third is that we revisit, we take old ideas that we'd previously dismissed and we revisit them when something changes, whether it's our users, our product, or our hypothesis. So, let's start with the first thing: borrowing. Now, whenever you're in your brainstorm or your vision phase or your discovery phase, it's very, very tempting, I'm sure as everyone knows to look to your direct competitors for inspiration because likely, probably they're very smart probably and the second thing is they're working on exactly the same problems as you are, and sometimes this works out fantastically well, this is Instagram stories.
But, generally, at Duolingo this is discouraged because one you're going to be, you can lose sight of your own product vision if you're, if you're hell-bent on copying and, and really chasing what others are doing, and then the second is that if you are copying, you are going to end up having a product that looks no different from, from everything else out there. So instead, what we do is we try to look outside of our own industry, which is education and we particularly learn from the gaming world, we learn from games. Why? Because Duolingo is in the business of motivation and some of the most effective techniques in motivation come from games like Super Mario Brothers or Candy Crush. They do something very special, which is when you fail a level, you immediately go back and try again and again and again until you actually succeed.
And that's something really important to cultivate in students, but it's pretty rare to actually find that in the educational context. So Duolingo broadly speaking feels, like a game as, as I'm sure you guys have seen for yourself, each of these circles are skills, which are a group of lessons and you have to complete one skill in order to progress. Each lesson is interactive, there's no passive learning, you have to actually answer things. And then we also have little touches, like a progress bar at the top to show you how much you have left in a lesson. But I want to talk about two particular features we've directly borrowed from games to illustrate this point about borrowing.
The first is streaks, now streaks are pretty common I would say, especially I've seen them in games, such as first-person shooters, where you might have a kill streak or say you have a login bonus. The way that we do streaks at Duolingo is that you pick a daily goal first to say it's one lesson or two lessons per day. And for every consecutive day that you come back and meet it, your streak is extended. And this has ended up being one of our most powerful retention mechanics, because one, it gives a meaningful, relevant reason to send people notifications and emails, telling them to come back, that doesn't feel too spammy and the second is that people get very, very, very attached to their streaks.
The longest streak ever, it's 2088 days, I assume that because this guy has probably done his streak, is John Arnold, who is a chemist and horse farmer and to put this in perspective, John Arnold, the chemist and horse farmer has done Duolingo every single day for the last five and a half years, which is inconceivable. The second feature that we directly borrowed from games was this concept of levels. A game that we love and are deeply inspired by Duolingo is Clash Royale, this mobile free to play game by Supercell, many of you might have played it. And in Clash Royale a very core mechanic is taking these different cards of characters and levelling them up and levelling them that makes them more powerful, which you can then use in combat.
And we like this for a couple of reasons. One, kind of in terms of the mechanics, individual to constantly had whenever you came back to the app, you had these progress bars on each of these different cards. So, lots of different progress bars, and every day you're trying to fill them up more in order to level them up. And it took this bigger concept of winning the game and broke it into these small discreet, manageable chunks, which we liked. The second is that you could actually kind of choose your own adventure. You could decide which cards to level up in a way that actually made sense to your particular strategy.
So, we, we kind of use this and we applied it to Duolingo. So instead of just having a skill, which is a group of lessons now, when you complete that skill, you actually level it up to the next level, which is these, these little numbers in the, in the crowns, over here and every time you level it up, you actually get access to harder content. And the reason we did that is because of this conundrum that we've had in sort of all of our time working on language education, but is bigger than us, which is whenever we try to add harder content, it was pretty discouraging to our more casual learners, especially because you know, they are learning on a phone, basically, they might be learning on the bus, might be learning at home.
But in order to actually teach a language in order to get people more fluent, you have to give them harder content, they have to struggle with it. So, this was a conundrum for us and this allowed us to basically work on this problem in a way that ended up being successful because now users could go through the app if they were casual users, but if they wanted to learn something harder, they could start levelling up. And this ended up being really positive on basically all our core metrics, whether it's daily active users, day 14 retention or lessons completed. Now just for the sake of transparency, of course, borrowing always does not work. So, I want to talk about an experiment that actually failed horribly.
And this is what we call moves, another game or a set of games that we're inspired by they're called match-three games. I'm sure many of you have played Gardenscapes or Candy Crush, these are mobile games and the way they work is very simple. You have all of these different objects on your screen and whenever you match three or more of them, you actually collect those. And the goal is to collect a certain number within a set number of moves. And these games are immensely, immensely, immensely successful. So, we try to borrow this and bring this to Duolingo with the idea of moves and moves made it such that at the beginning of every lesson, you had a fixed number of moves this number and in the corner, and every time you answer the question, it would use up one move and the goal was to actually complete each lesson within that fixed number of moves.
And the goal with this was that one, it's a bit more of an interesting strategic lesson experience, but also, we could use what I call difficulty curves, which are very common in the gaming industry, which is, we could vary the difficulty lesson to lesson in a way that was more interesting than just slowly ramping up difficulty over time. But as I mentioned before, this really did not work, this was either neutral or actually pretty negative on the metrics we cared about. So that's the first point borrow, particularly borrow from outside your own industry. And I would put in a quick plug to also just borrow from games because they do a lot of things that the tech industry just ignores basically.
Now when you have an idea that you borrowed and it's actually successful, what would you do next? And that brings us to the second point, which is to remix. When you have an idea that you've borrowed it's successful, you then want to remix it, to adapt it to your particular users and their specific needs and contexts. Now, we found so this, so I have a couple of illustrations of this that also have to do with streaks, which I mentioned before. Streaks can be incredibly motivating when you are actually keeping them, but when you lose your streaks, that is disastrous and people get very upset. You can imagine when John Arnold, the chemist and horse farm owner, what would happen when he loses his actual streak?
Particularly we went and looked a little deeper and we found that it's on Mondays, that people, or rather it's over the weekend, that people are actually losing their streak disproportionately, which, which makes sense to us. So, we came up with an idea to remix streaks with the weekend amulet. And the weekend amulet is something that you only see on Fridays and when you activate it, it allows you to keep your streak over the weekend, even if you don't practice. And that ended up being successful really people kept longer streaks, they ended up retaining for a longer period of time. The second way that we also remixed an idea borrowed from games was with the streak repair and this is an interesting one because, for the longest time when people lost their streaks, many would write to us saying, we will pay you, can you please bring them back, this is very useful to us?
And then at the same time, we had introduced something new, which was this premium subscription that had two features, one, it took away ads, and the second is that it allowed you to learn offline. So again, we weren't blocking content, but these were these extra features you could get. So, we decided to add this third feature, which is you can get a free streak repair every month, one time. So, if you lose a streak this month, it actually gets repaired and this is a perk of subscribing to Duolingo plus, and we like this idea because on the one hand users were literally writing to us saying, we want to pay you for this before it was even on our radar. And the second is that it didn't kill the integrity of the streak. The streak is meant to be a motivational tool and it would not have any value if you could revive it at any time without any price, monetary or otherwise.
So, we iterated on this a bunch of times to the point where now the streak repair, as it stands, actually contributes to 9% of our daily revenue. The crying duo also really helps. So you borrow ideas from outside your industry, not your competitors, and then you remix them to suit your user's needs and contexts. You know, streaks existed everywhere, but no one had really done anything like a weekend amulet or a streak repair before, and that made sense for our users.
These are externally facing ideas, but sometimes the best ideas are really good ideas actually come from within the company, within the team. And that brings us to the third and final point, which is to revisit ideas that have already been attempted because something changes. Now, this is something you're probably very familiar with this scenario where someone maybe it's, you comes up with an idea in a team meeting and the most senior team member goes ahead and dismisses it saying, oh, you know, we tried that before didn't work, there's no point in doing it again. So first off you should recognize HIPPOs, these are the highest paid person's opinion. Even if they look like elephants, you should recognize that they are HIPPOs because they are pretty insidious and they're going to lead to all of these ideas that you would have considered, you're going to dismiss them for good, because the highest paid person in the room decided that it wasn't a good idea.
But there are plenty of good ideas why you should revisit. There are plenty of good reasons why you should revisit ideas, right? The first thing is that your users could be completely different now than when you first attempted something. Andrew Chen, who is this guru of all things growth talks about the law of shitty click-throughs, which is basically saying, tactics decay over time, usually because user's behaviour actually changes. So, banner ads today have a click-through rate of about 0.05%, but the first banner ads have click-through rates of 78%. So, people really change in how they use your product.
The second is that your product itself could be different. You can imagine when you're first making an app, it's very simple. An onboarding experience would actually hurt because it adds one extra step before users get to the good stuff. But over time, your app could become much more complex and suddenly you need an onboarding experience because you need to drive users directly to the thing that's most valuable to them just as they're starting out. And third, your hypothesis could be different. You could just have more experience or a different take on why something didn't work and also a new take on what is the right way to iterate on it. So I want to illustrate this with achievement badges, which are a part of Duolingo and I think people are probably familiar with achievement badges for the longest time we hadn't added them to Duolingo because we try them once before they didn't work. So there's no point in trying again.
But when we actually looked and we dug deeper into that first experiment, things didn't make sense to us a year later when we were more experienced and that's because this experiment to validate achievement badges, and this was basically the minimum viable test we did only showed you a screen at the end of your registration process saying, congratulations for registering. And of course, now we know that that's not how achievements work, that's not what makes them effective because in order to have effective achievements, one, you need to have a bunch of them and they have to be for meaningful actions. Registration is super meaningful to us, but not meaningful at all to a user who's trying to learn a new language.
And the second is that you want to actually show them all together in a nice way so that you're preening over the ones that you've already earned and then you're coveting the ones that you haven't earned yet. That's what makes effective achievement badges. So, when we ran this experiment, this is what achievements look like in, in Duolingo today, it ended up having really positive impact on our core metrics. And finally, I just want to end with an experiment and a feature that we've where we're actually working on right now in my team, which is the acquisition team. And this is something that we've tried a bunch of times, basically over the last couple of years.
And we, Duolingo actually doesn't pay to acquire users. All of our growth comes from word of mouth, basically, and some PR, but we're always trying to, to optimize and kind of accelerate this organic growth channel that we have. So, we've, we've long wanted to master referral. And this is something that you've probably seen in many apps, stuff like Deliveroo, Dropbox, Uber, you refer a friend, they sign up and you both get something, whether it's free space, it's free food, it's rides, whatever it is, so we wanted to try this as well. So this first experiment from a couple of years ago, was kind of like the first achievements experiment, actually, where at the end of the registration process, we just say, Hey, invite your Facebook friends to Duolingo, and then there's a button to invite them.
And this got us a few hundred installs per day, but Duolingo gets several orders, more magnitude of app installs per day, just through organic means. So this wasn't going to kind of meet our mark, and we realized, and the reason for this seemed to be obvious to us because we didn't actually have something to offer unlike Uber which is giving you $20 for your free ride, we didn't have anything to offer at the time within the app. We were just literally asking people to refer their friends. So fast forward a year, and now we have a new referral experiment and this time we have something to offer users because our product has changed, this time its gems, which is this in-app currency that we have that now you can actually spend real dollars to purchase.
So we thought, okay, now we're going to offer this and this has monetary value and it's a good incentive and people will invite their friends and they're going to earn gems. But yet again, this didn't actually work out. It's a little better than the first experiment, but again, nowhere near where we wanted it to be. And this led us to kind of a new hypothesis, which was well, we knew two things. The first is that people are most likely to refer their friends who right at the beginning of their journey. So it's basically within the first day that most people want to refer others and because they're excited about the product. The second is that gems actually take a few days for you to understand as a user they're not immediately valuable or useful to you.
So fast forward one more year now, and we are having a retrospective and this is something that you might be doing at your own companies, but at Duolingo retrospectives are a time when a team will present on an idea that, on a previous project that either completely exceeded expectations or fell short of them. And they're pretty useful because they take the siloed information from that one team's head and it disseminates it to others so that they can either not make the same mistakes, or if they're gonna work on the same project, they have a big headstart. So, it's during this retrospective that we realized, okay, the product has changed yet again, we can do, we now have a record that is immediately meaningful to users and that's Duolingo plus.
So now this experiment that's actually running today on Android offers Duolingo plus for free for every person that you end up inviting that signs up and hopefully it works out. So, if you have any ideas about this, please like tweet at me or see me after the presentation and let us know what we should be doing, so that's, that's revisiting.
So just to summarize again, one more time, borrow, but borrow from outside your own industry, remix and make it relevant to your users and their particular needs and revisit, revisit ideas that have previously been dismissed. You know, don't take no for an answer, dig deeper, read the experiment report if it exists and see if your product has changed, whether it's the users, the product itself, or the hypothesis that you have.
So, I started with a story about Irish, so it feels right to end with a story about Irish as well. This is Shannon O'Neil, who is studying music education in LA a couple of years ago. And her health began to precipitously decline, her health and cognitive abilities. She had viral meningitis and during this time, because she was a fifth generation Irish American, she thought, okay, I'll learn Duolingo, and I'm going to choose Irish. And what started as just a way to work on her short-term memory and pass the time, ended up becoming a full-blown passion. So, she got better, she's healthy now and she ended up pursuing Irish studying it, moving to Ireland for a little bit and actually now is pursuing a career in teaching Irish. And on this whole journey, as she said, I'm really proud to call myself an Irish speaker and I hope to inspire others. Learning Irish has allowed me to reclaim or claim my heritage. And that, that's the power of free language education. Thank you.