How To Drive Data Driven Change In A Legacy Organization

Hi, everyone. Today, I'm going to talk about how to drive data driven decision making in a legacy organization. In today's world, you can imagine data is available everywhere. It's very cheap and it's very widespread but that makes it even more difficult to make sure that we are using data the right way. So, today, I'm going to talk about how we have used data so far in our organization and how we make our product decisions using data and also the learnings from it and the failures and successes of it. I'm Subhasree Chatterjee, I'm a Lead Data Analyst with LexisNexis. I lead a team of product analysts who work with the North America business unit. I haven't only eight years of experience in this industry and I also have a master's degree in Business Analytics from University of Cincinnati. I'm very passionate about data. I like to talk about data. I like to work with data but other than that, I am a long-distance runner. So, I have run a few marathons and generally I keep training for new long distance running. So, before I dive into today's topic, let me talk a bit about who LexisNexis is and what we do. So, LexisNexis is a pioneer in digitalization of legal data and also government data. And we have the largest content as of now for all this legal data. So, we try to support and innovate all the customers research and make sure that they are comprehensive and efficient. We also combine analytics and information to help customers make more informed decisions and achieve better outcomes. And we also try to advance the rule of law around the world. How do we do that? We help lawyers with comprehensive and efficient research. We collaborate with universities who are the future of legal research and we also work with government and codes by making laws accessible and strengthening the legal infrastructure. And we also participate in leading global
associations to advance the rule of law. LexisNexis has a lot of products in the portfolio to help with the different stages of legal procedure. But today, I'm going to talk about Lexis advanced research, which is the research tool and which is also the flagship product. This shows you the very high level and board eye view of how the product workflow looks like. So, as you can imagine, being the research tool, the main idea is to be able to search. So, you can search for something as broad as a legal topic, like maybe first degree murder or you can search for a very specific citation number to help you with research or to get all the data that you might need to prepare your case. So, once you have search for something, the idea is that you are going to get some documents back that relates to your search results. So, if you have search for a citation then likely you are going to get back one document, but if you are trying to do research then that means that you are going to get a lot of document back and you are probably going to read through it, try to find out which document works best for you. And once you have found a document, you will likely to retrieve that document. Well, what do I mean when I say retrieve? It might be something for your future reference, so you can download it in your systems or you can print it to have a hard copy with you, or you might even email it to somebody who is working with you or who you might be reporting. So, as you can imagine, being the research tool the success criteria. So, when do we know that the users were able to find something is when they are retrieving document. So, we considered this as the success of the session when we see our users retrieving lots of documents. Now, initially, when there was talk about product enhancement or new designs or new product creation, most of those were done by gut feel or received wisdom from our lawyers or from our users as assumptions around how they use our product. But we quickly realized that it's very important to actually see what they are doing in the product or what our users are actually talking about when they talk about our product. So, this is the process that we came up with to make sure that data is being used in each part of the product development. No, this is not something we have invented. As you can imagine it, you probably use it on your daily product development life cycle. So, the phases are identification and prioritization phase, hypothesis formation phase, and then prototype and delivery phase. Now, identification and privatization phase id very important for a large company like Lexis Nexis. There generally are lots of problems, or if I should say enhancement that needs to be made in our product or new product development. But you need to make sure that you prioritize the problems because we have limited resources and limited budget. So, how do you do that? You can use data to define that. First, ask the right questions. So, make sure that whatever you are trying to develop is in alignment with your overall value of the product. And also, what the customers are asking. And it cannot be that one customer is asking you something and you go ahead and build that into the product. You need to make sure that it's something that overall, all or most of our users are interested in and you cleanse different sources of data. So, it's not always possible to have access to usage data as such but you can get data from customer support calls. You can get data from survey data to make sure that whatever you are trying to build is in line with what our customers want. Now, when you have defined the customer problem, the next phase is to define the hypothesis. So, once you were trying to build something, it should be able to solve a user problem. So, then comes the hypothesis, make sure that your hypothesis is quantifiable because if you are trying to develop something, you should be able to track it in the future and make sure that it has solved the problem that you are trying to solve. So, that makes it even more important to define the right set of metrics and also make sure that you have all the proper tracking mechanism present in the product to be able to track all those hypothesis and matrix. Now, after that phase comes the prototype and delivery phase. In prototype and delivery phase, how you can improve data, you can AB test your different designs. You can create a dashboard with all the success matrix you have already defined to make sure that it is all the metrics are trending in the direction that you want it to trend. And you can also get some user feedback around your new design, how it is performing or how it can be improved even more. And you can get the satisfaction score for those new designs to make sure that it has solved the customer problem that we are trying to solve. Now, again, this has very simplistic view of how the design process looks like, it can be cyclic. So, you might have to go back from the hypothesis formation phase to the identification and prioritization phase again, or maybe from the prototype delivery phase. You might need to iterate among it to make sure that you are solving the right problem, or you are getting the matrix in the right range. You want it to be. Another thing, which is very important is to think about data as a comprehension of qualitative and quantitative data. So, in today's world, it's very usual to say data whenever you are thinking about quantitative data, but if you are not using qualitative data as well, you are losing on a comprehensive picture It's very normal to have your product analytics or data analytics team and the UX team working in silos without the communication. So, what we try to do in this process is make sure that the communication is present. So, as you can see, we have probably lots of hypothesis from the customer problem. Then the qualitative team and the quantitative team start to work on it parallelly. And there is lots of communication going on to make sure that you are getting the comprehensive picture of what the customer problem looks like and what kind of solution we can create from it. So, once you are comparing and contrasting your results that you are getting from different sources, you can again go back to define the hypothesis in a different way. You might have to redefine the hypothesis to make sure again, that you are able to solve the customer problem. And once you are done with the final set of hypotheses, you can then go on about validating those and this process should be present in each of the steps that I've mentioned in the previous slide. No, let me try to explain this whole process using a key study. So, as I've already mentioned, that document retrieval is considered as the success for our product. So, as it's a top predictor of the success of sessions that users are in. We always assume that higher is better because if the users are able to find something in the product, then they should be retrieving them. So, it makes sense that if they are retrieving more and more documents, then it means they are able to find what they're looking for in our product. So, why, when we started getting some feedback from our customers around. the delivery of the document retrieval. We started to look around it because it was one of the doc predictors. So, what we got data from our customer and support calls from NPS and also from UX feedback that users are having trouble with document retrieval. And again, document retrieval is the top predictor of success. And again, document retrieval is the top predictor of success. It was very easy to prioritize that as our next project. So, the customer problem we defined and it is very important step to be able to define the customer problem in such a way that you can leader track it, that you will be able to solve it. The customer problem we defined as "users are experiencing high document retrieval failure and cancellation rates". So, once we have defined this customer problem, the next step was to be able to define some hypothesis and see if we can disprove them. So, the hypothesis we came up with are three. First is users have difficulty distinguishing between documentary to icons. In the toolbar, so causing them to accidentally choose the wrong action. So, we have all of these different type of achievement that you can do from document page. Those are print, email, download, safe to work folder downloading cloud and things like that. So, it looked like that the icons we have wasn't distinctive enough. So, users are clicking on probably they want to email, but they are clicking on print. So, then when they realize that they are taking on the new wrong button, they are then canceling out of it. The second option was "users are drawn to the strong affordance of the quick action icon" and I like to use the example of Amazon here. So, if you are trying to buy something, then you have two options. One, you can add to the cart, or you can directly use the quick bell icon, similar goals for our documentary retrieval option as well. So, one option you have is directly keep on the retrieval form you want, and you just click on that and then you get the document back. The other form is to be able to modify some of the settings. Like you can change the name of the document. You can change the format; you can change how many pages you want up out of the document. So, that's done with settings options. So, as you can imagine, people being so busy and trying to get the document as fast as they can, they were using the quick icon mode. The last one was "pop-up blockers are causing the retrieval to fail". Now, if you try to retrieve any documents from any kind of product, you will realize that there is another step that's required from the browser site. So, if you have the popup blog on for your browsers, then that option might not show up. So, even if you are clicking on retrieval icons in our product, you are not getting the document finally, in your system or in the printer. So, that's a problem that we get to solve. I will go through each of the hypothesis and tell you how we went about disproving those hypotheses. So, the first one was interest have difficulty distinguishing between the retrieval icons like toolbar causing the to accidentally choose the wrong icon. What they saw in the data that 36% of our users are retrieving via more than one method on a single document. So, on a single document in the same session, they are probably using print and email or email and download to make sure that they are getting the document they want. When we asked our users, what did they say? That "one of my biggest problem is printing". I typically have to print a document multi times before I can get a full download that will print and even when it works, it's slow. The next hypothesis was "users are drawn to the strong affordance of the quick action icon". What we saw in the data is that 80% of all users retrieved via the quick action icon. And why do they do that? I actually did not know. You could do all the items in the survey. I usually just press email and okay. It is interesting to know that I could customize how the document is emailed. So, some of our users are not even aware of all the settings options they have available to them. So, that's a problem that we need to solve. The third one "pop-up blockers are causing the document retrieval to fail". What we saw in the data is cancellation on failure rate was around 10%. And why is that the case? How do I print a case? when I print, it does not go to my printer, nor does a printer dialogue box pop up so that I can choose a printer. And I can't find anywhere to set printer settings. Now 10% is quite a high document failure rate. When you are ready to retrieve the document, when you are at the final stage, you want the document but you are not able to get. So, this is also a problem that we need to solve. So, we had the iterations around UX design and the developer and they came up with three stages prototype. First, was the ambiguous buttons were cleared up. Second the document retrieval with settings was made the default option. And third was, a processing buddy was introduced so that they know that something is happening in the background. And then don't cancel out of the delivery option. So, once these different changes were made, the next step was to AB test the solution. So, when we put the different changes to the AB test, we saw that for the treatment group, the cancellation rate actually went down which is a good sign. So, then we went ahead with the full productionization of all the changes that we made. But when we launched those changes, there was panic. Why? Because the document retrieval rate went down and all with that that's total success core of the product as well. And everybody panicked because document retrieval was one of the top predators. So, everybody was assuming that when all of these changes will go live, everybody will probably go up or like the document retrieval rate will probably go up or it will remain the same. But when we looked at it closely, we realize that the problem we had that people were trying to deliver or trying to retrieve the document using my different methods for a single document that went down. So, as you can see, that these changes were made in the product and that this line is going down. Which is a good sign because we don't want them to have to retrieve through multiple methods to be able to get the document they want. And the next thing, what we realized was the customer calls that we were getting around document retrieval complaining about that they are not able to get the document they want that went down as well without release. So, both of these are good things. But because we were looking at the one metric, which is the number of document retrieval everybody started to panic. So, what does this tell us that it's very important to be able to define the right matrix. What we learned from all of this process. First, importance of defining the right metrics. So, you cannot just define few vanity metrics and just go about creating a new product or a new feature and try to understand if it is solving the right problem or not. It is very important to go through the process of defining your hypothesis and defining your success metrics, very particular to your customer problem. Second more is always not better. So, the assumption we had that document retrieval, if it goes up then it's great. It's probably not always the case because we don't want them to be able to go through multiple delivery methods before they can actually retrieve the document. Third is the importance of proper tracking of user actions. So, as I said in the process definition as well, it is very important that you are tracking all of these different things that the users can do in our product. So, that later you know that all of the problems you are trying to solve in the product you have actually solved. The last is importance of working together with other teams to rule out the opposing ways behavioral data can be interpreted. So, if you are looking at one single type of data or one single way of solving the problem, then you might have some blind spots that you are not able to understand the comprehensive picture but working with different teams with different point of view helps you remove those blind spots and able to get the comprehensive idea of what our customer's problems are and how we can solve it. Now, while this is great. You might ask me that this is good for you, but how do I use this for our process or for our product? So, this is a slide that you can probably take a screenshot of or just create a checklist with. And then you can use it for your product development process or feature development process. First, it's very important to get involved at the very beginning. So, don't wait for the product or the feature to go into production that users start using it, and then track the data. Start from the very beginning of when you are even thinking about creating a new feature or a product. Data both qualitative and quantitative is very important at all stages of product development. And I would argue it's the most important at the very beginning of the development cycle. Second, ask the right questions. So, is there really a problem to solve if you are hearing from one customer is that enough evidence that you need to create a new feature or you need to change something? Try to first understand how big is the problem and be able to pull data and help you understand that. Third, What changes are we trying to bring about and why. Of course, it's very important to listen to our users and what they want from our product but it's also very important to understand if it aligns with our overall value of our product or where we want to see our product in the next few years. And so it's absolutely important to make sure that we are creating products or features, which we want the user behavior to look like. So, how do we drive the necessary change in user behavior? Third. Eyes on the prize. So, how do we measure success or failure? You can create lots of different features. You can enhance your feature but how do you know if you have really solved the problem that you were trying to? So, it's very important to define the right set of matrix. Again, one composite matrix or one vanity matrix is not going to help you. You need to make sure that all of your matrix is very specific to your customer problem. Next, make sure to have proper tracking mechanism to track those different matrixes. Fourth point is there is no such thing as no data even when there is time pressure. So, we often see that we only try to use data when it's convenient, when we have lots of data available but you cannot do that. You need to make sure that you are trying to get the evidence from all different points of view, right. So, even if you don't have usage data for that particular feature of the product, you can talk to the customers, you can see what the survey data says. You can see what their feedbacks are from NPS. So, you will be able to get data if you are looking for it. And the most important point is collaboration is everything having product UX and data science, constantly talking to each other is very important to avoid blinkers and to work efficiently and going forward without any blind spots. And the last few things I want you to remember are these key questions. So, even if you're not a data nerd, even if you are not a data analyst trying to gather data or a UX researcher trying to research on the customer problem, you can always ask these questions to make sure that data is being included in your product development. So, what user problems are we trying to solve? What kind of evidence do we have on the problem and what's unusual as a reaction on the solution and how will we know that we have solved it? So, all of these questions are going to help you at different stages of the product development to make sure that you are using data in all these different stages. So, that's all from me. Thank you so much for listening in. This is my LinkedIn. If you want to contact me later or if you have any more questions. Thank you.