This session will discuss methods for involving patients, such as focus groups and usability studies, and share best practices on how to use them at different stages of study design and set-up.
So today I was thinking that since we’ve had a lot of discussion around patient involvement, engagement in very broad contexts, especially in the workshops that we had this morning and yesterday. And I was thinking that I would take more like a narrower look into how we can involve patients in terms of usability. How can we use different usability and user research methods in eCOA solution design to make sure we come up with something that patients actually can use and enjoy using. And in addition to the methods, like what to use, why, when, I was thinking that we could take a look a the pitfalls to avoid. I think we’ve had interesting experiences during the years. And then a few words about how to start building the routine, since I think that’s the question for many. There are all these wonderful ideas of what people could do but how to do it in practice is another question. And then I will try to understand how to be strategic around those questions and instead of just transactional usability considerations.
But before we jump to that, if Paul Margerison was here, he would be listening to this with ears very red, as he’s the true expert and I’m the one who takes it into use in clinical trials with the whole team. But there are different aspects to usability that I think are really key in understanding what we are trying to achieve when we talk about a usable solution. And these are very generically quoted in the user experience field.
So it’s about learnability, how easy it is and how fast it is to learn to use a solution. And because of this, when we do usability testing we don’t give them training, we don’t give them user guides. We just present them with the solution and we say, oh well imagine yourself needing to report, let’s say a hypoglycemic event, what would you do. And they are like, hm, and it starts from there. So we get a bit of a true understanding of what the up-front usability of the solution is and how quickly they learn to use it.
And then another thing is efficiency. So once the users have learned to use the solution, how quickly can they use it? And this is one of the key considerations in clinical trials because we tend to think that a solution needs to cater to all the users, to all patients who are included in the clinical trial. And obviously, hat’s the correct way of thinking about it. But we easily tend to then compromise efficiency. So something that we as professionals in this space need to really pay attention to, is that we don’t create solutions that are very inefficient to use, because on the other hand that’ll reduce the kind of satisfaction that users have with the systems.
Memorability—so when they go back to the solution, when they’ve done something once, are they likely to be able to do it the next time around, especially if there’s a gap between those two scenarios. Errors—how likely are they to do errors, can they always choose the right option, or do they sometimes make mistakes, and what can we learn from them. And satisfaction—how satisfied are they with the solution in general.
Now, one of the key challenges is, we think about usability quite a lot, but it’s very short-term thinking. And it’s challenging, it’s creating challenges for us. The challenges are that there is limited time to involve patients, so when we come to, oh there’s a concern, what shall we do, we need to make sure this is usable. It’s usually that question comes up pretty late in the process, and then limits the options, limits the time that we can spend involving patients and learn from that involvement. And it also means that if we are thinking about being able to have a data-driven approach—and I’ll talk about that a little later—if you have too little time to plan for something, you aren’t able to establish proper mechanisms for collecting data about what you want to learn.
So a typical scenario that we would encounter is that there is a study team concerned about an upcoming Phase III trial, about their patient population. Can they really use this solution? Are we putting too many things into the diary? Or are we putting in enough training and that sort of thing? Are they able to use it? And now we are talking about the setup time, that’s usually three months at maximum, including some localizations already. So for a design of an eCOA solution, a three-month setup time usually means something like four weeks. If you are trying to involve patients in a four-week setup, that’s going to be really challenging. You can do some things, but in order to really understand patients in depth, you need more time than that. And that’s the challenge.
So now, if we think about moving to a more strategic approach when it comes to user experience and understanding and planning it, one approach we can take is that of looking at the clinical trial as a program. So starting to build in usability from the first study. So we set up the research questions, concerns, try to understand what we want to know, what we need to plan for already when clinical trials start. And then when we get to the Phase III trial, we’ve actually been able to even get feedback from live trials about usability and are still able to refine the approach. So that’s much more efficient. In the end, we’ll be much more successful.
But what can you do with all the information, all the learnings that you’ve gathered about patients, not only perhaps the use of the eCOA solution but about patients in general? If you’ve planned for the clinical trial phase well, you will be able to utilize the learnings in any post-marketing activities that you do. And thinking about it even more strategically, about the whole drug life cycle, and even perhaps across a specific organization working on a specific indication, you can have an even more strategic approach where you tie user experience activities into the whole drug life cycle, wherein the beginning of the life cycle, you actually do patient research that feeds into all the different aspects of developing a new drug, including an eCOA solution. But that’s what we rarely see. And that’s where we would like to go.
So a lot of the time we start from usability testing, we need to usability test this so we can confirm that it works. But what we really would want to start from is understanding the patients, so learning what people do with a solution, in order to get information about that for solution design. So user research is a critical area. In user experience, it’s one of the key methodologies to learning about users. And I’ll give a few examples of what that means in practice. And then, one of the key aspects of usability testing, I’m sure there are plenty of you who have been involved in some sort of usability testing project because we typically do it as part of a cognitive debriefing study. But one of the key things to understand about usability is that it’s more an art than a science. So you will learn during the process, and the key thing is to allow enough time for implementing solutions for what you have learned. And a lot of the time, if a usability study has not been successful, that has been related to the fact that we haven’t had time, or the sponsor hasn’t had time, to make the necessary changes to actually solve the problems the usability studies identified.
So a lot of this work is done before a trial starts. And that’s partly because the focus is on the trial at hand. But if you think about learning and being able to establish a good base of knowledge about usability in real life, you have the trial, right. Patients in that trial will be using the solution. If you can think through what kind of evidence you would like to collect about usability in more real-life scenarios, you can build that sort of metadata collection into the diary, and at the end of the trial, in interim analysis, during the trial, you can see how the solution is actually performing for you, and you can feed that real-life feedback to the next trial you are doing in that area.
So there are some—if this feels too theoretical—I put some example questions there. So for example, we spend a lot of time thinking like, for example, studies in diabetes could be easier for patients. So for example, how much time do patients spend in a diabetes diary on a daily basis? Do they spend a lot of time in the beginning of the study but then it goes down as they learn to use the solution? Or does the burden stay approximately equal during the study? And how do the patients behave? Is it likely that patients who spend a lot of time in the diary are also likelier to be non-compliant? That sort of questions, you can only really understand what’s happening when you look at the real data being collected.
So kind of as a summary of the approaches that I’ve been talking about, there’s user research. And user research feeds into user design. Usability testing then is used to verify and validate the design and find more things to fix. And then real-life analytics can help you then see how the solution is performing in trials.
Any questions at this point? Or critical observations? Okay. We were actually planning with Karl to take questions again in the endend because we’ll be looking at a lot of examples as Karl goes through his presentation.
So what kind of user research methods can you use? So if you have the time to understand the users, to learn more about them, what could you do? We’ve used, one way or the other, all of these three methods, and they are the prevalent ones in the user experience build. So you have different types of field studies. Field studies are great because they give you the richest information about patients, and any type of users really. They can be interviews, they can be just visiting for example a patient at their home, if it’s difficult for them to go somewhere else, observe them through their daily tasks. You know, you can understand what their life is all about. You can also do different types of other interviews in the context of a clinical trial. The problem is that these studies can take quite a lot of time. So even though they are the richest method, they do require time.
Now focus groups are like—how would I say—they are a form of a field study in one way. You are basically trying to achieve the same thing but you only have time for a focus group, that’s how it usually ends up being. They are easy to arrange, so you can use recruitment agencies, you can use KOLs, patient advocacy groups to contact patients with suitable characteristics to become part of focus groups. And then you can even do usability testing, you can bring everything that happens in the study to the focus group, you can have all sorts of discussions and they can be very useful. One downside is that it’s very much up to patients who are participating what they want to share. And especially if it’s an indication where they might not be that comfortable sharing everything, that will really make your ability to learn about those things.
Now personas are not really perhaps a method, but they are actually very useful. And I have an example of the persona work we’ve done. So personas are like archetypes of users. So for example, patients in different indications or patients in different age groups or in different countries, you can create a persona for them. And personas are great because you can always have them there for your team. So whenever you are making a decision or considering something, and you need to somehow involve a patient aspect into what would this mean for a patient, that persona enables you to keep the patient concerns, motivators, lifestyle on the top of your mind and validate all of the decisions you make against that archetype. So it kind of is an easy way for you to ensure you’re going to the right direction.
So we haven’t yet done patient personas. And part of that is because, to some extent—this sounds really funny—for us at CRF, patients are not the most difficult users. But sites can be relatively challenging for an eCOA solution provider like us, as well as the study team with all of the different roles included in the typical study team. So here’s an example of Cathy, a clinical project manager who’s managing the study. And what we have internally is this material of who is Cathy. Who is she? What does she do? What are her concerns? What motivates her? What kind of a person is she? How would she behave in the team? How would she behave with us, and how would she behave with sites, and so on.
Now usability testing then, it’s a very different sort of an activity. So user research is more about understanding or learning and using those findings in your work. Usability testing is all focused on finding issues. So it scrutinizes what you have developed in order for you to gather feedback and refine what you have created. Expert reviews, I wanted to include that, even though to some extent I feel that’s like a—how would I say—I always want to do something more than just an expert review. But it’s such a simple method of collecting our user experience designers into one room to evaluate and assess a solution that we have created against certain guidelines. And it is powerful too. It’s easy to arrange. The problem is that the findings are within known parameters. So you will only find out things that you are already thinking about. And in a lot of usability testing you will find out things that you never imagined finding. So that’s the downside of it.
Now, usability testing—and this is the standard of what everyone does and it’s really powerful—so you do one set of usability tests with a certain population of users. Typically ten, but sometimes we do up to let’s say 20 for specific reasons. So it’s a very qualitative research method. So you have a patient perform real-life type of scenarios with the solution. And that could also include other things in the study, so it doesn’t always need to be an eCOA solution, it could be how to use the drug, how to use the packaging, how to use any other things that the patient is exposed to during the journey in the trial. The good thing is that it reveals problems, but the bad thing is that if you do only one usability study, you aren’t able to validate the solutions. And because of that, today we do formative usability testing in product development. So we organize a set of usability studies, which are usually one week apart, which enable us to collect the feedback, refine the solution, collect feedback again, refine the solution, and then have final test. And even then it’s sometimes quite challenging to get it right.
So these look to be just an example of a usability test report that we would receive from a usability agency that we typically use in usability studies for eCOA solutions, like in product development. So it’s a bit different from doing instrument validation.
So the typical thing would be that they would highlight a user interface. This one is a concomitant medication diary in Finnish, was tested in Finland. And they would lay out the findings, severity on the upper right corner, and then some solution recommendations. And then we would have a meeting with them to go through all the issues in more detail, all the recommended solutions in more detail, and then we would do this refinement for the next set of tests.
Stats about usability testing, and I won’t go into this in more detail, but there is one additional set of methodologies in user experience that could really help clinical trials. And we are only starting to explore this. So they are usually called customer journeys or user journeys. The idea of these journeys is to understand all the interactions that the patient or any type of user or customer has with a solution, service, people, any type of aspect of a clinical trial, and understand how these interactions work from the user’s perspective. And this provides a much more holistic understanding of what the total experience of the patient is during the clinical trial. So here you can think of all the things, how they interact with an eCOA solution, but also how they interact with the site. How does the training work, how does the informed consent work, what are all the interactions the patient has with the site. And how do they all play together to create a great experience for the patient, or perhaps sometimes a maybe sub-optimal one.
Then a lot of the time, sponsors, institutions, research institutions, as well as us, we work with partners to complete all of this work. And the typical partners that we work with are these three. So we would work with key opinion leaders. They are wonderful. They can facilitate all the aspects of usability as well as many other things that are actually really useful in designing the clinical trial. So the problem sometimes is that there are different expectations that the KOLs have versus what for example sponsors have, and one of the challenges that we face in for example engaging with KOLs for usability testing is that at the same time as we are doing usability testing with them, they also want evidence that the solution will work in the trial for their patients and for them. Now because usability tests are designed to reveal problems, KOLs need to be really well prepared for the outcome of the usability test. Instead of the usability test results showing no problems, everything will be wonderful, the test report will actually list issues that need to be solved. A relatively simple thing, but it has caused a discussion or two with our sponsors and KOLs.
Patient advocacy groups, equally wonderful to work with. Also very rewarding for everyone who works with them. The only downside is that the engagement is really time consuming. So you really need to prepare a lot of time so they are perhaps more useful for the user research phase rather than usability testing, because you need to reserve sometimes months for this engagement. And sometimes they are limited in terms of how they want to engage their members, which is very understandable, but then you of course are kind of limited in your research options to the ones that they would support and agree with.
And then usability agencies, so within our industry we would of course engage with experts in for example instrument validation and usability. But we also engage with usability agencies outside of our own immediate field, and that’s typically very useful. The only thing is, if you need to be sure that you work with people who are not too junior. You would easily find very junior people in usability research agencies. And one good thing in them is that there are agencies that partner with other agencies globally. So they form these networks. So if for example want to ensure your solution works in different geographies, they can typically arrange that.
And last but not least, building on current practices. So what can you do as the first thing. And I always think about it from the perspective, we do quite a lot of cognitive debriefing and usability studies in the industry. So one key thing that almost everyone can do is to focus on those studies and make sure that also other types of concerns or questions about usability are included in those studies, because that’s a key point, when you do have access to patient feedback, typically usability is a consideration, but there could perhaps be even more focus on those aspects. And then, making sure we are able to do these studies early enough for the feedback to be incorporated. I think many of you who work in this field have probably seen that sometimes there isn’t enough time to actually do the changes. And of course, not only frustrating for the people who are doing the research, but also for the study team who perhaps didn’t know it beforehand that that would happen.
And then of course one final thing is to share the learnings across the organization so they are again available for everyone planning a new study. Sometimes perhaps not easy to do in a global pharmaceutical organization, but still a very useful thing if one is able to pull it off. For example, using the central eCOA organizations as the central point, gathering all of this wisdom for upcoming projects.
That was everything that I had. Now to hand it over to Karl.
[END AT 27:25]