Process overview and some lessons learned during the design of an event driven eCOA solution to be used in a pediatric population.
Next up we’d like to introduce Mags Kelly-Homer. Mags, like Paul, hails from Ireland. I think there’s a theme developing here. Mags started a career at a clinical research organization in Ireland more than 15 years ago, and more recently, since 2005, has been working at the Astellas in the Netherlands. She’s worked in a variety of different roles but typically within the data science discipline. And Mags’s current position, she’s a Principle eCOA Manager and is responsible for the implementation of eCOA solutions in clinical studies globally. And this includes participation in improvement efforts, as well as in the evaluation of new technologies. Mags, welcome.
Thanks for the introduction, John. Just before I start, I would like to state that I didn’t volunteer to present at today’s session. Paul thought it might be a good idea and it might be interesting to share with you some of our recent experiences with working with CRF Health in trying to develop an eCOA solution that could be used in a pediatric population. So I guess time will tell whether or not he was correct in that assumption.
So the agenda for today’s meeting, so we’ll look at the concepts, so really our requirements, what we were trying to achieve. The expectations, nd there were some wild and wacky expectations within the wider team. The actual reality, so what we sort of ended up with in our study. The challenges we faced throughout the whole process, and also some of the feedback we’ve already received.
So the concept was to develop an event-driven eCOA solution for a paediatric population with overactive bladder. So at the very beginning there were some meetings and then there were some more meetings. And that was followed by even more meetings. And at these meetings, there were decisions made and then at subsequent meetings, different decisions were made. So in the internal team, we had quite a few areas represented, so we had our medics, clinical stats, health outcomes, DM, and also the dedicated eCOA function as well. But we also involved our key opinion leaders, so some experts in the field. So in the field of urology, dealing with children with OAB. So we also reached out to them on quite a regular basis to get their feedback on where we were going, were we moving in the right direction, is our end solution feasible, is it workable in this sort of population. And that was even before we reached out to the vendor. So when the vendor side, as usual we worked with the project manager and applications specialist. But we also had quite a lot of input from both the user experience designer, the health outcomes team, and as well sort of an external graphic designer was employed as well to sort of help us with the overall look.
And so in the beginning stages, we sort of looked at the must-haves, what we really needed to capture. Because of course we need to capture data so that we can prove that our drug is efficacious, so there were certain parameters that we had to capture. And of course, our medics would also have liked to capture additional information that was nice to have, but not necessarily going to add any value to the analysis at the end of the day, or even use in the analysis at the end of the day. So it was already decided up front it would only be requirements that were captured.
Now, the ideal solution would have been electronic everything. So basically, nothing for the patient to do, no additional burden, they just go around their normal routine, and everything miraculously appears via Bluetooth in the device and we have our data. So the team did look at what was out there by way of technology, and there are some things, not everything is fully developed. Some things were really only for use sort of in real world, and not necessarily validated sufficiently to be used within the confines of a clinical trial. So the dream solution was either an electronic toilet or even diaper or pants. So what could these things do, they could measure the urine volume. And maybe detect incontinence, and as I said, everything would miraculously go via Bluetooth to our device and we’d have all our data.
We had to adjust our expectations slightly, once they had actually done a little bit of additional research and had a look at what was actually out there and available for use. And so in the end we decided, okay, so what can we do realistically for this study. So if there was any technology we could use, the thing we were looking at was maybe some sort of integration with a volume measurement tool, so that would go directly to the device for measuring urine. Make it look appealing to our patients, so for the children and adolescents. Keep the text simple, and use more icons and, additionally, reduce the burden as much as possible, so make it a little bit more flexible.
Okay. Making the technology as seamless as possible, an integration with volume measurement tools, so we did look to see if there was some sort of scales or other volume measurement tools that were out there that you could use for our trial. Unfortunately, we didn’t find such a tool, at least not one with Bluetooth enabled technology that would go directly to the device. Some things were developed all right, that maybe we could have looked at a little bit further, but that would involve a lot of extra time, a lot of extra effort, and a lot of extra cost. So in the end, it should be a tick, manual input into the device. So the subjects—patients—children were unfortunately going to have to measure their urine volume and enter it into the device.
Now the look. Okay, so this is something we could work on, we hoped. So we had asked CRF Health to get a graphic designer to work with us. With most of our studies, we do have themes. So the theme that was given to the graphic designer was a dolphin theme. So what we wanted were possibly two different designs, one would work for the younger children, and one would be more suitable for the sort of mid-range or adolescent children. So dolphin theme, so here we go.
So this was our first little cartoon dolphin. And this was aimed at the younger population. So we had two variations on the screen, one would be used for the welcome and sort of thank-you screen with the full dolphin image, but then one with less graphics for the actual questions. So it wouldn’t detract from the information we were trying to collect. The second one was geared towards adolescents. Now, we saw these and actually nobody really liked the second one, the one that was geared towards adolescents. I’m not sure what the feeling is here in the room, but we decided no, we were going to get rid of that one. So that left us with just one theme and we decided okay maybe this isn’t going to appeal to everyone. We had to consider the older children and the adolescents as well, so CRF Health came up with a good idea, of maybe just using additional graphics that were already readily available, and this is what we ended up for our final cut. So we have the tie-dye, hearts, and this one is called splatter but as you can see it is a rugby player, I believe. So we thought, at least with the four options that were available, something should appeal to each of the children taking part in the study.
Okay, we weren’t happy with just the main sort of daytime questions. We also had a nighttime question that’s asked in the morning relating to the night before, and we asked if there’s anything additional we could do with those screens to make it sort of look, you know, just add that additional dimension to the overall look. And so we sort of made a nighttime look/feel to each of the designs.
So the next thing we wanted to work on was language. So here I’ve just shown an example of our adult OAB and sort of how that translates then to the paediatric OAB diary. So, please indicate the type of episode, or what did you experience. So urination, incontinence, and we also have a both option, where they have a bit of both. So we have the same idea in the child study, so we have pee in toilet, leakage, and yeah, pee in toilet and leakage. And we try to use icons as much as possible as well. So limit the amount of text that was used and increase the number of pictures. Now I know somebody asked earlier about using pictures in sort of global studies and how these would translate. And we did actually look at—we had similar discussions within the design team, and we did receive a range of different icons that could potentially be used, and these were sort of our safest options, basically, ones that we thought could be used globally on a global study. So this is what we went for in the end.
Also throughout the study, you know, if they enter impossible or improbable information, so we have, you know, checks that would fire or little popups that would fire. So we tried to speak directly to the children, so instead of is this correct, it was are you sure.
Also when it came to the training, so we have a mandatory training built into all our eCOA solutions, so it’s immediately available once the subject is set up in the diary. And they have to do that training before they’re actually able to continue with any regular entry. So again, we wanted to keep this simple. We didn’t want to put in everything that we wanted to capture but sort of just focus on the main themes, the main topics and sort of, again, speaking directly to the children here. So we did think that maybe some of it was a little bit simplistic and may not necessarily be related to the adolescents, but in the end we sort of ran out time, as you do, and we said, we’ll just let it go.And this is some feedback we’ll expect to receive later on in the study. So it’s sort of follow me and I’ll show you what to do. Sort of, your mission is to enter pees and leakages. And did you have to rush. And then during the weekdays grade your worst leakage. So dry, slightly wet, quite wet, fully wet.
We continued this simple theme to the quick reference guide. So we decided we would create a quick reference guide aimed directly at the children taking part in the study. So we keep it simple, and again just focus on those more important aspects of the trial and the data that we really needed to capture. But obviously we needed sort of a more technical version as well, so that was available for either the parents or the adolescents who would have been able to cope with that QRG.
Next, we looked at flexibility. So we wanted to reduce the burden as much as possible, considering we still wanted to have a seven-day voiding diary. But within that voiding diary, we have a five-day weekday diary, where there’s only two questions asked of the subjects and the two-day sort of weekend diary, as we called it, would capture all the additional information that we needed. So in relation to urine volume, number of incontinence episodes or leakages, the rush questions, yes or no. So within that, we did allow some flexibility in completing the so-called weekend diary. So we expected that during the week, the kids are at school. Somebody mentioned earlier maybe it’s not so nice that they have to take this device to school with them. So that’s where just the two questions come into play. The morning question, the evening question. So one they completed before they go to school, and the other one they completed in the evening when they come home. And also then the weekend diary, we expect them to be around home, I guess, at the weekends for the most part, so that’s when the other questions would kick in. But within this, we also realize that, okay maybe sometimes they’re on holidays, and so we allowed a little bit of flexibility as to when they can complete their weekend diary. So it had to be two consecutive days, but they could choose those days within that seven-day period.
We also used alarms within the study. So we had a morning alarm and an evening alarm, but we also had a weekday morning alarm, a weekend morning alarm, and a weekday evening alarm, and a weekend evening alarm. And what we did was we allowed the morning alarms to be optional, because our weekday questions—one was how was the night before—they could choose not to have that alarm. I guess, you know, if they’re getting ready for school in the morning, parents are rushing around trying to get everyone ready, dressed, up, out to school, maybe the last thing they want is this alarm going off in the background to ask them how was their right before. And because it was really just a wet-dry question, we didn’t think there would be any issue with recall answering that question later in the evening instead. So we also allowed the different alarms at the weekend, so we figured that although maybe a five-year-old might be still up six, seven in the morning at the weekend, maybe our 17-year-olds might like a little lie-in and don’t necessarily want an alarm going off to wake them from their slumber.
So that was pretty much as much we could do by way of flexibility, considering we still need to capture our information, we still needed to prove efficacy. And so yeah, that’s what we ended up with.
Now our biggest challenges during the process were really sort of the expectations versus reality. So I guess technology is evolving so much and sometimes it’s evolving quite rapidly, but maybe not as rapidly as we need in some areas so that we can actually use it within a clinical trial. And some of our medics, you know, had seen all these apps and all this technology and they wanted it all. And unfortunately, when push came to shove it’s just not something that we could actually have used in these situations. So again, the technology limitation based on the expectations of some of the team.
Now we had quite a large team to begin with. So when you’re designing something, you know, a lot of it is personal opinion. So it’s very subjective when it comes to an actual design. And we had a lot of opinions. And a lot of changing opinions. So actually, the process probably could have taken a lot less time if there weren’t so many opinions, and we did learn that a little bit later on in the process, so we did remove some of the team members sort of halfway through and then switched to the more operational side. And then things did speed up a little bit and we actually got to our final design in the end.
All right, so what I may not have mentioned so far—and apologies—is that the whole idea of this part of the process was to actually create a validated voiding diary that could be used in a paediatric population. So the intention was to create a solution that could be used in a usability study. So when we came up with our final design, and the smaller operational team were happy with that, we received the devices from CRF Health, we passed all around to the wider team, we passed those around to the KOLs, and we got some feedback there as well. So this was all pre-deployment to the actual usability study. So there were actually no changes requested on the design. Some minor tweaks to some text, just to make it less confusing I guess. And as I mentioned, the usability study is still in progress at the moment. And the main idea here is to assess the usability and acceptability of the device, ensure that all the components are understood by the patient, so they know what we’re asking them and they know what sort of responses they should provide, or they understand how they’re supposed to answer the question. And then of course, maybe we’ve gotten it wrong, so maybe some aspects are wrong. So we would like them also to propose some modifications or adaptations to the design.
Now the usability study is still ongoing. So it’s 15 subjects in total, five from each age range, so we have the lower age range, 5-8, 9-11, and 12-17. So we have 10 out of 15 patients completed actually at the moment. And I believe we’re expecting the last to be recruited by the end of this month, I believe, so we should have the study completed sort of mid-November and then we’ll be able to analyze everything we get.
But we did ask for some feedback already, based on those initial interviews that have taken place. So we were curious to actually know what themes the subjects chose. And actually there seems to have been an even split between the tie-dye, the dolphin—nobody liked the hearts, nobody liked the splatter. But there you go. Maybe we should have a poll later to see which design everyone here would like to go with. We’ll have a poll later maybe, John.
There was no reported problems in the eDiary usage, so that is the general feedback so far. So all of the children enjoyed using the device. They enjoyed that interaction. And there have been no technical issues reported either, so you’ll be happy to know that on the CRF Health side. They also liked the look. And also the icons. You know, because we were a little bit dubious maybe maybe about the leakage icon, so you know, all the questions in the questionnaires provided by the interview, both at the start and the end of the study, really every single detail had a question associated with it. And so far, we haven’t received any negative feedback in relation to the look, including the icons.
Now we did have some definitions in there as well, so to provide some clarification on what you’d consider slightly wet, quite wet, fully wet. And apparently there is some confusion over those, so we do expect some modifications to be made to those when we actually go to the proper clinical trials. Some other sort of minor issues were identified, but these could be handled sort of from the training aspect, just sort of focus more on those, either in the online training or in the QRGs.
And overall, completing the diary was not a burden. What was a burden was collecting and measuring urine volume, but we knew that was going to be a burden and unfortunately we do still require them to collect and measure the urine volume. But nobody at all had any problems with actually completing the device. And it was quite a pleasant experience, or so we are informed so far. I think a lot of kids think they’re sort of a little bit cool because you know, it’s a nice smartphone, and it looks pretty with all the pictures and everything so, you know, it’s a story for them to tell their friends then as well I guess.
So we didn’t get our dream solution. So no magic nappies, pants, talking toilet, or even our volume measurement. But we do hope we’ve at least made some small steps, so to make it look more appealing, to keep the language simple and use more icons, so pee, leakage, did you rush. And also to introduce a little bit of flexibility. Okay, not a lot of flexibility, but we tried to do as much as we possibly could within the confines of what we needed to collect. So the alarms and time to complete.
And I believe that’s it. Any questions?
Thank you very much, I think that’s a really good illustration of all this wonderful theory we’re talking about here about how it actually happens in the real world when you’re trying to make this work in a clinical trial. And I think it’s particularly indicative of maybe where we are as an industry, where including simple background themes in a questionnaire is considered kind of almost revolutionary, when that would be quite simplistic compared to a lot of the other stuff that’s being done with smartphones. So I think that’s almost a bit of a wakeup call for where we are as an industry and some of the things we maybe need to look to do to really drive ourselves forward. Did anyone have any questions for Mags?
AUDIENCE MEMBER: Thanks for a really great presentation. I was just wondering, you got good feedback about icons from the paediatric patients. At least I speak for myself, I always have a background image on my phone, and I was just wondering if you would consider using themes and icons for adult voiding diaries as well in the future.
Actually some of our clinical study managers that were also involved in sort of the operational team and were setting this up, they will be working on future OAB studies in adults, and they definitely think sort of this is the way to go. Why not just design a solution for a paediatric population, and then you can use that in all populations, because I think there’s quite a high level of education in this room, I guess. And sometimes we overestimate the literacy of some of our patient populations. And some of the text is quite complex, so urination, incontinence, why can’t we use pee and leakage with our adult population as well and make it look a bit prettier while they’re completing it. So I think that’s sort of definitely the thinking within Astellas anyway, that we would maybe just create a single solution that can be used for all populations. Okay, some modifications I guess would be required, but not too many. Like, the idea is maybe you would offend sort of the highly educated participants. But I certainly wouldn’t be offended completing the child diary. Again, that’s a personal opinion, but I think the same would be true of more participants.
AUDIENCE MEMBER: I think especially with BYOD, and we don’t know what size screens people will have in terms of phones, icons are a good way to kind of overcome localization battles, particularly with global studies.
Absolutely. And we could increase the button size as well by keeping the text simple.
AUDIENCE MEMBER: So I was involved in some of the early discussions for this particular trial, and there was a lot of talk at the beginning about bring your own device, so use an app. Obviously given the regulatory issues that Paul discussed and timelines—they’re always an issue—we didn’t get to go that route. But do you think that studies like this where we’re kind of treading the line between a traditional eCOA, slightly bland very close to paper approach, and something where we have something that’s a little bit more tailored to a specific patient population, do you think this is the baby step towards BYOD? It’s kind of treading a line between what the regulators would look at and accept and being different from a traditional provisioned device?
It’s possibly a step in the right direction. We are still using provisioned devices provided by yourselves for this usability study and also for the future clinical studies. I think, yeah, we’re still going to end up with provisioning as Paul mentioned earlier, we’re still not quite there yet with the BYOD, but maybe this is a step in the right direction.
Any other questions for Mags? I was wondering was there any teams or maybe individuals in particular within your company who had particular pushing power when it came to making those decisions? Obviously you were juggling a lot of opinions, as you said.
Our medics were quite vocal, I will admit. But as long as there were good arguments about why we couldn’t go one way or another, they were quite happy to accept that. And so when we got to the operational team, and that team sort of understood what we could and couldn’t do, they were able to sort of relay that information to the wider team, and then it was more acceptable.
Any other questions? All right Mags, thank you so much for your presentation.
[END AT 27:40]