Methodological and Practical Considerations for Developing Electronic Clinical Outcome Assessments (eCOA) in Rare Diseases

June 14, 2017



So we’re going to move on. We’re going to bring up Sarah and Nicola, who are both research managers at Adelphi Values. And they’re going to be talking about the methodological and practical considerations for developing electronic clinical outcome assessments, specifically focused towards rare diseases. They both represent the patient-centered outcomes practice and specialize in qualitative research plus the development and validation of COA. So if we could bring up Sarah and Nicola please. Thank you very much.


So everyone, my name is Nicola Williamson, and as Paul just said, I’m a Research Manager at Adelphi Values.


And I’m Sarah and I’m also a research manager at Adelphi Values.


So just to give everyone a bit of background, if you’re unsure what Adelphi Values is, we’re a healthcare consultancy, we’re based in the UK, we’ve also got offices in the US. So we work with pharmaceutical companies to conduct research, and primarily focused on patients, so we do both qualitative and quantitative studies. And we specialize in the development of COAs.

So just to give a bit of background today, obviously as evident that we’re all here today, eCOAs is now widely accepted method of data collection in both clinical trials and observational studies. But there are challenges associated with collecting data in COAs, particularly in populations with rare diseases. And this is mainly related to the heterogeneity of the symptomatology of the patients. So patients can experience a wide range of symptoms. So there’s challenges associated with collecting data both from a COA perspective and also from an eCOA perspective.

And then to go into some of the topics that we’re going to cover today. So firstly I’ll give an overview of rare diseases and why we’re focusing on them, and then talk about some of the challenges and solutions associated with measuring eCOAs in this population. And then after that, Sarah is going to talk about how to overcome some of these issues using eCOA training, and then the use and benefits of pilot studies.

So firstly, just to give an overview of rare diseases. So a disease or disorder is considered rare in the USA if it affects fewer than one in 1500 people, and then in Europe if it affects fewer than one in 2000. And 50% of rare diseases affect children, so it’s a large proportion. Globally, over 7000 rare diseases are characterized or acknowledged, and of these obviously there are a range of disorders and a range of symptoms that patients experience. So these vary from disease to disease, as well as patient to patient. And as there’s normally a lack of information about rare diseases, patients can often be misdiagnosed, which can delay treatment. And obviously that can impact on their quality of life and their experience generally. And just that last point about rare diseases, there’s often no existing effective treatment, so it’s really important to explore the patient experience.

So firstly, this challenge focuses on what I was talking about before in terms of the symptom presentation. So in a rare disease, often we’re dealing with small sample sizes, but given that it’s a rare disease patients often have different experiences of that disease. So some patients might experience one symptom whereas others wouldn’t. So it’s really important to capture that kind of variation in PROs and response options, to ensure that we accommodate that. And then also, there are issues to do with if there are a number of symptoms that are related to each other, ensuring that patients complete questions in the correct order.

Okay so solutions to dealing with that in eCOAs rather than paper is that we can incorporate skip patterns to overcome these issues, so patients would only complete questions that are related to their experience. And then also we can adapt the format of an eCOA, so using visual progress bars or adjusting the size and colour of the text so that it’s easy to understand and patients are aware of what they need to complete. And another thing we can do is to use an “other” response option, so assuming that not all patients fit within pre-specified categories, we could use on some items an “other” response option where patients can give some more information about what they are experiencing related to that symptom.


And then going on to challenges associated with patients by age and cognitive ability, so I’ll go into each of these separately on subsequent slides. As I mentioned before, up to 50% of rare diseases affect children, so paediatrics. And also rare diseases, commonly symptoms are cognitive ability. So therefore, often it’s useful to have multiple reporters involved if a patient can’t complete, or in this case of paediatrics. So a solution to this is to use multiple versions of an eCOA, so you can have one version that’s developed for a patient, and then another version that’s developed for a caregiver. You could administer them independently, so a patient would complete it at a different time to a caregiver. Or if the patient is unable to complete one, you could just rely on a caregiver report.

And then going on to specific challenges with paediatrics. So often paediatrics have maybe a short attention span, particularly younger children. So they’re not going to be potentially engaged in the study or focused on what we’re doing, they may not be motivated to use the device, or they might have difficulties reading or understanding the items and what’s asked for them. Also there might be issues with remembering how to complete the questionnaire. So for example, if they’re asked to complete on a daily basis, they might forget to do that. And then that last point about the experience. Their symptoms might actually hinder them completing the device completely.

So some solutions to this. So using an eCOA compared to paper is a lot quicker, and that definitely helps with the short attention span, particularly in paediatrics. In my experience when I’ve debriefed eCOAs, paediatrics have been really engaged and involved in using the eCOA and found it a fun way to kind of complete a questionnaire. The eCOA can also be adapted to support paediatrics. So as I mentioned before about visual aids, you can use visual response options as opposed to verbal descriptors. Skip patterns can be incorporated to ease the flow. And then instructions and simplified wording can be used to aid their completion. And then there also can be formatting things that we can do to increase their engagement, so for example, using themes throughout, personalizing it so it mentions their name or giving positive feedback once they’ve completed an eDiary.

And then going on to challenges with patients with a rare disease with cognitive impairments. So these patients may be less digitally proficient, so they may have less experience using a device. They might not have access to one at home or they might not even have access to a computer. So they may have issues with that. And then obviously there might be, in terms of the cognitive impairment, issues with understanding complex items, attention span, and their cognitive ability, which would hinder their completion of an eCOA.

And then in terms of solutions to this, simply with patients with patients who have difficulty using their fingers, eCOAs can incorporate what’s called a knuckle-tap function which means that larger button displays on the screen and they can tap that to provide a response rather than having to use their fingers. And clear instructions can be used to help reduce confusion. So displaying just one instruction on each screen or just one item rather than multiple items. In some studies we can provide devices rather than having the option of bring your own device, and that’s really useful for these patients, especially if they don’t have a device at home. And then obviously having the screen flow so that it’s obvious what’s going on and flowing from instructions to the items. And then I suppose in terms of engagement we have to think a bit more about following up with these patients, ensuring that they complete it, if they’ve got any questions, there’s normally a help desk number that people can phone and stuff like that. And then I guess if there are any issues that they’re having, and they’re really struggling, we can get a caregiver to complete it instead of the patient.

So now I’m going to hand over to Sarah.



So just the last challenge that we wanted to talk about today, and Nicola already mentioned about the complex symptom presentation of rare diseases. So often the instruments that we develop, we’re not able to kind of use generic ones for some of these rare diseases, we actually have to start from scratch and develop a new instrument. So we have to be careful in terms of the terminology that we’re using within that instrument. Even though the symptoms are very complex we have to make sure that we’re really aware of the language that we’re using. And also the PROs that we use may be addressing factors such as frequency, duration, and severity of symptoms. Again, this can be quite complicated to understand, particularly for those more complex symptoms. And just the final point there, there’s complex concepts that we may be addressing also, so we have an example there of movement disorder events. So it may not be as straightforward as just listing out the symptoms that the patient’s experiencing, but actually being able to kind of record the frequency of an episode of certain movement disorder events in a given day.

So in terms of solutions and benefits, the format of the screens can be adapted to help improve the flow of information. So we are often having to provide large sections of instructions for the instruments that we’re developing just to explain some of these complicated concepts, so in terms of eCOA we can actually display those on separate screens. We often find that on paper the participants generally skip to the first item without reading the instructions in depth. We can also provide training sessions on the device, and this is something that I’ll go into in more detail. But basically the ones we’ve used in previous projects are actually slide shows that have a voiceover and the participant can take the device home and actually complete the training in their own time. And this can really help kind of consolidate understanding and improve familiarity with the device before they actually enter the live study. Another solution is pilot testing, so again in rare disease we find this really useful because it helps participants to actually take part in a mock-up of the study before they then go on to the live study.

And so starting with training, just some of the benefits really. So it’s built into the device that the participant uses in the live study. They can complete it at home at a convenient time for themselves. And hopefully this kind of covers any issues that they may cover in completing the PRO, cover any complex terminology that may be used, just to check that they understand it and give them an opportunity to then report back to the researchers if there were any major issues. And this can allow changes to be made to the PRO or even the usability of the device before entering the live study. It’s great because it doesn’t affect the logistics of the study, so the participants will be sent a device anyway, so they’ll have that for the live study. And actually it means that the participants are required to complete the training before they then go on to the live study. It means that this is kind of a requirement that they have to complete.

We can incorporate screenshots of items within the training, so again this helps with familiarity so that they’re able to see some of the questions and response options that they may be presented with and also have a go at answering some of those. They can test the functionality of the device, so things like moving on to the next question and saving answers. And again, just general testing and consolidation of knowledge. So in some studies we’ve actually used a quiz at the end of the training, so we’ve asked questions around you know, how often should the instrument be completed, what’s the recall period, what should they be thinking about when completing the items. So again, it just helps to consolidate that knowledge and understanding, which definitely helps for the live study.

So these are just some example screens from a project that we recently conducted. So the project was in a rare disease called glucose transporter type 1 deficiency syndrome. And this is a rare condition which involves movement disorders. So there’s just some screenshots, so the first one on the left is just to show kind of some examples of objectives that may be given to the participant just so that they’re fully aware of when they should be completing the diary and what they should be recording in the diary.

The second shows an example screenshot, so again when I was talking about kind of more complex completions, this question is actually asking the participant to record with an open-ended response the duration of a given movement disorder event. So again, it can help them understand what they’re going to be required to respond to.

The third one there just shows an example of a quiz question, and again, it’s great to consolidate knowledge before they go into the live study. We’ve then got an example which shows tips for success, so just basically any final tips or considerations that they should be making when completing the instrument. And then that’s just the closing screen as well, just to verify that they’ve completed training and are therefore ready for the live study.


So in terms of pilot study, this is just an example of some methodology that we may use. So firstly the participant will be trained on using the device, which we’ve just run through. The participant will then be required to complete the device at home every day for kind of X number of days. And this data can then be collected and transmitted via a secure online server. So we’re actually being able to collect quantitative evidence about completion of the instrument prior to actually entering the trial.

We then follow up with the cognitive interview, and obviously we can then collect some more qualitative information about the actual usability and feasibility of completing the instrument, as well as understanding of the items.

So just some of the benefits associated with pilot studies. The main thing is that it highlights any underlying issues at an early stage so that these can be addressed before entering the trial, which can save huge amounts in terms of time and cost at a later stage. So we can focus on things like usability, so things like saving answers and moving to next screens, feasibility of completions, so if we’re saying that the participant has to complete the diary twice per day, just to check that that’s actually feasible for the participant and fits in with the time that they have. This relates back to the kind of patient burden aspect that we were talking about in previous presentations. And also again, just to check that they understand the items and response options.

And again, as I mentioned, because we’re not necessarily using generic instruments, we’re actually developing new ones, this information is really important for us. So it allows any edits to be made to the diary and formatting before entering the live study. It also provides the opportunity to collect that quantitative information before entering the live study. And again, this can give kind of a snapshot of information of what the data might look like from the actual trial, so we can see if there’s any particular ceiling or floor effects on the kind of response distribution as well.

And then there is the live study. So again, it kind of avoids anyone expecting findings or issues in later studies. And it also allows to test the logistics of actually administering devices. As Nicola was saying, in rare diseases, patients can be living in different countries, so we can just check that that process of data collection is actually feasible. So ultimately the pilot studies are kind of ensuring that the eDiary or instrument that we’re using is as easy as possible for participants to complete, and therefore more likely to generate valid and reliable data.

So these graphs are just some examples of the types of quantitative data that we might be able to collect in the pilot study. So the first one is just showing there that we collected information from child- and parent-completed diaries. We can then start to compare and contrast the responses from the two groups. Again, this is helpful for the live study just so that we can understand if actually the two groups would be submitting similar information or if there’s actually quite big differences in the two sets of information that they submit. And then the second graph there is showing the percentage of responses and kind of more focused on the response distribution. So again, we can see if any participants are not using certain response options and perhaps explore this before entering the live study.

So in conclusion, just to say that the very nature of rare diseases, the complex symptom presentation associated with rare diseases has huge implications for COA development. And while there’s many challenges associated with the development and use of COAs in rare disease, there’s actually a number of solutions that we can use, particularly with eCOA, to help overcome some of those. And things like device training and pilot studies can be really useful in generating that information to help validate the instrument and ensure that when we get to the live study, we’re actually generating the information that we require, that’s important to the participant.

So that was everything. Are there any questions at all?


So thank you very much, Nicola and Sarah. That was really thought provoking for me, actually. I hadn’t realized that there are so many rare diseases that have been identified, 7000 did you say, that’s a phenomenal number. It almost seems to me that you’re personalizing the experience for the patient with the devices and the instruments that you’re designing. It seems that this is taking—you’re going really forward thinking in this approach. So congratulations on the approach. Thank you.

Can I present it to anybody in the audience that might have questions, please.


Thank you very much, that was a really great presentation. Something that comes up occasionally when we’re talking with sponsors who are working in rare diseases who want to use electronic data capture, and tying back to Michelle’s presentation, might be considering doing some kind of equivalence testing. Obviously one of the issues with rare diseases is it’s pretty rare, there’s not many patients. Have you ever come across ways of maybe addressing, for example, not being able to target the exact kind of patient, or targeting patients in Phase II, or getting patients recruited into your Phase III and doing testing? How do you deal with the lack of patients out there for a lot of this testing when really the work requires getting access to the patients?


Yeah I think specifically for our studies we have to be mindful that the sample size is going to be small, and I think it’s good that the FDA recognize that the sample size isn’t going to be the size that we would expect from more common conditions. So that’s one thing that we keep in mind. We also tend to approach several different countries, so we’re not—you know, specifically targeted the US for instance, we’re actually having to go to different countries to collect as many responses and recruit as many participants as possible. So that’s a key thing, but obviously then that kind of causes issues of things like translations later down the line, so that’s something that needs to be considered. But with these studies, recruitment can take a lot longer than it does for your more generic studies, so we have to make sure that we allow that in the study timelines as well. And yeah, because we’re getting such varied responses from patient to patient, it’s more vital than ever for these studies that we’re really probing into their condition and making use of the concept elicitation phases of the interviews as well, just to make sure that all the items and response options that we generate are really valid and kind of reflect the experience of each patient.


Anybody else at all with any thoughts or questions?

Okay. Thank you very much once again, a really thought provoking presentation. Thank you.

[END AT 22:30]

eCOA and Older Users
eCOA and Older Users

eConsent: Enabling the Patient-Centered Clinical Trial of Tomorrow, Today
eConsent: Enabling the Patient-Centered Clinical Trial of Tomorrow, Today