Group cognitive rehabilitation to reduce the psychological impact of multiple sclerosis on quality of life: the CRAMMS RCT

Primary objective

The primary objective was to determine whether or not attending a cognitive rehabilitation programme (the intervention) in addition to usual care was associated with reduced psychological impact of MS on quality of life, as measured on the Multiple Sclerosis Impact Scale – Psychological Subscale64 (MSIS-Psy) compared with usual care alone (control).

Secondary objectives

The secondary objectives were to assess cost-effectiveness of the intervention, and whether or not the intervention was associated with improvements in participants’ attention and memory abilities or in self-reported memory problems in daily life, mood, fatigue, employment status and carer strain.

Sites

The trial was conducted in five sites in England (see Appendix 1, Table 25). Each site was an NHS trust providing neurology services for people with MS.

Three sites were opened to recruitment between March 2015 and May 2015. A fourth site opened in February 2016 and a fifth site in September 2016. All sites were open to recruitment until March 2017.

Inclusion criteria

People with MS were eligible for the trial if they met the following inclusion criteria:

• Aged 18–69 years. The upper age limit was to ensure that memory problems were likely to be because of MS rather than age-related memory impairments.

• Reported having relapsing–remitting or progressive MS.

• Diagnosed with MS at least 3 months prior to the screening assessment.

• Reported having cognitive problems as determined by a cut-off score of > 27 on the patient version of the MSNQ. 66 The cut-off score was based on the original validation study by Benedict et al. 66 with 50 participants with MS. The cut-off score was used to identify those with cognitive impairment on a neuropsychological test battery.

• Had cognitive deficits, defined as performance more than one standard deviation (SD) below the mean of healthy control participants corrected for age and education67 on at least one of the Brief Repeatable Battery of Neuropsychological Tests (BRBN). 68

• Able to travel to one of the centres to attend group sessions. Participants had to live in the geographical area covered by the sites and be able to get to the sites independently. Travel expenses were offered to those who needed them.

• Willing to receive treatment in a group if allocated to intervention.

• Able to speak English sufficiently to complete the cognitive assessments and take part in group sessions.

• Gave informed consent.

Exclusion criteria

Potential participants were excluded if they:

• had vision or hearing problems, such that they were unable to complete the cognitive assessments

• had concurrent severe medical or psychiatric conditions that prevented them from engaging in treatment

• were involved in other psychological intervention trials.

Usual care

All participants received their usual clinical care. In the standard NHS care pathway, people with MS with cognitive problems may get general advice from MS specialist nurses and occupational therapists on how to manage any cognitive difficulties. There are information sheets available on web pages of MS charities, which include suggestions for coping with cognitive problems. However, usual care does not normally include any specific intervention for cognitive problems or cognitive rehabilitation.

Other clinical services available to participants included employment rehabilitation services, self-help groups and support from specialist charities, such as the MS Society. Any input, including psychological or medical interventions, that participants received during the trial was recorded on the UHSSQ.

Trial Management Group

The TMG comprised the co-chief investigators, members of NCTU responsible for the running of the trial and the health economists. They met regularly to monitor the day-to-day running of the trial.

Trial Steering Committee

The TSC was responsible for the overall conduct of the trial. The TSC had an independent chairperson and four independent members. The independent members were rehabilitation professionals and service user representatives who were not otherwise involved in the trial. Members of the trial team, including the chief investigator, co-chief investigator and trial manager, were also part of the TSC. The TSC advised on recruitment strategies, monitored progress with recruitment and checked adherence to the trial protocol. Representatives of the sponsor were also invited to TSC meetings.

Data Monitoring Committee

The DMC was an independent group, the members of which had no other involvement with the trial. Members of the DMC included two rehabilitation professionals and a professor of health service research with expertise in statistics. The role of the DMC was to safeguard the interests of trial participants, with particular reference to the safety of the intervention; monitor the overall progress and conduct of the trial; and protect the validity and credibility of the trial.

The TSC and the DMC met independently of each other, with the DMC providing reports to the TSC.

Attendance at sessions

Participants attended a mean of 7 sessions (SD 3.4 sessions) not including catch-up sessions, and a mean of 7.7 sessions (SD 3.5) when catch-up sessions were included (Table 3). Three or more sessions (including catch-up sessions) were attended by 208 participants (85%). The reasons that participants did not attend sessions are shown in Table 4. Twenty-four participants (10%) missed sessions as they did not want to continue and six participants (2%) stopped attending as they withdrew from the trial (not mutually exclusive); otherwise, sessions tended to be missed because of illness or other commitments.

Inclusion in primary analysis of the primary outcome

A total of 173 participants (85%) in the usual-care group and 214 (87%) participants in the cognitive rehabilitation group were included in the primary analysis of the primary outcome at 12 months. Three participants in the usual-care group and two participants in the cognitive rehabilitation group who completed questionnaires at 12 months were not included in the primary analysis of the MSIS-Psy because either they completed the questionnaire > 15 months after randomisation (n = 4) or there was more than one item missing on the MSIS-Psy (n = 1).

Participants in both groups with no primary outcome were slightly younger, more likely to report having relapsing–remitting MS and had poorer scores on cognitive assessments and questionnaires at baseline (see Appendix 6, Tables 30 and 31).

Relative/friend questionnaire follow-up

At 6 months relative/friend questionnaires were completed for 158 participants (77%) in the usual-care group and 185 participants (76%) in the cognitive rehabilitation group. All of the 6-month relative/friend questionnaires were completed within 9 months of a participant’s randomisation.

At 12 months, relative/friend questionnaires were completed for 147 participants (72%) in the usual-care group and 171 participants (70%) in the cognitive rehabilitation group. One relative/friend questionnaire in the usual-care group and three relative/friend questionnaires in the cognitive rehabilitation group were completed > 15 months after the participant’s randomisation.

Unblinding at follow-up visits

Research assistants reported being unblinded prior to the follow-up for 17 participants at 6 months (4% of completed visits) and for six participants at 12 months (2% of completed visits). The percentage of RAs who reported being unblinded prior to the visits were similar in the two groups at each follow-up time point.

Research assistants reported being unblinded during the visit more often in the cognitive rehabilitation group than in the usual-care group. At 6 months, RAs reported being unblinded during the visit for 36 participants (16%) in the cognitive rehabilitation group and for 16 participants (9%) in the usual-care group. At 12 months, RAs reported being unblinded during the visit for 17 participants (8%) in the cognitive rehabilitation group and for six participants in the usual-care group (3%).

Overall, there was little agreement between the RA opinion of treatment allocation and the actual treatment allocation as assessed using the kappa statistic. The kappa values were 0 at the start of the 6-month visit, 0.17 at the end of the 6-month visit, 0.02 at the start of the 12-month visit and 0.07 at the end of the 12-month visit. Details of the RA opinion of treatment allocation at the start and end of the visit are shown in Appendix 7, Table 32.

Primary analysis

A total of 387 participants were included in the primary analysis at 12 months (86%); 376 of these completed the MSIS-Psy at baseline. There was a mean of 4.8 participants per cognitive rehabilitation group who had MSIS-Psy data at the 12-month follow-up.

There was no evidence of a difference between the two groups in the MSIS-Psy score at 12 months (Table 6). The adjusted difference in means was –0.6, with 95% confidence interval (CI) –1.5 to 0.3, which does not include the 2-point difference assumed to be clinically important on version 2.0 of the MSIS-Psy.

There was no evidence that the assumptions for the model were not met. The estimated ICC coefficient for the MSIS-Psy score from the model was 0 in the cognitive rehabilitation group.

Sensitivity analyses

The sensitivity analyses for the primary analysis were all consistent with the primary analysis, showing no clinically important difference between the groups in MSIS-Psy score at 12 months (Table 7).

Secondary analysis

The complier average causal effect estimate of the adjusted difference in means using the observed data was –0.7 (95% CI –1.7 to 0.4, n = 387), and using the multiply imputed data it was –0.6 (95% CI –1.7 to 0.4, n = 449), based on attending three or more cognitive rehabilitation sessions.

Subgroup analysis for the primary outcome

There was no evidence of a difference in the effect of cognitive rehabilitation according to MS type (p-value for interaction effect, 0.79), cognitive problems at baseline assessed on the MSNQ (p-value for interaction effect, 0.71), memory function at baseline using the Doors and People overall age-scaled score (p-value for interaction effect, 0.92) or speed of information processing at baseline using the Symbol Digit Modalities Test from the BRBN (p-value for interaction effect, 0.38) (Figure 2; see also Appendix 8, Table 33).

FIGURE 2.

Adjusted difference in mean MSIS-Psy score according to subgroup. DP, Doors and People; SDMT, Symbol Digit Modalities Test.

The MSIS-Psy score at 6 months

At 6 months, the adjusted difference in mean MSIS-Psy score and 95% CI favoured the cognitive rehabilitation group (Table 8). However, the 95% CI does not include the 2-point difference that is assumed to be clinically important.

Secondary outcomes assessed on questionnaires

At 6 and 12 months, the differences between the two groups favouring the cognitive rehabilitation group were observed for participant-reported frequency of memory problems assessed using the EMQ-p and mood using the GHQ-30 (Table 9). The EMQ-r scores (relative/friend report of a participant’s frequency of memory problems) were lower for the cognitive rehabilitation group, with an effect size for the difference in means between the two groups of a similar magnitude to the participant report (see Table 9).

There was no evidence of a difference between the groups in fatigue, health status (based on the EQ-5D VAS score), physical impact of MS on quality of life, or carer strain at 6 or 12 months (see Table 9).

The percentage of participants in employment (full or part time) was ≈30% at baseline and at 6 and 12 months in each group (Table 10). Further details on employment status can be seen in Appendix 9, Tables 34 and 35.

The mean scores on the GNDS, to assess disability due to MS, at 6 and 12 months were similar to baseline in both groups (see Table 9). The percentage of participants reporting that they had had a relapse in the previous 6 months was also similar in the two groups at follow-up (see Table 10). Details on the number of reported relapses can be seen in Appendix 10, Table 36.

Cognitive abilities

Scores from the BRBN, Doors and People, and Trail Making tests are shown in Table 11. There was no evidence of a difference between groups at 6 or 12 months for six of the eights tests compared on the BRBN. On the selective reminding test, there was evidence of higher total recall scores at 6 months and higher delayed recall scores at 12 months in the cognitive rehabilitation group. However, there was no evidence of a difference between groups in the selective reminding test total recall score at 12 months or the selective reminding test delayed recall scores at 6 months.

On the Doors and People test, there was evidence that the combined verbal scores were better for the cognitive rehabilitation group than for the usual-care group at 6 and 12 months. However, there was no evidence of a difference between the groups for the combined visual score or total forgetting score at either time point.

There was no evidence of a difference between the two groups in the Trail Making Test.

Epistemological position

Participant perspectives provide us with unique access to understand their thoughts, feelings and experiences about the services they received and the impact this had on their daily life. With respect to the provision of cognitive rehabilitation services (or lack thereof), we assumed that a ‘truth’ about its effectiveness could be approximated through the investigation of patient feedback. This is in line with a critical realist epistemological position. This position allows for the acceptance of the existence of a reality independent of human conception; human experience can know aspects of this reality, but it is fallible. 134 Thus, we acknowledge that the self-reports collected as part of this qualitative study are fallible; however, they can be viewed as an approximation of reality until better information is available.

Sampling

We aimed to recruit participants from all participating sites to obtain a wide range of views and experiences. We recruited eight participants (four from the intervention group and four from the control group) each from four of the sites to interview, as planned. However, as an additional site was added part way through recruitment, we included four interviews from this site: two from the intervention and two from the control group.

Participants were selected based on the maximum variation sampling technique135 to recruit individuals with different self-reported cognitive impairment levels (based on the EMQ) and self-reported demographic characteristics. Once an initial group of participants was recruited, subsequent participants were selected on the basis of how ‘different’ they were from the initial group, in order to have a diverse sample.

Sample size

There is no agreed sample size calculation for qualitative studies, so we adopted a pragmatic approach to interview as many participants as we could over a defined time frame, and based on our available resources. However, in deciding what was ‘pragmatic’, we took account of the sample sizes reported in other similar studies. For a similar trial of memory rehabilitation in people with traumatic brain injuries,95 we interviewed 32 participants (16 from the control group and 16 from the intervention group), and this provided us with sufficient data to answer our research questions (which were similar to our current research questions). We also sought information from two meta-syntheses: (1) of cognitive rehabilitation for people with MS,136 which had sample sizes of between 31 and 98 participants (it must be noted that some of the larger studies used focus groups and content analyses of written texts) and (2) of group-based memory rehabilitation,137 which had sample sizes of 10 to 38 participants. Therefore, we were confident that with 36 participants we would have rich data to allow us to explore with sufficient depth the areas we were interested in. We did have the facility to recruit a few more participants if we were not satisfied with the richness of the data from the 36 interviews. We did not look for data saturation, but our conceptual depth criterion was one of sufficiency. 138 We operationalised conceptual depth based on work by Nelson138 in terms of (1) the range of evidence (or quotations from different respondents), (2) a complex network of connection between themes, (3) subtlety in concepts, (4) resonance with extant literature and (5) external validity. 138 We aimed to obtain sufficient data within each cell of our framework matrix that related to each of our domains (or ‘conceptual categories’).

Procedure

Participants who had consented to be contacted about the qualitative study from the CRAMMS trial were considered for inclusion. We conducted one-to-one feedback interviews with participants enrolled in the trial within 2 months of completion of the 6-month follow-up assessment. To reduce social desirability bias, the feedback interviews were conducted by two individuals who were not involved with the assessments or intervention aspects of the trial.

The interviewers were a PhD student (Olga Klein) who was researching cognitive rehabilitation implementation in MS, and a carer of a person with MS. We decided to include a patient and public involvement (PPI) member as an interviewer based on our PPI group’s feedback to consider more ambitious ways to include PPI in our research. We also wanted to determine how the interviews conducted by both parties differed. Both the PhD student and the PPI interviewer were trained to conduct the interviews and use the interview schedule by one of the co-applicants (RdN).

Two semistructured interview schedules were developed based on extant literature and in relation to the content of the groups that participants attended. The interview schedules focused on the following domains:

• general experienced cognitive problems

• advice received about managing cognitive problems (before and during the trial, but not as part of the trial)

• treatments received to manage cognitive problems from the point participants were recruited to the point of the interview

• strategy use before the trial

• changes of cognitive abilities in the past 6 months (if any)

• what would have made the participants’ experience of being involved in the trial better.

The interview schedule for the control group included two additional specific aspects:

1. whether or not the participants had received any form of cognitive rehabilitation during the trial

2. participants’ experience and thoughts on being part of the control group.

The interview schedule for the intervention group focused on some specific aspects:

• most and least useful aspects of the cognitive rehabilitation programme

• strategy use after the trial, and impact on daily life activities

• participants’ experience and thoughts on being part of the intervention group

• overall experience of the cognitive rehabilitation programme.

The interviews were conducted over the telephone. Before the interview began, the interviewers reminded the participants of the purpose of the interviews, reassured them about confidentiality and told them that they could pause the interview or choose not to answer questions if they did not wish to. They were also told that the interview was going to be audio-recorded and transcribed verbatim. Transcription was conducted by a professional transcribing service. Transcripts were checked against the original audio-recording and any errors were corrected.

Advice received

No cognitive rehabilitation

Besides the information given about external memory aids, no participants received any other form of cognitive rehabilitation:

Well no, b– but, er, apart from that, my – my usual care is just my annual review . . . Where the doctor sort of talks to me about how I’m feeling and sort of has a bit of a look at my sort of, um, motor function, if you like. Watch me walk, look you know . . . Checks my reflexes and what have you. And – and that, that’s the only sort of usual care apart from – from going to the – on the CRAMMS [. . .] that I received.

ID 02

No, not a single thing.

ID 16

Self-generated strategy use

Generally, the majority of participants used external memory aids before they were recruited to the trial. These strategies included writing lists, keeping a calendar with appointments, and asking family members or friends to remind them of appointments or tasks they needed to complete. These tasks were self-generated, in that participants decided to try them out themselves:

Well, as I said, I did already, um, have – use lists for everything [. . .] And, erm, just to not get upset by it really [. . .] (and look after) myself.

ID 02

You know, erm, yeah. I write on my hand, if there’s some emergency, if an appointment I have to keep physically.

ID 10

Participants changed tack when they found that some of the strategies they were accustomed to were no longer providing them with the results they expected:

So, ‘cause that [writing lists] was sort of starting to not work, erm, I’ll say to my friend I’m like, ‘Can you remind me?’

ID 22

Improved

All six interviewees who stated that their cognitive abilities had improved over the previous 6 months were from the intervention group:

I think I’ve improved.

ID 12

I say that the CRAMMS has helped.

ID 07

I think that actually since I’ve been on – on the CRAMMS sort of meetings, they’ve [cognitive problems] improved.

ID 02

Stayed the same

Some participants from both the control and intervention groups stated that their cognitive abilities had stayed the same over the previous 6 months. However, the numbers of those in the intervention and control groups endorsing this view differed: 11 participants from the intervention group and six participants from the control group. Furthermore, although participants in the intervention group reported that their cognitive abilities had stayed the same, they did feel that they managed better (using strategies) as a result of the intervention:

No, I don’t think so. I don’t think they’ve [cognitive problems] got any better or any worse, about the same. It’s just that I’ve put strategies in place to help me with those things.

ID 04

No. I don’t know. I don’t know [whether cognitive problems have got better or worse]. Since I’ve been able to do, since being on the course thing, . . . I’m able to remember . . . that I just could not before . . . so that’s really good.

ID 14

I think I’m remembering a bit more than I was before. But a lot of it is just writing things down, doing different things.

ID 16

What is interesting about these quotations, all from participants from the intervention group, is that although they reported not knowing or not thinking that their cognitive problems had reduced, they do report improvements in daily life (‘I’m able to remember’, ‘I’m remembering a bit more than I was before’). However, they minimise the impact of these changes because they are using cognitive strategies to compensate for their deficits (‘It’s just that I’ve put strategies in place’, ‘But a lot of it is just writing things down’). Thus, they do not see this as improvement.

However, by contrast, those in the control group only reported not seeing any change over time:

I don’t think so, no. It might be slightly worse but nothing major. I’ve not noticed anything major.

ID 32

No, about the same.

ID 35

Actually, they probably stayed the same, to be honest.

ID 28

Worsened

Thirteen participants stated that they thought that their attention and memory had worsened; of these, all but one were in the control group:

. . . I’m forgetting more and more.

ID 31

Yes, it’s definitely got worse, yeah.

ID 36

Er, yeah, erm, my memory is getting worse definitely. I forget things within, within moments really.

ID 25

Many of these participants used words like ‘definitely’ to indicate that they had no doubt that their cognitive abilities had deteriorated. One person from the intervention group reported a worsening of their cognitive abilities:

I think they [cognitive problems] have got worse. [. . .] I am struggling with things, like I said, at work. And I am being now – I’m being performance managed because I am struggling because of my memory.

ID 08

Increased strategy use

All participants in the intervention group reported that they were using significantly more strategies as a result of the cognitive rehabilitation programme. Although several were using external memory aids prior to joining the programme, the majority reported that they were using considerably more internal memory aids after the intervention, to help them to memorise things and forget less. Some of the strategies that participants adopted were putting things in the same place (e.g. keys, mobile phone, wallet), using descriptions when they could not recall the name of an object, using pictures to recall events or things, grouping items on a grocery list (vegetables, fruits) and making little stories about things they wanted to remember.

We investigated further why they were using more strategies after having attended the intervention groups.

Psychoeducation

Participants commented that the intervention provided them and their families with information and support to accept, better understand and cope with their cognitive problems.

Effect of group format

All participants reported that the group format was a very useful aspect of the intervention. Participants commented that sharing experiences and giving advice in a group setting was very helpful to them. In addition, learning about other people’s issues made participants feel relieved that they were not the only ones who experienced these problems, and they felt comfortable to share their experiences because they felt understood by the others in the group.

Workbook

We enquired about how participants made use of the workbooks they were provided with as part of the intervention. Participants made use of the workbook in different ways.

Improvements seen in daily life activities

Participants in the intervention group reported that they had increased the level and types of activities they were involved in:

Yes, I think, remembering, erm, much more detail about things, and attacking, – attacking things, if that makes sense.

ID 10

Rather than me just sitting and thinking about things, it’s [the intervention] made me more active. Rather than thinking about something, it’s a case of rather than thinking I’ll do that later, I now think I’ll do that now. [. . .] So it’s made me get up and go really.

ID 05

I’ve started doing things myself more [. . .] I play games. It tends to helps me, certain games, trying to remember things and I’m writing things down more and I’ve got diaries and tend to use my phone a lot more now. My social life, I tend to spend a lot more time going to places now than my wife. I’ve just joined up as a member of the Man[chester] United fan club so I tend to go to a lot of football matches every other week. I tend to go to Liverpool to see my family and different things like that. And just spending time with the kids really, doing things with the kids.

ID 16

Positive appraisal of therapist skills

Many participants commented on the skill with which the therapists facilitated the group:

But I think the ladies [therapists], there were two of them over the 10 weeks who did it, who work for the CRAMMS units, were very good. They really were good at what they did, and they put the whole idea of the process to us very clearly and very succinctly, so I was very happy with that.

ID 03

No, I mean, no, the whole – as I said to you – the whole experience, I don’t think you could have done it any better. [. . .] And it was made really interesting [by the therapists] by having flip charts and you know things – there were little exercises that we could do that really were interesting. So, I think the whole experience was quite enjoyable.

ID 18

Suitability of venues

Although the venue was dependent on the site the participant was recruited from, most participants stated that the venue was easy to get to and that it was appropriate for them:

Lovely premises, oh thank you very much. For the rented room, oh yeah. They’ve got a balls room and oh, it was just beautiful. Erm, we were all these professionals, sat round this big posh table with these lovely chairs.

ID 10

Although some participants who used wheelchairs found that some venues were not easy to access, most participants found them easy to find and get to by public transport:

. . . there’s a taxi company that I use all the time. And they took me all the time and picked me up. Because I’ve got quite a good rapport with the taxi company, they’re always there to support me anyway. So I didn’t have any problems getting there [venue] at all.

ID 05

Having cognitive assessments

Participants in both the intervention and control groups mentioned that it was good to have the cognitive assessments completed both at baseline and at follow-up assessments. This allowed them to get an accurate idea of their cognitive abilities and profile:

I think it [cognitive assessment] highlighted certain issues I was not quite happy with in the first place, so it was good to actually do that, to do the study [. . .] With neurological things, you need to test yourself and see what your capabilities are with certain things. So it’s always good to test you. I quite enjoyed that, not enjoyed as such, because certain things I think I don’t like doing that but you have to do it. It was good to try them.

ID 34

It was just me and one other – she came to my house and did tests on me. Yeah that, yeah, and that took about – she came twice. [. . .] Yes, they [cognitive assessments] were very helpful, yes.

ID 38, talking about being in the control group

Possibility of receiving cognitive rehabilitation

Participants joined the trial with the expectation that, if they were allocated to the intervention group, this might be of benefit to them:

I was rather pleased I was in that [intervention group], because I thought, if you’re in a control group, there’s nothing going to make you improve. If you’re in the intervention group, there’s a chance that you’ll improve through the intervention.

ID 15

Actually, I was really looking forward to it, because I felt it was a chance to meet up with other people and for us to actually share our experiences together. So it was one of those times where you could actually speak to other people as well. So I was really looking forward to it. And I was really pleased when I was nominated to be on it.

ID 05

I was happy that I was going to get help [. . .] talking to different people that have got the problems I have. I think being there [in the intervention] helped a lot. I learned quite a lot from there so it’s not like, even though I wasn’t expecting much from it, but it helped in a way that I knew people out there were trying to do stuff and it, like, encouraged me to try and do more there as well. It made me feel a lot better.

ID 16

Altruism

Participants reported that, although they preferred to be part of the intervention group, they recognised the need for some people to be in the control group. For such participants, it was a sense of altruism that motivated them to join the trial and remain in it even when they were allocated to the control group:

I understand that [there had to be a control group]. That [being part of the intervention group] was the only thing I would say that would have helped me. I’m fine being in either group just to try and help and everything, but, yeah that [not being part of the intervention group] was the only thing really. That [finding out that he was part of the control group] was fine. I don’t mind because as long as I can help in some way, that’s why I do these trials, to help further with MS and help whatever way I can.

ID 24

Refresher/booster sessions

Some participants felt that a refresher/booster session would have been useful at a later point because they felt that the 10 weeks went by quite quickly. A few such sessions, they felt, would enable them to sustain what they had learnt:

But I’m just trying to think whether there could be anything else just to keep it going for a short, not another 10-week course or anything like that, just something that could have followed it up. Perhaps another meeting or another two meetings just to go through it again and even think of anything new that might come up in terms of techniques.

ID 03

Option to download workbook

Some participants also suggested that it would be useful to have access to the workbook online, but also reiterated the importance of having someone to go through the workbook with them:

I think it would [be a good idea to have a downloadable workbook], but I think it would have made a lot more sense for someone to explain it when you couldn’t understand certain things.

ID 16

But we ask, erm, that if there is a workbook that could be printed on the web, on the NHS, you know on the website, that will be able to help people, the research programme. It will be useful for people to dip in and look at memory issues and how they can be addressed and how they can probably manage them, yeah so that would be useful [. . .] Yes, I don’t know whether its licensed or not, but it would be really useful, erm, yeah. No I think it, I think, – I think it would be a shame to lose that effort of compiling it and the knowledge that was used to do it, and the fact that I felt that, for me personally, I really benefited from it and, – and use it on a daily basis, and now its hardwired with myself and equally for any others, erm, you can. ‘Cause, er, lots of people have memory problems, you don’t just have to have MS.

ID 13

Adding mindfulness to help focus

Two participants highlighted the importance of mindfulness and how it could help participants to be more in the ‘here and now’ and focus on the things they are doing. They felt that it would help to incorporate this aspect into the intervention programme:

Having mindfulness to call is really useful to keep focus on what your aim is to do, so that’s where I come with that really . . . ‘cause, I think, I think that if we all, well I don’t mean that I’m like a Buddha, but if we [are] all calm and serene and the, we not on edge or we not all frustrated, we, we just accept, erm where we are, a bit like acceptance and commitment therapy really, where we accepting what is happening to us, err, or, yeah, we accept what’s happening to us and then the different stage down the line, we know how to accept that second stage, we know and it continues, and so I think that, errm, part of that is having all the tools if you want to use in your tool bag.

ID 13

I don’t really utilise them [strategies to compensate for cognitive problems] as I should do really. I know deep down I should literally pause and think about it and just clear my mind and use mindfulness and stuff like that, but in the grand scheme of things I don’t as such. I’m fully aware I can do more.

ID 34

Effectiveness data

The economic analysis used the primary clinical outcome (MSIS-Psy score), as summarised in Chapter 2, in a cost-effectiveness analysis. The EQ-5D-5L and MSIS-8D (Multiple Sclerosis Impact Scale – 8 Dimensions; derived from the MSIS, as described in Benefit measurement and evaluation) were used for cost–utility analyses. Cost-effectiveness was evaluated at the primary end point of 12 months and at 6 months to evaluate any short-term effects of cognitive rehabilitation.

Benefit measurement and evaluation

Quality-adjusted life-years were generated using utility scores derived from the EQ-5D-5L index. The EuroQol-5 Dimensions (EQ-5D) is a standardised instrument for use as a measure of HRQoL. 106 The study employed the five-level version of the questionnaire (EQ-5D-5L). Individual-level utility scores were obtained at each assessment point using the EQ-5D-5L questionnaire. In August 2017, NICE issued a position statement advising that the five-level valuation set for England is not recommended for use in deriving utility values and instead advise that the validated mapping function to the three-level version be used for reference-case analysis. 145 The utility scores derived from the direct valuation set for the five-level version were used in a sensitivity analysis. Utilities were summarised and QALYs for each participant in the intervention and control groups were calculated over 6 and 12 months using the area under the curve approach (assuming linear interpolation). The mean difference per participant in QALY gain/loss between the cognitive rehabilitation and usual-care groups is presented over 6 and 12 months.

The Multiple Sclerosis Impact Scale (MSIS-29) was used to derive the MSIS-8D using published methods. 146 MSIS-8D is an 8-dimensional (general physical function, mobility, employment, social function, fatigue, cognition, depression and general emotional well-being) HRQoL measure used to produce a MSIS-8D utility score. Each item of the MSIS-8D has four levels, describing 65,536 health states. A descriptive analysis of MSIS-8D responses was undertaken; the responses were summarised and used to generate QALYs at 6 and 12 months.

The primary outcome for the CRAMMS trial was the psychological impact of MS on everyday life, as a reflection of HRQoL measured using the MSIS-Psy, version 2.0. As higher scores indicate greater psychological impact of MS on everyday life; an improvement would be represented by a negative difference in the MSIS-Psy score.

Intervention costs

Resource use resulting from the implementation of the CRAMMS trial intervention (including materials, consumables, staff time and training) was estimated through interviews and direct communications with clinical staff involved in the trial and the trial team, and by using data collected during the trial (e.g. supervision records). No travel costs were included for training or delivery of the CRAMMS trial intervention because, in routine clinical practice, cognitive rehabilitation training would be delivered at the base site of the clinical psychology team.

The resource use and costs associated with the delivery of the CRAMMS trial intervention were estimated from published sources and discussion with the trial team. Unit costs of materials and consumables were obtained directly from financial records collected by the trial team. Staff costs were obtained from published unit costs. 147 The costs of additional catch-up sessions delivered to participants who missed one or more previous sessions were based on the assumption that these required 30 minutes and were delivered as a group, rather than on an individual basis.

The assumptions were made from consulting with clinical experts on the trial team and the impact on the results was tested as part of the sensitivity analysis. For the base-case analysis, the costs of the cognitive assessment to determine eligibility for the CRAMMS trial were excluded as these were assumed to be part of usual care. The impact of including the cognitive assessment as part of the CRAMMS trial intervention was tested in a sensitivity analysis.

Costs of health-care and Personal Social Services resource use

Health-care resource use [including primary care consultations, accident and emergency (A&E) visits, outpatient appointments and inpatient stays] and Personal Social Services use (e.g. home adaptations) were collected using data from a health and social services questionnaire that had been developed and used in a previous MS trial. 107 Information was collected at baseline, and at 6 and 12 months’ follow-up to assess the difference in profile of health-care resource use in the intervention group compared with the usual-care group.

Costs were assigned using published unit costs. 147–149 Outpatient visits and inpatient stays were costed individually according to the reasons for health-care contact, length of stay and specialty/department visited recorded in the UHSSQ.

A description of unit costs associated with health and personal and social care resource use are presented in Appendix 13, Table 42.

Costs of medications

Participants were asked to recall the medications (including drug name, dosage and frequency) they had used over the previous 3 months at baseline and at follow-up assessments. The UHSSQ did not specify when medication was started or stopped, or how long the medication had been taken for. It was assumed that the medication was taken or used for the whole 3-month period [or, when applicable, the prescribed length from the British National Formulary (BNF)149]. This may have overestimated the cost of medications for some participants, but accounts for non-compliance to the length of prescription in both the usual care and the cognitive rehabilitation groups. For medications with multiple indications from the BNF,149 an assumption was made regarding the most probable indication applicable to the patient group or most common indication for the general population. Clinical members of the trial team checked all medication data and assumptions made before analysis.

The health-care costs in both the intervention and control groups were summated and presented for each of the time points with differences in resource utilisation between the CRAMMS trial intervention at 6 and 12 months calculated, using methods consistent with the statistical analysis of outcomes (as described in Chapter 2). The difference per participant in costs (including 95% CIs) was calculated, using non-parametric bootstrapping.

Missing data

Issues concerning missing data are particularly prevalent to health economic analyses. Missing items relating to health-care resource use may undervalue costs, whereas missing outcome data may be intrinsically linked to effects. Missing data may relate to item non-response, where a questionnaire is partially incomplete, or unit non-response, where all information is missing. 150

Missing questionnaire items in outcome measures

Partially completed questionnaires for the health outcomes used in the economic analysis were handled and the total score was calculated using the same approach as in the statistical analysis (see Chapter 2). If the number of missing items exceeded that for which total scores were calculable, the questionnaire was treated in accordance with unit non-response procedures. Any missing items in the EQ-5D-5L descriptive questionnaire were not individually imputed; utility values were instead imputed in accordance with unit non-response procedures.

Missing baseline data

If baseline scores for the outcome measures remained missing (e.g. the questionnaire was not usable, even taking into account the above rule for missing data), data were imputed using the mean baseline score at each study site. This enabled all participants to be included in the regression analyses of the outcome score, ensuring consistency with the statistical analysis. These simple imputation methods are superior to more complicated imputation methods when baseline variables are included in an adjusted analysis to improve the precision of the treatment effect. 116

Missing outcome data at follow-up

For the base-case health economic analysis, data were assumed to be missing at random (MAR). Multiple imputation using chained equations was undertaken to impute missing items. Imputations were combined following Rubin’s rules. 151 Owing to the predefined range of outcome scores, including utility derived from the EQ-5D-5L, truncated regression methods were used to ensure that imputed values were consistent with the valid range. Predictive mean matching (PMM) was used to impute missing cost data, ensuring that imputed values were consistent with observed data.

An available-case analysis (ACA) was conducted on both outcomes and costs as part of a sensitivity analysis. ACA is generally biased under MAR or missing not at random assumptions, with the ACA mean unbiased only if missingness in the parameter of interest is independent of its value. ACA examined the robustness of conclusions from the imputed data compared with those from the observed data, after conducting the procedures outlined in handling missing questionnaire items.

Missing service use data

The UHSSQ was used to estimate the resources used and costs of health service usage by participants because of MS. Participants were asked to complete the relevant boxes with the number of contacts received over the previous 3 months for each item, or to place a ‘00’ in the relevant box should no usage of this item have occurred. Item non-response may, however, represent either the individual failing to record the non-zero number of contacts or no contact. Missing UHSSQ data were thus treated according to the following rules:

• If one or more items were completed (values of ‘0’ or greater) on the UHSSQ, then the questionnaire was assumed to be fully completed and any missing items were imputed with zeros.

• If the UHSSQ was marked as ‘not done’ or otherwise fully incomplete, appropriate imputation methods were used. At baseline, data were imputed using the site mean akin to the process outlined above. At the 6-month and 12-month follow-ups, data were imputed using multiple imputation or ACA methods, with imputation conducted on the total for each contact type (e.g. general practice and community services, social services, medications).

Because of the typically skewed distribution of cost data, in which the UHSSQ was marked ‘not done’, PMM was used to impute missing costs under multiple imputation using the methodology outlined in Missing service use data.

Adjustments for timing of costs and benefits

As the trial period did not exceed 12 months, discounting was not applied to the within-trial analysis of either costs or outcomes.

Cost-effectiveness analyses

Cost–utility and cost-effectiveness analyses were undertaken to estimate the incremental cost per QALY gained and the incremental cost per improvement in MSIS-Psy score based on the primary clinical end point at 12 months. The multiple imputation approach to missing data for costs and QALYs was used for the primary cost–utility analysis. For the cost-effectiveness analysis, a similar methodological approach to the statistical analysis outlined in Chapter 3 was conducted to ensure consistency with the analysis and reporting of the primary outcome.

Results of the comparative analysis of incremental costs and effects can be summarised in terms of ICERs. An ICER can be represented as:

where C₁ and E₁ are the costs and effects of the intervention arm, C₀ and E₀ are the cost and effects of the control arm and ΔC and ΔE are the incremental costs and effects of the intervention compared with the control. For the cost–utility analysis using QALYs, ΔE > 0 would represent an improvement favouring the intervention group. For the cost-effectiveness analysis on the MSIS-Psy, higher scores represent a greater psychological impact of MS on everyday life; therefore, ΔE < 0 would represent an improvement favouring the intervention group.

If the intervention is less costly and more effective than usual care, then the intervention is classified as dominant and no ICER would be calculated. Conversely, if the intervention is more expensive and less effective than usual care, the intervention is considered to be dominated by usual care. ICERs are calculated when a trade-off between cost and effect is evident, for example when the intervention is more costly but generates an improved health effect (QALY gain) compared with usual care.

The ICER is reported to determine the cost-effectiveness of cognitive rehabilitation compared with usual care and to aid decision-making. NICE reports a base cost-effectiveness threshold of £20,000 per QALY, but cost-effectiveness is a spectrum rather than a dichotomy, with the threshold changing according to the circumstances. The ICERs are presented to assist with the decision-making process and are not an absolute statement on whether or not the intervention can be deemed cost-effective. No established willingness-to-pay (WTP) threshold for the MSIS-Psy or other secondary outcomes is available.

Net monetary benefit (NMB) was also calculated. NMB is a summary statistic that represents the value of the intervention in monetary terms when a WTP threshold for a unit of benefit is known. 152 NMB for the cognitive rehabilitation and control groups is based on the following equation:

where ΔE and ΔC represent the change in effects and costs and λ represents the WTP threshold. Incremental NMB was calculated as the difference in NMB between the intervention and usual care groups. If the incremental NMB is positive, the intervention can be identified as cost-effective at the given WTP threshold relative to usual care.

Allowance for uncertainty

Sensitivity analyses were undertaken to account for the uncertainty in the parameters used in the cost-effectiveness analyses. Deterministic one-way sensitivity analyses (OWSAs) were undertaken to examine the impact of changes in key parameters on ICERs by modifying the value of one parameter at a time within a plausible range (e.g. upper/lower 95% CIs, ±30% to key parameters). A threshold analysis was conducted to determine the required changes to affect base-case findings (e.g. changes in QALY gains to provide a cost-effective result).

Bootstrap resampling was undertaken to analyse the joint uncertainty of parameter estimates. A minimum of 1000 resamples was used. The results of the probabilistic sensitivity analysis are illustrated using cost-effectiveness planes and cost-effectiveness acceptability curves (CEACs). The probability that the intervention is cost-effective at various WTP thresholds was examined and expressed as a percentage. Where dominance is observed, the probability of dominance is presented.

Subgroup analysis

In the health economics plan, subgroup analysis was to be conducted only with sufficient evidence obtained from the exploratory analyses of interaction for the primary outcome (see Chapter 3) to justify further investigation on the cost-effectiveness findings. The trial team concluded that no subgroup analysis was required.

Longer-term cost-effectiveness

It was originally proposed to construct a decision-analytical model to extrapolate the findings from the trial to estimate the cost-effectiveness of cognitive rehabilitation compared with usual care beyond the trial horizon of 12 months. The modelling exercise was based on extrapolating the results from the within-trial analysis to longer-term cost per QALY estimates supplemented with data sources from the literature and, when necessary, clinical opinion from the trial team. A model plan is presented in Appendix 14, Figure 11.

Results

The costs for the economic evaluation consisted of two broad categories:

1. the implementation and delivery of the group-based cognitive rehabilitation programme

2. health and personal and social care resource use.

Intervention costs

The resource use and associated costs of delivering the intervention are summarised in Table 20; a detailed account of each component is presented in Appendix 15, Table 43. The primary analysis was conducted excluding costs associated with cognitive assessment, which the trial team agreed to be incurred by both usual care and cognitive rehabilitation groups. The total cost of cognitive rehabilitation was calculated at £209.

The main cost drivers of the intervention cost were training, the delivery of the sessions, supervision and administration. One clinical psychologist (band 8a, £62 per hour) delivered the training across all sites to APs (band 5, £33 per hour, one per site) on a one-to-one basis. Training was assumed to require 4 hours. No provision was included for travel costs and room hire associated with training as, in standard practice, this would take place on NHS premises. As detailed in Chapter 2, an AP delivered 10 cognitive rehabilitation sessions to each group individually. Each session lasted for approximately 1 hour and 30 minutes. An additional 30 minutes was required for both set-up and close-down; provision for 30-minute catch-up sessions was included for 137 of 450 sessions for those participants who missed one or more sessions previously.

To ensure that cognitive rehabilitation was delivered appropriately, provision for supervision is included in the cost of the intervention. Each AP routinely received supervision sessions with a clinical psychologist lasting 1 hour on a weekly basis, of which it was assumed that 30 minutes was attributable to the delivery of cognitive rehabilitation.

The delivery of cognitive rehabilitation required one administrator to set up each group, send invitations to participants and send reminders of upcoming sessions. It was assumed that an administrator (band 3, £25 per hour) would be required for 30 minutes per session. Additional costs of stationery, refreshments and providing an intervention manual to each participant were included.

Cognitive assessment is used to determine patient eligibility for cognitive rehabilitation and was a key cost driver in providing the intervention. From discussions with the TMG and clinicians, it was determined that cognitive assessment would form part of usual care and was not exclusive to the intervention group. This may differ on a site-by-site basis. It was decided that cognitive assessment would not be included in the primary analyses, but would be included in the sensitivity analyses.

Resource use and costs

The resource use for available cases is presented in Appendix 16, Tables 44–46. Resource use costs were summarised by:

• general practice and community nursing services

• hospital and community services

• therapy services

• social services

• A&E services

• other hospital services

• medications.

The key cost drivers for this group of participants were medications, and social services; however, for each component of the resource use questionnaire, outliers had a substantial impact on the mean value, with a few high-cost users increasing the average cost, resulting in high SDs. t-tests on differences between groups for each component of resource use were typically not statistically significant.

At 12 months, the cognitive rehabilitation group had medication costs that were £320.45 cheaper (95% CI –£928.61 to £287.71) than the medication costs of the usual-care group; however, the usual-care group had a much larger SD and a maximum medication cost of £34,929, compared with a maximum of £7395 for the cognitive rehabilitation group. Similarly, at 6 months, the cognitive rehabilitation group had medication costs that were £313.55 cheaper (95% CI –£860.86 to £233.77) than those of the usual-care group. However, the usual-care group had a much larger SD and a maximum medication cost of £26,516, which was over three times larger than the maximum medication cost for the cognitive rehabilitation group.

Excluding the costs of disease-modifying therapies resulted in a small increase in the difference in medication costs to –£332.33 (95% CI –£812.80 to £148.14) at 12 months, and a reduction at 6 months to –£181.13 (95% CI –£522.38 to £160.12); at both follow-ups the difference was not statistically significant.

Multiple imputation was used to impute missing costs. Imputation was performed on total costs for unit non-response after the procedures outlined in Chapter 2 regarding item non-response were performed.

The multiple imputation results in Table 21 show that, at 6 months, cognitive rehabilitation had total costs of £3144 and was about £20 cheaper than usual care. However, the difference was not statistically significant, with wide CIs observed. At 12 months, the mean total costs associated with cognitive rehabilitation were £5623; the difference between the two groups increased at 12 months, with cognitive rehabilitation about £575 cheaper than usual care. This indicates that cognitive rehabilitation was cost-saving compared with usual care; however, the differences in costs were not statistically significant at either 6 months or 12 months, with wide CIs at both follow-ups.

Further analysis was conducted to explore the drivers of costs in the two groups. Figure 3 presents a histogram of cumulative total costs for available cases at 12 months by intervention group.

FIGURE 3.

Histogram of cumulative total costs at 12 months. (a) Usual care; and (b) cognitive rehabilitation.

Figure 3 shows that at 12 months approximately 40% of participants in both the cognitive rehabilitation and usual care groups incurred total costs of ≤ £2500. Similarly, > 75% of participants in both groups incurred costs of ≤ £10,000. A few participants incurred costs in excess of £20,000, of whom nine were in the usual-care group and five were in the cognitive rehabilitation group. No participants in the cognitive rehabilitation group had costs in excess of £30,000, whereas there were two notable outliers in the usual-care group, with costs of £53,784, and £81,074. These outliers were investigated further. The participant with total costs of £53,784 incurred £24,809 in medication costs at both 6 months and 12 months, attributable to daily administration of teriparatide (Forsteo^®; Eli Lilly and Company, Indianapolis, IN, USA). The participant with total costs of £81,074 incurred £77,250 of costs due to 30 nights spent on a hospital ward as a planned admission (based on NHS reference costs148 for elective inpatient medical care of patients with MS). Two further participants in the usual-care group had total costs at 12 months in excess of £30,000. Excluding these two outliers from the ACA (see Appendix 17, Table 47) reduces mean total costs for the usual-care group to £5849.49 (SD 6190.98) and the adjusted difference in total costs is –£38.85 (95% CI –£1162.14 to £1084.45) (i.e. cognitive rehabilitation was slightly cheaper than usual care).

Outcomes

Utility values derived from EQ-5D-5L and MSIS-8D scores were converted into QALYs using linear interpolation and the area-under-the-curve method, with adjustment for baseline covariates including baseline utility, gender, MS type and site. In accordance with the methods for total costs, multiple imputation was used to impute missing outcome data for utility scores derived from both the EQ-5D-5L and the MSIS-8D. Multiple imputation was performed on items for which utility scores could not be calculated after the item non-response procedures detailed in Chapter 2 were performed. Results are presented in Table 22.

Table 22 shows the EQ-5D-5L scores at baseline and at 12 months. The mean utility value was slightly higher in both groups at 12 months than at baseline; however, there was no evidence of an observable QALY gain as a result of cognitive rehabilitation compared with usual care. The CIs ranged from –0.02 to 0.02, with no statistically significant difference between the two groups. Similar results were observed for EQ-5D-5L scores at 6 months.

We also examined HRQoL derived from the MS-specific MSIS-8D instrument using the modelling methodology presented by Goodwin et al. 146 Results are presented in Table 23.

Table 23 shows the MSIS-8D scores at baseline and at 12 months. The mean utility value was slightly higher in both groups at 12 months than at baseline. There was a numerically small QALY gain of 0.01 (95% CI –0.01 to 0.03) in the cognitive rehabilitation group compared with the usual-care group. A small baseline imbalance in MSIS-8D scores was noted, with the cognitive rehabilitation group having a slightly higher utility score than the usual-care group, and was included as a covariate in the analysis model. There was no statistical difference in QALY gains between the two groups. At 6 months, utility values in both groups were higher than at baseline, with a larger increase observed for the cognitive rehabilitation group than for the usual-care group. The adjusted difference in means for MSIS-8D and QALY gains, favouring the intervention group, was nearly statistically significant. This corresponds with the results for the MSIS-Psy in Chapter 3, with statistical significance observed at 6 months but not at 12 months.

The cost–utility analysis required the calculation of ICERs using the total costs and outcomes presented in Tables 21–23. The results for the cost–utility analysis are presented in Table 24.

Table 24 requires careful explanation. Cognitive rehabilitation shows numerical differences in costs at 6 and 12 months compared with usual care. With no evidence of any incremental effect on QALY gain between the groups (when the EQ-5D-5L is used), cognitive rehabilitation could be considered to be less expensive against no difference in effect (QALY gain) than usual care. When the MISIS-8D was used to generate QALYs, there was a numerically small QALY gain in cognitive rehabilitation compared with usual care, which (technically) presents cognitive rehabilitation as dominant over usual care (i.e. cognitive rehabilitation is less expensive and has a greater effect). A similar picture is presented when available cases are used to generate QALYs (EQ-5D-5L or MSIS-8D; see Appendix 17). For the MSIS-Psy, the negative effect is representative of an improvement for the cognitive rehabilitation group. This would, again, present cognitive rehabilitation as dominant over usual care for the primary outcome. However, across all scenarios, the CIs for both incremental costs and incremental effects span zero, and for the costs, CIs are wide. Given this, caution should be applied in interpreting these results.

Cost-effectiveness planes and associated CEACs were constructed using non-parametric bootstrapping (5000 replications with replacement) for the primary and secondary cost–utility analyses (cost per QALY) derived from the EQ-5D-5L and MSIS-8D, respectively.

Cost-effectiveness planes and CEACs were also constructed for the multiple imputation analysis on EQ-5D-5L and MSIS-8D QALY gains, and for the change in the MSIS-Psy. These are presented in Figures 4–6.

FIGURE 4.

Total costs and EQ-5D-5L QALYs at 12 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC. Reproduced with permission from Lincoln et al. 122 This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Reproduced with permission from Lincoln et al. 122 This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

FIGURE 5.

Total costs and MSIS-8D QALYs at 12 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC.

FIGURE 6.

Total costs and change in MSIS-Psy at 12 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC.

At 12 months, the cost-effectiveness planes (see Figures 4 and 5) show that cognitive rehabilitation was less costly and more effective than usual care; the distribution of bootstrapped estimates are centred in the south-east quadrant, which is consistent with the intervention dominating usual care. The estimates for MSIS-8D are centred further eastwards than those for the EQ-5D-5L. As higher scores on the MSIS-Psy indicate a greater psychological impact of MS on everyday life, a negative incremental effect represents an improvement in favour of cognitive rehabilitation, with the interpretation of the cost-effectiveness plane reversed on the x-axis compared with the analysis of QALY gains. The cost-effectiveness plane in Figure 6 has the largest proportion of estimates in the south-west quadrant; this is, again, consistent with the intervention being more effective and less costly than usual care.

The CEACs in Figures 4–6 show a high probability that cognitive rehabilitation was cost-effective compared with usual care at WTP thresholds of between £0 and £100,000 per QALY gain. For QALYs derived from the EQ-5D-5L, cognitive rehabilitation has an 84.8% probability of being cost-effective at a WTP threshold of £20,000 per QALY gain, and an 85.7% probability of being cost-effective at a WTP threshold of £30,000 per QALY gain. In comparison, for QALYs derived from the MSIS-8D, cognitive rehabilitation has an 86.7% probability of being cost-effective at a WTP threshold of £20,000 per QALY gain, and an 88.8% probability of being cost-effective at a WTP threshold of £30,000 per QALY gain. For both HRQoL measures, the probability that cognitive rehabilitation was cost-effective is very high at all WTP thresholds. For the MSIS-Psy, at a WTP threshold of £20,000 per point improvement, cognitive rehabilitation had a 60.6% probability of being cost-effective, and a 59.8% probability of being cost-effective at a WTP threshold of £30,000. However, caution should be given to interpreting these thresholds for the cost-effectiveness analysis as no established threshold exists for improvement in MSIS-Psy score.

Cost-effectiveness planes and CEACs were also constructed on multiply imputed data at 6 months to investigate earlier effects of the intervention on HRQoL; results are presented in Figures 7–9.

FIGURE 7.

Total costs and EQ-5D-5L QALYs at 6 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC.

FIGURE 8.

Total costs and MSIS-8D QALYs at 6 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC.

FIGURE 9.

Total costs and MSIS-Psy at 6 months (multiply imputed analysis). (a) Cost-effectiveness plane; and (b) CEAC.

At 6 months, the cost-effectiveness plane estimates in Figures 7 and 8 for the EQ-5D-5L and MSIS-8D are, again, primarily clustered on the x-axis, and to the east of the y-axis. The distribution of bootstrapped estimates for the MSIS-8D was slightly further east of the y-axis than for the EQ-5D-5L, consistent with the greater level of effectiveness observed for the disease-specific measure of HRQoL. The CEACs show that, at WTP thresholds of ≥ £20,000, a high probability of cost-effectiveness of cognitive rehabilitation is observed. For the MSIS-Psy, (see Figure 9) the distribution of bootstrapped estimates is located mainly to the west of the x-axis, consistent with the intervention being more effective than usual care. At a WTP threshold of £20,000, the intervention has an 83.9% probability of being cost-effective, which rises to 84.1% at a WTP threshold of £30,000.

Sensitivity analyses

To assess the impact of the base-case findings to both methodological and parameter uncertainty, a series of sensitivity analyses were conducted, including an ACA and a multitude of OWSAs.

The base-case analysis used multiple imputation to address missing data for both costs and outcomes. To examine the impact that imputation had on the findings, an ACA was conducted on the data directly observable from the questionnaire responses. Tables summarising cumulative costs, and QALYs derived from the EQ-5D-5L and MSIS-8D are presented in Appendix 17, Tables 47–49.

For available cases, Appendix 17, Table 47, shows that, at 12 months, the mean total costs associated with cognitive rehabilitation [£5885 (n = 208)] were over £800 lower than those of usual care [£6574 (n = 170)]. Consistent with the primary multiple imputation analysis (see Table 21), this indicates that cognitive rehabilitation was less costly than usual care. Again, the difference was not statistically significant, with wide CIs observed.

The incremental cost-effectiveness analysis for the ACA is presented in Appendix 17, Table 50. For the primary cost–utility analysis (incremental cost per QALY on EQ-5D-5L at 12 months) and all secondary cost–utility analyses, negative incremental costs and positive incremental QALYs are observed. This is consistent with the multiple imputation analysis and represents dominance of the intervention, with cognitive rehabilitation being less costly and more effective than usual care. As CIs for both costs and QALYs span zero and are not statistically significant, the interpretation of cognitive rehabilitation as cost-effective requires some caution. For the MSIS-Psy, negative incremental effects observed at 6 and 12 months are representative of an improvement in favour of the intervention group. The intervention can therefore be considered dominant at both 6 months and 12 months; however, as noted in Chapter 3, the difference at both follow-ups is not considered to be of clinical significance.

Cost-effectiveness planes and CEACs were also constructed for the ACA on EQ-5D-5L and MSIS-8D QALY gains. These are presented in Appendix 17, Figures 12–15. As observed for the multiple imputation analysis, at 12 months, the cost-effectiveness planes (Figures 12 and 13) show that the distribution of bootstrapped estimates were centred in the south-east quadrant, which is consistent with the intervention dominating usual care. The estimates for MSIS-8D are centred further eastwards than those for the EQ-5D-5L. Figures 12 and 13 show a high probability that cognitive rehabilitation was cost-effective compared with usual care at WTP thresholds of between £0 and £100,000 per QALY gain.

Cost-effectiveness planes and CEACs were also constructed at 6 months to investigate earlier effects of the intervention on HRQoL; the results are presented in Appendix 17, Figures 14 and 15. At 6 months, the cost-effectiveness plane estimates in Figures 14 and 15 for the EQ-5D-5L and MSIS-8D are, again, primarily centred in the south-east quadrant. Points are clustered closer to zero, with smaller incremental costs and QALY gains than in the 12-month analysis. In addition, all four quadrants of Figures 14 and 15 are populated, indicating greater uncertainty regarding the cost-effectiveness of cognitive rehabilitation at 6 months. In accordance with the CEACs at 12 months, Figures 14 and 15 show a high probability that cognitive rehabilitation was cost-effective at all WTP thresholds up to £100,000.

A series of OWSAs were undertaken to assess the impact of parameter uncertainty on the estimations of cost-effectiveness. These sensitivity analyses were conducted on the multiple imputation data set, providing an objective approach in handling uncertainty regarding missing data. The results are presented in Appendix 18, Tables 51 and 52.

As the 95% CIs for both incremental costs and incremental effects span zero, the OWSAs for the EQ-5D-5L QALY gains using a combination of these bounds provide four separate conclusions regarding the cost-effectiveness of cognitive rehabilitation. At 12 months, using the lower bound for cost and the upper bound for effect, intervention dominance is observed to be consistent with the base-case analysis, whereas when using the upper bound for cost and the lower bound for effects, usual care is dominant. Using the upper bound for both incremental costs and incremental effects produces an ICER of £31,055 per QALY gained, whereas using the lower bound for both incremental costs and incremental effects produces an ICER of £89,306 per QALY lost. Similar results are observed at 6 months and for the MSIS-8D at both 6 and 12 months. These sensitivity analyses emphasise that, despite the base-case analysis suggesting dominance of the intervention, these results are subject to uncertainty and are not conclusive.

Including the cost of cognitive assessment increased the intervention cost from £209, used in the primary analyses, to £333 (see Appendix 19, Table 53). At 12 months, the calculated ICERs for the QALYs derived from both the EQ-5D-5L and the MSIS-8D continued to be representative of intervention dominance. At 6 months, the inclusion of cognitive assessment resulted in an incremental cost of £104.09 (95% CI –£653.13 to £861.31). When cognitive assessment was included as a component of the intervention and used to estimate the incremental cost per QALY gain at 12 months using the EQ-5D-5L, an ICER of £39,826 was calculated. The ICER for the QALYs derived from the MSIS-8D would be £14,559.

As noted in Resource use and costs, two notable outliers with total costs in excess of £40,000 at 12 months were identified. Removing these two outliers from the ACA at 12 months reduced the incremental cost from –£808.33 to –£38.85 (95% CI –£1162.14 to £1084.45). This illustrates that the two outliers have a large effect on mean costs, despite not resulting in statistical significance. As cognitive rehabilitation continued to be less costly than usual care, albeit this cost difference was considerably reduced, there is no change in the interpretation in the ICER of intervention dominance.

Chapter 2 detailed the use of upper and lower 95% CIs for incremental costs and effects; we assessed the impact of changes in key parameters to the ICERs.

Results of model-based analysis for longer-term cost-effectiveness

The original trial protocol65 and health economics plan proposed a model-based analysis to evaluate the longer-term cost-effectiveness of cognitive rehabilitation at horizons > 1 year. In the proposal, criteria were determined to ensure that any model-based analysis would be rigorously undertaken and produce reliable and informative estimates of the longer-term cost-effectiveness of cognitive rehabilitation. Given the results of the within-trial analysis the conclusions are as follows:

• Cognitive rehabilitation does not show sufficient evidence of clinical effectiveness. Neither a statistically significant nor clinically important difference was observed between the cognitive rehabilitation and usual-care groups in the MSIS-Psy score at 12 months.

• Although the primary cost–utility analysis indicated that cognitive rehabilitation was dominant compared with usual care, differences in costs and QALY gains were not statistically significant.

• There was a lack of available evidence for the longer-term effects of cognitive rehabilitation. Clinical opinion deemed that, in the absence of any observed effect in the short term, no effect would be observed in the long term, during which patients no longer received the intervention; therefore, the decision was taken not to proceed with the model-based analysis.

• Although a robust mathematical model could be constructed, the data inputs would yield an extrapolation of the within-trial results and not produce plausible, reliable or informative estimates of the longer-term costs and consequences of cognitive rehabilitation.

One potential area of future consideration would be to re-examine the evidence in the light of extending the meta-analysis reported in the Cochrane Review29,31 to include the CRAMMS trial findings. This would provide more robust and reliable data to inform a model-based analysis in the future.

Clinical effectiveness

The results indicate that there was no benefit of this cognitive rehabilitation programme for this group of people with MS on their quality of life, with no clinically important difference in MSIS-Psy score (i.e. the primary outcome) between the two groups at the 12-month follow-up. However, there was a small difference in MSIS-Psy score at 6 months follow-up, favouring the intervention group. There was a difference between the cognitive rehabilitation and the usual-care groups in terms of subjective participant and relative reports of memory problems, as assessed on the EMQ, favouring the cognitive rehabilitation group at both 6 and 12 months. A similar finding was observed on mood outcomes, indicating that the cognitive rehabilitation group showed less psychological distress than the usual-care group at both 6 and 12 months. There were, however, no important differences between the groups on objectively assessed cognitive abilities, fatigue, employment status or carer strain at the 6- or 12-month follow-ups. No safety concerns were raised and no deaths were reported. The evaluation of treatment fidelity indicated that cognitive rehabilitation was delivered as intended. Interviews with a subset of participants found that the treatment was well received and was different from usual care. During interviews, the majority of participants reported that they found the intervention helpful in reducing cognitive problems and improving confidence in daily life.

Cost-effectiveness

The health economic analysis illustrates that cognitive rehabilitation had lower costs and higher QALY gain than usual care; however, incremental differences in both these measures were small. Although this represents dominance of the intervention, at 12 months differences were not statistically significant and sensitivity analyses demonstrated uncertainty concerning the base-case analysis. Therefore, definitive conclusions cannot be drawn regarding the cost-effectiveness of cognitive rehabilitation compared with usual care.

Choice of primary outcome measure

The commissioning brief indicated that quality of life was an important outcome; therefore, a quality-of-life measure was used as the primary outcome. The MSIS was chosen as it was developed specifically for people with MS, provides a patient’s perspective on the impact of their disease on daily life and Rasch Analysis has confirmed the validity of two separate subscales: physical and psychological. 84,85 The MSIS-Psy subscale includes items on mood, and coping and cognition, has good psychometric properties and has been shown to be sensitive to the effects of rehabilitation interventions (see Chapter 2). Most of the MS-specific quality-of-life scales include many items that reflect the MS disease process, such as balance, stiffness and spasms. However, cognitive rehabilitation does not alter the disease process; therefore, the physical symptoms of MS are not expected to change. The aim is to enable people to cope with their symptoms better. Although the MSIS-Psy subscale does include items related to cognition, such as problems concentrating, it also includes items not likely to be influenced by cognitive rehabilitation, such as feeling unwell and having problems sleeping.

Another option was one of two modified versions of the Short Form questionnaire-36 items (SF-36), which has been amended to use as outcomes for pharmacological trials in people with MS: the Multiple Sclerosis Quality of Life-54 (MSQOL-54),155 and the Multiple Sclerosis Quality of Life Inventory (MSQLI). 156 However, both are very long157 and include a large proportion of items that are unlikely to be influenced by cognitive rehabilitation. A shorter version, the Multiple Sclerosis International Quality of Life questionnaire (MusiQoL),158 may have been appropriate but, as noted by Bandari et al. ,157 there is a lack of information on the responsiveness of the instrument compared with other clinical end points. In addition, Baumstarck et al. 159 found that the MusiQoL was not responsive to changes in disability as recorded on other measures.

There are two outcome measures that have been developed to assess the outcome of cognitive rehabilitation. The Adaptation to Memory Difficulties Outcome questionnaire (AMEDO)160 was developed to assess the effects of memory rehabilitation, and the validation studies included people with MS. Patchick et al. 161 developed an outcome measure for cognitive rehabilitation after stroke. Although these measures seem appropriate, their responsiveness to the effects of intervention and their psychometric properties had not been fully evaluated at the time of starting the trial. However, they might be considered in future evaluations of cognitive rehabilitation.

Therefore, the MSIS-Psy subscale was probably the most appropriate quality-of-life measure. However, even on the MSIS-Psy subscale, many of the items are unlikely to be directly influenced by cognitive rehabilitation. Quality of life is, of necessity, a multidomain construct, and changing quality of life, although desirable, may be unrealistic for interventions that focus on specific symptoms of MS. Future triallists may also consider a more direct measure of cognitive abilities in daily life as a co-primary outcome for cognitive rehabilitation trials. Co-primary outcomes are useful to provide a more ‘comprehensive picture of the intervention’s effects’,162 but have implications for the trial design.

Long-term versus short-term effects of the intervention

The aim of the cognitive rehabilitation programme was to change the long-term effects of cognitive impairment and, therefore, the primary outcome at the 12-month follow-up. Although this time frame is useful to determine the longevity of treatment effects, not having interim assessment time points more proximal to the end of the intervention risks missing the immediate impact of the intervention. As with most other behaviour change interventions, it is probable that the impact of the intervention is greatest immediately after the intervention, with gains diminishing over time. There was some evidence in this trial to suggest that the programme improved quality of life in the short term (i.e. at the 6-month follow-up), but the effects were not maintained over time. This is consistent with previous memory rehabilitation research that shows that there are short-term benefits in quality of life, which are not maintained. 29 Therefore, future research should consider having more outcome time points. However, the addition of further assessment time points has the potential to add to participant burden, risking lower numbers consenting to participate in the trial and increasing the likelihood of practice effects on some measures. Future research should also focus on ways in which the effects of cognitive rehabilitation can be maintained. This might include booster sessions, telephone calls to review progress or online support; an evaluation of booster sessions in a previous trial of improving learning in people with MS52 indicated little benefit from these, but the sample sizes were small.

Subjective cognitive impairment and mood

There were effects of cognitive rehabilitation on the reported frequency of everyday memory problems and mood. This could be because cognitive rehabilitation improved participants’ everyday memory, which had a beneficial effect on mood, or it could be that the intervention improved mood, which led to a decreased frequency of reporting cognitive problems.

Previous research has shown that there is a strong relationship between subjective cognitive problems and mood in people with MS. 102,103,163–166 Therefore, some of those recruited may have reported cognitive problems associated with low mood, and this is consistent with the improvement in mood that also occurred.

These effects may be partly attributable to the social contact afforded by attending for cognitive rehabilitation rather than the cognitive strategy training aspects of the intervention per se. The group component was reported to be a very useful aspect of the intervention by all participants in the intervention group who were interviewed. Future research on this type of cognitive rehabilitation programme should control for the therapeutic effect of the group interaction itself. However, creating an acceptable attention placebo control for such programmes is difficult:59 they run the risk of disproportionate attrition from the control group and there is no established gold standard for implementing an attention placebo control in psychosocial intervention trials. 167

It seems more probable that the change in mood was in response to changes in everyday memory, rather than the cause of the changes in everyday memory. The reason is that relatives’ reports of the frequency of everyday memory problems also showed a difference between groups, and relatives’ reports would be unlikely to be influenced by the participants’ mood. 102

One of the reasons that the EMQ was included as an outcome measure was to provide comparison with other studies of the same intervention. 59,60,95 The difference in EMQ scores at 6 and 12 months was about 5 points. The differences in the participant-reported EMQ are slightly smaller than those observed in previous studies of the intervention in people with MS,60,62 but consistent with the difference observed in people with traumatic brain injury. 95 The smaller difference in relation to previous studies of people with MS may be due to the broader recruitment strategies used and the difference may be comparable to the study of people with traumatic brain injury because similar recruitment strategies were used.

It is, therefore, also probable that teaching people with MS the cognitive strategies improved their confidence to use these in daily life. Improved confidence was also reported during the interviews by those who had received the intervention. Increased strategy use, the benefits of the group environment and improved confidence have all been reported in other qualitative studies of perceived effects of cognitive rehabilitation among people with MS. 136

Effect on cognitive impairment

The results, overall, indicated that the intervention had no effect on cognitive impairment. Scores in both the intervention and the usual-care groups tended to improve over time, which is probably as a result of practice effects and familiarity with the tasks. This finding is consistent with some studies of interventions to improve strategy use,58,168 whereas change in cognitive abilities tends to be observed in studies that provide specific training on cognitive tasks through computerised activities that are closely linked to the cognitive outcomes. 32,35,36,43,45,169

Other outcomes

Fatigue was assessed to determine whether or not the intervention exacerbated fatigue-related problems, but there was no indication of any negative effect on fatigue. Several previous trials of cognitive rehabilitation have included fatigue as an outcome measure and none has shown any effects of cognitive rehabilitation on general levels of fatigue. 35,38,40,42,46,58,170 Two studies have shown positive effects on cognitive fatigue,35,170 but in both these studies the intervention was computerised retraining of cognitive abilities. Therefore, it is improbable that cognitive rehabilitation has any effect on levels of general fatigue.

Guy’s Neurological Disability Scale and the number of relapses were recorded to determine whether or not any differences between the two groups could be attributed to relapses or increasing disability. There were no differences in the MSIS-Phy subscale score or the GNDS between the groups, so there is no suggestion that the findings were due to differences in the physical effects of MS.

High levels of unemployment are common among adults with MS. 171 In this trial, employment status was no different between the groups at baseline or at the follow-ups. The majority (97%) of those who were not employed at baseline were not employed at follow-up. Only a few participants gained employment during the trial and this was equally balanced between the groups. Therefore, the intervention did not facilitate return to work. This finding is likely to be partly due to the low rate of employment at baseline, and because the sample had relatively severe cognitive problems at the time of recruitment. There was also no evidence that the intervention enabled people to continue to work. This finding, however, needs to be considered with caution. The questions around employment and education did not consider whether or not participants had taken up voluntary positions. Although, to our knowledge, there is no research on the effect of volunteering on people with MS, there is good evidence that volunteering has a positive effect on psychological well-being and mood and slows self-reported decline in health and functioning levels among older adults. 172,173 Furthermore, absenteeism or presenteeism were not investigated. Research has demonstrated that there is a link between cognitive impairment and levels of absenteeism. 174 Future research could consider more nuanced ways of collecting information about employment status.

There was also no indication of an effect of the intervention on carers and relatives. Previous studies of people with MS have not examined the effects of cognitive rehabilitation on carers. However, one study with patients with acquired brain injuries found that holistic neuropsychological rehabilitation did reduce carer strain. 175 This study, however, used a pre–post repeated-measures design, and is therefore susceptible to bias.

Consistency with previous research

There are relatively few trials that have been conducted using a similar intervention with people with MS. The results support the pilot study,60 which demonstrated a beneficial effect on mood. A similar finding was observed in the ReMIND trial,62 which used a similar treatment approach, whereby people with MS who were taught the use of compensation strategies had better mood outcomes than those who had only restitution approaches. The finding that perceived cognitive deficits improved in response to group treatment is consistent with two studies with people with MS. 58,168 However, comparison of the present trial with the ReMemBrIn trial,95 which evaluated the same intervention in people with traumatic brain injury, suggested that those with MS responded better than those with traumatic brain injury. The most probable reason for this discrepancy is that usual care for people with traumatic brain injury often includes cognitive rehabilitation whereas it does not for people with MS. Those with traumatic brain injury may have been taught many of the strategies in the intervention before they were recruited and so they responded less well to the intervention.

The cost of providing cognitive rehabilitation in this trial was slightly higher than the costs of providing the intervention to people with traumatic brain injury (£209 vs. £167). 95 This was due to higher administration costs and the inclusion of the manual, stationery and refreshments in the cost of the intervention in the CRAMMS trial. If these are excluded, then the cost of providing the intervention is very similar in the two trials. However, the overall costs were higher for those with MS than for those with traumatic brain injury, largely because of the much higher medication costs for those with MS. The cost of providing cognitive rehabilitation was also similar to other group psychological interventions for people with multiple sclerosis. 90,107 Overall, the cost of providing cognitive rehabilitation is very low in the context of the overall costs of the disease. 176 The finding that there was little evidence to support the cost-effectiveness of cognitive rehabilitation was also consistent with the ReMemBrIn trial. 95

Trial design

The strength of the trial lies in its methodological quality and rigour. The trial was prospectively registered and the protocol was published. 65 There was allocation concealment, meaning that those recruiting participants had no control over the random allocation or, therefore, who would receive treatment. Randomisation was conducted by a web-based system prepared in advance of the trial. The clinically relevant end points were defined a priori. The primary outcome was a questionnaire, so it was completed independently. The face-to-face outcome assessments were conducted mainly by a researcher who was blind to group allocation. Treatment sessions were video-recorded in order to assess the fidelity of the intervention. The SAP was agreed in advance of the data lock stage. The trial was reported in line with the CONSORT guidelines. 177 The content, format and delivery of the intervention was reported in line with TIDieR and other relevant guidelines. 109,178 The trial was adequately powered and recruited the required sample, and the attrition level was as predicted in the sample size calculation. There were independent trial steering and data monitoring committees to ensure that the trial was conducted as planned and that participant safety issues were considered.

One possibility for the small differences in subjective reports of cognitive problems in daily life and mood is that the effects observed were not due to the strategies taught as part of cognitive rehabilitation, but were due to additional attention and meeting others with MS. The ideal design would have been to also have a control condition in which participants met as a group for the same frequency and duration of sessions as the cognitive rehabilitation group but did not receive the active component of cognitive rehabilitation. In one of the pilot studies,59 there was an attention placebo self-help group as the control condition. In this condition, participants met to discuss their health and memory problems and were taught relaxation exercises. The facilitator did not initiate any memory talk. The fidelity analysis of these sessions130 confirmed that the self-help control group had significantly less discussion of memory than the active treatment groups. However, the self-help group was difficult to run, as participants wanted to discuss their memory problems and they felt that the problems they faced were not being addressed in the sessions. This may have had a negative effect, rather than controlling for the positive effect of being in a group.

In NHS clinical practice, very few people with MS receive treatment for their cognitive problems. They may receive some advice from MS nurses and occupational therapists and may be directed to information on MS charity websites. Therefore, it was decided to use usual care as the control condition rather than attempting to devise an attention placebo condition, which would be difficult to deliver. This means that the results are valuable for informing NHS practice, but the mechanism of the changes observed remains unclear.

Recruitment

Recruitment took place in five sites across England; thus, the participants are probably representative of those attending clinical neurology services in England. The recruitment strategy was to invite as many people as possible from each site to consider taking part in the CRAMMS trial. This was to ensure that all those who thought that they had cognitive problems were aware of the trial. As a result, the letters of invitation were sent to many who were not eligible. There were also a high proportion of people who did not respond to the invitation. The most probable reason is that they did not consider that they had cognitive problems, but there is no way of verifying whether or not this is the case.

Multiple recruitment sources and strategies were used to identify potential participants, including self-referrals. This is a strength, as it is inclusive, but is also a limitation. Some people may have received information from more than one source. As there is no demographic data available on all those who were approached and invited, it is not possible to ascertain the exact proportion of those who were invited who were eligible for the study and whether or not those recruited differ in any way from those who did not take part. Also, because some people were recruited through posters and publicity on websites, such as the MS Society and MS Trust web pages, it is not known how many people viewed the information and, therefore, what proportion of people chose to self-refer as a result.

The lack of information on the demographic characteristics of those who did not wish to take part in the CRAMMS trial limits the ability to determine the proportion of people who would be eligible for the treatment if it were available in the NHS.

Selection of participants

The inclusion criteria were broad in order to involve all those who might be treated in clinical practice, thereby increasing the generalisability of the findings. The most common reason for exclusion of those who were screened for eligibility was that they did not meet the criterion score on the MSNQ. Almost half of those excluded (116/283, 49%) were excluded for not meeting this criterion, yet they wanted to take part in a trial of cognitive rehabilitation, and so believed that they had cognitive difficulties.

In the pilot study,60 participants were recruited if they reported cognitive problems in daily life, and no standardised assessment of subjective cognitive impairment was used. In order to maintain consistency across sites, the MSNQ was used in the CRAMMS trial with a predefined cut-off score. As a result, some people who reported cognitive problems in daily life were excluded from the CRAMMS trial because they did not meet the cut-off score on the MSNQ. This may have led to a sample with relatively severe self-reported cognitive problems. This is supported by the difference in EMQ scores at baseline, which were lower in the pilot study (intervention, mean score 27; control, mean score 30)60 than in the CRAMMS trial (mean score 46).

A score of > 27 on the MSNQ was used as an inclusion criterion to identify those with cognitive problems in daily life, based on the original validation study. 66 Benedict et al. 66 included 50 participants with MS and used the informant version of the questionnaire rather than the patient-reported version. This cut-off score was also used by Vanotti et al. ,72 who found high sensitivity and specificity for the informant version. However, lower cut-off scores have been recommended subsequently in other studies. 69,179,180 Benedict et al. 69 found that the optimum cut-off score was > 23 on the patient-reported version and > 22 on the informant-reported version to identify those with cognitive impairment. O’Brien et al. 179 evaluated these cut-off scores in a sample of 48 people with MS, but found that the optimum cut-off score was > 8 to indicate impairment, using the same criterion for cognitive impairment as both studies by Benedict et al. 66,69 Comparison of cut-off scores for an internet version of the MSNQ181 showed that the MSNQ had a sensitivity of 69% using 27 as a cut-off score, in contrast to 79% using the lower cut-off score of > 22. In comparison, the specificity reduced from 42% to 17%. Therefore, the criterion used in the CRAMMS trial required a high level of reporting of problems in daily life; consequently, some people with fewer problems in daily life, who may have benefited from the intervention, were excluded.

It is possible that those with very high frequency of cognitive problems in daily life may not benefit from the intervention. It may be that > 10 sessions are required to address all their difficulties and to ensure that they are able to retain the information. The subgroup analysis of the MSNQ score at baseline in relation to 1-year outcomes on the MSIS-Psy, although not providing evidence of an interaction effect, suggested that the more severe the self-reported cognitive problems, the lower the response to intervention. Future trials should consider whether or not those with fewer problems in daily life who are less severely cognitively impaired may benefit more.

The second most frequent reason for exclusion was that participants were not able to travel to the treatment site (33/238, 14%). Travel expenses were offered but funding was not sufficient to provide taxis for all those who could not manage on public transport and who lived a long distance from where the treatment sessions were held. It is improbable that these people would have attended this treatment if it were offered in NHS clinical services, as transport would probably not be provided. Treatment sessions were held at several centres surrounding each site, so those unable to get to these centres would probably not have been able to get to NHS rehabilitation settings either.

Only a few people were excluded on the basis of being aged > 69 years (11/238, 5%), but in clinical practice these people probably would have been offered the treatment. In this respect, the group is slightly more homogeneous that those who would have been offered treatment in NHS clinical practice, but as the proportion is low it is unlikely to have had any real effect on the results.

Participants were recruited who failed one subtest of the BRBN. As a result, participants with problems in attention, memory or executive abilities were included. The BRBN was chosen as it is a relatively short screening test with appropriate normative data. The Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS)182 is similar but takes longer to administer. A shorter alternative was the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS),183 which has the advantage of being brief, but at the time of designing the study there was limited information of the psychometric properties of this measure. 184 More information is now available,185–189 and a direct comparison between the two190 suggested that the BICAMS is an acceptable alternative, but is less sensitive to cognitive problems, than the BRBN. The BICAMS may be a suitable alternative in future studies as a brief measure to detect cognitive impairment.

The use of a criterion of impairment relative to a healthy control population meant that some people were excluded who were impaired relative to their own pre-morbid level, but within the average range of the general population. This problem, highlighted by Sumowski et al. ,191 means that not all those with cognitive impairment from their MS would have been included, even though they may have reported problems on the subjective measure.

Participant characteristics

Participant demographic characteristics were largely consistent with what would be expected in a sample of people with MS. The average age was 50 years, with the majority in their late 40s and early 50s. This reflects the likelihood that they were not recruited very soon after diagnosis, when the average age is more likely to be < 40 years,192,193 and that some recruitment was through charities, such as the MS Society, whose meetings tend to be attended by a higher proportion of people with secondary progressive MS, than those attending neurology clinics.

The proportion of women in the trial was 73%. This is consistent with the expectation that women are more likely to have MS than men. 192,193

Most participants were of white ethnicity, consistent with the general UK population,194 which partly reflects the communities where participants were recruited. 195 The south-west and north-east of England have relatively low numbers of people from ethnic minority groups. 194 However, even in the East Midlands there were very few participants from an ethnic minority background. Some ethnic minorities may have been excluded because we required people to have a sufficient level of English proficiency to allow them to independently complete the standardised tests and questionnaires and because the intervention was delivered in English; however, the under-representation of ethnic minority groups in health-care research in Western countries has been observed more widely. 196 In addition, the incidence of MS is recognised to be lower in some non-white ethnic groups than in those who are white. 141

Participants were, on average, recruited 11 years after their MS diagnosis; most were recruited > 5 years after diagnosis. Therefore, they had established problems and were not adjusting to the effect of diagnosis at the same time as engaging in the intervention. For some people, their cognitive problems may have been so long standing that they had established coping strategies and found it difficult to adapt to new ones. Feedback in the interviews suggested that some people thought that they should have received cognitive rehabilitation sooner after the onset of their problems, so that they could learn to use strategies to prevent their cognitive problems affecting their daily life.

Most participants (65%) reported having relapsing–remitting MS. However, this may be an overestimate, as some commented that this was the diagnosis they had been given and now they thought that they were probably secondary progressive. As many participants were recruited through community settings, they may not have received an updated diagnosis from a neurologist. This is consistent with the majority (76%) reporting no relapses in the previous 6 months at the time of recruitment. Although it would have been desirable to have the type of MS confirmed by a neurologist at the time of recruitment, there was no indication of differential effects of cognitive rehabilitation according to reported type of MS.

The cognitive rehabilitation programme was intended to teach participants strategies to cope with both attention and memory problems. However, the content of the intervention was much more focused on coping with memory problems and only two of the 10 sessions were devoted to problems of attention. It is possible that those who primarily had problems with attention did not have enough intervention for their specific problems to enable them to benefit. It is also possible that attention problems respond to restitution strategies rather than compensation strategies. Most of the studies of computerised cognitive rehabilitation have targeted the attention deficit in people with MS, rather than the memory impairment. Mitolo et al. 28 reported that most recent studies have focused on retraining attention, speed of processing and executive functions, rather than memory, and these have mainly involved computerised practice. It may be that those with attention problems benefit more from a restitution strategy and that compensation strategies should be reserved for those with memory problems. This is supported by the finding that there was a trend for those with more severe attention problems, as assessed on the Symbol Digit Modalities Test, to respond less well to the intervention than those with milder problems. Future studies of this rehabilitation programme could select those with memory problems, rather than also including those with exclusively attention problems, even though the programme does include consideration of strategies to improve attention.

Intervention

Attendance at sessions was relatively good; therefore, most people received most of the intervention. The ability to provide catch-up sessions for those who could not attend some sessions also ensured that people received most of the intervention and reflected procedures used in clinical practice. The attendance rate [mean 7.7 (SD 3.5) sessions] was consistent with the pilot study60 [mean 7.9 (SD 0.23) sessions] and higher than in a similar trial of the cognitive rehabilitation programme for people with traumatic brain injury64 [mean 6.3 (SD 3.5) sessions]. However, many studies do not report attendance rates,197 so direct comparisons of attendance rates across trials are limited.

Many of the reasons for non-attendance reflect aspects of daily life unrelated to cognitive problems, such as illness and holidays, rather than withdrawing or not wanting to continue in the group. The optimum ‘dose’ of the intervention remains unknown, and so it is not clear whether or not more sessions would have improved the outcome or fewer sessions would have maintained the outcome. More Phase II trials are needed to establish the optimum dose of the intervention.

The focus of the intervention was on teaching strategies to cope with cognitive problems, rather than addressing the underlying cognitive deficit. The strength of this approach is that the effects of rehabilitation are more likely to persist over time as cognitive abilities deteriorate.

This cognitive rehabilitation programme may not be appropriate to all those with cognitive problems and some clinicians may select treatments according to participant characteristics or to the nature of their cognitive problems. However, there is little information available on who might benefit the most from specific types of cognitive rehabilitation.

The cognitive rehabilitation was conducted in groups. An advantage of this is that people may benefit from the experiences of others in a similar situation. Some of the qualitative feedback in previous studies suggested that this is an important element of the intervention. 136 Indeed, the qualitative data from the interviews did suggest that participants highly valued the group format. However, the groups included people with a range of cognitive impairments. Those with severe problems may have found that too much material was covered too quickly; those with mild problems may have found the pace too slow and not sufficiently tailored to their specific problems. However, there was nothing in the qualitative data that supported this assumption. Furthermore, in clinical practice, it is unusual to be able to create groups based on level of ability and so the results reflect the way the intervention would be delivered in clinical practice. Individual treatment has the advantage that the intervention can be specifically tailored to an individual’s pattern of cognitive deficits, and may also improve attendance compared with group formats. 29 However, evaluation of such individualised treatment becomes problematic as it is difficult to describe the content of the intervention in such a way that it can be replicated.

Some people may have declined to take part because they did not wish to be treated in a group. Providing a few individual sessions before participants are allocated to a group may allow their specific concerns to be addressed and they can be prepared for sharing their problems with others. A combination of individual and group treatment may maximise the benefits to be gained from cognitive rehabilitation. This strategy has been used previously,57,168,170 but there is no clear indication that outcomes are any better using this format.

The intervention was conducted over 10 sessions, which were held approximately once a week. There may be advantages in spreading the intervention over a longer period of time so that participants can practise between session tasks for longer, and to get more familiar with one strategy before learning about another. However, this frequency of sessions is similar to a previous study of cognitive rehabilitation delivered in groups,57 which provided eight sessions over 13 weeks. In addition, the provision of refresher sessions after the end of the structured programme may facilitate the maintenance of any gains obtained from the intervention. There was support for this suggestion in the participant feedback interview data.

Fidelity of the intervention

There were several mechanisms in place to ensure the fidelity of the intervention. The treatment was standardised by having a manual outlining the content of each session and facilitator notes were provided to guide the AP. The APs were all trained to deliver the intervention by one of the co-applicants. The co-applicant also provided one-to-one monthly supervision in addition to the monthly AP peer supervision. This enabled the APs, who delivered the intervention, to share experiences and to clarify any uncertainties relating to the manual and facilitator notes.

The video analysis suggested that the cognitive rehabilitation programme was delivered as planned and as described in the manual. However, the main limitation of the video analysis was that recordings were incomplete and did not include all sessions at all sites. Some were only partially recorded, as a result of the camera battery running low or the camera not being turned on again after the break. This limited the recordings that could be included in the analysis. Therefore, it is not possible to ascertain whether or not all sessions were delivered consistently in accordance with the manual at all sites. However, based on feedback obtained during monthly peer and supervisor meetings with the APs, it is improbable that the unrecorded sessions were different from the recorded ones.

It was not possible to check for drift over time as not enough sessions were recorded from each site. It would also have been better to observe the videos over the course of the trial, rather than in the final analysis, so that any minor discrepancies in the delivery of the intervention could have been addressed as the trial progressed. The video analysis focused on the content of treatment, but there was no assessment of the skills of the therapists. Important aspects of the group sessions, such as group cohesiveness and the skill of the therapist in managing group dynamics, were not monitored. However, most participants were complimentary about the therapists’ skills at facilitating the group.

Outcome assessment

The outcomes included a broad range of assessments. The primary outcome was a measure of quality of life, so that it would detect the general effects of receiving cognitive rehabilitation. The more specific effects of cognitive rehabilitation were assessed using subjective measures, which reflect the effect of cognitive problems in daily life, and objective tests of cognitive abilities. The potential negative consequences of the intervention were assessed through measures of fatigue and monitoring the safety outcome. The effect on relatives/friends was assessed through the MCSI.

Having patient-reported outcomes that rely on recall of information over time can be a challenge for people with cognitive problems. However, given that the primary outcome had to be a quality-of-life measure, it was inevitable that this was a patient-reported outcome measure.

The primary outcome questionnaire was completed and returned by post or online, and so was mainly conducted independently of an outcome assessor. However, when participants failed to return their questionnaires, they were reminded at the home visit and some opted to complete the questionnaire with the outcome assessor at the visit. The outcome assessments were conducted by researchers not involved in delivering treatment so that they were blind to group allocation. However, some unblinding of outcome assessors occurred. This was largely due to participants telling the assessor about their treatment, despite having been prompted not to. The cognitive tests were completed by outcome assessors at home visits and some unblinding occurred. However, the rate of unblinding was low. Comparison between the guessed group and the allocated group showed very weak correspondence and, in some instances, those who thought that they were unblinded guessed the allocation incorrectly. However, as noted by Bang,198 it is impossible to eliminate bias even when perfect blinding is achieved.

The follow-up completion rates were good and consistent with planned loss to follow-up. The main reasons for non-completion of follow-ups were health problems, non-response to postal assessments and failure to contact participants to arrange assessment visits. The rate of completion of cognitive tests was slightly lower than the rate for completion of questionnaires. Some participants refused to do some of the cognitive tests. This was most marked on the Paced Auditory Serial Addition Test (PASAT). It was also not clear from some records whether the PASAT was missing because it was not offered as a result of time constraints or technical problems or whether it was refused. If participants were offered a cognitive test but refused to answer or were unable to provide any responses, they were scored the minimum, consistent with standard administration procedures, but the outcome assessors did not always make this distinction clear in their records and some missing data may have occurred when participants were offered the test but were unable to complete the task. These were recorded as protocol deviations.

There is an inherent problem with using attention and memory tests as outcomes of cognitive rehabilitation, in that although the rehabilitation focuses on teaching people to effectively use attention and memory aids to reduce their cognitive problems, these standardised tests themselves do not allow the use of such aids. Therefore, it is improbable that there will be changes observed on these tests if participants are not allowed to use any aids, but equally, if aids are permitted, the tests would not have been administered as per the test manual.

The cognitive assessments were lengthy to administer and, as a result, could not always be completed in one session. If participants ran out of time or were too tired to continue, they were offered the opportunity to complete the cognitive assessments at a second visit; however, this sometimes proved difficult to arrange and did not always occur.

Questionnaires could be completed online or on paper versions that were returned by post. Only 86 (19%) participants chose to complete outcome questionnaires online, but the completion rate was higher (6 months: 98% online, 92% postal; 12 months: 92% online, 83% postal) than for the postal questionnaires.

Outcomes were assessed at both the 6- and 12-month follow-ups. Therefore, the long-term effects of cognitive rehabilitation could be determined, although it would be desirable to monitor whether or not the benefits persisted for longer.

Sample size

Most previous studies of cognitive rehabilitation have been small and underpowered. The median total sample size for MS memory rehabilitation studies is 42 (range 19–240). 29 The target sample size of 400 was exceeded, with a total of 449 randomised. The loss to follow-up for the primary outcome of 15% was consistent with that predicted in the calculation of sample size needed. The baseline characteristics of participants included in the primary analysis were similar in the two groups, as were the characteristics of the participants not included.

Economic analysis

Few studies have considered the cost-effectiveness of cognitive rehabilitation. Overall, providing cognitive rehabilitation cost less than usual care, due to a reduction in the use of other health and social services while receiving cognitive rehabilitation and differences in medication costs. This occurred mainly in non-disease-modifying medications, and so could not be attributed to differences in the disease progression between the two groups. Costs varied widely across participants in both groups, and lower costs for the intervention group were not robust to sensitivity analysis.

One limitation of the calculation of the costs of the intervention is that it relied on self-report of services used and medication. There is evidence from people with stroke, who are also likely to have cognitive impairments, that reporting of health and social services received is not accurate when compared against formal records. 199 Although the inaccuracies would have occurred in both groups, it may have reduced the ability to detect differences between the groups. In addition, participants were asked to self-report on medications used and many opted to show the RA their prescription list. However, some disease-modifying medications would not be listed on prescription lists, as they were issued directly from hospitals. Although this under-reporting would have affected both groups, it may have also diluted the ability to detect differences.

The cost-effectiveness analysis relied on the EQ-5D-5L to calculate QALYs. However, as highlighted by others,90,200 this is relatively insensitive to changes in HRQoL in people with MS and no difference in QALY gain was observed from the EQ-5D-5L. This is supported by the greater effects observed on the MSIS-8D, which was designed to assess QALYs in those with MS. However, even when the MSIS-8D was used as an alternative method for deriving utilities, there was a numerically small, significant, incremental QALY gain. However, only two of the eight items of the MSIS-8D would be expected to change with cognitive rehabilitation, and so any overall changes are likely to be small.

In addition, the cost-effectiveness analysis was considered in relation to the primary clinical outcome, another quality-of-life measure, the MSIS-Psy. This may not be sensitive to the effects of cognitive rehabilitation, as discussed above; therefore, cognitive rehabilitation was deemed unlikely to be cost-effective as incremental costs and effects are not statistically significant. Consideration of the incremental cost per point gained on the EMQ or GHQ-30 may be useful to inform the cost of provision of cognitive rehabilitation in clinical practice.

Patient and public involvement

The trial was actively supported by service user representatives. They provided suggestions to make the trial documents and lay summary clearer and more easily understood by service users. They made suggestions for the recruitment of participants, particularly from MS charities and local MS Society branches. One service user contributed to the qualitative study by interviewing participants and was involved in discussions of the analysis of themes. This service user also assisted with the dissemination of results by contributing to a video for service users. The contribution helped to ensure that the recruitment target was met. It also demonstrated that service users can be actively involved in data collection. The limitation was that some service users were not able to maintain their input over the full 4 years of the study. Two had relapses that affected their ability to attend meetings. Although six people originally agreed to take part in the video to disseminate the findings, three dropped out and could not be replaced in time for filming to take place before the end of the trial.

Implications for health care

• People with MS have problems with attention and memory and are seeking help for these problems.

• This trial has not shown any long-term effect of this cognitive rehabilitation programme on quality of life for people with MS, measured using the MSIS-Psy subscale, but there was an effect in the short term.

• There was some evidence that cognitive rehabilitation improved both memory problems in daily life and mood.

• Participant feedback was positive, with some participants reporting daily life benefits of attending the cognitive rehabilitation programme.

• Providing cognitive rehabilitation cost no more than usual care.

• It may be appropriate to offer this intervention in clinical practice given the clinical effects and lack of alternative effective treatments, until there is evidence for alternative effective treatments.

• Further evaluation of those with MS who may benefit most from cognitive rehabilitation is needed.

Recommendations for research

• There needs to be more small-scale efficacy studies to establish the appropriate selection criteria for cognitive rehabilitation programmes, so that the intervention is tailored to those who may benefit most.

• Future studies should attempt to control for the effects of the group environment, in order to ascertain whether it is the contact with others with similar problems or the content of the programme that is most important.

• Further research is needed to explore how short-term benefits of cognitive rehabilitation can be maintained in the long term.

Sites

• Nottingham University Hospitals NHS Trust: Cris Constantinescu (Principal Investigator), Holly Chappell (AP), Sara Clarke (AP), Kara Crossley (AP), Cara Knight (AP) and Kristy Martin (AP).

• The Walton Centre NHS Foundation Trust: Perry Moore (Principal Investigator), Carolyn Young (Neurology Adviser), Alexandra Cunliffe (AP), Catherine Pollard (AP) and John Wilson (AP).

• Sheffield Teaching Hospitals NHS Foundation Trust: Basil Sharrack (Principal Investigator), Claire Isaac (Clinical Neuropsychology Adviser), David Griffiths (Clinical Neuropsychology Adviser), Emma Trigg (AP) and Serena Vanzan (AP).

• North Bristol NHS Trust: Vera Fixter (Principal Investigator), Laura Hanley (Principal Investigator), Joanna Dalton (AP) and Stephanie Pegnall (AP).

• South Tees Hospitals NHS Foundation Trust: Stephen Evans (Principal Investigator) and Miranda Wheeler (AP).

Participant Identification Centre leads

Lena Palmer, Tracy Tyrell, Shannon Gaughan, Gemma Elliot, Sophie Keogh, Helen Oldknow, Catherine Edwards and Nigel Schofield.

Outcome assessors

Hannah Carpenter, Heather Cogger, Rachel Harnell, Jacqueline Mhizha-Murira, Katie Powers and Luke Squires.

Qualitative interviewers

Susan Evans and Olga Klein.

Fidelity analysis

Jacqueline Mhizha-Murira.

Trial and data management

Kirsty Sprange (Senior Trial Manager), Amy Evans (Trial Manager from 2015 to 2016), Florence Day (Trial Manager from 2016), Jo Hobbs (Trial Co-ordinator from 2015), Natalie Wakefield (Trial Administrator/Co-ordinator from 2016), Brian Barnes (Data Manager), Monica Crone (Data Co-ordinator), Matthew Foster (Data Administrator), Chris Rumsey (IT Programmer) and Alexandra Erven (Research Facilitator).

Trial Steering Committee (independent members)

Rona Moss-Morris (chairperson), Audrey Bowen, Rory O’Connor, Julia Scott, Susan Evans and Shirlee McKeown (until September 2015).

Data Monitoring Committee (independent members)

Jenny Freeman, Rod Taylor and Anita Rose.