A web-based, peer-supported self-management intervention to reduce distress in relatives of people with psychosis or bipolar disorder: the REACT RCT

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 14/49/34. The contractual start date was in October 2015. The draft report began editorial review in February 2019 and was accepted for publication in November 2019. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Fiona Lobban reports grants from the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme during the conduct of the study. Fiona Lobban and Lesley Chapman were involved in the design and development of the Relatives’ Education And Coping Toolkit (REACT); hence, this is not an independent evaluation. Bruce Hollingsworth reports that he was a NIHR Health Services and Delivery Research Commissioned Board member (2013–15). Paula Williamson reports grants from NIHR during the conduct of the study, and that the Clinical Trials Research Centre at the University of Liverpool was in receipt of NIHR Support Funding during the conduct of the study.

Disclaimer

This report contains transcripts of interviews conducted in the course of the research and contains language that may offend some readers.

Copyright statement

© Queen’s Printer and Controller of HMSO 2020. This work was produced by Lobban et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2020 Queen’s Printer and Controller of HMSO

Challenges faced by relatives of people with psychosis or bipolar disorder

Psychosis is an umbrella term that covers many different conditions, the common feature of which is that people perceive or interpret the world around them very differently. The most frequent ways this manifests are having beliefs that are not shared by others and do not have a basis that is understandable to others (often called delusions); not being able to think clearly, and so sounding muddled and hard to follow (often called thought disorder); or experiencing, for example hearing or seeing, things that other people cannot (often called hallucinations).

As well as the presence of these unusual experiences, many people with psychosis also report a loss of valued experiences, most notably pleasure in everyday activities (anhedonia) and loss of motivation (apathy). These losses are sometimes referred to as ‘negative symptoms’, and are particularly challenging for relatives, not least because they are hard to differentiate from normal ‘teenage angst’, side effects of medication or depression. It is difficult to report exact figures on the number of people who will experience psychosis as many may never have contact with mental health services, but most recent estimates have calculated worldwide prevalence to be approximately 1 in 13 people (7.7%),4 and up to 10% of people in the UK report some kind of psychotic experiences. 5 Only a fraction of these people (approximately 1% of the general population) will ever come into contact with mental health services and receive a diagnosis of a mental health condition. In general, these are likely to be people for whom these experiences are particularly distressing or cause significant changes in behaviour.

Bipolar disorder is the third most common mental health cause of disability globally,6 affecting 1–4.5% of adults7 and costing the English economy £5.2B annually, largely owing to inadequate treatment. 8 BD is characterised by episodes of extreme low mood (depression) and extreme high or irritable mood (mania, or hypomania in its milder form). Challenging behaviours, such as increased self-harm and suicidal behaviour, excessive spending, sexual disinhibition and heightened irritability, can all occur during mood episodes, which are often accompanied by psychotic symptoms. Between episodes, functioning may return to normal levels, although many people report problematic subsyndromal levels of depression that affect their functioning and relationships. 9

Psychosis and BD present significant challenges to relatives, particularly in recognising and understanding what is happening, living with the elevated risk of suicide, the impact on relationships within the family, and having to balance caring, work and other family commitments. These challenges are exacerbated by difficulties in accessing mental health services, which delay access to effective treatment, leading to worse long-term outcomes. 10,11 It has been estimated that more than one-third of relatives of people with psychosis experience clinically significant levels of distress and burden,12 and estimates from more recent studies with relatives from early intervention in psychosis (EIP) services are even higher, at > 60%. 2,13

Almost half of relatives of people with psychosis experience post-traumatic stress symptoms associated with their caring roles,14 particularly linked to episodes of violence, disruptive behaviour and forced admission. 15

Key factors that increase the negative impact of psychosis on carers include being a female carer;16 living with the person with psychosis; young patient age and awareness of suicidal ideation;17 reduced social support and family resources;17,18 use of emotion-focused coping strategies;19 and beliefs that relatives hold about the psychosis, particularly those concerning cause and control. 20–22

In BD, symptoms associated with depressive episodes result in increased burden of caring, poorer general health, and depressive symptoms in carers. 23–25 The frequency of suicide attempts among the BD population is higher than in many other populations affected by mental health issues, creating a distressing situation for carers. 26 During periods of mania, extravagant spending, irritability and inappropriate and disproportionate behaviour become more frequent and extreme. 27–29 The challenge of learning to cope with symptoms associated with manic and depressive episodes can not only negatively affect the service user, but also diminish carers’ and their families’ quality of life, with carers expressing feelings of helplessness, anger and anxiety. 30,31

Importantly, many relatives also report positive aspects of caring for someone with a severe mental health problem, including identifying personal strengths; feeling a sense of love, caring and compassion; developing new insights about their lives and living; and greater intimacy with others as a result of their journey coping with mental illness. 32,33

We have chosen to focus on the needs of relatives of people with psychosis or BD together in the REACT project, as they face many common challenges. These include how best to support someone in their recovery journey, how to deal with a mental health crisis, how to manage difficult situations, how to manage stress and how to understand and navigate mental health services and the treatments they offer. Mental health services are often structured such that people with a diagnosis of psychosis or BD are managed in the same teams (e.g. community mental health teams, EIP teams). Therefore, having interventions to support relatives that work across these conditions also makes practical sense.

Support for relatives

Relatives of people with psychosis or BD provide a large amount of unpaid care,34,35 but at high personal cost in terms of distress and burden. 12,36,37 Without this unpaid care, the NHS would not be able to cope. Historically, the impact of severe mental health problems on relatives has been ignored, or included as a secondary outcome to service user outcomes. 38 However, more recently, there has been increasing recognition of the importance of understanding the impact on the carer of supporting someone with a mental health problem, and of the need for effective interventions to address this impact. 39

There is now good evidence that interventions that support relatives can improve outcomes both for service users38,40,41 and for carers. 39,42–44 The exact nature of these interventions for carers varies. Some are psychoeducational, aiming to empower relatives with information and coping strategies. 45 Others are more systemic and, in addition to psychoeducation, work with the family on approaches to problem-solving and communication (e.g. Miklowitz46). Interventions also vary in how many sessions are offered, over what period sessions are offered, whether or not the service user also takes part in the intervention and whether the programme is offered to individual families or in groups. To date, there are no good-quality data sets that determine which content or format is most effective.

The exact mechanism underlying the improvements in carer outcomes as a result of family-focused support is not well understood. However, studies have identified several factors associated with improved outcomes for relatives, including the following:

• the development of an effective cognitive ‘working model’ of psychosis that increases empathy and understanding towards the service user47

• a greater sense of being able to cope with the problems they face (self-efficacy)13,48

• a less emotionally charged relationship49–51

• a sense of feeling better supported by others. 32

Implementing support for relatives

The UK government recognises the need to support relatives in a caring role,52 and the National Institute for Health and Care Excellence (NICE) recommends that all relatives be given carer-focused education and support and offered a structured family intervention to enhance family coping and communication. 53,54

However, a recent national audit of EIP services in the UK showed poor implementation, with only 50% of relatives receiving a carer-focused education and support programme and only 31% being offered a structured intervention (of whom only 12% took up the offer). 55 A survey of > 1100 families across 22 European countries showed that half were dissatisfied with how they were involved by mental health services,56 suggesting that it is unlikely that services are better elsewhere.

The challenges services face in providing structured family interventions are likely to include cost and the practicalities of delivery. Many require two trained therapists, available at the same time weekly for up to 9 months with regular access to supervision, as well as a family whose members can all commit to attending face-to-face sessions, often during office hours, and have a sufficiently open and robust relationship with the service user.

However, these factors do not account for the lack of more straightforward psychoeducational approaches and emotional support for relatives. It is likely that additional organisational factors also play a role in poor implementation.

For example, staff workload is often measured in terms of the number of service user contacts, which may not adequately recognise time spent with family members. Under pressure, this may be the first thing to go. As well as highlighting the need for organisation-wide change, a study by Eassom et al. 57 identified staff reservations about the level of involvement family members should be given in the recovery process, fear of negative outcomes as a result of involving family members and the need for an exclusive patient–professional relationship.

Attempts have been made to improve access to support for relatives. Achieving Better Access to Mental Health Services by 2020,58 published in October 2014 by NHS England and the Department of Health and Social Care, sets a key standard that ‘more than 50% of people experiencing a first episode of psychosis will be treated with a NICE approved care package within two weeks of referral’ (contains public sector information licensed under the Open Government Licence v3.0). This includes carer-focused education and access to structured family interventions. Clinical Commissioning Groups (CCGs) have been directed to allocate funding to EIP teams to support efforts to meet this standard, the success of which is being audited by the College Centre for Quality Improvement in the Royal College of Psychiatrists. 55

In addition to government policy, the Triangle of Care59 has been developed by the Carers Trust and the Mental Health Development Unit as a ‘guide to best practice’ to help mental health services improve collaboration and partnership with carers. The Triangle of Care sets out six key standards, which have already been adopted by a significant number of NHS trusts. However, the extent to which these are being met is currently monitored through self-audit only, which may limit their impact on practice.

In this context, our aim was to develop a user-friendly, easily accessible self-management intervention, based on the principles of psychoeducation and family intervention, that could overcome some of the identified barriers to implementation, and be made available to all relatives of people with psychosis or BD across the UK. Specifically, we were keen that the intervention would target key appraisals and coping strategies, be empowering for relatives, not require extensive staff time or training, be low-cost and require little input from mental health services. Self-management interventions that have the flexibility to be used alongside other work and family commitments and augment other forms of support are ideally suited to meet the needs of relatives, and have the potential to be widely available in an increasingly resource-restricted health service.

Initial development of REACT

REACT was first developed as a paper-based self-management intervention for relatives of people with psychosis in EIP services as part of a National Institute for Health Research (NIHR) Research for Patient Benefit-funded study. 60 The content was informed3 by:

• our systematic review of interventions for relatives of people with psychosis,39 highlighting the distinguishing features of effective interventions

• focus groups of relatives3

• cognitive–behavioural therapy models highlighting the importance of helping relatives to understand psychosis and build on existing strategies

• the research team’s own expertise.

REACT consisted of 13 sections, designed to be used flexibly depending on the individual needs of relatives. The 13 sections were as follows:

1. Introduction to REACT

2. What is psychosis?

3. Managing positive symptoms

4. Managing negative symptoms

5. Dealing with crises

6. Dealing with difficult behaviour

7. Managing stress: thinking differently

8. Managing stress: doing things differently

9. Understanding mental health services

10. Treatment options

11. The future

12. Resource directory (RD)

13. Jargon terms.

Through this combination of modules and the directory, the toolkit aimed both to support users directly, by providing advice, information and case studies, and to connect them to other sources of advice and support. The toolkit was printed in A5 format and given to each participant. It could also be read online or downloaded from a website made available to participants, although this was very little used and relatives made clear in feedback that they preferred a resource that they could hold in their hands. Most sections were < 24 pages: the shortest was 11 pages, the longest was the RD, at 43 pages. The large number of links to other services and resources meant that the toolkit required constant editing and updating to maintain its accuracy.

Support in using the toolkit came from EIP support workers, who conducted one-to-one introductory sessions with each participant, with ongoing support available for 6 months by telephone or e-mail, as preferred, limited to 60 minutes per week. Participants who did not contact their support worker or who missed appointments were called at least monthly to maintain engagement. The intention of this personalised support was to help relatives to identify their greatest challenges and to then navigate the toolkit to find information or useful strategies that could be used. The focus was on practical engagement rather than transmission of information, with relatives encouraged to adopt and adapt new strategies, acquire and use new skills and reflect on the results: an active rather than passive approach.

The feasibility and acceptability of the toolkit were tested in a randomised controlled trial (RCT), in which the toolkit was offered in addition to treatment as usual (TAU). Relatives’ distress and well-being outcomes were compared at 6 months’ follow-up against those receiving TAU. This trial showed that relatives were very keen to engage with the intervention, that staff could integrate delivery of REACT into existing EIP services, and that relatives who received REACT were significantly less distressed than those receiving TAU at 6 months, as measured by the General Health Questionnaire-28 items (GHQ-28)61 [regression coefficient –6.59, 95% confidence interval (CI) –12.55 to –0.64]. They also had higher levels of perceived support [measured using the Carer Well-being and Support (CWS) Questionnaire:62 regression coefficient 4.86, 95% CI 0.77 to 8.96] and perceived ability to cope (measured using the Family Questionnaire:63 regression coefficient –4.89, 95% CI –9.34 to –0.44) than those receiving TAU.

As well as strongly preferring the printed toolkit over the online version, relatives showed a slight preference for accessing support by telephone rather than e-mail. There were practical challenges in retaining trained NHS staff for the trial and in maintaining their allocated time to support the intervention.

Qualitative feedback from relatives who used REACT was extremely positive. 64 Relatives reported feeling less isolated and more supported as a result of REACT:

Say it were a Saturday night and they aren’t available, I still know, come Monday morning, I could ring that number, and that could help me through that weekend.

When asked what they felt REACT had changed for them, relatives reported changes in their ways of thinking about the service user’s behaviour:

[It] made me see things differently from a different point of view.

They also reported changes in their coping strategies:

I now know how to say things to David – rather than, I would have said, ‘Oh David, do you have to smoke? Your teeth are going to be yellow,’ I will say something now like, ‘Oh David, you know, you are so nice looking, you would be even nicer if you didn’t smoke because smoking can actually make your teeth go yellow.’ So I know how to word things more to make him feel not guilty about the smoking.

They also reported feeling more supported as a result of the intervention:

Oh, very reassuring. It [support] saved my life. I know that sounds melodramatic, but it saved my life; I feel as if it saved my sanity in a way.

Subsequent development of REACT

Based on qualitative feedback from relatives using REACT in the feasibility trial, as well as input from the patient and public involvement (PPI) group and from clinical academic experts in our research team, we proposed the following goals for the next version of the intervention:

1. to adapt REACT for a broader range of relatives, including those outside EIP and those supporting people with BD

2. to make a more interactive version of REACT, available online to relatives

3. to have expert relatives offer REACT support rather than NHS staff.

The reasons for these goals are explained in the following sections.

Have expert relatives offer the REACT support rather than NHS staff

Offering REACT to all relatives, including those not in contact with mental health services, and putting it online allowed us to develop a single dedicated team to support the intervention. This gave us greater control over the nature and design of the support. We chose to employ peer workers – relatives with lived experience of supporting someone with a mental health problem – for several reasons.

Our PPI group strongly favoured this approach. Our co-applicant (LCh) had worked with the charity Rethink for many years, sharing her knowledge and experience to support other relatives. There was a shared perception that peer workers would be highly knowledgeable, empathetic and motivated to support other relatives. This was reiterated in the design workshops that we conducted to develop the online version of REACT (see Chapter 2).

Qualitative and observational studies have identified a number of benefits of using peer workers, both for those receiving support and for the peers themselves. 73 For service users, these benefits include reduced admission rates; greater empowerment, empathy, acceptance and hope; better social functioning; and less stigma. Benefits reported by peer workers include support for their own recovery and personal growth, acquisition of new skills and the therapeutic effect of helping others. For paid peer workers, there are also financial benefits. 74,75 Employing relatives as REACT supporters to support the REACT website was an opportunity to develop the strategic role of service users and carers in delivery of services and explore the benefits and challenges of this role. It also circumvented the difficulty of identifying existing staff who had the expertise, time and support to take on this role. Developing the peer worker role as part of the NHS workforce was also consistent with recommendations from NICE in 2016. 76

Securing funding to evaluate REACT

In January 2013, we applied to NIHR for Health Technology Assessment (HTA) programme funding to deliver these adaptations and test the clinical effectiveness and cost-effectiveness of the new version of REACT in a large definitive trial. The application was rejected (13 December 2013), as the trial was seen as too expensive to deliver, at £2,032,667. Given the positive outcomes we had already demonstrated and the correspondence of the intervention to NICE guidance requirements to offer all carers psychoeducational and emotional support, the HTA programme funding committee suggested that we consider how best to deliver REACT in the NHS.

We then responded to the HTA programme efficient design call for RCTs that tested efficient trial designs to address the problem of the escalating costs of large-scale definitive trials. This offered the opportunity to test the online REACT intervention in an entirely online trial. Advantages of online trial design include the potential to reach a greater number and range of participants; to reach a population more representative of those likely to use an online intervention; to recruit more people over a shorter time frame; to simplify protocols for secure randomisation and data entry; and to deliver a trial much more cheaply because fewer staff are needed. 77 However, retention rates for such trials can be low,77,78 compromising internal validity of the trial.

This efficient design reduced the costs of the trial to £633,404. We were awarded funding for 36 months, to begin in October 2015.

In parallel, we applied to the NIHR Health Services and Delivery Research (HSDR) programme for funding for an implementation study to identify the factors affecting successful delivery of an online intervention within EIP services. This study was also funded for 30 months, starting in March 2016. The two studies were complementary. The HSDR study explored factors affecting implementation of a clinician-supported intervention in EIP services in the NHS, in contrast to this study’s peer worker-supported intervention. It also included an evaluation of outcomes for the relatives, using the same measures and follow-up period as this study. We thus had the potential to compare both the reach and the outcomes achieved by providing REACT through two very different study designs. This allowed us to answer questions about which was likely to be the most effective service provision model, as well as to compare the effectiveness of peer worker- and clinician-supported approaches. The HSDR report was submitted for publication in December 2018.

Trial design

The REACT trial is a single-blind, parallel online RCT to determine the clinical effectiveness and cost-effectiveness of REACT, including an online RD, compared with the RD only, for relatives of people with psychosis or BD. No changes were made to any other support that relatives received outside the trial. See Chapter 3 for the specific objectives of the trial.

Research team

The team included relatives, clinicians, academics and methodologists from a range of disciplines, with a common interest in developing and evaluating new ways to support people with mental health problems and their relatives. Team members were all UK based, and, although some had worked together previously, the team in its entirety came together specifically for this project. Although the team all differed in background, training, epistemological and ontological stance, some important factors underpinned the team in working together successfully:

• a commitment to improving the lives of people with mental health problems and their relatives in non-stigmatising, empowering, and recovery-focused ways

• a recognition of the huge role that relatives play in supporting people with health problems, and of the current lack of adequate support available to them

• a belief in the importance of evidence-based health care and the need to carry out high-quality research to inform how NHS funding is spent

• an interest in testing the potential of digital technology to increase accessibility and reduce costs of clinical interventions, while also exploring the challenges of and barriers to this approach

• a commitment to identify efficient ways to carry out publicly funded research to provide value for money to the UK taxpayer.

Trial monitoring

Patient and public involvement

Consistent with A Framework for Mental Health Research,79 this project was developed and conducted in partnership with service users and carers.

Introduction

In this chapter, we describe how the paper-based REACT intervention (see Chapter 1) was adapted to an online version, and describe the resulting content. The three key adaptations, based on learning from the feasibility trial, were to:

• broaden REACT to make it suitable for relatives outside EIP services and for relatives supporting people with BD

• make REACT more interactive and directly available online to relatives

• offer REACT with support from trained relatives.

In the feasibility trial of the paper-based REACT, modules were posted online as PDFs to aid availability. Clearly, this is not the same as an online intervention, which would typically be more structured, self-guided, interactive, visually rich and personally tailored. Therefore, considerable work was needed.

Following guidance from the Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth (CONSORT-EHEALTH) checklist, version 1.6.1,81 we describe the history, development and initial formative evaluations of REACT, details of how the dynamic components worked and how the site was updated. We provide relevant screenshots and have archived the site (access available on request). We also present the development and content of the RD that was offered as our active control intervention. As required by the Consolidated Standards of Reporting Trials (CONSORT), we identify the sponsorship and ownership of the site.

The aim was to involve relatives, clinicians and the clinical academic team in co-developing content and design to ensure that both would be of high quality, user friendly, meet the needs of relatives and engender confidence in potential referrers. The importance of understanding the needs of users and involving them in design has been well reported. 82 We carried out a series of workshops with users (relatives) and drew on the expertise within our clinical academic team to develop a list of design features, which guided our development of REACT.

Understanding user perspectives: method

We held two workshops (workshops 1 and 2, with 13 and 11 participants, respectively) to explore the needs of relatives outside EIP services, and their views about how REACT should be developed. We aimed to recruit participants from diverse backgrounds in terms of age, gender, relationship with the service user, length of experience as a caregiver and computer literacy. Workshop participants were invited if they identified as supporting a relative or close friend with psychosis or BD. Our recruitment strategy was to advertise locally to obtain a convenience sample that could attend face-to-face workshops. Advertisements were circulated via Lancaster University’s Spectrum Centre for Mental Health Research,83 using e-mail, social media and post. The Spectrum Centre is a multidisciplinary research centre focusing on the development and evaluation of psychosocial interventions for people with long-term mental health problems.

In accordance with our approval from the Lancaster University Ethics Committee, those who indicated interest were sent an information sheet by e-mail or post before the workshop, allowing them to raise any questions in advance. This information included the purpose of the workshop, its location and duration, that participation was voluntary, and what would happen to their audio-recorded data.

Participants were offered a £20 Amazon (Amazon.com, Inc., Bellevue, WA, USA) voucher in recognition of their time and input. Two researchers facilitated each workshop, using a semistructured topic guide to lead discussion and ask open questions to elicit a range of views.

First, the participants were encouraged to reflect and share their lived experiences as caregivers of people with psychosis or BD. This included how their relative was first diagnosed, how they became involved as caregivers, the impact of mental health problems on their family and daily life, and their current sources of support, as well as the types of support or strategies they found most useful.

Second, they were asked their views on gaps in the support system for caregivers, how to fill these and whether or not online support could play a role in this. Finally, they reviewed the REACT booklets together and discussed whether or not and how these could be redesigned as a web-based intervention, and what additional support might be needed to facilitate REACT’s use online. Each workshop was audio-recorded with participants’ permissions, and transcribed verbatim for later analysis (see the data analysis section).

We conducted an additional workshop (workshop 3) with two participants to scope the types of features that could be included in the web-based REACT intervention. Both participants were comfortable using computers, used social media frequently and were interested in helping with the design of REACT. Before the workshop, participants were given access to the website of the PDFs. They were also asked to identify other websites that they liked or disliked, both related and unrelated to mental health. We explored their aesthetics and functionalities together. This was followed by a visual demonstration of a series of design prototypes for the web-based REACT intervention, based on our findings from workshops 1 and 2. The prototypes were developed in conjunction with a web design company and mainly focused on aesthetic aspects of the web interface, such as logo, font style and size, navigation menu, colour scheme and multimedia choices.

We wanted to know how to translate the values and needs that participants had highlighted in workshops 1 and 2 into functionalities. In particular, we wanted to understand how best to give users a positive experience. We therefore asked participants in workshop 3 to discuss how to design the web-based REACT to be more engaging, what features might motivate users to keep returning and what types of support could be offered only online.

A full description of the findings of all three workshops is reported elsewhere,1 and is summarised in the next section.

Understanding user perspectives: results

The majority of participants in workshops 1 and 2 were female, aged > 45 years, parents and infrequent computer users. Most had had many years’ experience of caring for relatives with severe mental illness (on average, about 10 years). In workshop 3, one participant had taken part in workshop 1 and was typical of that group, whereas the second participant was aged 21 years and new to the study. In total, 25 participants took part in this qualitative study across all three workshops (18 females, age range 21–75 years, n = 20 parents). Participants comfortably shared their experiences, and no prompting was required for conversation to flow. Key findings covered their caregiving experiences, the support that they believed was needed, the potential role of online interventions and the design of such interventions.

Creating the design brief

Based on these findings, we drew on the expertise of the TMG to create a design brief for building REACT. The TMG included a relative with extensive experience of supporting other relatives through a leading charity, and a consultant psychiatrist, two consultant clinical psychologists and a GP, all with expertise in supporting people with mental health problems and their families, and some with expertise in designing and delivering digital health interventions (SJo, EM). The brief was as follows.

Building and hosting the REACT intervention

REACT was initially built in WordPress (Automattic Inc., San Francisco, CA, USA) by a web design and hosting company local to Lancaster; initially, the plan was for the company to host and maintain the REACT site during the trial. However, owing to difficulties in establishing a sustainable plan to do this, the site was brought in-house and is now located on a dedicated virtual server at Lancaster University. It is maintained and updated bimonthly by a digital technology developer from the REACT team (AW).

To meet the design brief, the WordPress site required a number of plug-ins. These included standard ones, such as bbPress (Automattic Inc.) to run the REACT group forum, and some built bespoke for the REACT site.

Appearance

Screenshots in Figure 1 show the look and feel of the REACT intervention.

FIGURE 1.

Screenshots of the REACT intervention website.

The REACT supporters

The role, training and supervision of the REACT supporters are briefly summarised in the following paragraphs. The detailed manuals for the REACT supervisors and the REACT supporters, as well as the REACT risk protocol, can be found on the project web page [www.journalslibrary.nihr.ac.uk/programmes/hta/144934/#/ (accessed 28 October 2019)].

Logic model for REACT

For any intervention, it is important to make explicit the underlying theoretical basis. Figure 2 outlines the process by which we hypothesised that REACT would benefit relatives.

FIGURE 2.

Hypothesised mechanism for REACT.

Reflections on developing REACT

REACT drew on evidence-based cognitive and behavioural psychological theories and clinical practice. The content had been developed and refined over many years, with extensive input from relatives, clinicians and service users in an early-intervention context. Therefore, in building the online version, we already had a good idea of the problems faced by relatives and an understanding of the processes underlying these difficulties and the strategies that could be successfully used to address them.

Even so, in broadening the reach of REACT to relatives of people outside EIP services and those with bipolar experiences, we felt that it was important to explore further what challenges these groups experienced and what support they valued. We also wanted to investigate relatives’ views about online interventions. This information was invaluable in helping us to build a design brief, and, ultimately, to develop the online version of REACT in time to start the trial. However, there were many challenges along the way, and we learnt a lot in overcoming them.

The drive to translate REACT from a paper-based to digital intervention came primarily from the practical need to have something that was easy to update, cheap to deliver and consistent with the NHS’ direction of travel towards digital NHS. 89 However, the extent to which service users and relatives wanted to receive support online was not clear. Many studies showed support for digital health interventions (DHIs) in mental health, but none of the studies was from representative samples of the service-using population, and most studies had recruited small convenience samples of people interested in digital health. 90,91 The relatives in our workshops were also a convenience sample, but were invited to the study to talk more broadly about their experiences of supporting someone with a mental health problem, and so may have been less biased in their views about online support. They had many reservations about the use of online interventions, but also suggested that this might be a cohort effect, and that a younger population would perhaps be less concerned and more comfortable with this mode of delivery.

Although all the relatives in our sample could access online support, some were limited in their access to private spaces to use computers, and in their connectivity, skills and confidence. There is some concern that, rather than broadening access, online interventions might, in fact, narrow access, dividing those with and those without easy access to online computers, and also those with and those without the skills, confidence and motivation to go online. 92 In many ways, it was helpful to hear the concerns of these relatives, as it made us think about how to address these in our design. However, involving more regular users of online technology might have generated ideas for additional functionality. For example, live social media feeds are now a standard feature of many online interventions and, with hindsight, might have been of benefit by ensuring constant new material that brought people back to the site.

Drawing on the extensive work done to develop the paper-based REACT also gave us a text-heavy starting point, which required a lot of reverse engineering to produce content more relevant to an online intervention. Starting from scratch might, in some ways, have been easier and led to a more digitally friendly product.

We wanted REACT to be built by experts who could work face to face with the relatives in our RAG. We did not have an existing relationship with any digital health companies, and Lancaster University did not, at the time, have a list of preferred providers. We therefore put the build of REACT out to tender among local companies. We had a limited budget, which proved to be well below the costs quoted by many companies offering bespoke builds. We knew that REACT would need to evolve, with edits made over time, and that staff would need training in how to use the site, all of which would make us reliant on those who could write the code, further increasing ongoing costs. We also recognised that many web-development businesses do not survive, and so we were wary of stranding ourselves with a bespoke site built in a code we could not then edit.

Therefore, we employed a local digital technology company that quoted within our budget, with some experience of building attractive websites using the open-source WordPress programme. However, during the build process, we became concerned about the company’s ability to deliver secure storage and regular updates for the website; therefore, we decided to bring the project in-house. The site was moved to the university server, we ensured that all identifiable user data were stored separately at the Liverpool CTRC, and all edits and updates were done by IT experts in the REACT team. This proved to be a wise decision as the company ceased to trade fewer than 12 months later.

Although a genuine partnership with a digital technology company would have offered many advantages, including greater investment in the product design and support for wider dissemination, there were also many challenges with this model. Digital technology companies are rarely run as not-for-profit social enterprises; therefore, they will look to maximise profit from the NHS. Given the NHS’s increasing reliance on digital technology, costs to the NHS are likely to rise exponentially. This may cause serious financial problems in the future.

Finally, we know that having people to support digital interventions is crucial to the success of such interventions,93,94 but how and by whom this is best done remains to be answered. The relatives in our workshops talked extensively about the power of peer-to-peer support. This is consistent with feedback from our feasibility trial, and with growing support for the development of peer worker roles in mental health services around the world. 95–100 In developing the design brief for this role, we identified a range of new skills that need to be learnt (e.g. managing online forums and communicating effectively in text format) and existing protocols that need to be adapted, such as risk management. These were new to and challenging for not only the REACT supporters, but also supervisors. Considerable time and effort were therefore needed to provide training, support and development for all members of the team.

Introduction

This chapter outlines the methods used to assess the clinical effectiveness of REACT. This chapter follows the structure of the CONSORT statement,101 including the extensions for pragmatic trials102 and for eHealth interventions. 81 As our data were collected online, we have also reported data collection in line with the Checklist for Reporting Results of Internet E-Survey. 103

Trial design

This was a primarily online two-arm, pragmatic, single-blind, individually randomised controlled superiority trial. The trial is described as primarily online because all of the intervention and most of the data collection occurred online. However, some text, telephone and postal reminders were used to support the registration process and to maximise retention.

Setting

This trial took place online in the UK. It was hosted by one NHS foundation trust, and other trusts and CCGs were eligible to take part as participant identification centres (PICs). Recruitment also took place through local and national mental health charities, media, social media and Google Ads (Google Inc.). REACT was built using WordPress open-source software, and hosted and maintained at Lancaster University. Data were collected using a bespoke web-based system at the Liverpool CTRC.

Participants

Eligibility criteria were designed to be as broad as possible. Inclusion criteria were (according to self-report):

• Aged ≥ 16 years.

• Living in the UK.

• Relative or close friend of someone with psychosis or BD.

• Currently experiencing distress due to their relative or close friend (determined as selecting ‘rather more than usual’ or ‘much more than usual’ on the GHQ-28 item ‘Have you recently been feeling nervous and strung up all the time?’). This was included to avoid a floor effect on levels of distress at baseline, and was the item that correlated most highly with the GHQ-28 total score in the REACT feasibility trial.

• Currently seeking help (self-identified), and, therefore, likely to engage with support offered.

• Access to an internet-enabled computer.

• Sufficient English fluency to comprehend the intervention content.

Exclusion criteria included living in any of the six geographical areas by postcode taking part in the parallel IMPART study104 of the same intervention. In addition, only one relative per service user was allowed to participate, to avoid a clustering effect.

Recruitment strategy

Recruitment took place from 22 April 2016 to 30 September 2017. Before this, relatives who visited the website were invited to leave a contact e-mail address. A range of online and offline recruitment strategies was used, all directing potential participants to the trial home page (www.reacttoolkit.co.uk; accessed 1 June 2019), which provided information about the trial and how to take part.

At registration, participants were asked how they had found out about the REACT trial. The number of participants coming into the trial through each avenue was monitored by the TMG at its monthly meetings, and the recruitment strategy was adapted accordingly.

Social media

Twitter (Twitter, Inc., San Francisco, CA, USA)

Studies of Twitter (www.twitter.com) as a recruitment tool for online studies have been mixed: some found it unsuccessful,105 others successful. 106 Given its large reach and free nature, we set up a REACT Twitter account. The research team posted several times a week about the trial and relevant mental health news, and encouraged colleagues with Twitter accounts to tweet or retweet. We posted or retweeted 325 times and gained 711 followers.

Facebook

Facebook (www.facebook.com) has been used successfully for recruitment in previous studies. 107 We set up a REACT Facebook page to post relevant mental health and trial-related news. After an expensive and fruitless attempt at creating Facebook advertisements ourselves, we enlisted an agent for this purpose, funded by one regional clinical research network for 13 months and by another five networks across the UK for the last 7 months.

In all, 70 carousel and standard-type advertisements were used, targeting seven geographical areas. We split test each variant, reallocating the spend to the best-performing advertisements. The standard advertisement type outperformed the carousel advertisements, and the more obvious images, containing text that reiterated the messaging, outperformed other advertisements. The best-performing advertisements are shown in Figure 3.

FIGURE 3.

Screenshots of Facebook advertisements.

Advertisements were viewed on mobile devices over desktops by 40 : 1. Over 13 months, we reached 873,096 individuals; 53,216 people engaged with an advertisement (liking, commenting or sharing) and there were 71,026 clicks through to the REACT website.

Consent

We adhered to the British Psychological Society’s guidelines for taking informed consent online. 109 These included taking a record of valid consent; including checkboxes relating to specific consent statements; limiting the number of consent items; and ensuring participants were fully informed of trial procedures, risks, confidentiality and right to withdraw. Capacity to consent was assumed, as there is little scope to assess it in an online trial. All participants were e-mailed a Microsoft Word (Microsoft Corporation, Redmond, WA, USA) copy of the completed consent form, with a version kept on a secure server and in the trial database.

Participants were informed that they were free to withdraw at any time, but that online questionnaire data could be removed only within 2 weeks of data collection, and web usage data, online forum posts and direct messages could not be removed and would be used for research purposes. Participants were informed that data would be stored on secure servers at the Liverpool CTRC and Lancaster University. Note that the trial predated the European Union General Data Protection Regulation. 110

Interventions

The design and delivery of the REACT trial were described in detail in Chapter 2. Participants were randomly allocated either to the REACT arm (TAU plus toolkit including the RD) or to TAU plus the RD. Those in the REACT arm could access the toolkit with a username and password, without charge, whenever they wished throughout the trial (for a minimum of 24 weeks after the final participant was recruited). REACT supporters were available on weekdays from 9.00 to 16.30, excluding public holidays and university closures (1.5 weeks at Christmas and 1 week at Easter). Participants were advised to use the intervention as they needed. No changes were made to existing support and treatment.

Participants allocated to the RD-only arm logged on to the same website, but could see only the ‘meet the team’ and RD pages. At the end of the trial, they were given access to the modules, without the forum or direct messaging.

Objectives and hypotheses

The aim of this RCT was to determine the clinical effectiveness and cost-effectiveness of REACT (including online RD) and TAU, compared with TAU and the RD only. This comparator was chosen to test the effect of offering REACT as an additional intervention to the support that relatives could already access. Relatives were likely to access support from a wide range of services. In this context, TAU was used to indicate that no attempts were made to make any changes to any of these other sources of support.

The objectives were to determine the following:

• the impact of the REACT intervention on relatives’ distress

• the impact of the REACT intervention on relatives’ well-being and support

• the impact of the REACT intervention on hypothesised mediators of change, including relatives’ beliefs, perceived coping and amount of use of REACT

• the delivery and maintenance costs of the REACT intervention

• the incremental cost-effectiveness ratio (ICER) of the REACT intervention

• the key issues for which relatives seek support.

The primary hypothesis was that there would be a significant difference (p < 0.05) between the two arms of the trial in the GHQ-28 scores at the 24-week follow-up.

Outcomes

All outcomes were collected online using a closed system available only to participants with an account on the REACT site. Questionnaires were all validated self-reporting measures, although not originally designed for use online. These were presented in order of priority to the study (primary outcome first) and used drop-down options and checkboxes. Participants were required to respond to all items before moving on to the next measure. At baseline, participants were required to complete all measures before being randomised. Following completion of each questionnaire, the data were submitted and participants were unable to edit them.

Mediators

To test the proposed mediators of change in relatives’ outcomes, participants also completed online versions of the Brief Illness Perception Questionnaire (Brief IPQ),117 a 15-item Likert scale assessment of beliefs about psychosis and BD, with an additional single item to assess perceived coping; and the Brief Coping Orientation to Problems Experienced (Brief COPE),118 a 28-item measure widely used to assess coping styles.

The REACT was designed using a cognitive–behavioural framework, which proposes that the way in which relatives think about the challenges they face, and the things they do in response, will determine their levels of distress. 119 The Brief IPQ allowed us to assess the way in which relatives made sense of the psychosis or BD, namely their cognitive models. The Brief COPE inventory allowed us to assess the specific strategies they were using to manage their distress.

Sample size

We aimed to recruit 666 relatives of people with psychosis or BD to accurately test the primary hypothesis that there would be a significant difference (p < 0.05) between the trial arms in distress, measured by GHQ-28 scores at the 24-week follow-up.

Our feasibility trial2 had shown a mean difference between arms at 6 months (controlling for baseline) of 6.59 units [standard deviation (SD) 16.6 units] in favour of the REACT arm.

To build a degree of protection against pilot results proving optimistic, and to accommodate adaptations to the design of the study and the intervention, we reduced our estimate of the mean difference in this trial from 6.59 to 5.0 units. A detailed qualitative and quantitative analysis of our feasibility data suggests that a (within-relative) reduction of 3 units on the GHQ-28 can indicate clinically meaningful change; however, the minimum difference required for the between-arm comparison (of change in GHQ-28 score from baseline) was set at 5 units, to justify the staff and resource costs associated with delivery of the intervention.

We retained our estimate of the SD of 16.60 from the feasibility study, consistent with other studies using this measure with relatives in EIP services13 and somewhat higher than those from other mental health or dementia services. 120,121 A total of 666 (333 per arm) participants provided 90% power to reject the null hypothesis (p < 0.05) with an effect size of 5.0 units, assuming a 30% dropout rate by 24 weeks. Although dropout was only 17% in our feasibility trial, it tends to be higher in online trials. 78

Data collection

Baseline measures

Demographic information and baseline measures were completed before randomisation. To make the process less daunting, questionnaires were paginated rather than appearing as one long web page. Validating algorithms ensured that all questions were completed and inappropriate answers minimised. These methods were reviewed by the RAG to ensure that they were fit for purpose. Demographic data included age, gender, ethnicity, marital status, education, employment, living arrangements (including dependents), the primary diagnosis of the service user, length of time in caring role, number of people they were caring for, relationship to person(s) with mental health problem, whether or not they lived with the person(s), level and type of contact, whether or not they received support from NHS services and internet access.

On completion of the baseline questionnaires, participants were randomised to either the REACT arm or the RD-only arm, using a list generated by the statistical team.

At the 12- and 24-week follow-ups, up to three reminder emails were sent at 5-day intervals. If data entry had still not started, the trial manager was prompted to send a personalised e-mail, and, finally, to try to make contact by telephone to ask if the participant would complete the GHQ-28 (primary outcome) over the telephone (with the trial manager entering the data) or in writing, in which case the researcher would download and post a personalised letter and questionnaire. Returned data would be entered manually by the researcher. Towards the end of the trial, the researcher was also able manually to trigger a SMS with a unique URL for the GHQ-28. To identify participants who had completed some of the measures but stopped mid-way through the process, the trial manager’s dashboard displayed how many questionnaires had been completed at each stage.

The REACT participant information sheet, available on the website, explained the withdrawal process. A link to withdraw from the trial featured in all reminder emails, and in the toolkit. Withdrawing participants were required to select from a drop-down menu of options:

• ‘I don’t want to complete the 12-week follow-up questionnaires but I am happy to be contacted for the 24-week follow-up’.

• ‘I don’t want to complete ANY more follow-up questionnaires but I would like to continue to use the website. Note that any data inputted to the website will continue to be used for research purposes’.

• ‘I don’t want to complete ANY more follow-up questionnaires and DO NOT want to use the website anymore. Note that your access to the website will cease’.

Participants were also invited to provide an optional reason for withdrawing:

• ‘I don’t have time due to other commitments’.

• ‘I didn’t like the website I was given’.

• ‘I don’t like filling in the questionnaires’.

• ‘I don’t feel well enough to take part’.

• ‘Other (please specify)’.

Researchers would ask the same sets of questions of participants who withdrew over the telephone or by e-mail, and enter this information into the system. Withdrawal also blocked further automated reminders.

The data collection process is shown in Figure 4.

FIGURE 4.

Data collection process. a, Accessed 1 September 2018.

The REACT trial registration process incorporated a number of strategies to enhance engagement:

• lay-language speech bubbles

• a progress bar appeared at the top of each registration page, and there was an indication of how long each stage should take

• an automated e-mail reminder with detailed instructions was sent if participants had not progressed within 24 hours of giving consent

• an automated e-mail reminder was sent 24 hours after participants completed activation and 7 days after starting the baseline questionnaire

• e-mail reminders were copied to the trial manager to check that the registration process was working correctly

• the option to e-mail the trial manager during registration to get e-mail or telephone support through the process

• a dashboard that allowed the trial manager to monitor progress and identify those who needed chasing.

Retention strategy

Dropout from follow-up is a particular problem in online trials. 78 Drawing on lessons from previous studies,77,107,122–124 we used the following strategies to maximise follow-up:

• We randomised participants only once baseline assessment measures were completed.

• We included detailed explanations in our recruitment materials about why data completion at follow-up was so important.

• We obtained multiple contact details at registration.

• We sent multiple reminders by different methods and with different options for completing online measures, based on PPI feedback, striking a balance between cost, data and burden on participants.

• We gave £10 or £20 vouchers as incentives to complete follow-up questionnaires (see Appendix 2).

• We informed RD-only participants that they would be able to access toolkit modules after the final follow-up.

Blinding

All data were self-reported and predominantly entered online by participants. Outcome questionnaires submitted by post were recorded as such and inputted by the trial manager, who was blinded to allocation. Data were uploaded directly to the CTRC database. The system allowed only valid values to be entered. To prevent any bias in the conduct of the trial, the chief investigator, trial manager and statisticians were blinded to treatment assignment. The REACT supporters, clinical supervisors, qualitative interviewer, one CTRC analyst of web usage data and technical staff (AW) were unblinded. Participants were also unblinded.

To minimise unblinding, all contact with participants was prefaced by a reminder not to disclose their trial arm. If the trial manager was unblinded regarding a particular participant, another blinded team member delivered any non-automated reminders and carried out any data entry for that participant. All instances of unblinding were recorded.

Statistical analysis

A full statistical analysis plan was published on 22 March 2017, before the start of data collection, and updated on 20 December 2017, before the end of data collection. Both versions are available online. 125

Updates to the plan are listed in Appendix 1. All analyses were done using SAS® statistical analysis software version 9.4 (SAS Institute Inc., Cary, NC, USA; SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration) or Stata® version 14 (StataCorp LP, College Station, TX, USA).

Causal analysis

To investigate the relationship between website use and outcome, we recorded data on baseline covariates (correlated with website use and outcome) and relevant website use (from participants in both randomised arms). Instrumental variable (IV) regression was used to estimate the impact of intervention use on the outcome using the IV regression (ivreg) command in Stata.

Intervention use (i.e. web page downloads from the REACT intervention site excluding the RD) was summarised as a single continuous covariate derived from web usage data. Given the extensive password protection of the REACT site, it was assumed that participants in the control arm had null intervention use.

Exploratory analyses of the total number of log-ins and the total time spent logged in over 24 weeks as other measures of use were also conducted, and the suitability of randomisation as the instrument in this regression was assessed (see Report Supplementary Material 1).

Participants’ experiences of the intervention

Participants in the REACT intervention arm were asked to respond to the following statements at 12 and 24 weeks post randomisation (based on previously published studies):127

• ‘I always feel supported by the REACT supporters’.

• ‘I always feel supported by the REACT group’.

• ‘I always feel the REACT site is a safe and confidential environment’.

Options for each answer were ‘strongly disagree’, ‘disagree’, ‘agree’, and ‘strongly agree’.

Missing data

Participants were asked to complete the primary outcome measure (GHQ-28) before being presented with any other measures, to maximise primary outcome data collection. In addition, participants were unable to submit any questionnaire with a missing field; therefore, by design, no data were missing from questionnaires completed online. (This was not true for some items assessing the relatives’ caring role owing to an error in design and testing; hence, there are some missing values for these variables.)

The only instances of missing outcome data occurred when the GHQ-28 was returned by post. These questionnaires were included in the analysis if more than half of the questions in each of the four subscales had been completed; missing answers were given the mean value of given answers within that subscale for that participant. This approach was chosen following discussions with the TSC, with guidance provided by the Patient Health Questionnaire-9 items. 128

This simple approach was chosen as there were very few missing data entries (four missing questions from three patients at 12 weeks, and one missing question at 24 weeks) on which to base multiple imputation.

Data entered up to 18 weeks post randomisation were considered to be 12-week data; data entered beyond 18 weeks and up to 30 weeks post randomisation were considered to be 24-week data. Any duplication of 24-week data entered by a given participant (i.e. if the participant provided ‘12-week’ data beyond 18 weeks and later entered 24-week data) was addressed by choosing the data entered closest in time to 24 weeks post randomisation.

Missing completion dates for postal GHQ-28s were estimated using the mid-point between the date sent and the date received. If not recorded, the date of receipt was imputed using the mean number of days between sending and receiving questionnaires for all accurately recorded postal GHQ-28s.

Joint modelling of the longitudinal outcome data and the time to dropout was carried out using the stjm command in Stata, to demonstrate any association between these two processes. A longitudinal trajectory plot was produced using the subsidiary command ‘stjmgraph’; the graph showed dependence between the longitudinal profiles and dropout.

Monitoring of adverse events

Adverse events were assessed in terms of the number of participants for whom it was necessary to trigger the risk protocol, for both low-risk and high-risk events [see the project web page: www.journalslibrary.nihr.ac.uk/programmes/hta/144934/#/ (accessed 28 October 2019)]. Risks were identified in one of three ways:

1. ‘red flags’ raised by the system in response to answers to GHQ-28 items D3, D4, D6 or D7 that indicated possible risk to self, or to CWS Questionnaire questions 29 or 30 indicating possible risk from the person cared for, with the trial manager notified via the dashboard

2. by a REACT supporter through the forum or direct messaging

3. by the trial manager when contacting non-responders for follow-up (in both arms, with strategies to ensure that blinding was not broken).

Low-risk alerts triggered a standardised e-mail to the participant, expressing concern, checking they were OK, and pointing them to appropriate support. High-risk alerts, defined as clear evidence of immediate and serious risk to life or to child welfare, and referred to as ‘serious adverse events’, led to immediate contact with police or social services, as appropriate, including sharing participant details. Safeguarding concerns were also covered by the protocol.

Add-on studies

We conducted a study within a trial (SWAT) to investigate factors influencing the impact of incentives to participants (see Appendix 2) and a qualitative study to understand participants’ experiences of using REACT (see Chapter 6). We also plan to conduct a qualitative analysis of the posts on the REACT forum to identify the key issues that relatives raised in the online forum (see Chapter 6 for initial themes).

Ethics approval and research governance

Ethics approval was given by Lancaster National Research Ethics Service committee (reference number 15/NW/0732) on 21 September 2015. The eight ethics amendments (current version 1.8, dated 6 December 2017) can be found on the project web page: www.journalslibrary.nihr.ac.uk/programmes/hta/144934/#/ (accessed 28 October 2019).

Patient and public involvement strategy

One of the study investigators is a parent of someone living with psychosis and was extensively involved in the development of REACT, the RD and data collection processes. She was part of the supervisory team for REACT supporters.

We established a RAG, working primarily online, to give detailed feedback on REACT, the online data collection processes and the recruitment strategy.

Our REACT supporters were people with lived experience of supporting someone with psychosis or BD; as well as supporting the REACT intervention, they were involved in promotion, recruitment, interpretation and writing up the findings. A REACT supporter presented a paper as part of a symposium about REACT at the World Congress for Behavioural and Cognitive Therapies in Berlin in July 2019. The TSC included a relative and service user.

Data management, storage and security

All participant trial data were collected through an online system at Liverpool CTRC and stored on secure servers physically located in access-controlled server rooms and backed up nightly to a separate physical location. All identifiable data were encrypted using a 256-bit encryption algorithm. CTRC servers were subject to penetration-testing audits undertaken by University of Liverpool central IT staff. Website usage data, qualitative data from the REACT group forum and direct messages to REACT supporters were taken from the REACT website and hosted on a dedicated virtual private server at Lancaster University. All communication with website users was limited to Secure Sockets Layer- (SSL-)protected Hypertext Transfer Protocol Secure (HTTPS) protocol, to protect passwords and data in transit over the internet.

The REACT data are stored on a secure server at Lancaster University, which complies with relevant statutory provisions including the Data Protection Act 2018129 and the EU General Data Protection Regulation. 110 Data held on Lancaster servers are stored in a resilient storage infrastructure that is dual-housed in the university’s data centres (on site). There are multiple levels of redundancy built in to these storage arrays: snapshots and backups are automated and taken regularly.

University single sign-on (SSO) credentials are required to access the shared network drive; the principal investigator controls access to specific folders and ensures that this is monitored regularly.

Introduction

This chapter reports the findings from analyses to test the clinical effectiveness of REACT. The structure follows the CONSORT guidelines and the methods outlined in Chapter 3. A full statistical analysis is available in Report Supplementary Material 1.

First, we describe the flow of participants through the trial. We then list any deviations from our published protocol. 130 We describe the levels of intervention use in both arms of the trial. Next, we present our rate of recruitment, report the findings of the internal pilot, test the effectiveness of the online and offline strategies to maximise recruitment and explore the impact of the reminders protocol on retention.

We then present the baseline data for each of the key outcomes: distress (measured using the GHQ-28), carer well-being (measured using the CWS Questionnaire) and carer support (also measured with the CWS Questionnaire). We test the impact of the REACT intervention on each of the main outcomes (primary outcome: GHQ-28 score at 24 weeks; secondary outcomes: GHQ-28 score at 12 weeks and CWS Questionnaire scores at 12 and 24 weeks), and explore evidence for any causal impact. We test whether or not illness perceptions and coping strategies mediate any relationship between the intervention and GHQ-28 score, as hypothesised.

We present two additional analyses: testing the impact of participants’ baseline characteristics on the primary outcome and the impact of ‘lurking’ on the REACT group forum. Finally, we report the adverse events that occurred during the trial.

Technical issues that arose during the trial are set out in Report Supplementary Material 1, Table 6–6. The resulting deviations from protocol are set out in Appendix 1, Table 54.

Participant flow

The total number of visits to the REACT trial page from 22 April 2016 to 30 September 2017 was 451,832. The total number of visits to the trial registration page was 4348.

Figure 5 shows the flow of participants through each stage of the trial.

FIGURE 5.

The CONSORT flow diagram. Reproduced from Lobban et al. 131 © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (https://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. The figure includes minor additions and formatting changes to the original figure.

Reproduced from Lobban et al. 131 © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (https://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. The figure includes minor additions and formatting changes to the original figure.

Of the 3287 people who completed the eligibility screening, 1528 (46%) consented to the trial. Of these, 343 failed to proceed through the registration process and 1416 people failed on at least one of the eligibility items. The number failing on each eligibility item is shown in Table 1 (note that 55 people failed on more than one criterion).

The most frequently failed item was the requirement for participants to score 2 (‘rather more than usual’) or 3 (‘much more than usual’) on the GHQ-28 item ‘Have you recently been feeling nervous and strung-up all the time?’.

Of the 1528 who consented, 1089 started the baseline measures but 282 did not proceed; 807 completed the measures. Seven did not proceed at this point; the remaining 800 (52% of those consenting) were randomised.

Use of the intervention (compliance with treatment)

Table 2 shows the level of intervention use in each arm of the trial. These data were available for only 700 of the 800 people, as page view numbers began to be recorded only part-way through the trial. The number of participants who logged in to the site at least once was similar in both arms of the trial, but the total number of log-ins after the first visit was much higher in the REACT arm than in the RD-only arm. The median time spent on the REACT site in the REACT arm was 50.8 minutes (IQR 12.4–172.1 minutes), with a large amount of variation between individuals (range 0.1–4505.5 minutes). The median time spent on the RD in the RD-only arm was 0.5 minutes, again with large variation, ranging from 0 to 42.9 minutes.

Detailed statistics on the use of the intervention, split by module, are provided in Report Supplementary Material 1, Tables 6–9.

The most popular module (i.e. most people visited at least once) was the REACT arm: the online forum. More than half of the people allocated to the REACT visited this module (n = 207, 52%). The least popular modules were ‘Recovery: looking to the future’ (n = 108, 27%) and ‘Managing stress; thinking differently’ (n = 108, 27%).

Recruitment

Rate

Recruitment took place from 22 April 2016 to 30 September 2017. The rate of recruitment is shown in Figure 6. Recruitment was steady and close to target for the first 8 months, and then rose above target. Ethics approval was granted to continue recruitment throughout the recruitment phase, despite the target having been reached, to ensure we had an adequate sample size to test the primary and secondary hypotheses. The increase in recruitment in July 2016 may have been due to circulation of a Bipolar UK e-newsletter at this time, and the accelerating rate from December 2016 may have been as a result of increasing Facebook activity.

FIGURE 6.

Recruitment rate.

Effect of reminders on retention

With ethics committee approval, participants received the following series of reminders to complete follow-ups (until follow-up was completed):

• Up to three automated e-mail reminders at 5-day intervals.

• A manual text message from the trial manager 3 days after the final automated e-mail, with the participant’s log-in details and instructions on how to complete the follow-up.

• A telephone call from the trial manager 3 days after the text message, asking the participant to complete the questionnaires online, but offering to complete the primary questionnaire over the telephone. If the participant did not answer, a second attempt was made; an answer-machine message left if this too was unsuccessful.

• A postal pack containing a letter, the primary questionnaire and a reply-paid envelope, and an automated text message triggered by the trial manager through the trial dashboard. The message contained prespecified text and a link to complete the primary questionnaire on their mobile phone.

Participants who indicated any reluctance to remain in the sample were withdrawn from the trial immediately. Table 7 shows the number of participants who completed the primary outcome measure after each reminder for each time point. Most participants who completed follow-ups did so after the first automated e-mail reminder (n = 399). However, all reminders (online and offline) increased completion, suggesting that multiple reminders and different options for completion (online and offline) have a positive, cumulative effect on follow-up rate. There was no obvious difference in the impact of reminders at the 12- and 24-week follow-ups.

Baseline data

The demographic and situational characteristics of the participants are presented in Table 8 (a fuller version can be found in Report Supplementary Material 1).

Participants were typically middle-aged (53% were aged between 40 and 60 years), white British (91%), female (81%), mothers (48%), highly educated (55% were educated to university level) and supporting a young adult aged ≤ 35 years (61%). More than half (58%) were supporting someone with BD. Most were supporting only one person with a mental health problem, but 26% reported supporting two or more people, and 57% had other dependents. A total of 61% were married or in a civil partnership. Most were in full-time, part-time or voluntary work (64%), but 8.5% reported being unable to work specifically because of their caring responsibilities. All but four participants had home internet access.

Baseline scores on the main outcomes are presented in Table 9. These scores showed very high levels of distress. Using Likert scoring, a mean total score of 40 points on the GHQ-28 (SD 14) suggested that the majority of the sample scored above the 23/24 cut-off point for psychiatric caseness. 132,133 The highest scores were on the anxiety/insomnia subscale.

Well-being and support scores measured using the CWS Questionnaire were also very low compared with other studies using this measure with groups of relatives of people with psychosis, although there are no clinical thresholds or established norms for this scale. The mean well-being scores were in the 50s at baseline (possible range is 0–128; higher scores indicate greater well-being). The mean support scores at baseline were < 20 in each arm (possible range 0–51; higher scores indicate greater support). Other studies of relatives of people with mental health problems report CWS Questionnaire mean scores in the 70s for well-being and 30s for support. 2,62,134 This pattern is likely to be a result of the stringent eligibility criteria used to prevent a floor effect on distress scores.

There are no clinical thresholds or published norms for scores on the Brief IPQ or the Brief COPE. These measures are used to assess change over time, and to assess the relationship between illness perceptions and coping styles and relevant outcomes.

Outcomes and estimated effect sizes

Table 10 sets out summary scores for the overall GHQ-28 and for each of its four subscales at baseline and at the 12- and 24-week follow-ups (primary end point).

Causal analysis

Mediation analysis

Additional analyses

The following analyses were conducted in addition to the testing of our main trial hypotheses, which were prespecified in the published statistical analysis plan.

Adverse events

During the trial, 363 participants received a low-risk response (an automated e-mail sent in response to a report of distress, but with no threat to self or others) and 185 participants had more than one. No high-risk events (serious adverse events) were reported. The trial manager identified three low-risk events, all of which were in the RD-only arm. The REACT supporters reported 16 low-risk events (eight forum posts, five direct messages and three in e-mail correspondence from people outside the intervention). Table 20 shows the number of participants scoring a low-risk item in each arm, and across each time point.

Introduction

This chapter presents the health economic analysis of REACT: a within-trial incremental cost-effectiveness analysis of using REACT and the RD compared with access to the RD only. Both interventions were offered in addition to TAU.

In accordance with NICE guidance,135 our analysis adopted an NHS and Personal Social Services perspective. Costs for REACT and the RD were considered, as well as costs related to health care provided in a hospital setting, primary care, community health, emergency services, Personal Social Services and community mental health services. For health outcomes, relatives’ distress was assessed using the GHQ-28, and quality of life was assessed using a generic measure of health-related quality of life: the EuroQol-5 Dimensions, five-level version (EQ-5D-5L). Intervention costs were measured with a microcosting approach. Results were also presented for the societal perspective, including relatives’ time off work due to their caring role.

In addition, we also reported the burden to carers as time spent in caring role, using data about participants’ unpaid hours spent looking after their relatives and time off work due to their caring role.

Information about the use of medicines was also collected, but, owing to errors in the data collection process that resulted in data about the length of time each medicine was used for, these data could not be included in the analyses.

All costs are presented in Great British pounds using 2018 available costs.

Aims and objectives

The aim of the health economic analysis was to determine the cost-effectiveness of TAU plus REACT and the RD in reducing distress for relatives, compared with TAU and access to only the RD. The specific objectives were to determine:

• the cost of delivering TAU plus REACT plus the RD, compared with NHS and productivity cost savings in the use of health services, and paid work adapted from the Client Service Receipt Inventory (CSRI)

• cost-effectiveness – the cost of significant unit changes (defined as 3-point reduction) in the primary outcome (GHQ-28 score)

• cost–utility – the marginal cost of any marginal change in quality-adjusted life-years (QALYs), measured by the EQ-5D-5L.

Method: costs

Information about the trial design, setting, participants, recruitment, randomisation, data collection and effectiveness outcomes are reported in Chapter 3. Following NICE guidance,135 the economic analysis was undertaken from an NHS and Personal Social Services perspective. Costs of real-world delivery in the trial were used. A microcosting approach136 was undertaken to value REACT and the RD.

Costs of REACT and the RD were calculated separately. Health-care and Personal Social Services use, and health outcomes, including a generic measure of health and well-being (GHQ-28) and a generic health-related quality-of-life outcome (EQ-5D-5L), were assessed online.

Methods: analysis

A health economics analysis plan was developed and approved by the TMG before the analysis began (see Report Supplementary Material 2).

Results

Primary outcome

Results for outcomes and incremental costs and effectiveness based on imputed data sets are presented in Tables 38–40 as means with standard errors (SEs). Except at baseline, the mean costs were always higher for the intervention arm (Table 38). Overall, participants in the intervention arm had higher costs than participants in the control arm, but the difference was not statistically significant. Incremental costs, adjusted for participants’ baseline age and gender, came to £286.77 (95% CI –£858.81 to £1432.36; p = 0.624).

TABLE 38 - Cost estimates for health and Personal Social Services analysis per participant from imputed data, by arm

SE, standard error.

Cost	Trial arm (£)
	REACT	RD only
Mean intervention cost	142.95	0.84
Baseline, mean (SE)	651.29 (116.37)	1263.70 (402.50)
12 weeks, mean (SE)	681.61 (110.48)	505.41 (27.55)
24 weeks, mean (SE)	998.72 (209.77)	450.14 (41.13)
Total mean cost	2474.57	2220.09
Mean unadjusted incremental cost	254.47
Incremental cost adjusted for age and gender, mean (95% CI)	286.77 (–858.81 to 1432.36)

At baseline, estimated GHQ-28 scores were similar in the control arm and in the intervention arm. At 12 and 24 weeks, GHQ-28 scores were lower in the intervention arm.

The incremental GHQ-28 score over the 24 weeks of the trial, unadjusted, was –1.067. Adjusted for baseline GHQ-28 score, it was –1.160 (95% CI –3.371 to 1.051), indicating that participants in the intervention arm were more likely to have a lower GHQ-28 score than participants in the control arm; however, the difference was very small and not statistically significant (p = 0.304). Further adjusting GHQ-28 scores for age and gender, the difference between arms decreased slightly, to –1.152 (95% CI –3.370 to 1.065), but was still small and not significant (p = 0.308).

In terms of GHQ-28 scores, the REACT intervention was costlier and more effective than the RD only (Table 39), suggesting that REACT users had a marginally better outcome than RD users. However, the 95% CI for costs and GHQ-28 scores include zero, making this conclusion uncertain. For a reduction of 3 points in the GHQ-28 score, an additional £746.80 per relative had to be spent (calculated as if the cost per –1.152 is £286.77; thus, a reduction of 3 points corresponds to £746.80).

TABLE 39 - The GHQ-28 estimates per participant from imputed data, by arm

SE, standard error.

GHQ-28	Trial arm
	REACT	RD only
Baseline score, mean (SE)	40.25 (0.5145)	40.05 (0.4942)
12-week score, mean (SE)	31.27 (0.4882)	32.80 (0.4882)
24-week score, mean (SE)	30.36 (0.5123)	31.43 (0.4844)
Mean unadjusted incremental score	–1.067
Incremental score adjusted for baseline score, mean (SE)	–1.160 (1.127)
Incremental score adjusted for baseline score, age and gender, mean (95% CI)	–1.152 (–3.370 to 1.065)

Estimated EQ-5D-5L scores (Table 40) were always higher in the control arm than in the intervention arm.

TABLE 40 - The EQ-5D-5L index score and QALY estimates per participant from imputed data at baseline and at 12 and 24 weeks, by trial arm (mean, SE)

SE, standard error.

EQ-5D-5L and QALY	REACT arm	RD-only arm
EQ-5D-5L index score, mean (SE)
Baseline	0.7570 (0.2113)	0.7637 (0.1908)
12 weeks	0.7743 (0.1886)	0.7849 (0.1676)
24 weeks	0.7812 (0.1890)	0.7925 (0.1666)
Mean QALYs
QALYs 12 weeks	0.176	0.178
QALYs 24 weeks	0.179	0.181
Total QALYs	0.355	0.359
Mean incremental QALYs
Unadjusted	–0.0045
Adjusted for baseline EQ-5D-5L index score	–0.0021
Adjusted for baseline EQ-5D-5L index score, age and gender, mean (95% CI)	–0.0024 (–0.0088 to 0.0039)

The incremental QALY over the 24 weeks of the trial, unadjusted, was –0.0045. Adjusted for baseline EQ-5D-5L score, it was –0.0021 (95% CI –0.0084 to 0.0042), indicating that participants in the intervention arm were more likely to have lower QALY gains than participants in the control arm; however, the difference was very small and not statistically significant (p = 0.517). Further adjusting QALYs for age and gender, the difference between arms increased slightly, to –0.0024 (95% CI –0.0088 to 0.0039), but was still small and not significant (p = 0.448).

In terms of EQ-5D-5L scores, the REACT intervention was costlier and less effective than the RD only (Table 41), suggesting that REACT users had health-related quality-of-life losses at extra cost compared with RD users. However, the 95% CI for costs and QALYs include zero, making this conclusion uncertain.

TABLE 41 - Incremental cost-effectiveness estimated from imputed data

Cost and QALY	Mean (95% CI)
Incremental cost (£) adjusted for baseline age and gender	286.77 (–858.81 to 1432.36)
Incremental QALYs adjusted for baseline EQ-5D-5L score, age and gender	–0.0024 (–0.0088 to 0.0039)
ΔCost/ΔQALY	Intervention dominated by control as being costlier and less effective

The cost-effectiveness plane (Figure 7) illustrated this point: the uncertainty surrounding the point estimates of the incremental costs and incremental QALYs from the primary analysis was important, with the majority of the distribution scattered in the northern quadrants of the plane, supporting the conclusion that the intervention was costlier. The magnitude of the incremental QALYs was rather small.

FIGURE 7.

Cost-effectiveness plane, intervention arm vs. control arm.

The CEAC (Figure 8) was constructed using the bootstrapped data and showed that the probability that REACT was cost-effective within the existing willingness-to-pay threshold of £20,000–30,000 was ≈50% when compared with the RD.

FIGURE 8.

The CEAC for the intervention.

Societal perspective

The societal perspective was added to the health and social services analysis to assess productivity losses associated with the caring role of the relatives. To account for those losses, we considered the number of hours people currently working had to take off work because of their caring roles.

The EQ-5D-5L scores were the same as those estimated for the NHS and Personal Social Services perspective. In this section, we added to the costs estimated for the health and Personal Social Services perspective the costs associated with productivity losses of those in work.

The pattern observed for costs in the societal perspective (Tables 42 and 43) was similar to that observed for complete cases in the REACT arm: increasing costs from baseline to week 24. In the control arm, the opposite behaviour was observed: costs decreased from baseline to week 24. Costs reported by participants in the REACT arm were higher than the costs reported by participants in the RD-only arm, as expected; however, the adjusted incremental costs were similar to the ones observed in the NHS and Personal Social Services perspective, at £309.93 (95% CI –£861.04 to £1480.90), but were not statistically significant.

TABLE 42 - Imputed cases mean (SE) costs for societal perspective data at baseline and at 12 and 24 weeks, by trial arm

SE, standard error.

Cost	Trial arm (£)
	REACT	RD only
Costs (£)
Mean intervention cost	142.95	0.84
Baseline, mean (SE)	891.52 (167.60)	1507.46 (403.24)
12 weeks, mean (SE)	909.81 (110.76)	734.61 (28.01)
24 weeks, mean (SE)	1243.32 (211.15)	662.96 (40.94)
Total cost	3187.60	2905.86
Incremental costs
Unadjusted incremental costs	281.73
Adjusted incremental costs for age and gender, mean (95% CI)	309.93 (–861.04 to 1480.90)

TABLE 43 - Societal perspective incremental cost-effectiveness (mean [95% CI)]

Cost and QALY	Mean (95% CI)
Incremental cost (£) adjusted for age and gender	309.93 (–861.04 to 1480.90)
Incremental QALYs adjusted for baseline EQ-5D-5L index score, age and gender	–0.0026 (–0.0089 to 0.0037)
ΔCost/ΔQALY	Intervention dominated by control as being costlier and less effective

Figures 9 and 10 present the cost-effectiveness plane and CEAC for the societal perspective using imputed cases. It is clear from the distribution of the dots in the cost-effectiveness plane that the intervention can be costlier and less effective than the control.

FIGURE 9.

Cost-effectiveness plane, intervention arm vs. control arm (imputed cases, societal perspective).

FIGURE 10.

The CEAC for the intervention (imputed cases, societal perspective).

The probability of the intervention being cost-effective (see Figure 10) was similar to what had been estimated from imputed data in the NHS and Personal Social Services perspective. The probability of the intervention being cost-effective at £20,000 was 46%.

Sensitivity analysis

Complete-case analysis

The extent of completeness varied for the outcomes used in the primary analysis (Table 44). We considered complete cases to be participants who had information for all three data points.

TABLE 44 - Number of complete cases for costs and EQ-5D-5L score

	Trial arm
	REACT	RD only
Total costs (n participants)	196	236
EQ-5D-5L (n participants)	197	238

The EQ-5D-5L was completed at all three time points by 193 participants in the REACT arm and 233 participants in the control arm (Table 45). The REACT participants who completed the EQ-5D-5L at all three time points had higher mean scores at all three time points than the REACT participants who did not complete the EQ-5D-5L at all three time points. This difference was not seen in the RD-only arm, where index score values were similar.

TABLE 45 - Complete cases: EQ-5D-5L index scores and QALYs at baseline and at 12 and 24 weeks, by trial arm

EQ-5D-5L index scores and QALYs	REACT (n = 193)	RD only (n = 233)
EQ-5D-5L index score, mean (SD)
Baseline	0.781 (0.186)	0.764 (0.196)
12 weeks	0.797 (0.181)	0.788 (0.185)
24 weeks	0.806 (0.182)	0.793 (0.183)
QALYs, mean (SD)
QALYs 12 weeks	0.182 (0.040)	0.178 (0.040)
QALYs 24 weeks	0.184 (0.040)	0.182 (0.040)
Total QALYs	0.366 (0.078)	0.361 (0.079)
Incremental QALYs
Unadjusted (mean)	0.0057
Adjusted for baseline EQ-5D-5L index score (mean)	–0.0011
Adjusted for baseline EQ-5D-5L index score, age and gender, mean (95% CI)	–0.0013 (–0.0079 to 0.0053)

The pattern observed for costs for complete cases (Tables 46 and 47) was different from that obtained with imputed cases. Costs reported by participants in the REACT arm were higher than the costs reported by participants in the RD-only arm, as expected, and the adjusted incremental costs were only somewhat higher than the costs observed for imputed cases, at £715.83 (95% CI –£328.48 to £1760.14).

TABLE 46 - Complete cases: mean (SD) costs at baseline and at 12 and 24 weeks, by trial arm

	Trial arm (£)
	REACT	RD only
Costs, mean (SD)
Intervention cost	142.95	0.84
Baseline	252.00 (587.92)	613.52 (3675.92)
12 weeks	507.15 (3026.46)	263.44 (681.02)
24 weeks	935.07 (5851.86)	225.10 (553.60)
Total	1837.17	1102.90
Incremental costs
Unadjusted incremental costs	734.27
Adjusted incremental costs for age and gender	715.83

TABLE 47 - Complete cases: incremental cost-effectiveness [mean (95% CI)]

Cost and QALY	Mean (95% CI)
Incremental cost (£) adjusted for age and gender	715.83 (–328.48 to 1760.14)
Incremental QALYs adjusted for baseline EQ-5D-5L index score, age and gender	–0.0013 (–0.0079 to 0.0053)
ΔCost/ΔQALY	Intervention dominated by control as being costlier and less effective

The unadjusted incremental QALY gain was 0.0057 (95% CI –0.0091 to 0.02067; p = 0.447) when we compare the REACT participants with the RD-only participants. When we adjust for baseline EQ-5D-5L index score, age and gender, there is no difference between the arms.

Figures 11 and 12 present the cost-effectiveness plane and CEAC as for the primary analysis, but using only data from the 426 complete cases. It is clear from the distribution of the dots in the cost-effectiveness plane that the intervention can be costlier and less effective than the control.

FIGURE 11.

Cost-effectiveness plane, intervention arm vs. control arm (complete cases).

FIGURE 12.

The CEAC for the intervention (complete cases).

Taking this into account, it is not surprising that the probability of the intervention being cost-effective (see Figure 12) was much lower than estimated from imputed data. The probability of the intervention being cost-effective at £20,000 was only 8%.

Discussion

Introduction

The main aim of the trial was to assess the clinical effectiveness and cost-effectiveness of REACT. However, we also wanted to understand participants’ experiences of REACT, how they interacted with the website and how both could be improved. Therefore, we conducted a qualitative study to understand relatives’ experiences of using REACT. The qualitative interviews explored how people experienced REACT, the factors that influenced their levels of use, which areas of the toolkit were most used and how people experienced the support offered via the forum and personal messaging.

Method

We conducted a qualitative study using thematic framework analysis to understand participants’ experiences of REACT, how they interacted with the website and how it could be improved. In addition to the formal qualitative interviews with participants described in this chapter, we also explored how people used the REACT forum and the types of topic covered; this was a preliminary analysis conducted during the trial. As per protocol, we are seeking additional funding to conduct further analysis on these data.

Results

Discussion

Chapter 2 reported on the development of REACT as an online toolkit, drawing on the insights of experts by experience at a series of workshops and the design brief created from this by the TMG. The logic model for REACT, based on well-established cognitive–behavioural principles, mapped out the challenges faced by relatives, the key features of REACT, the anticipated experiences of users and outcomes in response to such experiences. The in-depth interviews that were analysed were broadly consistent with the logic model.

Relatives reported multiple challenges in understanding the behaviours of the relative they cared for and how best to respond to these, resulting in distress, isolation, stigma and shame. REACT offered them psychoeducational information and interactive support through the forum and direct messaging, as well as additional information on where to seek further help on specific topics. Participants reported that they used REACT to learn more about BD and psychosis, to develop more effective ways of dealing with challenging situations, to share problems and solutions and to feel more able to manage their experiences.

A consistent message was that participants felt that REACT would be most useful early on in their relative’s recovery journey, when they were likely to be seeking information and strategies. Relatives later in this journey might need more emotion-focused interventions.

Relatives found prioritising time to use REACT difficult. The advantage of an online intervention is that it is always accessible. However, the corollary is that there is no dedicated time to engage. Overall, relatives reported feeling less isolated, more supported and, through better self-care, more able to attend to their own well-being without feeling ashamed to do so.

There were helpful suggestions about potential improvements and updates for REACT, consistent with digital formats rapidly evolving to become more interactive and integrated. Peer support and REACT supporter contributions were seen as crucial features of REACT, but not all participants accessed these. It is therefore crucial that future iterations of REACT optimise access to these elements. Linguistic analysis of the REACT forum indicated this platform was used to share information about living with mental health issues in a broad context, including, but not limited to, the clinical and behavioural features of psychosis and BD. Consistent with REACT’s recovery perspective, there were also discussions about how both a relative and the individual with the mental health diagnosis could best live well alongside BD and psychosis.

A key limitation of our qualitative findings was the difficulty in recruiting low-level users for interview, resulting in a high proportion of high-level users in the final sample. This was despite the invitation’s emphasis that we were very interested in talking to all participants in the trial, even those who had not used REACT at all. Although the views of medium- and low-level users have contributed to each of the key themes, the predominance of high-level users is likely to be linked to the primarily positive views we received. There remains a real need to understand better the views of those who do not engage with DHIs, to ensure that DHIs do not increase inequalities in access to mental health services.

Introduction

In this chapter, we summarise the main findings from the study and interpret these in the light of other relevant research and the clinical context. We highlight the key strengths and limitations of the research that need to be taken into account when considering these findings. We draw out implications for policy, practice and future research. As this study was funded by a specific call by the HTA for efficient trial designs, we comment on how the lessons from this study can be used to inform future online trials. Finally, we draw out the main conclusions from this work.

Summary and interpretation of key findings

To our knowledge, this was the first definitive RCT testing an online DHI for relatives of people with severe mental health problems. The aim was to develop and test the clinical effectiveness and cost-effectiveness of an online, supported self-management toolkit for relatives of people with psychosis and BD (which included a comprehensive RD), compared with a comprehensive RD alone. At baseline, participants reported very high levels of distress and low levels of well-being, both of which improved significantly, but there was no evidence of a difference between the arms. There was some evidence that those with access to REACT felt better supported than those with access to the RD only, but the difference was small and, after accounting for missing data, failed to reach statistical significance (p = 0.051).

REACT appeared to be highly acceptable, with users reporting feeling safe and supported. No serious adverse events were reported. REACT cost an estimated £142.95 per participant, including full development and delivery costs. These costs would reduce as the number of users grew. There was no evidence that REACT increased QALYs (incremental QALYS adjusted for baseline EQ-5D-5L score, age and gender: –0.0024, 95% CI –0.0088 to 0.0039) or would save money (incremental cost adjusted for baseline age and gender: £286.77, 95% CI –£858.81 to £1432.36; p = 0.624).

REACT was originally designed as a paper-based intervention for relatives of people with psychosis in EIP services;2 we were able to adapt it to create an online DHI that also supported relatives of people with BD. Modules relevant to understanding and managing BD experiences were added, and a forum to facilitate a peer support community was established, along with a confidential direct messaging system, both of which were successfully moderated by trained relatives with lived experience of supporting someone with psychosis or BD. The textual content was updated, and made more interactive through the use of videos of service users, relatives and professionals sharing their experiences and knowledge, and the use of reflective exercises to help relatives to think about how to apply the information they were gaining to their specific situations. Making the toolkit available online made it easier to disseminate widely, and to update content as required.

REACT was made available to relatives across the UK who agreed to take part in an online trial. Recruitment, eligibility, consent, randomisation, data collection and follow-up all occurred primarily online (with some face-to-face recruitment and some telephone or postal reminders for follow-up). A bespoke data collection system was built. Despite the huge increase in development and testing of DHIs for mental health problems across the world, to our knowledge, this was the first definitive RCT test in Europe of an online DHI for relatives of people with severe mental health problems. In the USA, Glynn et al. 150 and Rotondi et al. 151 both report proof-of-concept trials for online psychoeducation programmes for relatives of people diagnosed with schizophrenia, but neither arm reported a fully powered definitive evaluation.

There was a lot of interest from relatives in using REACT. Our aim was to recruit 666 relatives over 18 months. We recruited 800 relatives fairly easily during the recruitment phase of the study. This might indicate substantial unmet need in this area, and the potential for widespread use of an online support tool. Many more relatives were keen to take part, but 43% of those who completed the eligibility criteria questionnaire failed on at least one item, most commonly (81% of those failing) on the requirement to report ‘being strung up and nervous all the time’, ‘rather more than usual ‘ or ‘much more than usual’. This item was chosen as having the highest item-total correlation with total GHQ-28 scores in our feasibility study and was included to avoid a floor effect at baseline. However, as a consequence, most relatives in the trial were in crisis at the beginning of the study, but relatives who had been distressed for longer periods and so responded ‘no more than usual’ were ineligible.

Baseline GHQ-28 scores [mean 40.2 (SD 14.3)] were much higher than those reported for previous samples in studies of relatives of people with psychosis or BD who were seeking help,13,120 including our feasibility trial [mean 34.14 (SD 24.63)]. 2 The same pattern was evident for well-being and support scores, which were significantly lower than in our feasibility trial.

Over the study’s 24-week follow-up, there was a big reduction in distress and an increase in well-being and support in all relatives across both arms. This is likely to be because circumstances change over time and, for people recruited in crisis, the crisis is likely to have abated by follow-up, at least for some participants. This statistical regression to the mean makes it harder to see any impact from an intervention, as any difference needs to be significant, over and above the effect of time.

Furthermore, reducing distress using an online psychoeducation approach might have been an overly ambitious goal. Although psychoeducation has been reliably shown to improve knowledge outcomes, and is recommended by NICE,53,54 there is no evidence that it reduces distress, even when delivered face to face. 43 There is evidence that face-to-face interventions designed to improve outcomes for relatives are generally less effective for those with higher levels of distress. 39 Although many people flourish alongside psychosis or BD, these are potentially chronic mental health problems that, for some people, can have a devastating effect on their lives and the lives of their family and friends. REACT provides information and strategies to support relatives, but did not offer a cure for either condition, which might be what relatives were initially seeking to reduce their distress.

Although REACT showed no benefit over only the RD in reducing distress (as measured using the GHQ-28), relatives who had access to REACT did report feeling more supported than those who had access to the RD only at 12 weeks (2.50, 95% CI 0.87 to 4.12) and 24 weeks (1.65, 95% CI 0.04 to 3.27). However, this was a small effect and, after accounting for missing data in a longitudinal model, the difference between arms was no longer statistically significant at the cut-off point of p < 0.05 (1.51, 95% CI –0.005 to 3.01; p = 0.051), and unlikely to be of clinical significance. This is because, although similar numbers of relatives dropped out of follow-up in each arm of the trial, dropout in the RD-only arm was not linked to outcome, whereas those in the REACT arm who felt less supported were more likely to drop out. This pattern of informative dropout was consistent across all the outcome measures, and highlighted the importance of adequately accounting for missing data in RCTs.

We were able to test the hypothesised logic model that REACT would reduce distress by providing relatives with a working cognitive model of psychosis or BD (assessed using the Brief IPQ) and more adaptive coping (using the Brief COPE). Relatives in both arms of the trial developed more benign perceptions of illness and reported feeling more able to cope over the 24-week period. However, these changes did not differ between those receiving REACT and those receiving the RD only, and there was no evidence that these changes played a causal role in reduced levels of distress. Coping strategies assessed using the Brief COPE showed little change over time, but already favoured active coping, planning and acceptance at the outset.

REACT was valued by relatives for being comprehensive, relevant, easy to access, private and anonymous. The proactive support from REACT supporters was appreciated, as was the opportunity to learn about how best to support someone with a mental health problem through a variety of different mediums (text, video, forum). Relatives provided useful feedback about the content and design of REACT, all of which should be used to further develop the toolkit. A consistent message was that REACT would be most useful to relatives early on in the recovery journey, when they were likely to be seeking information and strategies. This suggested that more recruitment through GP surgeries and EIP teams would be beneficial. Some relatives found seeking help for their own needs difficult, and most found it hard to prioritise time to use REACT.

The problem of low levels of use is common across many DHIs. The flexibility of DHIs, in that they can be used any time, anywhere (with an internet connection), may paradoxically limit their effectiveness. The median time spent on REACT (excluding the RD) in the REACT arm was 50.8 minutes (IQR 12.4–172.1 minutes), with a large variation between individuals (range 0.1–4505.5 minutes). The median time spent on the RD in the RD-only arm was 0.5 minutes (IQR 0–1.6 minutes), again with wide variation from 0 to 42.9 minutes. However, as the purpose of the RD was to signpost relatives to other sources of support, and we do not have data on how much they used these alternative supports, these figures cannot be directly compared.

Causal analysis suggested a small but non-significant association between website use and outcome, suggesting the need to better understand factors that affect engagement with REACT, and DHIs more broadly.

All relatives were free to use REACT at their own convenience. The modules were openly accessible and we did not specify any ideal levels or order of use (although there was a logic to the presentation order). Although relatives were sent periodic reminders in response to low levels of activity, they did not report their levels of use to anyone directly. The REACT supporters moderated activity on the forum and responded to direct messages from relatives, but did not support use of the psychoeducation modules. The content was easy to follow, and each module had self-reflection exercises built in, to help relatives to think about how the information might apply to their specific situation. However, as Mohr et al. 152 reflect, it may be that users will more effectively engage with DHIs if there are clear expectations of use, and the presence of a respected individual to whom the individual is accountable. This may be similar to the effect of personal trainers for exercise: it’s not that we do not know how to exercise, or that we need to exercise, but that, for some people, being accountable to a respected person at a dedicated time and specific place makes us more likely to do it. This might, in part, account for evidence to suggest greater efficacy of supported interventions,153 and suggests that broadening the role of the REACT supporters to support all relatives to engage with all parts of REACT, including the modules, and as occurred in the feasibility study, might increase levels of use.

There were several ways in which this study differed from the REACT feasibility trial, which might account for the differences in findings. These differences might provide valuable insights into factors that affect the effectiveness of DHIs, but all require further testing:

• Relatives in this study were more distressed at the outset and more likely to be in crisis.

• Relatives had been caring for longer, and the information in REACT might have come too late.

• REACT was online, rather than in paper form, and so might have been less accessible or visible, reducing triggers for use.

• Relatives did not receive as much support to use REACT, in general. Only the forum and direct messaging were supported, and relatives not engaging with these components did not receive any support in using REACT.

• REACT was offered in addition to TAU, but not as an integral part of a clinical service, in which clinicians working with relatives were also aware of the nature of the problems experienced by the people they were caring for.

• REACT was supported by trained relatives rather than members of the clinical team.

• REACT was compared with an active control (the RD).

Strengths of the trial

Despite the recent interest in use of DHIs in mental health, and the government’s backing of this agenda,154,155 there is still a lack of a robust evidence base to support the effectiveness of DHIs in mental health. This trial was rigorously conducted, with a large, broadly recruited sample, a clearly defined and theoretically based supported intervention, an active control arm, a good follow-up rate for an online trial, web-based randomisation, a robust blinding protocol to manage any direct contact with participants and a pre-published analysis plan that appropriately addresses missing data.

As a result of these strengths, our findings highlight three key dangers of current research in this area. The first is assuming that interventions adapted from those delivered face to face will be equally as effective as those delivered face to face: REACT was shown to be effective when offered in paper form and supported by telephone by staff linked to the relevant clinical team.

The second is the use of uncontrolled pre–post comparison to argue for effectiveness: REACT would have appeared very effective without comparison with an active control arm.

The third is the need to appropriately manage missing data. Dropout in online trials is often higher than in offline trials, and cannot be assumed to occur at random: relatives in the REACT arm who were more distressed were also more likely to drop out of follow-up. This may be true for other online interventions, as it may be harder to adapt the intervention to better meet individual user needs than would be the case in face-to-face support.

Limitations of the trial

The trial also had several limitations that need to be taken into account in interpreting the findings.

Accepting the need for rigour, there were limitations in selecting a traditional RCT design to evaluate REACT. RCTs were developed to test the effectiveness of static stand-alone interventions that can be easily defined (e.g. dose of a drug). DHIs, including REACT, are often designed to be one component of a comprehensive package of care, and their effectiveness should be evaluated as such. Their impact is likely to be very different when offered as part of a service, rather than in isolation. For example, REACT is one part of the support that NICE recommends to relatives, and may only be effective for relatives able to access other elements of this support (e.g. appropriate clinical and social support for the person with psychosis or BD, and/or family intervention). REACT educates relatives about the services they should be receiving. If they are unable to access these, it could increase rather than decrease distress. Alternatively, awareness that a relative is accessing REACT could lead to changes in behaviour among clinical staff, who may reduce other forms of input that they perceive as no longer necessary. Testing an intervention that is designed to be part of a package of care, using a RCT design, maximises internal validity, but is inherently problematic in ignoring the impact of the changes throughout the wider system. As recommended by the new guidance being developed by the Medical Research Council and NIHR for developing and evaluating complex interventions, study designs that take a more systems-based approach to the evaluation of interventions like REACT, which are just one component of a more complex health-care package, may be more appropriate than traditional RCTs. 156

Digital health interventions also need to be able to evolve and adapt at pace. The REACT trial took 3 years to deliver; by the end of the trial, the toolkit looked dated, but had not been adapted during the trial in an attempt to standardise the intervention across all participants. The way in which people used technology had evolved, such as increased use of mobile applications, and the content needed to be updated to reflect advances in the way in which mental health problems were understood and managed over time. REACT participants provided excellent ideas for updates as part of the qualitative data collection, described in Chapter 6. However, health research currently adopts a model of ‘definitive’ testing, using a large expensive trial lasting several years, and testing between-arm differences. If a trial shows no overall group benefit for the intervention after a defined period, it is likely to become very challenging to attract additional investment to develop it further.

This is in contrast to the iterative dynamic model of design, evaluate, adapt157 seen in other areas of digital development, in which real-time data collection informs continuous adaptations over time. Because DHIs can similarly facilitate the collection of large numbers of real-time data, within-person variation in the impact of the intervention can be explored, to identify what works for particular individuals in particular contexts.

A further limitation of this trial included the failure to recruit a broader range of relatives. Despite designing a broad recruitment strategy, incorporating a wide range of online and offline recruitment strategies, our sample predominantly comprised white British females. There was some evidence of a greater proportion of partners than in previous studies. 2,13,120 It is possible that the predominance of female participants partially reflects the continuing burden of care falling to women, but the lack of relatives from ethnic minority groups is likely to result from lack of cultural adaptation in content and language, both of which were beyond the scope of this trial. Cultural adaptation should be a key focus of future work in this area, to ensure that DHIs do not exacerbate existing inequalities in access to health care.

Treatment as usual was not well measured and, therefore, cannot be reliably defined. There were also some technical issues in data collection online using the CSRI, which meant that information about medication use and some service use was not reliable and could not be used. This made it difficult to understand exactly what services relatives were receiving, and if and how these changed as a result of their being offered REACT or accessing the RD. Given that both interventions were designed to be one component of a care package, and a key function of both is to inform and signpost relatives to other sources of support available, more rigorous and detailed data on how these additional data sources were accessed and used would have allowed us to better understand the impact of both REACT and the RD.

Retention to follow-up played a very important role in this trial. Online trials have historically reported higher dropout rates than those conducted face to face. 77 In anticipation of this, we over-recruited to the trial to ensure that the final analysis was adequately powered. This strategy was successful and there was no evidence of differential rates of dropout between the two arms. However, those who dropped out of the REACT arm were more distressed than those who remained. This was not the case in the RD-only arm. This meant that data could not be assumed to be missing at random in both arms. When missing data were taken into account, any statistically significant benefits of REACT over the RD were lost. We do not know why relatives who were more distressed were more likely to drop out of the REACT arm, but it may have been low satisfaction with the intervention. Those in the RD-only arm may have been less dissatisfied, or had lower expectations of the RD and remained involved for the promise of access to the REACT modules at the end of the trial (even though these were also made available to those who dropped out).

Recruitment to the nested qualitative study in this trial was also limited by a failure to interview those in the RD-only arm and those who dropped out of the trial prior to the 24-week follow-up. The focus was on understanding relatives’ experiences of using REACT; as the RD was one component of REACT, we did not recruit additionally from the RD-only arm. However, we acknowledge that their experiences of the RD may have been very different. We recruited only those who had completed the 24-week follow-up as we were keen to prioritise collection of the primary outcome data and so did not want to overburden those from whom we were still seeking data. However, given that retention was not random (see Chapter 4), this biased us to interview relatives who were less distressed at baseline.

Our intention was to recruit a sample across the range of levels of use of REACT so we could understand what factors might affect engagement. Inevitably, it proved easier to recruit those with higher levels of use; hence, our findings are likely to be positively biased in their impressions of REACT.

Practical limitations of this trial included underestimating the amount of IT resources needed to build and host REACT and the online trial methodology. Part-time support meant that we could not always address issues immediately, and led to protocol deviations (outlined in Chapter 4). Finally, PPI in the REACT trial was challenging. Despite extensive involvement of relatives in the clinical delivery of REACT (from REACT supporters and supervisors), it was much more difficult to engage relatives in the RAG or the oversight committees. Consistent with the online design of the trial, this work was conducted remotely via digital or telephone communication, and it was harder to create a sense of engagement.

Finally, this trial is not an independent evaluation as members of the research team were also involved in the design and delivery of REACT. This may have led to bias in interpretation of the findings. Extensive peer review has been used to mitigate this, but it is important that this remains acknowledged.

Implications for online trials

This trial was funded as part of a specific call by the HTA programme for efficient trial designs.

The key challenges and our suggested solutions are provided in Table 53. These recommendations build directly on previous research in this area. 77

Implications for health-care delivery

Services aim to offer support and education to relatives of people with psychosis or bipolar disorder, as recommended by NICE guidelines. We assessed the effect of including REACT as one component of a comprehensive care package that addresses these guidelines. Relatives felt safe and well supported using REACT, which might facilitate better engagement with other aspects of the service. However, adding REACT to treatment as usual was no more effective than adding an online Resource Directory in reducing the level of relatives’ distress, as measured with the GHQ-28 items.

Key developments should include redesigning the content and presentation using feedback from participants; making the technology more interactive and user friendly; increasing the role of REACT supporters to include support to use the modules; specifying recommended levels of use; and offering REACT to relatives earlier in the recovery journey, alongside other components of care, particularly for those with high levels of distress. REACT can also help services to meet the NICE recommendation54 to increase the involvement of service users (including relatives) in peer worker roles to support the delivery of health-care services. However, in its current form, REACT is unlikely to reduce relatives’ distress or save money in terms of the use of other health-care services.

The need for DHIs to enhance, but not replace, face-to-face support is consistent with findings across other groups,158 including young people,159 who are often assumed to be more likely to embrace digital technology.

Recommendations for further research

• Given the apparent unmet need and high acceptability of REACT, further work is needed to make the content of REACT more effective, by iteratively co-designing and testing each of the key developments, including redesigning the content and presentation using feedback from participants; making the technology more interactive and user friendly; increasing the role of REACT supporters to include support to use the modules; specifying recommended levels of use; and offering REACT to relatives earlier in the recovery journey, alongside other components of care, particularly for those with high levels of distress.

• Psychoeducation and support are important and valued by relatives, but distress (as measured by the GHQ-28) may not be the most appropriate outcome to evaluate their effectiveness. Understanding more about a chronic health problem for which there is no immediate cure is important, but is unlikely to reduce distress without additional therapeutic input. Therefore, the effectiveness of psychoeducation interventions may be better tested against alternative outcomes, such as whether they support relatives to feel more knowledgeable, more empowered, better able to cope and more engaged with services, rather than on reducing distress. Further research could focus on working with relatives to understand their views about what should be the primary outcomes for future psychoeducation programmes.

• Research is needed to understand how to increase the amount of engagement and use of REACT and other DHIs, to maximise their potential to improve outcomes.

• Research is needed to understand how to improve uptake and reach of REACT (and other DHIs) by groups that currently show low levels of use of mental health services and support, including ethnic minority groups and men.

• The impact of effective psychoeducation interventions for relatives on service user outcomes needs to be tested.

• Randomised controlled trials can be delivered online at a lower cost. However, this methodology presents new challenges in keeping participants engaged throughout long-term follow-ups, and managing high-quality PPI; both need to be addressed by further research.

Conclusions

Despite being informed by evidence-based theory and extensive user-informed design, and having previously shown positive outcomes on relatives’ distress in a pilot study, REACT did not reduce relatives’ distress significantly more than a comprehensive RD listing nationally available support services for relatives, nor did it reduce resource use in other areas of health and social care. Distress reduced significantly in both arms, but, as there was no TAU-only arm, it is possible that both interventions were effective, but equally possible that neither was, and that this change was a function of time.

There are several reasons why REACT may have failed to show the benefits suggested in the feasibility study: the relatives taking part included those caring for people with BD, those who had been caring for longer and those who were more highly distressed at baseline; REACT was online rather than in paper form; REACT was supported by trained relatives rather than staff in the participant’s clinical team; REACT was tested against an active control (the RD) of unknown effectiveness; and there was less support targeted at using the modules rather than the forum. It is impossible to know which of these factors was crucial in determining these findings.

Notwithstanding these findings, relatives using REACT also reported feeling safe and supported, and no serious adverse events were reported. Qualitative feedback was very positive. Relatives need access to information and emotional support,53,54 even if it does not reduce their distress, as assessed using the GHQ-28.

Contributions of authors

Fiona Lobban (https://orcid.org/0000-0001-6594-4350) was the chief investigator.

Nadia Akers (https://orcid.org/0000-0002-1276-6092) was a REACT supporter and provided administrative support.

Duncan Appelbe (https://orcid.org/0000-0003-1493-0391) provided IT design, and managed and supported the online data collection of trial participant data.

Rossella Iraci Capuccinello (https://orcid.org/0000-0002-6875-4867) conducted the health economics analysis.

Lesley Chapman was a co-investigator and an expert relative, responsible for training, supervision and support of the REACT supporters.

Lizzi Collinge (https://orcid.org/0000-0001-5297-4112) was a REACT supporter.

Susanna Dodd (https://orcid.org/0000-0003-2851-3337) (Institute of Child Health, Alder Hey Children’s NHS Foundation Trust) was the supervising statistician.

Sue Flowers (https://orcid.org/0000-0002-9302-3492) was a REACT supporter.

Bruce Hollingsworth (https://orcid.org/0000-0002-4314-6523) was a co-investigator and the lead for the health economics analysis.

Mahsa Honary (https://orcid.org/0000-0001-5966-2197) contributed to the development and design of the content for the online REACT.

Sonia Johnson (https://orcid.org/0000-0002-2219-1384) was a co-investigator, advising on intervention content and the design and implementation of the trial.

Steven H Jones (https://orcid.org/0000-0002-8801-5113) was a co-investigator, responsible for training and supervising the REACT supporters.

Ceu Mateus (https://orcid.org/0000-0001-6219-219X) conducted the health economics analysis.

Barbara Mezes (https://orcid.org/0000-0002-0799-2423) provided trial management and administrative support.

Elizabeth Murray (https://orcid.org/0000-0002-8932-3695) was a co-investigator, contributing to setting the research questions, designing the research methods and protocol, analysing and interpreting the data, and writing the final report.

Katerina Panagaki (https://orcid.org/0000-0002-0383-4890) conducted and analysed qualitative interviews with relatives in the REACT arm of the trial.

Naomi Rainford (https://orcid.org/0000-0002-5876-3946) (Institute of Child Health, Alder Hey Children’s NHS Foundation Trust) is a statistician who supported the management of the trial data.

Heather Robinson (https://orcid.org/0000-0002-3566-2253) was the trial manager.

Anna Rosala-Hallas (https://orcid.org/0000-0001-8012-9995) (Institute of Child Health, Alder Hey Children’s NHS Foundation Trust) is a statistician who supported the analysis of the trial data.

William Sellwood (https://orcid.org/0000-0001-8260-9503) supported the supervision of the REACT supporters.

Andrew Walker (https://orcid.org/0000-0001-6486-6135) is a digital technologist who was involved in the design and delivery of REACT and in data extraction, management and analysis.

Paula Williamson (https://orcid.org/0000-0001-9802-6636) (Institute of Child Health, Alder Hey Children’s NHS Foundation Trust) was a co-applicant and advisor on trial design and statistical analysis.

Publications

Robinson H, Lobban F, Honary M. Relatives’ Education and Coping Toolkit (REACT) Lessons learnt from developing a national online trial for relatives/friends of people with psychosis or bipolar disorder. Early Interv Psychia 2016;10:70.

Lobban F, Robinson H, Appelbe D, Barraclough J, Bedson E, Collinge L, et al. Protocol for an online randomised controlled trial to evaluate the clinical and cost-effectiveness of a peer-supported self-management intervention for relatives of people with psychosis or bipolar disorder: Relatives’ Education And Coping Toolkit (REACT). BMJ Open 2017;7:e016965.

Appelbe D, Robinson H, Dodd S, Walker A, Williamson P, Lobban F. Implementing and running an online trial – a case study using the REACT trial. Trials 2017;18(Suppl. 1):230.

Honary M, Fisher NR, McNaney R, Lobban F. A web-based intervention for relatives of people experiencing psychosis or bipolar disorder: design study using a user-centred approach. JMIR Mental Health 2018;5:e11473.

Lobban F, Collinge L, Robinson H, Flowers S, Minns V, Jones S. Approaches to service user engagement in the Relatives’ Education and Coping Toolkit (REACT) RCT. Bipolar Disord 2018;20:31.

Honary M, McNaney R, Lobban F. Designing Video Stories Around the Lived Experience of Severe Mental Illness. In NordiCHI ’18 Proceedings of the 10th Nordic Conference on Human–Computer Interaction. New York, NY: ACM; 2018. pp. 25–38.

Lobban F, Akers N, Appelbe D, Chapman L, Collinge L, Dodd S, et al. Clinical effectiveness of a web-based peer-supported self-management intervention for relatives of people with psychosis or bipolar (REACT): online, observer-blind, randomised controlled superiority trial. BMC Psychiatry 2020;20:160.

Data-sharing statement

Ownership of copyright and intellectual property rights for all research conducted for the REACT trial will ultimately be held by Lancaster University. We intend to make available the individual participant data that underlie the results reported in this article, after de-identification. Data will be made available on request 12 months following article publication, and only to researchers who provide a methodologically sound proposal and where the proposed use of the data has been approved by an independent ethics review committee (‘learned intermediary’) identified for this purpose. Proposals should be directed to the corresponding author at rdm@lancaster.ac.uk. Data will be available for 10 years at Lancaster University’s Research Directory (10.17635/lancaster/researchdata/306). The study protocol130 and statistical analysis plan125 are already published and freely available.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care.