Digital smartphone intervention to recognise and manage early warning signs in schizophrenia to prevent relapse: the EMPOWER feasibility cluster RCT

Early warning signs monitoring and interventions

A further well-established approach for relapse prevention in research and practice is early warning signs (EWS) monitoring, established by Birchwood, Herz and colleagues. 48,49 EWS monitoring is based on the assumption that it is possible to intervene to reduce the likelihood of a relapse of psychosis through its timely prediction. This is generally achieved by comparing ongoing assessments of EWS indicators against an individual’s ‘baseline’ score. 50 Relapse in psychosis is now understood to be the end point of a process of change associated with early signs including changes in affect and incipient psychotic experiences. 50 EWS may be identifiable for as long as 5–8 weeks prior to a relapse,51 creating significant opportunity for early intervention. However, review evidence suggests considerable variation in the proportion of relapses correctly predicted by EWS (sensitivity 10–80%, median 60%) and non-relapses correctly predicted (specificity 38–100%, median 81%), with more frequent monitoring and the inclusion of both affective and psychosis symptoms found to improve prediction. 52

Eisner et al. 52 identified 17 papers involving early signs interventions in their 2013 review. Early signs interventions involve responding to EWS changes with a plan designed to prevent or minimise relapse. Just one of these studies used a purely psychological approach, with two or three sessions of relapse prevention focused CBT administered in response to an assessed increase in EWS from baseline. 53 People in the CBT arm had significantly fewer relapses than people in a treatment-as-usual arm, but the findings may have been influenced by the lack of masked assessment of relapse. Four studies included interventions in response to EWS increases that Eisner et al. described as multicomponent relapse prevention techniques. These included stress management and problem-solving methods, increased practitioner contact and antipsychotic medication increases,54–57 as well as three interventions that involved a relative or friend. 54,55,57 In two out of the three studies that assessed relapse, those in the intervention arm had significantly improved outcomes compared with those receiving treatment as usual,54,55 and the other study was potentially underpowered to show an effect on relapse. 57

The remaining studies in the Eisner et al. 52 review related to interventions in response to EWS that involved changes in medication alone. In one study, targeted medication or placebo was given in addition to maintenance medication on the emergence of EWS. However, no differences were observed between the arms in relation to time to symptom exacerbation at 2-year follow-up. 58 The remaining 11 studies temporarily used targeted medication on the emergence of EWS but in the absence of other ‘maintenance’ antipsychotic medication. 59–70 Although there were variations in the method of treatment and in comparators, in all but one65 of the seven studies of targeted medication approaches where relapse was assessed the outcomes were better for people in receipt of a moderate dose of maintenance medication than for people receiving targeted medication alone. 61,62,64,67–69 Two studies59,60 measured hospital admission alone, finding no significant differences in admission rates between people in receipt of maintenance medication regimes and those in receipt of targeted medication regimes.

Across all of the included studies, comparison was complicated by the heterogeneity of EWS assessment and the definition of relapse alongside a number of methodological weaknesses, including potential sampling bias and underpowered studies. Despite this, a number of conclusions were drawn about EWS-informed interventions. Targeted medication in response to EWS was found to be less effective than maintenance for relapse prevention, and replication with blinded assessment of relapse was recommended for the lone psychological intervention53 before firm conclusions could be drawn about its seeming potential. Multicomponent responses to EWS were described as showing promise for relapse prevention, but methodological weaknesses were highlighted, suggesting the need for further research.

Additional evidence of the potential for EWS interventions was provided in a 2013 Cochrane review71 that compared the effectiveness of EWS interventions with that of treatment as usual on time to relapse, hospitalisation, functioning and symptoms. The review found that significantly fewer people relapsed with EWS interventions than with usual care [23% vs. 43%; risk ratio 0.53, 95% confidence interval (CI) 0.36 to 0.79; 15 randomised controlled trials (RCTs), n = 1502] and significantly fewer people were readmitted to hospital (19% vs. 39%; risk ratio 0.48, 95% CI 0.35 to 0.66; 15 RCTs, n = 1457). There was, however, no effect on reducing time to relapse. However, the review found low overall quality of evidence, which limited generalisability to usual care, making it impossible to recommend the use of EWS interventions in routine practice until higher-quality evidence becomes available.

Service user barriers to relapse detection and prevention

Uncertainty about the prognostic validity of EWS52 brings with it a risk of unnecessary interventions from services, which may, in turn, heighten the fear of relapse in people with experiences of psychosis and their carers. 17 Feelings of fear, helplessness and depression are commonly experienced prior to full relapse. 72 Fear of illness or relapse is also associated with poorer insight,25 emotional dysfunction,73 suicide risk,74 more traumatic experiences of psychosis, and fear of psychosis symptoms and hospital. 16 It is also a barrier to coping and relapse prevention for family members. 18 Our 2015 RCT17 of relapse detection showed that fear of recurrence contributed independently to the prediction of relapse (sensitivity 72%, 95% CI 52% to 86%) compared with EWS (sensitivity 79%, 95% CI 62% to 89%) and should, therefore, be included in EWS monitoring. Fear of recurrence was also associated with increased depression and feelings of entrapment, self-blame and shame and was a significant predictor of time to relapse. This suggests that fear of recurrence may be a risk factor for relapse and for increased distress arising from psychosis experiences and that it may play some role in accelerating the process of relapse. All in all, this suggests that fear of recurrence is a key barrier to, and potential target for, interventions to predict and prevent relapse.

Concurrently, people affected by psychosis are more likely than those in the general population to adopt avoidant coping styles,75,76 which may be an attempt to prevent relapse and minimise the effects of public stigma. 73 Avoidant coping in psychosis has been associated with higher neuroticism and lower extraversion,77 reduced insight,78 negative early childhood experiences, insecure identity,79 cognitive deficits80 and a generally greater insecurity in relationships and reluctance to seek help in a crisis. 81,82 In the context of active psychosis symptoms, avoidance may represent a safety behaviour based on the perceived threat of other people. 83 Review evidence suggests that greater difficulties in forming relationships among people affected by psychosis is associated with poorer service engagement and poorer relationships with practitioners, as well as longer and more frequent inpatient admissions. 81 Reluctance to seek help may be an understandable response to fear generated through previous negative, and potentially traumatic, experiences of inpatient admission. 16 In totality, this suggests that identifying and responding to EWS may be constrained by a variety of factors. These include avoidant coping styles and an associated reluctance to seek help or disclose EWS, poor-quality relationships between people in receipt of services and those providing them, and a wider fear of relapse.

Service barriers to relapse detection and prevention

One potentially important means of improving the implementation of EWS approaches without necessarily increasing fear of relapse is using interventions to increase shared decision-making for relapse prevention and risk management. Joint crisis plans (JCPs), which allow for the expression of service user preferences in the event of a future relapse, are one such example in the UK. The CRIMSON study84 compared the effectiveness of JCPs with that of treatment as usual for people with a diagnosis of schizophrenia but found no significant impact on the reduction of compulsory treatment, raising questions about the wider application of JCPs. An associated process evaluation85 examined how stakeholders perceived JCPs and shared decision-making as well as the barriers to implementation. The main barriers identified related to practitioners and included a general ambivalence towards JCPs and the perception that shared decision-making was already happening, as well as concerns about service users’ expressed wishes for JCPs. These barriers led to poor clinician engagement, which in turn undermined service users’ contributions to the process. It was also noted that, in times of crisis, JCPs were largely ignored, with practitioners reverting to standard practice. As a result of feeling unable to influence practitioner views through JCPs, people using services felt that there was a lack of respect for their views and that they were not able to influence shared decisions. Clinicians themselves experienced their ongoing clinical interactions as ritualised, particularly in relation to risk, while wider evidence highlights the need to be conscious that EWS monitoring for relapse prevention is also contingent on the person receiving services initiating help-seeking from a position of vulnerability and perceived threat (linked, for example, to previous experiences of coercive treatment). 86

People affected by psychosis may have had difficult or traumatic experiences of relapse,16 and there can be many barriers to help-seeking,87 which reduce the possibility of early intervention at times of crisis. This may in turn increase reliance on coercive practices, potentially building on already negative expectations in a vicious cycle. This means that there is some urgency to develop and evaluate an intervention that can facilitate safe and honest disclosure of possible EWS while also encouraging collaborative and non-coercive responses from mental health practitioners.

Conceptual framework for improving relapse detection and prevention

Our conceptual framework for improving relapse detection and prevention aims to understand the awareness of and response to EWS in the context of relationships. Figure 1 illustrates a cognitive–interpersonal model of EWS in which fear of recurrence drives negative emotions including fear, anxiety and shame. This distress may be regulated by coping strategies including avoidance and hypervigilance. Such responses may in turn influence practitioner responses at times of increased risk of relapse, for example interpreting avoidance (of appointments, telephone calls, etc.) as evidence of the need to focus more on risk management. For someone using services, such shifts in practice may in turn simply reaffirm existing negative expectations in a cyclical manner. Therefore, interventions that can enhance positive emotional awareness, choice and autonomy (through self-management promotion) and improved communication (through increased understanding) could provide a means to disrupt and change these negative interpersonal cycles. Given that the modal time window to intervene in the context of EWS is 2 weeks,52 interventions that can enhance access to data to inform help-seeking and shared decision-making could contribute to relapse detection and prevention. Digital technology may offer such an opportunity for this kind of timely intervention.

FIGURE 1.

Cognitive–interpersonal framework for EWS.

Objective

The objective of phase 1 was to conduct task group interviews to (1) evaluate the acceptability and usability of mobile symptom reporting using smartphones among service users, carers and mental health staff; (2) identify incentives and barriers to use by service users and carers and to implementation by mental health staff; and (3) identify pathways to relapse identification and prevention. These interviews informed modifications to the EMPOWER mobile app, which was then subjected to a 5-week beta-testing phase.

The aims of the work packages (WPs) that constituted phase 1 are as follows.

• WP 1: (1) evaluate the acceptability and usability of mobile symptom recording using smartphones among service users and their carers; and (2) identify incentives and barriers to use.

  Deliverables: software and protocol updates in response to feedback from service users and carers.

• WP 2: (1) evaluate the acceptability and usability of mobile symptom recording using smartphones among professional mental health care staff; (2) identify incentives and barriers to implementation by mental health staff; and (3) identify relapse prevention pathways and whole-team responses.

  Deliverables: (1) carry out software and team protocol updates in response to feedback from professional care staff; (2) develop care pathways and identify operational barriers and enablers; and (3) identify the training needs of teams participating in the phase 2 pilot cluster randomised controlled trial.

• WP 3: finalise the EMPOWER app for its implementation in phase 2.

  Deliverables: (1) carry out software and protocol updates in response to feedback from service users, carers and staff; (2) agree on final modifications to the EMPOWER app to enhance its usability; (3) and finalise measurement methods for the assessment of self-reported acceptability and usability.

Settings

Parallel arms of data collection for phase 1 took place in NHS Greater Glasgow & Clyde, UK, and NorthWestern Adult Community Mental Health Services in Melbourne, Australia.

Methods

Study population

Eligibility criteria

Sample size

The numbers projected for WPs 1 and 2 (i.e. 30 service users, 30 carers and 20–30 professional mental health care staff) and WP 3 (i.e. 10 service users) were to provide sufficient data to create the framework of analysis. No formal sample size calculation (e.g. power analysis) was considered appropriate for these WPs, as their aim was not to evaluate treatment effects.

Analytic methods

Task groups (WPs 1 and 2) and follow-up interviews (WP 3) were digitally recorded, transcribed and anonymised before being entered into NVivo version 12 (a computer-assisted qualitative software package; QSR International, Warrington, UK) to organise the data and enable progression to analysis. Analysis drew on framework analysis, which is a qualitative approach specialising in pragmatic, generalisable qualitative methods designed for real-world implementation. 126,127 This approach has been developed specifically for applied or policy-relevant qualitative research in which the objectives of the investigation are typically set in advance and shaped by the information requirements of the funding body. The time scales of applied research tend to be short and there is often a need to link the analysis with quantitative findings.

Participant safety and withdrawal

Data security and management

The confidentiality of all study data was ensured using the security mechanisms outlined below.

Protocol amendments

Protocol amendments are described in Table 1.

Protocol breaches

The Post-Study System Usability Questionnaire was omitted from WP 3 follow-up interviews.

Objectives

Our phase 2 protocol is published elsewhere. 131 The overall purpose of this study was to establish the feasibility of conducting a definitive cluster randomised controlled trial (cRCT) comparing EMPOWER with treatment as usual (TAU). We sought to establish the parameters of the feasibility, acceptability, usability, safety and outcome signals of an intervention as an adjunct to usual care that would be deliverable in the NHS and Australian CMHS settings. The EMPOWER intervention aimed to:

1. enhance the recognition of EWS among people using services and their carers

2. provide a stepped-care pathway that was either self-activated or in liaison with a carer and/or a community health-care professional, which then

3. triggered a relapse prevention strategy that could be stepped up to a whole-team response to reduce the likelihood of a psychotic relapse.

Specifically, we aimed to:

1. determine the rates of eligibility, consent and recruitment of potentially eligible participants (people using services, carers and care co-ordinators) to the study

2. assess the performance and safety of the EMPOWER class 1 medical device (CI/2017/0039)

3. assess the feasibility, acceptability and usability of the intervention, including feedback on suggested enhancements from people receiving the intervention, peer support workers and clinicians

4. assess primary and secondary outcomes to determine the preliminary signals of efficacy of the EMPOWER intervention as a basis for assessing the feasibility of collecting follow-up measures, determining primary and secondary outcomes, and determining probable sample size requirements for a future main trial

5. undertake a qualitative analysis of relapses to refine the intervention in the main trial

6. establish the study parameters and data-gathering frameworks required for a co-ordinated health economic evaluation of a full trial across the UK and Australia

7. enhance and tailor our mobile phone software app to deliver EWS monitoring, self-management interventions and access to a relapse prevention pathway that was embedded in whole-team protocols and action.

Trial design

We evaluated EMPOWER using a multicentre, two-arm, parallel-group cRCT involving eight purposively selected CMHS (two in Melbourne, Australia, and six in Glasgow, UK) with 12-month follow-up. The CMHS were the unit of randomisation (the cluster), with the intervention delivered by the teams to people using services and with the outcomes assessed within these clusters. The study was planned and implemented in concordance with the Consolidated Standards of Reporting Trials (CONSORT) cluster trial extension132 and the extension to randomised pilot and feasibility trials. 133 We chose a cluster design as the EMPOWER intervention aimed to enable a team-based response to people in receipt of services whose real-time EWS monitoring activates a relapse prevention pathway.

Ethics and governance

The study received approvals from the West of Scotland Research Ethics Service (GN16MH271 reference 16/WS/0225) and Melbourne Health Human Research Ethics Committee (HREC/15/MH/344). The study sponsors were NHS Greater Glasgow & Clyde in the UK and the Australian Catholic University in Australia. The trial also received approvals from the NHS Health Research Authority and a notice of no objection for a trial of a medical device (CI/2017/0039) from the UK Medicines and Healthcare products Regulatory Agency (MHRA), and was prospectively registered (ISRCTN99559262). The trial methods of enrolment, interventions and assessments are summarised in Table 2.

A trial Study Steering Committee (SSC) and a Data Monitoring and Ethics Committee (DMEC) were constituted in accordance with National Institute for Health and Care Research (NIHR) guidance. The DMEC reported to the SSC and the SSC reported to NIHR. Protocol amendments were reviewed by the DMEC and SSC before being submitted to the relevant Research Ethics Committee (see Table 4). The study was conducted in accordance with the recommendations for physicians involved in research on human participants adopted by the 18th World Medical Assembly, Helsinki 1964,134 and later revisions.

Preliminary work: patient and public involvement

Phase 1, as described above, was included to ensure extensive consultation with key stakeholders, namely mental health care staff, people with lived experience and carers, in a series of task groups across Glasgow and Melbourne. Stakeholder consultation directly shaped the development of the EMPOWER intervention. The Scottish Recovery Network also played a key role in shaping further consultation with people with lived experience to further refine intervention development and planning.

Eligibility criteria

Participation was sought from CMHS in NHS Greater Glasgow & Clyde in the UK and NorthWestern Mental Health services in Melbourne, Australia. All participants (mental health care staff, service users and carers) were approached for their informed and written consent prior to assessment and randomisation. Research assistants were responsible for recruitment and taking informed consent. Written informed consent was obtained from all trial participants.

Interventions

In describing the EMPOWER intervention we utilise the Template for Intervention Description and Replication (TIDieR) checklist. 135

EMPOWER intervention

Rationale

The rationale for the EMPOWER intervention was informed by the cognitive–interpersonal model outlined in Chapter 1 and was designed to enable participants to monitor changes in their well-being on a daily basis. In EMPOWER, we referred to changes in well-being as ‘ebb and flow’ as a means of moving away from risk-orientated monitoring that can sensitise individuals to increased fear of relapse. The terminology also attempted to convey a normalising framework for understanding changes in emotions and psychotic experiences in daily life. Clinical triage of changes in well-being that were suggestive of EWS was enabled through an EMPOWER algorithm that triggered a check-in prompt (ChIP). This enabled the prompt identification of any EWS and triggered a relapse prevention pathway if warranted.

The EMPOWER app was developed in part through consultation with people using services, their carers and mental health care professionals. Service user participants had access to the EMPOWER app for up to 12 months of the intervention period. EMPOWER was developed as a flexible tool for users to:

1. monitor daily the ‘ebb and flow’ of changes in well-being

2. incorporate personalised EWS items

3. receive automated EMPOWER (self-management) messages directly

4. review their own data and keep a diary of their experiences via a smartphone user interface.

Materials

Daily monitoring of well-being was initiated by pseudo-random mobile phone notifications to complete the EMPOWER questionnaire (see Report Supplementary Material 2). Notification reminders were sent after 5 minutes (when there had been no response), after which app users were allowed a 5-hour time window within which they could respond to questions. The questionnaire contains 22 items reflecting 13 domains (e.g. mood, anxiety, coping, psychotic experiences, self-esteem, connectedness to others, fear of relapse and personalised EWS). Items included both positive (e.g. ‘I’ve been feeling close to others’) and negative (e.g. ‘I’ve been worrying about relapse’) content. Each item was completed using a simple screen swipe, which enabled quick and efficient completion by users. Each item was automatically scored on a scale of 1–7. Where particular items scored > 3, users were invited to complete supplementary questions to enable a more fine-grained assessment of that domain. This provided up to a maximum of 56 questionnaire items.

Peer support working in EMPOWER

The EMPOWER intervention was blended with peer support. Peer support workers are people who bring lived experience of mental health problems and recovery to their work in mental health services, with practices underpinned by a set of values and principles. 120,136 The role is increasingly well established in mental health policy and practice in many countries, including the UK and Australia. 137 Peer support is also being increasingly integrated with digital mental health interventions. 124 Two peer support workers were employed in Glasgow and one was employed in Melbourne, all on a part-time basis, to work with people in the intervention arm of the study. Their work was underpinned by a values framework developed by the Scottish Recovery Network,138 with supervision available at both sites and also across sites.

Peer support worker roles initially focused on introducing participants to the ethos and principles of the EMPOWER stepped-care approach and setting up and personalising the app on provided or personal handsets. Following this set-up and familiarisation period, peer support workers maintained regular contact with app users, at least weekly in the baseline period and approximately fortnightly following the baseline. Contact was maintained primarily through telephone calls, although text messaging and in-person meetings also took place, with the type and amount of contact determined by personal preference and need. Peer support worker roles included:

1. Supporting engagement with the intervention. This included checking in with people to assess whether or not there were any blocks to their successful use of the app and to advise on how usage might be adapted to individual needs, for example encouraging people to take breaks from self-monitoring as required.

2. Offering technical advice and support in relation to participants’ use of the phone. This included supporting people in becoming familiar with mobile phone functions and ensuring that they had mobile phone data connectivity and adequate credit for data.

3. Encouraging the general exploration of well-being and of how data generated through the app reflected life experience and aided self-management.

4. Reviewing and supporting participants’ use of different app functions. This included supporting participants to access and make use of chart and diary functions. In particular, peer support workers encouraged participants to review charts and sought to prompt curiosity about chart data and the implications of these for self-management and well-being.

5. Aiding the interpretation of messages generated by the app. This included giving app users additional information on message content, as required, for example providing further information on how to access support to develop an advance statement.

6. Sharing personal experiences of app use. Peer support workers had personal experience of using the EMPOWER app and were able to reflect with research participants on their own experiences and, if required, offer practical advice and share aspects of their experience that they felt might support participants.

7. Monitoring and reporting performance and safety issues. This included making routine enquiries in relation to any possible adverse events and the extent to which these were associated with the intervention.

Procedures

A peer support worker met with participants on an individual basis to introduce the rationale for using the app, to collaboratively set up the app on their participant’s mobile phone or a study phone, and to support the individual’s familiarisation with the handset and app functions. Participants were invited to choose up to three personalised EWS items to be included in the EMPOWER questionnaire and further personalisation of delusion-specific items could also be made. Where possible, an individual’s care co-ordinator and nominated carer were invited to contribute to this meeting. Participants were asked to undertake daily monitoring for an initial 4-week baseline period to help establish their personal baseline of the ‘ebb and flow’ of their well-being. During this period, additional support was provided by peer support workers through weekly telephone follow-up. This provided an opportunity to encourage use of the app, solve any technical problems and identify any adverse effects. At the end of the 4 weeks, a further meeting was arranged with the peer support worker or mental health nurse to review monitoring, encourage engagement with EMPOWER messages, agree participants’ preferences for actions in response to changes in well-being that were suggestive of EWS and encourage participants to continue utilising local CMHS for clinical care. All participants were offered ongoing fortnightly peer support worker support to encourage them to use the app, to support their reflection on changes in well-being and their broader context including, for example, stressful life events, and to encourage them to use self-management strategies prompted by EMPOWER messages.

Digital procedures

The analysis and handling of questionnaire data was governed by the EMPOWER algorithm. The EMPOWER algorithm is a class 1 medical device (CI/2017/0039) that forms one part of a broader system that was designed to identify and respond to changes in well-being that were suggestive of EWS. Figure 2 provides a graphical representation of the system’s high-level components and data flow.

FIGURE 2.

The EMPOWER system. Google and the Google logo are registered trademarks of Google LLC, used with permission.

Google and the Google logo are registered trademarks of Google LLC, used with permission.

Participants used a mobile phone app that prompted them to answer a daily questionnaire about the potential EWS of psychosis. The data were then submitted to the EMPOWER server and analysed by the ChIP algorithm. The algorithm, which is summarised in Figure 3, compared a participant’s latest data entry against their personal baseline. If changes exceeded predefined thresholds, a ChIP was generated for the participant. The consequences of the ChIP were that the research team, which included a registered mental health nurse (in the UK), clinical psychologists (in the UK and Australia) and a general psychologist (in Australia only), were e-mailed about the participant and that participant’s status was set to ‘alert’ and was highly visible on a secure web-based researcher interface. In addition to viewing and handling ChIPs, researchers could view longitudinal graphs of their participants’ well-being and possible EWS, filtered by question or by domain (group of questions). ChIPs were initially screened by a member of the research team, followed by (1), in Australia, sharing a summary of relevant ChIP data with the clinical team, and (2), in the UK, a member of the research team checking in with the participant. Based on the outcome of this triage assessment the researcher could share an update with the participant’s care co-ordinator, who could, if indicated, escalate increased support to the participant from their local CMHS to reduce risk of relapse. We aimed to respond to ChIPs in some way within 24 hours or the next working day.

FIGURE 3.

EMPOWER algorithm responding.

The alert algorithm also ran a separate process scanning for EWS changes against the baseline. Based on these changes, the logic selected a message from the most appropriate of several content-based message pools (i.e. one message pool contained helpful messages about ‘mood’, another had messages about ‘anxiety and coping’, etc.). This message was delivered back to, and displayed on, the participant’s EMPOWER app. Messages were intended to help people have a greater sense of control over their mental health and well-being and to support self-management. We set up a specific patient and public involvement group to assist us in the co-design of the message function in the app. The group advised on the curation, content and delivery of messages. This group met on four occasions and had public representation from four people with direct lived experience of psychosis.

Training and support to community mental health service staff

After CMHS had been randomised to EMPOWER, we aimed to provide training to those mental health care staff in teams based on our model of relapse prevention, which emphasised:

• therapeutic alliance

• barriers to help-seeking

• familiarisation with the app

• developing an individualised formulation of risk of relapse

• developing a collaborative relapse prevention plan.

Following this, we aimed to meet with care co-ordinators on a fortnightly basis to provide support in the implementation of EMPOWER. These meetings aimed to clarify and encourage formulation of any changes and participant responses within the model, and support clinicians to consider EMPOWER-consistent intervention options.

Background intellectual property

The background intellectual property and functionality was the pre-existing ClinTouch app developed in 2010 and funded by two grants from the UK Medical Research Council. The aims of the app were to help people with serious mental illnesses to manage their own symptoms and to prevent relapse. Software development and beta testing used an experience-driven design process in which service users with psychosis were involved in the design and development of the app, its functionality and its standard operating procedures. Health professionals were included in system design where it related to routine practice, and in the design of new, digitally enabled workflows. Randomised feasibility trials showed this method of active symptom monitoring to be safe, feasible and acceptable. 139–141 This personalised smartphone application triggers the collection of symptom self-ratings several times daily and wirelessly uploads these in real time to a secure central server. A semi-random beep (two to four times daily) alerts the user to complete a set of 12–14 branching items about current symptom severity using a touchscreen slider. A graphical summary of how symptoms fluctuate over time is assembled and displayed on the handset. A validation study compared face-to-face ratings on the gold-standard Positive and Negative Syndrome Scale (PANSS) with four-times-daily ClinTouch assessments over 1 week. This confirmed the validity of the self-reported items, with core psychotic symptom and mood items showing moderate to strong (r > 0.6) correlations between the two methods. 139 Non-core, behaviourally assessed items, such as negative symptoms, showed weaker correlations.

Based on this proof of concept, the standalone smartphone system was integrated via an application programming interface into NHS trust ICT (information and communication technology) platforms to allow the streaming of summary information into electronic health records, and to enable health professionals to track current symptoms on desktops at the team base and receive alerts when symptoms exceeded a pre-agreed personalised threshold. An open, randomised trial of ClinTouch active symptom monitoring compared with management as usual was conducted in NHS mental health trusts in Manchester and South London. 142 Recruitment to the trial took place between February 2014 and May 2015. The aims were to assess (1) the acceptability and safety of continuous monitoring over 3 months, (2) the impact of active self-monitoring on positive psychotic symptoms assessed at 6 and 12 weeks, (3) the feasibility of detecting EWS of relapse communicated to health-care staff via an application programming interface allowing data to be streamed into the electronic health record and (4) the impact of active self-monitoring on positive psychotic symptoms assessed at 6 and 12 weeks. Eligible participants with a DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) diagnosis of schizophrenia and related disorders and a history of relapse within the previous 2 years were enrolled from an early intervention team and a community mental health team.

The findings were that, of 181 eligible patients, 81 (45%) consented and were randomised to either active symptom monitoring or management as usual. Of those in the active symptom monitoring group, 90% continued to use the system at 12-week follow-up. In this group, adherence, defined as responding to > 33% of alerts, was 84%. At 12 weeks, active symptom monitoring compared with usual management was associated with no difference on the empowerment scale. The pre-planned intention-to-treat analysis of the primary outcome (i.e. positive symptom score on the PANSS scale) showed a significant reduction in the active symptom monitoring group over 12 weeks in the early intervention centre. Alerts for personalised EWS of relapse were able to be built into the workflows of the two NHS trusts, and 100% of health professional staff used the system in a new digital workflow. Qualitative analyses supported the acceptability of the system to participants and staff.

Outcomes

All outcome measures were administered at baseline and subsequently at 3, 6 and 12 months by research assistants who were trained in the use of all of the instruments and scales and had achieved a satisfactory level of inter-rater reliability (see Appendix 7). Research assistants had, as a minimum, a strong honours degree in psychology or a related discipline. Regular inter-rater reliability assessments were conducted during the trial. Research assistants entered anonymised participant data from paper records onto an electronic case record form hosted by the University of Aberdeen, with the exception of relapse data, which were entered by the trial manager. In Glasgow, data were collected in participants’ homes or in health service facilities. In Melbourne, all data were collected on health service premises. Data quality and error checking were conducted at each time point, namely baseline and 3, 6 and 12 months post randomisation.

Candidate outcomes

Across measures, a lower score generally represents a positive or more desirable rating. Exceptions to this are the Service Attachment Questionnaire147 and the Medication Adherence Rating Scale,148 on which a lower score represents poorer engagement and poorer adherence, respectively.

Process evaluation

In line with Medical Research Council guidance on the process evaluation of complex interventions,165 we produced a logic model for the EMPOWER intervention and conducted a process evaluation. The process evaluation will be used to explore the ways in which EMPOWER may operate to produce outcomes. Specifically, it will focus on intervention fidelity, exposure, reach, context, recruitment, retention and contamination, as well as the acceptability of study procedures. We will interview service users, carers, mental health care staff and research staff to ensure a multiperspective understanding of the intervention. The process evaluation has been published elsewhere. 166

Sample size

No formal sample size calculation was appropriate for this pilot phase. The proposed sample size of up to 120 service users across 40 care co-ordinators in eight CMHS was deemed to be sufficient to establish feasibility and obtain parameters, including the relevant intracluster correlation coefficients for the cluster design, to inform the design and size of a future definitive, pragmatic, multicentre and multinational cRCT.

Recruitment and randomisation

The unit of randomisation was the CMHS (the cluster). Participating CMHS were randomised within stratified pairs to the EMPOWER relapse prevention intervention or to continue their usual approach to care.

Researchers approached each eligible care co-ordinator and sought their consent to participate in the trial. Prior to randomisation, consenting care co-ordinators provided an anonymised list of their current potentially eligible caseload of people using services. This list was then randomly ordered by the Centre for Healthcare Randomised Trials (CHaRT). Researchers then approached identified people sequentially in blocks of up to five potentially eligible participants and sought informed consent to participate in the study. If there were further participants eligible for inclusion at the end of this block, the researcher moved onto the next block of five (if applicable). Care co-ordinators provided participants with an easy-to-read information leaflet about the study to enable potential participants to express interest in finding out more information. In Australia, information posters were displayed in the staff areas of participating sites to inform care co-ordinators about the study and the contact details of research assistants were provided for those who wanted to participate.

We aimed to approach and consent, on average, three participants per care co-ordinator (giving a total of up to 120 potential participants). Following delays resulting from Medical Device Registration procedures and consultation with the independent DMEC and SSC, this recruitment target was amended to 86 participants. After completing baseline assessments on all consenting service users in care co-ordinators’ and CMHS’ caseloads, the clinical trials unit (CTU) at CHaRT conducted randomisation of the CMHS. For Australia, with just two clusters, this was by simple randomisation by the CTU. For Glasgow, with six clusters, the CTU created three pairs of teams based on similarity of the catchment area in terms of social deprivation (Carstairs) score or CMHS type (e.g. early intervention service). The CTU randomly allocated one member of the pair to the intervention and the remaining member to control.

In this pilot phase we explored the best method of randomly allocating the clusters in the full trial, specifically to establish what matching factors (if any, and/or if matching at all is appropriate, methodologically) were suitable. Any violations of the study protocol were recorded and reported to the Research Ethics Committee, the SSC and the independent DMEC.

Statistical analysis

A full statistical analysis plan was written (and published on the CHaRT website167) prior to any analysis being undertaken. All analyses were carried out using the intention-to-treat principle with data from all participants included in the analysis including those who did not complete the intervention. Every effort was made to follow up all participants in both arms for research assessments. The analysis followed the guidelines of the CONSORT statement for cluster randomised trials and recommendations for the analysis of cluster randomised trials when presenting and analysing the data. Here, we had potentially repeated measures on individual patients nested within care co-ordinators who were nested within teams (the unit of randomisation) who were nested within region (Australia and Scotland).

A full trial analysis may consider adjusting for these factors using appropriate random (service user, if relevant; care co-ordinator; and team) and fixed (region) effects. However, for this feasibility study, analysis models for all outcomes used a simplified approach whereby we analysed the unmatched data.

Baseline characteristics were also examined for dropouts and completers in each treatment arm. Treatment arms were described at baseline and follow-up using means [with standard deviations (SDs)], medians (with interquartile ranges) and numbers (with percentages), where relevant, for demographics and outcome measures. We analysed the main primary outcome, one or more relapses within the previous 12 months, using a generalised linear model, as a modified random-effects multilevel Poisson regression with a log-link function and robust error variance. 168 This facilitates the estimation of covariate adjusted relative risks (RRs) and the derivation of the absolute risk difference. This model was adjusted for a fixed country effect and accounted for possible CMHS clustering using a random-effects robust variance for centres. As relapse was not assessed formally at baseline, this could not be incorporated into the model. Time to the first relapse was assessed using Cox regression.

The repeated measures aspects of the secondary outcomes meant that we used mixed random-effects generalised linear models with appropriate distributional forms depending on the outcome being analysed. These allowed the estimation of the treatment effects over time by including a time-by-treatment interaction for fixed (nominal) time points of 3, 6 and 12 months from randomisation. These were adjusted for the baseline measures for the outcome being assessed as well as having a fixed country effect and random centre effect. Owing to the large number of secondary outcome measures and the feasibility nature of the study, although we examined p-values (but only those with < 1% significance) and effect size (using Cohen’s d), this was not to judge the treatment impact but rather to assess the worth of each scale for inclusion in any potential full trial. The trial statistician remained blind to the primary outcome until the final data cut.

A full health economic statistical analysis plan was written prior to analysis being undertaken. As part of the within-trial economic evaluation, we tested two health-related quality of life measures that can be used to assess quality-adjusted life-years (QALYs), the EQ-5D-5L, and the Assessment of Quality of Life (AQoL-8D), in the feasibility trial. Although the EQ-5D-5L is very commonly used in the UK and Australian contexts, its sensitivity in and appropriateness for people with schizophrenia has been seriously questioned. 169 The AQoL-8D is a newer health-related quality-of-life measure that was developed to be sensitive to the quality-of-life domains that are important to people with mental health problems. A RUQ to capture the costs incurred was also tested. This questionnaire needed to be appropriate to both the UK and the Australian contexts but would possibly require some system-specific modules for services that differ between the two.

Work package 6: widening stakeholder engagement

Following the cRCT, we aimed to engage with mental health services and local service user and carer organisations in the potential centres for a main trial (Edinburgh, Manchester and Birmingham). Our plan was to host three knowledge exchange events in Edinburgh, Manchester and Birmingham and invite key representatives of NHS services, professional staff and local service user and carer organisations. At these events we aimed to identify the key learning outcomes from the EMPOWER project and work with stakeholders to develop plans for the main study phase. We aimed to follow up these knowledge exchange events with active engagement with local NHS services, CMHS and management, local research and development (R&D) and information governance departments. We aimed to identify potential changes to services that would threaten cluster randomisation in a future trial. The outcomes of WP 6 are summarised in Appendix 11.

Protocol amendments

The protocol amendments are described in Table 4.

Protocol breaches

There were no protocol breaches.

Study amendments

In September 2018 we were granted a 6-month extension to the study by the Health Technology Assessment programme team on the basis of delays incurred as a result of medical device registration. Within the extension request we made the case, with the backing of our SSC and DMEC, for a reduction in the target sample from 120 service user participants to 86. This revised target was based on our recruitment progress and justified with reference to sample sizes of similar feasibility studies of mHealth for psychosis. 117,170

System safety and privacy protection

Three general principles of information security (confidentiality, integrity and availability) were followed in the design and implementation of EMPOWER, which applied to phases 1 and 2 (WPs 3 and 4). All data transmitted to and from EMPOWER servers were encrypted over https with strong ciphers as detailed in Approved Cryptographic Algorithms: Good Practice Guideline. 128 Cipher suites were implemented in compliance with section 6 (‘Preferred uses of cryptographic algorithms in security protocols’) of the Good Practice Guidelines. In cases where participant data were downloaded from the EMPOWER site, these data were securely encrypted with a pass phrase of appropriate length and complexity. Data transfers were secured using standard web security protocols. Uploading study data to a central server in real time enabled them to be captured and so this protected against data loss, such as if a phone was lost or stolen. This removed the need for personal data to be stored on the device. The purpose of the server in this case was secure data storage.

Theory

In understanding a problem in health care, it is important to be pragmatic and use methodologies that create interpretive insights for researchers to theorise about how a problem is sustained. 189 Moore and Evans183 state that implementation theories are useful because they help to explain what happens during clinical trials. 190 Incorporating theory can also enhance understandings of barriers to research translation and how these barriers might operate. The Medical Research Council guidelines on evaluating complex interventions165 recommend using theory to understand implementation processes. In line with this, questions asked of participants during phase 1 were designed using normalisation process theory (NPT),191 which is concerned with the work that groups and individuals do when interacting with an intervention and how they make sense of it within their everyday practices. To differentiate EMPOWER from current relapse prevention practice, we also used NPT to characterise stakeholder discussions about both current practice and the proposed intervention. In particular, we evaluated expectations of the EMPOWER intervention using the following NPT constructs: coherence (e.g. did EMPOWER make sense to service users, clinicians and carers?), cognitive participation (e.g. did stakeholders think it was a good idea?) and collective action (operational work that people expect they would need to do to implement EMPOWER). Our rationale for using theory to analyse the data was the pragmatic goal of explaining stakeholder views as discussed during the preliminary qualitative phase, rather than building or testing theories. 192 Therefore, the results will be reported in line with whether or not the EMPOWER approach made sense to participants and how the intervention was expected to ‘fit’ in terms of relationships and the everyday work of relapse prevention.

Aims

The aims have been detailed in Chapter 2. To summarise, there were three WPs.

Sampling and recruitment

Analysis

Task groups with service users (work package 1)

Task groups with carers (work package 1)

Task groups with staff (work package 2)

Beta testing (work package 3)

Participants had access to the app for an average of 36.7 days (range 32–49 days) and completed the app questionnaire on an average of 25.4 days (range 16–35 days), giving an overall rate of engagement of 68.9%. These data are summarised in Table 6. All service users described the app as easy to use, including three participants who were not familiar with smartphones. Participants’ experiences were generally positive, and the majority of participants felt that the EMPOWER app was an accessible and useful tool to enhance self-management.

Interpretative phenomenological analysis (work package 3)

As part of the analysis of the transcribed interviews with seven participants, three inter-related superordinate themes were constructed: (1) generating evidence, (2) awareness and learning and (3) acceptability and feasibility. Figure 4 illustrates these three superordinate themes and their subordinate themes.

FIGURE 4.

Superordinate and subordinate themes.

Coherence

In WPs 1 and 2, it was important to understand if the EMPOWER app made sense, had a clear purpose, was distinct from existing practices, would bring benefits that stakeholders would value, and would fit with the goals and activities of stakeholders. In NPT, this is referred to as coherence: the extent to which participants make sense of an intervention. During WPs 1 and 2, all participants described relapse as a negative event leading to significant distress and/or disruption in their spheres and contexts. There was a clear and shared understanding of the function of monitoring EWS to identify changes in well-being suggestive of relapse. The EMPOWER app seemed to offer something distinctive from routine care and was valued by participants. Anticipated benefits included being able learn from the data and develop a better understanding of well-being, providing a memory aid to support personal decision-making or shared decision-making, providing a tool to enable carers to encourage learning, monitoring and help-seeking, and offering a means of developing greater self-reflection.

Cognitive participation

It was also important to understand if stakeholders thought that the app was a good idea, the extent to which they saw the app as having utility, and whether or not they would be prepared to invest time and effort in the EMPOWER app. In NPT this is referred to as cognitive participation, or the extent of necessary commitment and engagement by participants. Although the app was distinctive and made sense to mental health care staff, they also raised important and distinctive concerns, especially in relation to high caseloads and time pressures that already had an impact on their ability to spend time working with service users. Staff perceived that if the app increased their workload then they would not have capacity to work with it. Staff also were concerned that many of their service users would not use the intervention because of the severity of illness, cognitive impairments, the impacts of digital exclusion or other stresses arising from poverty and deprivation. These are common concerns that have resulted in digital triallists conducting research focusing on digital exclusion in the context of psychosis. 204 Staff reflected that often it was these same service users who were at highest risk of relapse, and therefore those most in need of relapse prevention would be least likely to utilise the intervention. Berry et al. 205 previously reported that mental health care staff have conflicting views about the use of the internet and mobile phones for self-management by people with psychosis. The authors found that mental health care staff were cautiously optimistic about using mobile phone apps for monitoring. However, they also found that staff had concerns about their responsibility when receiving symptom reports from service users as they were worried about missing important risk information. Therefore, staff expressed a preference for receiving reports on monitoring during sessions with service users. In other research,170 staff have found the number of data from digital interventions overwhelming. These findings resonate with our own finding that concerns about lack of staff time and capacity are key logistical barriers to the implementation of digital technologies in psychosis care175 and should be considered when designing interventions. By contrast, both service users and carers felt that the app provided an important opportunity to keep track of and identify changes in well-being as a way of improving self-management and control. In addition, service users saw messages as a potential means of additional support and encouragement to access the knowledge and skills to stay well, as long as those messages were salient and meaningful.

Collective action

In NPT, collective action relates to the work that participants need to engage in to make an intervention function in its context. This includes questions regarding how the intervention may promote or impede existing practices, and what impact the intervention might have on the division of resources or allocations of responsibilities among stakeholders. 206 Mental health care staff spoke about the importance of data from the EMPOWER app being reliable and valid in order to be useful for incorporation into routine practice and clinical decision-making. This included reflections on threats to the reliability and validity of data. Staff were concerned that service users may use the app to increase access to support and crisis care by exaggerating the severity of their experiences and that service users might lack the insight to reliably enter data that would be helpful to clinical decision-making. Staff were also concerned about data being viewed without broader awareness of the context of either the person (e.g. their access to sources of support, coping) or the service (e.g. its knowledge and expertise in delivering care to that individual). All of these factors were seen as potentially having an impact on existing practice in unhelpful ways, such as by increasing workload. Service users expressed concerns about help-seeking in the context of early signs of relapse, in particular about their data being shared and utilised to initiate unwanted interventions, including changes to medication and rehospitalisation. At the same time, service users said that the app should not be seen as a replacement for routine support from mental health services (a concern also expressed by mental health care staff and carers). In mapping out standard EWS-based monitoring from the task groups, it appeared that carers felt they were on the outside of the service user–clinician relationship, and many reported that clinicians could fail to take their concerns about their loved one’s well-being on board. These concerns manifested in relation to the app, and carers expressed a desire to be involved, where agreed with their loved one, in encouraging use of the app. In summary, different stakeholder groups had unique perspectives and concerns about current relapse management that appeared to act as a barrier to shared decision-making, echoing the qualitative work conducted following the CRIMSON study. 207 In terms of collective action, predictions about EMPOWER implementation barriers appeared to be clustered around whether or not the intervention was expected to have an impact on existing issues and concerns. For example, if the intervention increased workload, staff would be likely to resist using it, a phenomenon sometimes observed when novel digital interventions are implemented in clinical settings. 172

Translation to EMPOWER phase 2

Task groups identified that the proposed EMPOWER intervention (including the app) made sense to all of our stakeholder groups and was relevant to either existing mental health services practice or the concerns and values of service users and carers. We identified and theoretically framed important barriers to implementation in routine clinical practice. These barriers included the current context of practice, as staff reported carrying large and busy caseloads and were concerned that the app might create additional and unnecessary work as a result of the lack of reliability and validity of service users’ self-reported data. In addition, staff saw themselves as having strengths in their knowledge about and expertise in managing relapse and in their ability to tailor their approach to relapse detection and prevention to individual service users based on their knowledge of that person. Therefore, based on the NPT framework,190 there exists in current clinical practice potential structural barriers (e.g. lack of staff time) but also cognitive barriers (e.g. how staff think about their own role and skills in relation to relapse prevention). In response, we were careful to consider the role of triage in the design of the broader EMPOWER intervention. That is, rather than alerts triggered by changes in well-being going directly to mental health care staff, these alerts would be triaged by members of the research team and shared with staff members as part of a broader discussion incorporating relevant changes having an impact on the service user and relevant staff interactions with the service user. Sekhon et al. 208 stated that decisions about whether or not an intervention is appropriate for end-users are made through anticipated or actual cognitive and emotional responses to that intervention. Although service users and carers in this study valued relapse prevention as an important goal, they also raised important concerns around fears of help-seeking in the event of a crisis and the risk of exposure to unwanted interventions, particularly rehospitalisation. Mapping out the extent to which emotional responses such as fear may factor into hypothetical intervention acceptability was a key part of this research. The triage by research team members of changes in well-being was an important component of the EMPOWER intervention as a means of making sense of changes in well-being over time, linking changes to broader contextual factors and identifying patterns of change over time, all of which it was hoped would create an enriched understanding of changes in well-being. Therefore, the aim of the triage role was to reduce the fear and uncertainty of using a novel self-monitoring approach situated within existing fears about help-seeking that are inherent in current EWS-based monitoring approaches to relapse prevention. In addition, an important part of the EMPOWER intervention was the role of peer support. The results from task groups and beta testing also enabled the research team to carefully consider and refine the role of peer support workers in the intervention. We further emphasised the peer support workers’ role in helping service users (and, where relevant, carers) learn about the mobile app and its functions, providing support with using the app and reflecting on app-generated messages, prompting engagement and curiosity in these messages as a basis for developing greater autonomy and self-management. As part of this, peer support workers were also app users, enabling them to share their own experiences of mental health more broadly or their experiences of using the app more specifically.

Beta testing

User experience research in digital interventions for psychosis has been important in understanding and anticipating potential implementation barriers for interventions addressing smoking,209 paranoia185 and symptom management. 210 Conducting UX testing directly with end-users in advance of the trial was a further important step in anticipating specific implementation issues with the EMPOWER intervention. Through conducting beta testing, we identified a number of relevant and important technical issues. Fixes for these technical issues were included in software updates successfully delivered before phase 2. Technical problems resulting in poor UX, such as apps crashing, have been identified as key implementation barriers in clinical trials of digital interventions for psychosis,211 highlighting the importance of conducting short-scale UX testing to identify technical issues. There were a number of important findings in relation to participants’ experience of the app, including learning from the experiences of those who had never used a smartphone previously, themselves an under-researched subgroup. 204 Participants noted that the app was simple to use and that the questions were relevant to their well-being. Participants felt that the app was a useful tool that they would be able to use to self-monitor their own well-being, and this was viewed as an important basis for staying well. Having used the app, all participants in the beta-testing phase expressed a preference for being able to share data with others, particularly mental health care professionals. In this regard, the app was seen as a tool for generating evidence to support personal awareness and learning or, indeed, for sharing evidence with mental health care professionals. The well-being messages generated in response to user data were understood as credible and a helpful prompt for self-management, providing access to broader information about experiences relevant to staying well.

Although the hypothetical acceptability of the well-being messages appeared reasonable in both the task groups and the UX testing phase, service user participants nonetheless expressed that if they perceived the messages to be patronising they would not be likely to engage with them. Furthermore, previous research has demonstrated framing effects (e.g. a willingness to even continue to read messages) of the words that are used in well-being messages. 212 To ensure that the messages were most relevant and salient to service users, we set up a patient and public involvement group to steer the development of principles for the curation and refinement of messages, to provide advice and good sources of user-friendly information and also to review the content and delivery of messages for EMPOWER. In recognition of how participants spoke about how current relapse prevention practice operates within an overarching structure of fear and uncertainty, we also changed the language around EWS monitoring. The language of EWS monitoring used by the researchers during task groups suggested that all stakeholders identified strong associations with risk. By using the app as a way to monitor the ‘ebb and flow’ of well-being, the language took on a more normalising and engaging tone. This was reflected in changes to the ‘clinician interface’ (where app user data were monitored by the research team), where changes identified by the algorithm generated ‘alerts’ that also continued to evoke the language of risk. We modified this to the term ‘check-in prompts’ to reflect the shift towards triage as a basis for learning and recognising that there was significant uncertainty about the clinical relevance of changes detected by the algorithm and the extent to which these may reflect true or false positives.

Service user baseline demographic characteristics

Baseline characteristics are summarised in Table 7. Men and women were similarly represented in the sample, and the mean age of service user participants was 43 years. Thirty-seven per cent of participants described themselves as having an informal carer. We noted a higher proportion of service users in the TAU arm stating that they had a carer (n = 17, 55%) than in the EMPOWER arm (n = 10, 24%). On average, service users had been in contact with mental health services for around 12 years.

Clinical characteristics of service users at baseline

Clinical characteristics of service users at baseline are described in Table 8. Ninety-five per cent of service users were assessed as being in full or partial remission at baseline. The total scores on the PANSS indicate that this group of individuals had a moderate level of illness severity,213 and Personal and Social Performance (PSP) scale scores suggest a marked impairment of functioning across the group. 151 Levels of depression were also high, with 56% of the sample scoring ≥ 6 on the CDSS, suggesting the presence of major depression. 214 Levels of alcohol and other drug use were low at baseline.

Carer and care co-ordinator baseline characteristics

Carer and care co-ordinator baseline characteristics are summarised in Tables 30 and 31 (see Appendix 3). Carers were predominantly female (71%) and had a mean age of 49 years. Care co-ordinators were also predominantly female (74%) and had been qualified for 11 years on average.

App engagement

Of the 41 participants who had the app set up, 33 (80.5%) completed the 4-week baseline. These participants used the app for a mean of 31.5 weeks (SD 14.5 weeks; range 44 weeks). During that period, participants used the app for a mean of 64.1% (SD 22.5%; range 76.3%) of days. Therefore, according to our a priori criterion for acceptable engagement of 33% daily use, 30 (91%) participants met our criterion for adherence. Among those randomised to the EMPOWER arm, this represents 71% meeting the criterion. This range of 71–91% provides a broader estimate of overall engagement.

Fourteen participants (42.4%) were still using the app at their final follow-up assessment. Survival analysis methods are recommended for understanding attrition in digital interventions. 215 Attrition represents the amount of time to a relevant event occurring; in this case, the criterion was 4 sequential weeks of not meeting the intended adherence criterion of 33%. The analysis was completed using the survfit function in the survival package in R, with bootstrapping performed using the bootkm function from the hmisc package in R. Overall, the median survival for not missing 4 sequential weeks of 33% use was 32 weeks (bootstrapped 95% CI 14 weeks to ∞). We noted that the upper limit returned an infinite value, likely to be a result of the skewed data. In other words, for 50% of participants who had completed a baseline it took 33 weeks before missing 4 sequential weeks of intended intervention usage. The width of the CIs suggest that some level of uncertainty is appropriate when interpreting the result of this test (Figure 7). To summarise, the length of time at which 50% of participants no longer meet the intended adherence criterion is hard to predict within this sample (especially in terms of an upper limit) but it is likely not to fall below 14 weeks. Supplementary information on our approach to describing engagement with the EMPOWER app is provided in Appendix 5.

FIGURE 7.

Kaplan–Meier curve of time to 4 weeks’ app use < 33%.

We explored the number of peer support worker sessions delivered over the course of the study. A total of 522 peer support worker sessions were offered to 42 participants in the EMPOWER arm (mean 12.4, SD 6.8; median 11.0, interquartile range 10.0). We also explored the number of ChIPs triggered by the algorithm during the study. There were 558 ChIPs during the study across 37 participants (mean 15.1 per participant, SD 10.5 per participant; median 13.0, interquartile range 15).

Provision of study phones

A total of 28 mobile phones were provided to and used by participants: 22 in Glasgow and six in Melbourne. Six provided phones were either lost or stolen, four in Glasgow and two in Melbourne, and handsets were replaced on three of those occasions. The remaining participants in the EMPOWER arm chose to have the app installed on a personal handset. We did not ask for provided phones to be returned.

Adverse events

Adverse events monitoring and reporting included serious and non-serious adverse events, in line with the registration of the study as the Investigation of a Class 1 Medical Device. The medical device was specifically the algorithm that reviewed app data and determined responses. Across both arms there was a total of 54 adverse events, affecting 29 people, during the study. Around half of all events across arms were classified as serious adverse events, and the vast majority of these were anticipated. There was one death during the study. In terms of relatedness of adverse events, six events were assessed as being related to a study procedure, one of which was serious. This involved a threat made to a member of research staff in Melbourne in relation to the use of a study phone.

Adverse events are summarised in Table 10. There were no adverse events of any type related to the EMPOWER Class 1 Medical Device. However, to ascertain a fuller picture of possible harms than would have been available from focusing solely on medical device-related events, we additionally monitored adverse events related to people’s experience of the EMPOWER app more generally. There were 13 app-related adverse events, affecting 11 people, one of which was serious. This involved a brief hospital admission for a physical health complaint, which the service user described as being in part related to feeling overwhelmed by the recent installation of the app. The service user, who had not yet used the app, appeared to be experiencing depression based on their contemporaneous CDSS assessment, and subsequently withdrew from the study. Examples of non-serious app-related adverse events included four instances in which the app caused unhelpful rumination. In one of these instances, where the self-monitoring approach was described as counter to the service user’s usual coping strategy of ‘burying things’, the participant withdrew from the study. Other participants described feeling forced to think about being unwell because of questions in the app, with one person suggesting less frequent monitoring in future iterations. Unhelpful rumination of this type was identified by one participant as an issue when they were well, whereas a second person was affected when they felt more depressed. Two participants specifically cited increased paranoia as a result of the app. On one occasion this related to the timing of question alarms and on another it related to the specific content of personalised questions. A further participant identified that personalised question content unhelpfully triggered traumatic memories of psychosis. One participant reported experiencing increased anxiety after being asked a question that was new to them as a result of branching rules in the question set. In one case a participant reported increased worry as a result of losing their provided mobile phone, and, in a further event, a participant experienced distress as a result of a technical fault arising from a conflict between the app’s software and their personal phone. App-related adverse events and our responses to them are described in more detail elsewhere. 113

Device deficiencies

Two device deficiencies of the investigational medical device were identified during the study. The first was categorised as a manufacturing defect and was related to the following performance end point:

• Following 4 weeks of usage, the EMPOWER algorithm calculates participants’ individualised baseline of symptoms and experiences.

One participant’s initial baseline observations showed no variation. That is, they scored the same for each item on each day of the baseline period. As the result of an anomaly in the algorithm’s calculation of standard deviation, every time the participant responded to questions following the baseline period, a ChIP was automatically generated regardless of any relevant change in scores.

A second device deficiency, which was categorised as a device malfunction, involved an unexpected server shutdown at the University of Manchester. This lasted for around 5 hours, affecting the following performance end points during that time:

• Researcher accesses participants’ questionnaire responses and generates charts to observe changes over time.

• Researcher receives a record of alerts for each participant and is able to record actions in relation to these alerts.

Candidate primary outcomes

We collected complete relapse data on 67 (92%) participants at 3 months and on 61 (84%) participants at every follow-up time point. Research assistants screened electronic case records to identify potential episodes of relapse and exacerbation at 3, 6 and 12 months post randomisation. These episodes provided the basis for individual anonymised case vignettes that were reviewed by an independent and blinded adjudication panel, who determined what, if any, criteria for relapse were met, the date of relapse and the type of relapse. Following calibration on four relapse assessments, inter-rater reliability testing was undertaken on a further 12 assessments for which we noted substantial agreement (k = 0.80). These findings demonstrate the feasibility of collecting data relating to relapses and exacerbations from routine clinical data.

During the study, we identified 27 relapses over 12 months, with a larger number in the TAU arm (n = 19) than in the EMPOWER arm (n = 8). Among these relapses were 11 hospital admissions, nine in the TAU arm and two in the EMPOWER arm. Five relapses involved the use of mental health legislation, all of which were in the TAU arm. Details of the relapse characteristics collected at each follow-up time point are in Appendix 6, Table 37.

Our main approach was to include participants with complete 12-month follow-up data; there were 8 out of 33 (24%) relapses in EMPOWER and 13 out of 28 (46%) in TAU (Table 11). Using all available follow-up data in a time-to-first relapse analysis showed that relapse was less likely in the EMPOWER arm (hazard ratio 0.32, 95% CI 0.14 to 0.74). This is also illustrated in the Kaplan–Meier plot in Figure 8.

FIGURE 8.

Kaplan–Meier plot of time to first relapse.

Table 11 also shows the time to first relapse, with indications that this was longer for those in the EMPOWER arm (median 4.8 months) than for those in the TAU arm (median 1.4 months).

A key concern was that the EMPOWER intervention would lead to an increased fear of relapse because of the potential that monitoring possible early signs of relapse would lead to increased anxiety and hypervigilance. Table 12 summarises the outcomes of all subscales and the total scale of the Fear of Recurrence Scale (FoRSe) at all time points, including baseline. At 12 months, FoRSe data were available for 58 (79.5%) randomised participants. We did not observe any increase in the FoRSe subscale fear of relapse over time in the EMPOWER group. however, we did observe that fear of relapse appeared to be lower in the EMPOWER arm than in the TAU arm at 12 months (mean difference –4.29, 95% CI –7.29 to –1.28; Cohen’s d = –0.76). For intrusiveness the mean difference was –1.16 (95% CI –3.84 to 1.52; Cohen’s d = –0.18) and for awareness the mean difference was –2.17 (95% CI –4.47 to 0.12; Cohen’s d = –0.36); the FoRSe total mean difference was –7.53 (95% CI –14.45 to 0.60; Cohen’s d = –0.53).

We undertook an analysis of the eight relapses in the EMPOWER arm. Of these, seven occurred during exposure to the EMPOWER intervention, and six of these were associated with a ChIP. Among the seven relapses, help-seeking was initiated by a service user on four occasions and by a carer on one occasion. In the remaining two events, the relapse was identified by a care co-ordinator during a routine follow-up appointment.

In summary, fewer participants in the EMPOWER arm had a relapse and time to first relapse was longer, and at 12 months the EMPOWER participants were less fearful of having a relapse than participants receiving TAU. For a feasibility study, these are encouraging results.

Clinical outcomes

Mental health outcomes were assessed using the PANSS, the PSP scale and the CDSS at baseline and at 3, 6 and 12 months. Our approach to establishing and monitoring rater reliability is detailed in Appendix 7. Overall rater agreement was as follows: PANSS (84%), CDSS (96%) and PSP scale (86%). Table 13 shows that, at 12 months, PANSS data were available for 55 (75%), PSP scale for 55 (75%) and CDSS for 53 (73%) of randomised participants. Outcomes at 12 months on the PANSS scales were as follows: PANSS positive (mean difference 0.67, 95% CI –1.56 to 2.91; Cohen’s d = 0.11), PANSS negative (mean difference –2.82, 95% CI –4.75 to –0.89; Cohen’s d = –0.57), PANSS disorganisation (mean difference –0.48, 95% CI –2.01 to 1.06; Cohen’s d = –0.08), PANSS excitement (mean difference –0.04, 95% CI –0.45 to 0.37; Cohen’s d = –0.03), PANSS emotional distress (mean difference –0.65, 95% CI –2.26 to 0.95; Cohen’s d = –0.17), and PANSS total (mean difference –3.75, 95% CI –8.31 to 0.81; Cohen’s d = –0.23). The mean difference was –6.09 for the PSP scale (95% CI –0.23 to 12.41; Cohen’s d = 0.35) and –1.20 for the CDSS (95% CI –2.83 to 0.44; Cohen’s d = –0.25).

Substance use outcomes

Substance use outcomes were assessed using the Timeline Followback measure at baseline and at 3-, 6- and 12-month follow-up (Table 14). At 12 months, data on substance use were available for 62 (85%) randomised participants. At 12 months, the RR indices were as follows: alcohol (RR 1.09, 95% CI 0.94 to 1.16), cannabis (RR 0.96, 95% CI 0.91 to 1.02) and other drugs (RR 0.97, 95% CI 0.93 to 1.02).

Emotional distress outcomes

Emotional distress outcomes were assessed using the HADS and the PBIQ-R at baseline and at the 3-, 6- and 12-month follow-ups (Table 15). At 12 months, data were available for 58 (80%) randomised participants. At 12 months, the mean differences were –1.40 for HADS anxiety (95% CI –3.53 to –0.47; Cohen’s d = –0.28) and –2.00 for HADS depression (95% CI –3.05 to 0.26; Cohen’s d = –0.41). For PBIQ-R, mean difference was –1.07 for control (95% CI –2.12 to –0.02; Cohen’s d = –0.42), –0.60 for shame (95% CI –1.85 to 0.50; Cohen’s d = –0.23), –1.06 for entrapment (95% CI –2.28 to 0.16; Cohen’s d = –0.34), –0.45 for loss (95% CI –1.45 to 0.55; Cohen’s d = –0.17) and 0.00 for social marginalisation (95% CI –1.41 to 1.42; Cohen’s d = –0.00).

Service engagement and adherence

Service engagement outcomes were measured using the Service Attachment Questionnaire. Medication adherence was measured used the Medication Adherence Rating Scale. Both measures were administered at baseline and at the 3-, 6- and 12-month follow-ups (Table 16). At 12 months, data were available for 58 (80%) randomised participants. The outcomes at 12 months were mean difference of 0.04 on the Service Attachment Questionnaire (95% CI –0.09 to 0.16; Cohen’s d = 0.15) and mean difference of –0.84 on the Medication Adherence Rating Scale (95% CI –1.66 to –0.01; Cohen’s d = 0.50).

Carers

Descriptions of carer outcomes using the Involvement Evaluation Questionnaire at baseline and at 3, 6 and 12 months are presented in Table 18. At 12 months, data were available for eight (47%) consented carers. Only descriptive data are provided for information and transparency purposes.

Care co-ordinators

Data for care co-ordinators’ assessment of service user engagement with services using the Service Engagement Scale at baseline and at 3, 6 and 12 months are summarised in Table 19. At 12 months, data were available for 31 (42%) randomised service user participants. Lower scores on this scale are indicative of better service engagement. The outcomes at 12 months were mean difference of –0.16 for availability (95% CI –1.20 to 0.88; Cohen’s d = 0.59), 0.54 for collaboration (95% CI –0.77 to 1.85; Cohen’s d = 0.26), 1.03 for help-seeking (95% CI –0.49 to 2.55; Cohen’s d = 0.33), 1.15 for treatment adherence (95% CI –0.21 to 2.09; Cohen’s d = 0.58) and 2.50 for treatment total (95% CI –0.79 to 5.79; Cohen’s d = 0.37).

Small numbers of clusters

Despite the use of random-effects models, the estimates from these models may not be robust because of the small number of clusters. To investigate the impact of this, the models presented above for continuous (or pseudo-continuous) outcomes were compared with those using the Kenward–Roger correction216 in Stata^® version 15 (Stata Corp LP, College Station, TX, USA). The point estimates were similar to the original analyses, as were the Cohen’s d estimates. These were the measures used to assess the outcomes. Therefore, the interpretation of the original models remains unchanged, although this should be done with caution given that the models are based on a feasibility study.

Impact of possible informative withdrawal

The possibility of informative withdrawal was also investigated, with several scenarios considered. There are too few data points to conduct the usual type of pattern mixture with multiple imputed data sets using the full model. To get an initial feel of the impact, a simple pattern mixture investigation using a basic logistic model was used. This indicated that the point estimates were stable if around 50% of those who withdrew were set to ‘relapse’. Next, we examined several different scenarios using the more complex full model. This included (1) the best outcome – all who withdrew (without primary outcome data) were set as ‘no relapse’; (2) the worst outcome – all who withdrew were set as ‘relapse’; and (3) all points in between.

A conservative rate of ‘relapse’ for these withdrawals would be the rate observed in the TAU arm (46%) implying that no more than three might have been expected to ‘relapse’ had they remained in the study. However, a RR of 0.70 is a clinically significant effect for relapse reduction. Using the pattern mixture scenarios and the full models indicated that while standard errors increased there had to be 6 or more of these withdrawals set to ‘relapse’ for the RR to increase beyond 0.70. The original intention-to-treat (ITT) estimate was 0.50. Using the scenarios if 5 withdrawals were set to ‘relapse’ this still only increased the estimate to 0.69.

Overview

The report of this health economic evaluation followed the Consolidated Health Economics Evaluation Reporting Standards (CHEERS) guidance. 219 The economic evaluation was conducted using individual participant cost and effect data collected alongside the EMPOWER trial. The time horizon of the analysis was 12 months with four assessment points: baseline and 3-, 6- and 12-month follow-up.

Study perspective

Different perspectives were adopted for the economic analysis, reflecting the national health technology assessment guidelines in Australia and the UK. 220,221 The base-case scenario adopted three perspectives. First, a health-care payer perspective was adopted; this perspective was limited to costs borne directly by the UK NHS or Australian government (e.g. primary, secondary and community services) and outcomes related to service users (EQ-5D-5L217 and AQoL-8D157). Second, a health-care sector perspective was adopted; this perspective extended the health-care payer perspective to include medical costs paid for by patient out-of-pocket expenses and third-party payers. These additional costs are of particular relevance to the Australia health-care system. The final perspective adopted was societal; this includes all of the costs considered in the health-care sector perspective plus additional relevant costs outside the health-care sector. These additional costs include service user time for using the app, informal care, productivity loss, costs of online self-help services and contacts with the legal, criminal and justice departments. The societal perspective also includes additional outcomes; in the case of EMPOWER, this included outcomes for carers (EQ-5D-5L217 and CarerQoL-7D218).

Identification, measurement and valuation of resource use

Guided by the different perspectives adopted in this study, different cost components were identified: costs related to the intervention, health service use costs, costs outside the health-care sector and costs directly incurred by service users and their carers. All costs were collected based on the 2017 reference year and were expressed as Great British pounds. Australian unit costs were converted to pounds using the CCEMG–EPPI-Centre Cost Converter. 222 As the costs were collected over a 1-year period, discounting was not applied.

Identification, measurement and valuation of outcome measures

For service users, we tested two health-related quality-of-life measures, both of which can be used to estimate QALYs: the EQ-5D-5L217 and the AQoL-8D. 157

The EQ-5D-5L is used frequently in the UK and Australian contexts. It is a generic questionnaire with five questions covering mobility, self-care, usual activities, pain/discomfort and anxiety/depression. Each of these questions has a possible five levels of response. However, a previous review has indicated mixed findings in terms of the measure’s responsiveness to change and convergent validity. 169 Therefore, the EQ-5D-5L was complemented with the newer AQoL-8D, which was designed to be sensitive to the aspects of quality of life important to people with mental health problems. The AQoL-8D contains 35 items, each of which has between four and six possible responses. It has eight dimensions, which are further summarised as super-dimension physical and super-dimension psychosocial.

The outcome measures were valued using preference-based value sets. Although a value set has been published for the EQ-5D-5L for England,156 currently the National Institute for Health and Care Excellence (NICE) does not recommend the use of this measure. 246 Therefore, the EQ-5D-5L utility scores were mapped on to the EQ-5D-3L using the EQ5Dmap command in Stata. 247 As there is currently no Australian value set available for the EQ-5D-5L, the UK value set was used and then also mapped on to the EQ-5D-3L, reflecting the preferences of members of the public in the UK. Finally, the AQoL-8D responses were converted to utilities using the scoring algorithm developed by Richardson et al. 157

With regard to carer outcomes, two outcome measures were included in the carer survey to capture the impact of EMPOWER on carers’ health-related quality of life, using the EQ-5D-5L, and informal care, using the CarerQoL-7D. 218 Carers’ EQ-5D-5L responses were also mapped on to the EQ-5D-3L, whereas the UK utility weights were used for the CarerQoL-7D. 248

All outcome measures were administered at baseline and at 3, 6 and 12 months by trained research assistants.

Missing data

The extent of missing data was examined and assessed to establish the completion rates of the RUQ and outcome measures. Missing data are a well-known issue in cost-effectiveness analyses alongside clinical trials. Instead of conducting a complete-case analysis that is likely to result in bias in economic evaluation, multiple imputation by chained equations was applied to impute missing values for cost and effects. The analysis was performed in Stata using the ‘ice’ package and Stata codes developed by Faria et al. 249 In total, 34 data sets were imputed that reflected the percentage of cumulative missing values. The costs and outcome analysis models were then run across the 34 imputed data sets and combined using Rubin’s rules250 to produce a single set of results.

Analysis

The analysis carried out was according to the ITT principle, which implies that service users will be analysed in accordance with their allocated treatment arm, irrespective of the treatment they received. The differences in resource use between the groups is presented without costs attached to exclude any issues with valuing resource use. The results are unadjusted and include available cases to make use of all data. ‘Available cases’ include all complete data collected during the trial, irrespective of whether, for a particular participant, this included all follow-up points. All outcome measures were assessed for their suitability for collection during the definitive trial. The utility scores derived from the outcome measures are presented for available cases for each arm without adjustment to make use of all data.

Resource use and unit costs were then combined to give a total cost for each participant. The primary type of analysis was a cost–utility analysis, using QALYs as a measure of outcome derived from the EQ-5D-5L measure. For both arms, the mean values of costs and outcomes are reported, as well as the mean differences between the groups. Generalised linear models were used to assess the mean difference in costs and outcomes between the arms, adjusted for baseline characteristics (i.e. baseline utility, baseline costs, country, age and gender). For mean differences in total costs, a gamma distribution and an identity link were applied, whereas for the incremental effect in QALYs, a Gaussian distribution and identity link were used. The Glass’s Delta effect size was calculated (i.e. the difference between the mean scores of the EMPOWER and TAU arms divided by the standard deviation of the TAU arm) for all costs and outcomes, with the recommended minimum effect size defined as Δ = 0.41, moderate Δ = 1.15, and strong Δ = 2.70. 251

Once the difference in cost and QALYs had been estimated, a joint assessment of the two was reported using an incremental approach, with differences in mean costs and mean outcome scores expressed using a ratio of incremental cost per QALY [i.e. an incremental cost-effectiveness ratio (ICER)]:

The non-parametric ‘bootstrapping’ technique was then used with 1000 iterations of incremental cost and incremental effect so that sampling uncertainty could be graphically plotted on a cost-effectiveness plane. Cost-effectiveness planes are graphical displays of cost-effectiveness results plotted on a two-by-two-dimension graph where the horizontal axis represents the differences in outcomes (in this case QALYs) and the vertical axis represent the differences in costs. Results in the north-east quadrant show that the intervention costs more but also has greater benefits than the comparator, whereas results in the south-east quadrant show that the intervention costs less but has greater benefits than the comparator (this is referred to as the intervention being ‘dominant’). Results in the south-west quadrant show that the intervention costs less but also has fewer health benefits than the comparator and results in the north-west quadrant show that the intervention costs more but also has fewer health benefits than the comparator (this is referred to as the intervention being ‘dominated’). The proportions of ICERs falling in each quadrant of the cost-effectiveness plane were calculated, as were the proportions falling underneath the threshold of £20,000 per QALY (for results falling in the top right-hand quadrant and the bottom right-hand quadrant).

An additional cost-effectiveness analysis was carried out that combined the costs with within-trial relapse data (primary outcome) to present a cost per relapse case avoided.

Owing to the small number of carers recruited, carer costs and effects are presented descriptively and not as a formal incremental analysis or combined with service user costs and effects.

Sensitivity analysis

To explore the robustness of the results, sensitivity analyses were carried out. First, owing to the use of multiple imputation, we conducted a complete-case analysis. Secondly, the AQoL-8D questionnaire results were used as the outcome, rather than the results from the EQ-5D-5L.

Missing data

The number (and percentages) of completed economic measures are presented in Table 20 for both randomised service users and carers. For the resource use questionnaire at baseline, 97% (n = 71) of service users and 100% (n = 17) of carers had completed measures. At 3 months post randomisation, this fell to 81% (n = 59) and 65% (n = 11). At 6 months, the percentage of completed measures for service users reduced to 77% (n = 56), but for carers it remained at 65% (n = 11). At the final follow-up at 12 months, 75% (n = 55) of service users and 47% (n = 8) of carers completed measures.

Outcome measures

Service users

Outcome measure results for service user available cases are presented in Tables 21 and 22. In the EMPOWER arm, EQ-5D-5L-derived utilities increased at each follow-up point; the utility score was 0.644 at baseline (SD 0.258) and 0.732 at the 12-month follow-up (SD 0.231). In the TAU arm, EQ-5D-5L utilities decreased slightly over the follow-up period; at baseline they were 0.620 (0.268), at 3-month follow-up they decreased to 0.593 (0.285), but at 6-month follow-up they increased to 0.657 (0.209) and they finally decreased to 0.607 (0.254) at 12-month follow-up. These trends are illustrated in Figure 9.

TABLE 21 - The EQ-5D-5L service user results at all time points

Time point	EMPOWER		TAU
	Mean (SD)	n (%)	Mean (SD)	n (%)
Baseline	0.644 (0.258)	40 (95.2)	0.620 (0.268)	30 (96.8)
3 months	0.666 (0.234)	30 (71.4)	0.593 (0.285)	26 (83.9)
6 months	0.694 (0.197)	30 (71.4)	0.657 (0.209)	26 (83.9)
12 months	0.732 (0.231)	28 (66.7)	0.607 (0.254)	26 (83.9)

TABLE 22 - The AQoL-8D service user results at all time points

Time point	EMPOWER		TAU
	Mean (SD)	n (%)	Mean (SD)	n (%)
Psychosocial
Baseline	0.234 (0.162)	40 (95.2)	0.221 (0.160)	29 (93.5)
3 months	0.271 (0.207)	30 (71.4)	0.216 (0.149)	27 (87.1)
6 months	0.269 (0.179)	30 (71.4)	0.207 (0.156)	26 (83.9)
12 months	0.299 (0.176)	28 (66.7)	0.201 (0.132)	27 (87.1)
Physical
Baseline	0.595 (0.250)	40 (95.2)	0.553 (0.190)	29 (93.5)
3 months	0.576 (0.226)	30 (71.4)	0.551 (0.216)	27 (87.1)
6 months	0.600 (0.227)	30 (71.4)	0.582 (0.245)	26 (83.9)
12 months	0.649 (0.226)	28 (66.7)	0.569 (0.226)	27 (87.1)
Total
Baseline	0.522 (0.228)	40 (95.2)	0.500 (0.190)	29 (93.5)
3 months	0.545 (0.233)	30 (71.4)	0.491 (0.200)	27 (87.1)
6 months	0.557 (0.220)	30 (71.4)	0.489 (0.212)	26 (83.9)
12 months	0.609 (0.210)	28 (66.7)	0.485 (0.200)	27 (87.1)

FIGURE 9.

The EQ-5D-5L service user results, with 95% CIs.

Total AQoL-8D utilities show a similar trend to EQ-5D-5L utilities; in the EMPOWER arm AQoL-8D utilities increased at each follow-up point from 0.522 (SD 0.228) at baseline to 0.609 (SD 0.210) at 12-month follow-up. In the treatment-as-usual arm, the utilities decreased at each follow-up point, from 0.500 (0.190) at baseline to 0.485 (SD 0.200) at the 12-month follow-up point. These trends are illustrated in Figure 10.

FIGURE 10.

The AQoL-8D service user results, with 95% CIs.

The AQoL-8D psychosocial dimension scores increased over the follow-up period, from 0.234 at baseline (SD 0.162) to 0.299 at the 12-month follow-up (SD 0.176), but decreased between the 3- and 6-month follow-up points. In the TAU arm, the psychosocial dimension decreased at each follow-up point, from 0.221 (0.160) at baseline to 0.201 (SD 0.132) at the 12-month follow-up. In terms of the physical dimension, scores in the EMPOWER arm increased during the follow-up period, from 0.595 at baseline (SD 0.250) to 0.649 (SD 0.266) at the 12-month follow-up, with a decrease between baseline and 3-month follow-up points. In the treatment-as-usual arm the physical dimension score also increased between baseline and 12 months follow-up; 0.553 (SD 0.190) to 0.569 (SD 0.226), with decreases between baseline and 3 months and 6 and 12 months, and an increase between 6- and 12-month follow-up. Again, overall AQoL-8D completion was higher in the TAU arm (93.5–83.9%, n = 29–26) than in the EMPOWER arm (95.2–66.7%, n = 40–28).

Carers

Outcome measures for carer available cases are presented in Tables 23 and 24. In the EMPOWER arm, EQ-5D-5L utilities decreased over the follow-up period from 0.815 (SD 0.107) at baseline to 0.798 (SD 0.099) at 12-month follow-up, with an increase between 3 and 6 months and 6 and 12 months. In the TAU arm, EQ-5D-5L utilities decreased over the follow-up period, from 0.748 (SD 0.195) at baseline to 0.695 (SD 0.156) at 12-month follow-up. These trends are illustrated in Figure 11.

TABLE 23 - The EQ-5D-5L carer results at all time points

Time point	EMPOWER		TAU
	Mean (SD)	n (%)	Mean (SD)	n (%)
Baseline	0.815 (0.107)	7 (100)	0.748 (0.195)	9 (90.0)
3 months	0.775 (0.184)	5 (71.4)	0.678 (0.283)	6 (60.0)
6 months	0.797 (0.169)	5 (71.4)	0.686 (0.233)	6 (60.0)
12 months	0.798 (0.099)	4 (57.1)	0.695 (0.156)	4 (40.0)

TABLE 24 - The CarerQol-7D results at all time points

Time point	EMPOWER		TAU
	Mean (SD)	n (%)	Mean (SD)	n (%)
Baseline	85.9 (15.8)	7 (100)	78.7 (14.3)	10 (100)
3 months	86.1 (12.9)	5 (71.4)	79.5 (21.4)	6 (60.0)
6 months	86.0 (5.08)	5 (71.4)	77.8 (19.0)	6 (60.0)
12 months	91.8 (5.24)	4 (57.1)	91.5 (2.67)	4 (40.0)

FIGURE 11.

The EQ-5D-5L carer results, with 95% CIs.

For the CarerQoL-7D results, scores in the EMPOWER arm increased over the 12 months’ follow-up from 85.9 (SD 15.8) at baseline to 91.8 (SD 5.24) at 12 months, with a decrease between 6 and 12 months. This overall increase is mirrored in the TAU arm: 78.7 (SD 14.3) at baseline compared with 91.5 (SD 2.67) at 12 months, with a decrease between 3- and 6-month follow-up. These trends are illustrated in Figure 12. Similar to the EQ-5D-5L completion, numbers were greater in the EMPOWER arm (100–57.1%, n = 7–5) than in the TAU arm (100–40%, n = 9–6).

FIGURE 12.

The CarerQoL-7D results, with 95% CIs.

Resource use

Mean costs and quality-adjusted life-years

The multiple imputation base-case mean costs and QALYs are presented in Table 25. From the health-care sector and the health-care payer perspectives, the mean total costs for the EMPOWER arm were lower than those for the TAU arm, whereas the mean total societal costs for the EMPOWER arm were marginally higher than those for the TAU arm: £12,991 (SD £2622) compared with £12,820 (SD £2892), a difference of £170 (95% CI –£7783 to £8124). Key cost drivers include the intervention cost, health professional visits (mean of £2419 for EMPOWER and £2824 for TAU), hospital admission (mean of £2286 for EMPOWER and £3388 for TAU), stays at on-site accommodation (non-acute specialist residential mental health care in Melbourne only: mean of £2734 for EMPOWER and £3300 for TAU) and mental health medication (£836 for EMPOWER and £1106 for TAU). These cost drivers are illustrated in Figure 13, with a complete breakdown of total mean costs per arm. References to hospital admission costs and to ‘on-site’ costs reflect the differences in reporting of costs according to jurisdiction.

FIGURE 13.

Breakdown of the total costs of EMPOWER and TAU (societal perspective).

As discussed in Results, Outcome measures, there is a trend for the QALYs to be higher in the EMPOWER arm; this result is reflected in the multiple imputation results, with higher QALYs in the EMPOWER arm.

The results of the base-case cost–utility analysis are in Table 26. From both the health-care payer and the health-care sector perspectives, EMPOWER is dominant; it is both less costly and more effective than TAU. From the societal perspective, EMPOWER is more costly and more effective, resulting in an ICER of £3041. This ICER would be considered cost-effective using the NICE threshold of £20,000 per QALY gained. 220

A cost-effectiveness plane for the health-care payer perspective is presented in Figure 14. It shows that the majority of cost and effect pairs fall beneath the £20,000 threshold. A minority of pairs fall into the north-west and south-west quadrants, which would indicate that EMPOWER would be considered less effective than TAU. However, the majority of pairs fall into the north-east and south-east quadrants, where EMPOWER is more effective than TAU. Although the cost-effectiveness plane shows uncertainty around whether EMPOWER is more or less costly than TAU, there is less certainty around whether EMPOWER is more effective in terms of QALYs calculated using the EQ-5D-5L. At a willingness-to-pay threshold of £20,000 per QALY gain, there is around a 70% probability that EMPOWER is cost-effective. The probability that EMPOWER dominates TAU is 58%.

FIGURE 14.

Cost-effectiveness plane (health-care payer perspective).

Table 27 shows the results for the two sensitivity analyses for the health-care payer perspective. When including complete-case costs and QALYs only, the ICER is £2817, indicating cost-effectiveness. Including QALYs from the AQoL-8D instead of the EQ-5D-5L results in EMPOWER dominating TAU.

Mean costs and relapse cases

The results of the cost-effectiveness analysis are presented in Table 28. The number of relapse cases avoided is greater in the EMPOWER arm than in the TAU arm. The cost-effectiveness analysis results show that when using relapse cases avoided as the outcome, both the health-care payer and the societal perspective are dominant and when using the health-care sector perspective the ICER is £12 per relapse case avoided.

Candidate primary outcomes

A further aim of the feasibility trial was to assess primary and secondary outcomes to determine the preliminary signals of efficacy of the EMPOWER intervention as a basis for assessing the feasibility of collecting follow-up measures, determining primary and secondary outcomes, and determining probable sample size requirements for a future main trial. Our candidate co-primary outcomes were relapse and fear of relapse. Overall, the findings indicated that over 12 months, compared with TAU, EMPOWER appeared to reduce the number of relapses, increase the time to first relapse, and reduce fear of relapse. We successfully collected relapse data for 84% of participants at every follow-up point during the trial. We theorised that relapse events involve factors related to the service user (change in positive symptoms, duration of more than 1 week and reduction in functioning and/or increase in risk) and factors related to service responses (change in clinical management, hospitalisation and coercion into treatment). We successfully operationalised these criteria and measured relapse from routinely collected electronic patient record data. We subjected these data to blinded reliability assessments, finding substantial agreement between raters. Finally, all of these data were subject to a blinded outcome assessment.

In studies of interventions for relapse prevention in schizophrenia, there has been a lack of clear consensus about the definitions of relapse. For example, in a meta-analysis of antipsychotic drugs compared with placebo,229 the trial definitions of relapse varied and included clinical judgement (n = 26), need for medication (n = 17), rating scale-based definitions (n = 15), admission to hospital (n = 3) and dropout because symptoms worsened (n = 2). This lack of consensus is present in other meta-analyses. 261,262 In the Kishimoto et al. meta-analysis262 of first- compared with second-generation antipsychotics, nine of the authors’ included studies did not define relapse. A key strength of our study was that our definition was theoretically informed by our intervention model, was defined a priori, was capable of being extracted from routine health service data and was assessed blind with reliability data reported.

Relapse was less likely among participants in the EMPOWER arm, and we identified a RR of 0.50 (95% CI 0.14 to 0.74) over 12 months. This result, although encouraging, needs some cautionary interpretation given that the feasibility study had a small number of clusters, each with small numbers of participants, and that there were more withdrawals from the EMPOWER arm. The impact of the small numbers of clusters was examined and deemed not to be a major concern, although the overall small numbers prevent certainty about this. In terms of the potential impact of the withdrawals on the primary outcome, the number needed to change the benefit of the EMPOWER intervention would have to be set at six or more, a level felt to be unrealistically high. Sim263 and Eldridge et al. 264 have cautioned against the use of effect sizes derived from feasibility and pilot trials to be used as the basis for sample size estimation for a main trial. The imprecision arising from the smaller sample sizes in feasibility or pilot studies can increase uncertainty. Although our sample size (n = 73) is relatively good for a feasibility/pilot study,265 we note the wide 95% CI around our estimate for reduced relapse. Sim263 has recommended that effect size estimates for main trials be based on clinical judgement rather than on statistics. Our RR is comparable with meta-analytic evidence for early signs monitoring for relapse prevention71 (RR 0.53, 95% CI 0.36 to 0.79), family interventions for schizophrenia46 (RR 0.55, 95% CI 0.48 to 0.62: 12 months) and antipsychotic drugs compared with placebo over 12 months229 (RR 0.40, 95% CI 0.33 to 0.49: 12 months). Recently, Johnson et al. 123 has shown that peer-supported self-management reduced readmissions to acute care (odds ratio 0.66, 95% CI 0.43 to 0.99).

In a future main trial we envisage randomising between 300 and 500 participants. With a trial of this size, we would have 90% power to detect a range of important but potentially more realistic effects across a range of outcomes. In terms of our candidate primary outcome, if we assume a relapse at 1 year of 50% in the control group, we would have 90% power to detect a RR of 0.7 (i.e. 35% in the EMPOWER arm) and if we make allowance for a 20% dropout rate with a sample size of 500. For continuous outcomes we would have 90% power to detect effect sizes in the range of 0.3–0.4 SDs, or 80% power for 0.25 SD, which are worthwhile differences and are conservative in comparison with published data,46,71,229 reflecting high-quality findings on self-management interventions for schizophrenia. 123 Our choice of a RR of 0.7 also takes into account our analysis of the impact of the seven withdrawals in TAU and allows for the scenario in which five out of these seven withdrawals experienced a relapse. This is a conservative estimate given that if we allowed for the number of relapses in the withdrawal group to resemble our findings in TAU this number would be set at 3.

Our finding that fear of relapse was reduced more in the EMPOWER arm than in the TAU arm is consistent with our earlier empirical work in relation to fear of recurrence. 17 In that study we found that, controlling for baseline psychiatric symptoms and EWS, fear of relapse was the only predictor of time to relapse, in that greater fear of relapse was linked to shorter time to relapse. We also found that fear of relapse increased in the weeks preceding relapse and had the same sensitivity and specificity to relapse prediction as traditional measures of EWS. 17 In our theory, fear of relapse drives avoidance of help-seeking and potentially unhelpful coping strategies, thereby increasing vulnerability to relapse (see Figure 1). The finding that fear of relapse was reduced in those in the EMPOWER arm signals that this variable is likely to be an important co-primary outcome of a main trial.

Secondary and mechanism outcomes

Across our candidate secondary outcomes the rates of data completion among service user participants ranged from 73% to 85% (median 80%), demonstrating that collecting these outcomes is feasible. Again, although the impact of the small numbers of clusters was deemed not to be of major concern, as this was a feasibility study all of the results require cautionary interpretation. The profile of clinical and emotional distress outcomes was generally in favour of the EMPOWER arm, with some notable signals in relation to negative symptoms at 12 months (Cohen’s d = –0.57), self-rated depression (Cohen’s d = –0.41) and PBIQ-R control over illness (Cohen’s d = –0.42). We also noted that medication adherence was lower in the EMPOWER arm at 12 months (Cohen’s d = –0.50), signalling that the lower rate of relapse observed in the study may not be due to increased medication adherence. Although we would not rely on these outcome signals to infer efficacy, the measures provide important data that are relevant to the intervention and are feasible to collect in a main trial. We did note that rates of substance misuse were low and similar to those reported by Bucci et al. 170 We were successful in achieving good rates of follow-up using the Timeline Followback measure (85%) but, given the time-consuming nature of this measure, the low rates of substance misuse in the sample and the absence of any signals, we will carefully consider its inclusion in a main trial.

Across our candidate mechanism outcomes, rates of data completion were 85%. The profile of outcomes were generally in favour of the EMPOWER arm. We observed notable signals in relation to personal recovery at 12 months (Cohen’s d = 0.33) and general self-efficacy at 12 months (Cohen’s d = 0.33). These data indicate that these measures not only are feasible to collect but may provide important signals in a main trial in terms of mechanisms of action. In a future main trial we would also include fear of relapse as a mechanism of change, as discussed above.

As has been already noted, a minority of service users had a nominated carer and, overall, a small number of carers participated in the study. At 12-month follow-up, data were available for 47% of carer participants. Twelve-month data on care co-ordinators’ ratings of service user engagement with services were available for 42% of service user participants. During the trial we observed high rates of staff turnover and absence and changes to care co-ordinators. We also observed that care co-ordinators would often be unavailable when we were arranging appointments to collect data from them. In Chapter 3, care co-ordinators raised a number of important service-related factors including high caseloads and time pressures that had an impact on their ability to spend time with service users. These observations regarding the small number of nominated carers and the high attrition of carers and care co-ordinators have important implications for a main trial.

In terms of carers, this group expressed clear needs in relation to wanting to support family members with a diagnosis of schizophrenia and experiencing both satisfaction and stress in relation to their caring responsibilities. The EMPOWER intervention did not address the needs of carers directly, only through providing a resource to service users that they could choose (or not) to share with carers or friends and family. A recent systematic review of digital interventions for people with psychosis or bipolar disorder115 found that none of the 26 included studies targeted family or friends whereas the vast majority of digital interventions utilised support from staff or peers in delivery. There is clearly a need to develop interventions that focus on the needs of carers, families and friends. There are a number of emerging studies showing that online interventions that specifically target needs of carers are feasible, and future research aimed at involving relatives and friends should target their needs specifically. 266–268

In terms of staff, we have noted (see Chapters 1 and 3) the demands of routine practice, constraints of high caseloads and a focus on risk as constraints on joint crisis care plans and shared decision-making approaches. 84,85 In phase 1 of this trial (see Chapter 3), staff reported that the proposed EMPOWER intervention made sense but that if the app increased workload then they would not have the capacity to work with it. Based on our findings, and to scale up the intervention for a main trial, triage of EWS could be better placed within CMHS, rather than within the research team, thereby facilitating team-wide implementation in local services and better accommodating staff turnover and absences.

Strengths

The trial had a number of important strengths. We established a priori our theory of EWS and relapse; we clearly defined our outcome variable of relapse and established the feasibility of utilising routine clinical data to inform classification of relapse; and we demonstrated reliability in the blind rating of these data to classify relapse events. Linked to our theory of EWS, we established clear mechanisms of relapse detection and prevention, including fear of relapse, personal recovery and self-efficacy. Across these outcomes we demonstrated signals to suggest that improved outcomes were associated with the EMPOWER arm compared to TAU.

Our study also adds to the growing literature on digital technology for people with psychosis. We demonstrated the feasibility, acceptability and usability of a blended intervention combining digital technology with peer support and clinical triage to assess changes in well-being indicative of EWS, which then activated a relapse prevention pathway to local CMHS.

We delivered a feasibility trial of a digital intervention across two distinct health-care systems, in Scotland and Australia, demonstrating the potential scalability of the EMPOWER intervention across health-care systems internationally.

We overcame important limitations in relation to selection bias in cluster randomised trials by identifying, approaching, consenting and assessing participants prior to revealing allocation. Although this led to some participants dropping out prior to allocation, we learned important lessons for retaining participants in future studies.

This was the first digital technology mental health trial to be regulated by the MHRA. The trial developed significant strengths in the monitoring and detection of adverse events in digital technology trials. In addition, these adverse events have been important in helping shape the future development of the intervention.

Limitations

The study had a number of important limitations. This was a feasibility trial and the outcome signals detected during the study cannot be taken as evidence of efficacy or effectiveness. We require a larger-scale definitive trial to determine effectiveness.

There appeared to be a differential rate of attrition from the EMPOWER arm of the study. This appeared to be related to the initial burden and effects of the intervention in terms of the daily monitoring. In addition, installation of the app could have been improved. More time could also have been spent at the outset supporting engagement with peer support, exploring the benefits and difficulties of self-monitoring and the model underpinning the intervention and linking this to the valued goals of participants. This might have reduced the rate of withdrawal or, at the very least, facilitated planned withdrawal and improved opportunities for participants to consider remaining in the study to complete follow-up assessments or to provide explicit consent for their routine data to be collected from case notes to allow relapse assessment. In retrospect, we do not feel that our participant information sheet was clearly worded in a way that made continued collection of these data possible following withdrawal.

The psychometric properties of the EMPOWER questionnaire require investigation, particularly the sensitivity and specificity to relapse. Our finding was that ChIPs were not specific to relapse, and while this was helpful to enable blending of the monitoring with peer support, we cannot use the measure to reliably predict relapse. In addition, the questionnaire would benefit from further patient and public involvement to help shape the content and range of questions in order to improve its salience for service users.

A further limitation of the study is that we did not measure participants’ engagement with self-management interventions. Although we have an ongoing process evaluation, which included in-depth interviews with a purposive sample of just over 40% of participants in the EMPOWER arm, the lack of a quantitative measure of self-management limits how confidently we can understand how the EMPOWER intervention might work. Such a measure would be valuable to include in a main trial.