Digitally supported CBT to reduce paranoia and improve reasoning for people with schizophrenia-spectrum psychosis: the SlowMo RCT

Article history

The research reported in this issue of the journal was funded by the EME programme as project number 15/48/21. The contractual start date was in February 2017. The final report began editorial review in June 2020 and was accepted for publication in November 2020. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The EME editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© 2021 Garety et al. This work was produced by Garety et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaption in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2021 Garety et al.

Introduction

Paranoia, or fear of deliberate harm from others, is one of the most common symptoms of schizophrenia-spectrum disorders, and is associated with significant distress and disruption to the person’s life. 1 This results in increased service use, including inpatient admissions, and high costs to mental health-care providers. Developing effective interventions for paranoia is, thus, a clinical priority. The National Institute for Health and Care Excellence recommends cognitive–behavioural therapy (CBT) for psychosis, including paranoia. 2 Meta-analytical studies of first-generation CBT for psychosis have found small to medium sized beneficial effects for delusions (e.g. Turner et al. 3) and positive symptoms more broadly. 4 However, there remain significant challenges to access, engagement, adherence and effectiveness. 1,5

We approached developing the SlowMo therapy in two main ways: first, by adopting an interventionist causal approach6 to increase CBT for psychosis effectiveness and, second, by incorporating inclusive, human-centred design methods to improve the user experience of therapy and to enhance engagement and adherence. 7 The interventionist causal approach involves identifying mechanisms playing a causal role in paranoia (e.g. reasoning, worry and sleep dysfunction) and then developing tailored interventions. 8 SlowMo therapy focuses on reasoning processes that are robustly associated with paranoia: the highly replicated jumping to conclusions (JTC) bias (forming rapid judgements based on a small amount of information) and belief inflexibility bias (the metacognitive capacity for reflecting on one’s beliefs, changing them in the light of reflection and evidence, and generating and considering alternatives). 8,9 Meta-analyses have established that JTC is associated with psychosis, with some specificity for delusions. It also increases the risk of psychosis and predicts outcome in response to treatment. 10–12 Meta-analytic evidence supports an association between belief flexibility and delusions. 13 These thinking patterns are common: in 1800 patients with psychosis in NHS services, difficulties in slow, analytic thinking were present in 60.3% of patients with severe paranoia. 14 In summary, there is converging evidence for a causal role of JTC and belief flexibility in delusion development and maintenance, thereby providing a target for both prevention and treatment strategies. 15

Systematic attempts to ameliorate reasoning in people with psychosis to reduce positive symptoms have included group-based metacognitive training (MCT),16 with a focus on JTC as a causal reasoning bias. One meta-analysis of nine studies of MCT yielded small, non-significant, effects on reasoning, positive symptoms and delusions. 17 The most comprehensive meta-analysis included 15 studies and reported positive effects for delusion change. However, when restricted to studies at low risk of bias, the meta-analysis found small effect sizes for positive symptoms [g = −0.28, 95% confidence interval (CI) −0.50 to −0.06] and non-significant effects for delusions (g = −0.18, 95% CI −0.43 to 0.06). 18

Given the strong theoretical rationale and the mixed empirical findings, we built on this MCT research and on CBT for psychosis approaches for paranoia19 to develop a new cognitive–behavioural intervention. This intervention aims to enhance the impact on paranoia and reasoning, specifically by helping people to build awareness of their tendency to jump to conclusions and by intensively targeting belief flexibility. We have also used inclusive, human-centred design to address the needs of those for whom the content and process of standard therapy present barriers to engagement. The therapy redesign focused on improving its ease of use, enjoyment and perceived usefulness (known as the user experience) to support engagement and adherence for the widest possible range of people. 20,21 The SlowMo intervention is the first blended digital psychological therapy for paranoia [i.e. a face-to-face therapy supported by digital technology, for both in-session content and a tailored mobile telephone application (hereafter referred to as ‘mobile app’)]; it is the end point of a decade of development. 7,22–25 In proof-of-concept and feasibility studies, we found changes in reasoning and improvements in paranoia severity. 22–24 In an experimental study of 100 people with delusions, we compared a brief reasoning-focused digital intervention with an active control condition and found preliminary evidence that changes in belief flexibility mediated improvements in paranoia. 25 Further work validated the user experience of the digital interface for a diverse user population. 7 Over time, as we developed the intervention, we focused more on enhancing belief flexibility than on reducing JTC. We also adopted the term ‘fast and slow thinking’ to communicate the underlying reasoning concepts. 7,15,26 SlowMo therapy aims to assist people with paranoia by supporting them to identify their upsetting concerns and fast-thinking habits, then providing them with strategies to slow down for a moment to focus on new alternative information and develop ways of feeling safer. It aims to improve the appeal, ease of use and clinical effectiveness of CBT for paranoia. 7,27

The current study tested the SlowMo therapy in a fully powered, methodically rigorous, multisite randomised controlled trial (RCT). We aimed to test its clinical efficacy in reducing paranoia, to examine its acceptability, to determine the mechanisms through which it works and to identify participant characteristics that might moderate its effectiveness. We selected treatment as usual (TAU) as the comparator condition. This was because there is a low penetration of evidence-based psychological treatment in the NHS and, thus, the key efficacy question to address at this stage is whether or not SlowMo therapy confers benefits over and above standard care. We have previously established the superiority of an earlier brief version of the intervention against an active control intervention. We predicted that SlowMo therapy would improve paranoia and reasoning and might improve a number of other outcomes, such as self-concept, worry, quality of life and well-being.

An important secondary goal was to evaluate mechanisms of action; the trial hypotheses concern reasoning and are best tested where the control condition is inactive with respect to the targeted psychological processes. We hypothesised that SlowMo therapy would improve reasoning and that the primary mechanism for its treatment effects on paranoia would be mediated through reasoning, specifically belief flexibility and JTC. We also examined worry as a potential mediator, as it has previously been shown to mediate change in paranoia. 28 However, given that worry was not directly targeted by the treatment, we hypothesised that worry would not mediate the treatment effects of SlowMo on paranoia. The treatment effects of a previous version of our intervention were moderated adversely by negative symptoms and working memory. 25 We included hypotheses concerning these potential moderators to examine whether this study replicated these effects or if the redesign worked as intended and rendered the intervention equally accessible and effective across a wide range of users with different cognitive capacities and symptom profiles.

The research questions examined were as follows:

1. Is SlowMo efficacious in reducing paranoia severity over 24 weeks when added to TAU, compared with TAU alone?

2. Does SlowMo lead to changes in the following outcomes: reasoning, well-being, quality of life, self-schemas and others schemas, service use and worry?

3. Does SlowMo reduce paranoia severity by improving fast thinking (reducing belief inflexibility and JTC)?

4. Do participant characteristics (i.e. their cognitive capacities, specifically working memory and thinking habits; and their symptoms, specifically negative symptoms) moderate the effects of the intervention?

5. Does outcome differ by adherence to the intervention?

6. Is SlowMo therapy acceptable, as assessed by therapy uptake and adherence?

Methods

Statistical analysis

We powered the study to detect a clinically meaningful 10-point reduction in the GPTS total score with a standard deviation (SD) of 25 (effect size = 0.4). We accounted for the partial nested design because of clustering in the SlowMo group, assuming an ICC of 0.01 with 10 therapists, and for no clustering in the TAU group using the clsampsi command in Stata^® version 16 (StataCorp LP, College Station, TX, USA). 52 With the 1 : 1 treatment allocation and the 0.05 significance level, a simple two-tailed t-test with 150 people per group had 90% power to detect an effect size of 0.4 and 80% power to detect an effect size of 0.35. To allow for a conservatively high 20% attrition, we aimed to recruit 360 patients at baseline split equally across three sites (120 per site, 60 per group per site).

We report data in line with the Consolidated Standards of Reporting Trials (CONSORT) 2018 Statement for Social and Psychological Interventions,53 showing attrition rates and loss to follow-up. All analyses are performed using the intention-to-treat population, incorporating data from all participants, including those who do not complete therapy. The statistical analysis plan was agreed with an independent DMEC before any inspection of post-randomisation data by the research team. Statisticians became unblinded following all data collection only, and the statistical analysis was performed unblinded owing to the need to account for therapist effects in the SlowMo group. No interim analysis was performed. All analysis was conducted in Stata version 16.0.

Descriptive statistics by randomised group are presented for baseline values, with no tests of statistical significance or CIs for differences between groups.

To test the primary hypothesis that the intervention would reduce paranoia severity over 12–24 weeks, we fitted a linear mixed model that allowed for clustering by both participants and therapists to the repeated measures of GPTS. 31 The model included the following as fixed effects: randomised group, time (coded as categorical), time by randomised group interaction, treatment site, baseline paranoia severity and the corresponding baseline assessment for the outcome under investigation. The treatment effect (adjusted between-group mean difference) was estimated from the model for each time point separately. All secondary outcome measures were analysed using the same modelling approach: linear mixed models for continuous outcomes and logistic mixed models for binary outcomes and reporting conditional odds ratios. Cohen’s d effect sizes at 12 and 24 weeks were calculated as the adjusted mean difference of the outcome divided by the sample SD of the outcome at baseline. These are displayed in a forest plot for all continuous outcomes and as odds ratios for binary outcomes (see Figures 2 and 3).

The moderation analyses investigated whether or not the effect of the SlowMo intervention on GPTS,31 R-GPTS37 and PSYRATS34 was moderated by the following:

• the baseline measure of the outcomes

• reasoning – belief flexibility (possibility of being mistaken42 and alternative explanations43) and JTC (beads task9 85 : 15: more than two beads drawn, yes/no)

• negative symptoms (BNSS45)

• BAPQ46

• working memory (Letter–Number sequencing test47)

• set-shifting [trail-making task48 (B–A)]

• presence of a carer (yes/no)

• perceived criticism of carer,49 if present.

For a continuous moderator, the difference in treatment effect between unit levels of the moderator can be interpreted as the difference in the estimated treatment effect between a participant with a moderator value at baseline of α + 1 and a participant with a moderator value at baseline of α. For a binary moderator (e.g. presence of a carer), the difference in treatment effect can be interpreted as the difference in the estimated treatment effect between participants with a carer and participants without a carer.

The mediation analyses examined the potential mechanisms underlying the effect of SlowMo plus TAU compared with TAU on clinical paranoia outcomes, GPTS,31 R-GPTS37 and PSYRATS. 34 Jumping to conclusions,9 belief flexibility (possibility of being mistaken42 and alternative explanations43) and worry41 at 12 weeks were individually considered as mediators of the effect on the outcomes at 12 weeks and 24 weeks separately. The analysis used causal mediation analysis based on parametric regression models. 54 This involved estimating a linear model for each mediator with random assignment, baseline outcome, baseline mediator, site and paranoia cut-off point at baseline as covariates, and separately estimating a linear model for each outcome with the mediator, group assignment, baseline outcome, baseline mediator, site and paranoid cut-off point as covariates. The effect of group assignment on the mediator is multiplied by the effect of the mediator on outcome to estimate the indirect effect, and the effect of SlowMo on outcome in the model including mediator is an estimate of the direct effect. The indirect and direct effects sum to the total effect, and bootstrapping with 500 replications was used to obtain valid standard errors (SEs) for the causal effects; 95% CIs are based on the percentile of the bootstrap distribution. The proportion mediated is the indirect effect divided by the total effect.

To account for departures from random allocation in the SlowMo group who received therapy, we performed two compliance-adjusted analyses for a binary compliance measure (attending at least one session of SlowMo therapy) and a continuous measure of the number of sessions received. Both analyses used a two-stage instrumental variables approach. The first stage involved regressing the treatment receipt measure on randomisation, baseline GPTS,31 site and paranoia, and saving the predicted value of the treatment receipt measure. In the second stage, this predicted value was included in the analysis models in place of the randomisation variable. Both models were estimated in a single bootstrap procedure to produce valid SEs for the effect of treatment received, with resampling at the participant level.

The first measure of compliance indicates anyone who receives at least one session of therapy. The treatment effect is interpreted as the complier-average causal effect, where complier is defined as those participants randomised to the SlowMo group who received at least one session of therapy and those participants randomised to the TAU group who would have received at least one session of therapy had they been randomised to the SlowMo group (a counterfactual, based on predictions from a model). The treatment effect is the adjusted mean difference between randomised groups within this subgroup of compliers.

The second measure of compliance is the number of sessions of therapy attended. This is observed for all participants in the SlowMo group (ranging from no sessions to eight sessions) and is fixed by design at zero in the TAU group. The treatment effect is the effect of one additional session of therapy on the outcome, assuming a linear effect, for example going from s sessions to s + 1 sessions for any s between 0 and 8. Details of the statistical approach for mediation analysis and departures from random allocation are outlined in Dunn et al. 55

Missing data on individual measures were prorated if > 90% of the items were completed; otherwise the measure was considered missing. We checked for differential predictors of missing outcomes by comparing responders to non-responders on key baseline variables. Maximum likelihood estimation in the mixed models accounts for missing outcome data under a missing-at-random assumption, conditional on the covariates included in the model.

The numbers of serious adverse events (SAEs) and adverse events are presented as the number of events and number of individuals with events for each randomised group.

Results

From 1 May 2017 to 14 May 2019, we assessed 604 people for eligibility and, of these, recruited 362 participants. In total, 181 participants were allocated to the SlowMo group and 181 participants were allocated to the treatment as usual group (Figure 1). There was one post-randomisation withdrawal: a participant in the control group who withdrew fully from the trial and requested that no data were included. The final sample was, therefore, 361 participants (Table 2 and see Figure 1).

FIGURE 1.

Trial profile and participant flow diagram (CONSORT).

Data were available for over 90% of the sample at each follow-up point (12 weeks: n = 328, 91%; 24 weeks: n = 333, 92%) (see Table 2). The 12-week assessments were conducted at a mean of 13.5 weeks (range 8.6–19.6 weeks) and the 24-week assessments were conducted at a mean of 25.2 weeks (range 12.9–38.3 weeks). Unmasking without replacement of an assessor occurred in 22 participants (6.7%) at 12 weeks and 19 participants (5.7%) at 24 weeks; however, in some instances unblinding occurred after a number of measures, including the primary outcome measure, had been collected: at 12 weeks, only 12 (3.6%) and, at 24 weeks, only 11 (3.3%) of the primary outcome GPTS data were collected unmasked (Table 3).

The baseline characteristics of the participants are shown in Table 4. As is typical of samples of people with persisting psychosis, there was a preponderance of male participants (70%) and there was an average age of 42.6 years. Other clinical characteristics (diagnosis, years in contact with services and medication equivalent dosages) were also typical. Eighty per cent of participants had clinical diagnoses of schizophrenia or schizoaffective disorder. Approximately 70% of the total participant sample were of white ethnicity, 58% lived alone and about 80% were unemployed. There were no marked demographic, diagnostic or clinical differences between the groups. Their paranoia was severe: 94.4% and 93.9% of the TAU and the SlowMo groups, respectively, met the criteria37 for likely presence of persecutory delusions on the GPTS31 (see Appendix 1, Table 17). Table 4 also shows the stratification factors (paranoia severity and site) and Table 5 shows the baseline values of negative symptoms, cognitive measures and carer characteristics examined as putative moderators.

In terms of treatment uptake, the average number of SlowMo sessions attended was 6.8 sessions (SD 2.6 sessions), rising to 7.3 sessions (SD 1.9 sessions) for those who attended at least one session. In total, 145 (80%) of those participants randomised to SlowMo (n = 181) completed all eight therapy sessions, 13 (7%) did not attend any sessions and a further 23 (14%) discontinued between session 1 and session 7. The mean session duration was 75 minutes (SD 29 minutes), including web app delivery and out-of-clinic work. Adherence to the delivery of the web app content was high, with each of the eight sessions reaching adherence ratings of at least 90%. Adherence to the mobile app was also high: of those who attended at least one therapy session and, therefore, were given a mobile telephone with the app installed, 71.4% met criteria for adherent use.

Both the descriptive statistics for all of the primary and secondary outcomes (Tables 6 and 7), and the effect estimates (Figures 2 and 3) show that SlowMo led to greater reductions in paranoia than TAU, as assessed by the total score of our primary outcome measure (GPTS31) post treatment [–8.06 (SE 2.85), 95% CI –13.64 to –2.48; p = 0.005] and at 24 weeks [–5.27 (SE 2.84, 95% CI –10.83 to 0.29; p = 0.063]. The reduction in paranoia on GPTS31 part B (persecutory) was significant at both time points, as was the R-GPTS37 total, PSYRATS34 delusions and SAPS36 persecutory delusion (see Figures 2 and 3). However, effects were less consistent for ideas of reference (as measured by GPTS31 part A, R-GPTS37 social reference and SAPS36 ideas/delusions of reference), with significant effects found either post treatment or at follow-up, but not at both time points (see Figure 2). See Appendix 1, Figure 9, for outcomes by group at three time points for the GPTS, R-GPTS and PSYRATS. Effect sizes were small (approximately Cohen’s d = 0.3) on the GPTS31 total, and moderate (Cohen’s d = 0.5) for PSYRATS34 total and SAPS36 persecutory delusions.

Treatment effects were found for some but not all of our reasoning measures: of the measures of belief flexibility, the possibility of being mistaken,42 both as a continuous rating and as a dichotomous rating, improved significantly, but alternative explanations43 did not improve. Jumping to conclusions9 showed little evidence of improvement (with only one significant finding: beads drawn at 12 weeks). The fast scale of the Fast and Slow Thinking Questionnaire44 showed improvements at both time points and the slow scale showed improvements at 24 weeks. Significant improvements, with a small effect size of approximately Cohen’s d = 0.3, were found for SlowMo in the following secondary outcome measures at either or both time points, and most consistently at 24-week follow-up: well-being (WEMWBS38), quality of life (MANSA39), worry (PSWQ41) and self-concept but not other-concept as measured on the BCSS40 (see Table 7 and Figure 2).

FIGURE 2.

Forest plot of all continuous outcomes at (a) 12 weeks and (b) 24 weeks.

FIGURE 3.

Forest plot of binary outcomes at (a) 12 weeks and (b) 24 weeks.

The moderation analysis (Table 8) found no differential effects on the paranoia outcome as measured by the GPTS31 or the R-GPTS. 37 There was only one moderation effect, on the PSYRATS,34 with p < 0.05. However, given the number of tests conducted, this finding may have occurred by chance. These results show that the treatment is effective against paranoia at all levels of the moderators.

The results of the mediation analysis on the GPTS31 at 12 and 24 weeks are shown in Table 9. The results for the R-GPTS37 and the PSYRATS34 are presented in Appendix 1 (see Tables 18 and 19). Only the possibility of being mistaken42 (whether measured as a binary variable or as a continuous measure) and worry41 mediated the effects of the treatment on all paranoia outcomes at 12 and 24 weeks. Approximately 40% of the total effect was mediated through each mediator at 12 weeks and 56% was mediated at 24 weeks.

The results of the compliance-adjusted analysis on the GPTS,31 R-GPTS37 and PSYRATS34 are shown in Table 10. Given that there is no access to SlowMo therapy for the TAU group, the complier-average causal effect is an adjustment to the intention-to-treat effect for each outcome divided by the predicted proportion of those in the SlowMo group who were observed to attend at least one session of therapy. The results show significant treatment effects of SlowMo therapy compared with TAU in the compliers at all time points. The dose–response effect shows that the treatment effect increases as the number of SlowMo sessions increases.

Fifty-four adverse events were reported over the course of the trial, of which 51 were serious events, occurring in 19 people in the SlowMo group and 21 in the TAU group (Table 11). There were no deaths recorded. More than half of the SAEs were mental health hospital admission or crisis referrals (SlowMo, n = 13; TAU, n = 16) or physical health crises (SlowMo, n = 8; TAU, n = 2), none of which was rated as being related to participation in the trial. One SAE in the TAU group was rated as ‘definitely related’ to trial involvement: it involved a complaint made when the research team shared information with the clinical team under a duty of care (confirmed as such by an independent ethics review). (The participant subsequently requested that their data be withdrawn and is, therefore, a ‘post-randomisation exclusion’ in the data analysis.)

Concomitant treatments (psychosocial, psychological therapy and medications) and services (days in crisis care and in admission to hospital) that were provided to both groups as usual treatment were monitored from case note review and using a modified version of the Client Service Receipt Inventory33 (see Appendix 1, Tables 20 and 21).

Discussion

To the best of our knowledge, the SlowMo trial is the largest trial to date of a psychological therapy focused on fear of harm from others (paranoia) undertaken in a clinical population. We recruited the target of 362 participants, with over 90% followed up at each time point. We addressed two main goals: to improve effectiveness and to overcome user experience barriers to engagement and adherence. The study results suggest that SlowMo therapy is clinically effective and supports the intended user experience.

We found consistent significant effects of SlowMo therapy on paranoia when added to routine treatment, compared with routine treatment alone, over and above the generally improving trajectory of both groups. Improvements were demonstrated on all paranoia and persecutory delusions outcomes across the 6 months (ranging from small to medium effect sizes). The effects were less consistent for ideas of reference, with significant effects at either post treatment or follow-up, but not both time points. There were also improvements in the reasoning targets of belief flexibility (possibility of being mistaken,42 but not alternative explanations43) and self-reported reasoning on the Fast and Slow Thinking Questionnaire44 (the fast subscale at both time points, and the slow subscale at 24 weeks). There was, however, little evidence of improvement in JTC9 (a single significant effect of the eight measured). There were treatment effects with small effect sizes on improving well-being, quality of life, self-concept and worry, with these important gains seen most consistently at the longer-term (6-month) follow-up. These represent sustained improvements in well-being and quality of life, and are highly valued by service users. 56

The therapy uptake and adherence were all in the upper range, suggesting that SlowMo therapy is highly acceptable. Furthermore, the effects were not moderated by hypothesised characteristics, indicating that it is beneficial regardless of cognitive capacity, baseline symptoms, the presence of a carer, or carer criticism. The treatment targeted reasoning to improve paranoia; we found that this outcome was, as hypothesised, mediated by improvement in a fundamental aspect of reasoning, belief flexibility (possibility of being mistaken42) and also, but not as hypothesised, reducing worry.

We found no evidence of the intervention being harmful. Both groups generally improved across the course of the trial and there were similar numbers of SAEs across the two groups. The most common SAEs of hospital crisis and inpatient admission were smaller in number of days at follow-up in both groups (see Appendix 1, Table 21). No SAE was considered definitely related to trial participation in the SlowMo group by the independent DMEC; the only SAE definitely related to trial involvement in the TAU group involved a complaint that information was shared under duty of care (see above). The relatedness of adverse events to the digital hardware and software supporting SlowMo therapy is reported in Chapter 3.

We used a range of measures of paranoia and a limitation of the study is that our primary outcome measure, the self-reported GPTS,31 was revised during the course of the trial. 37 Therefore, we report outcomes using both versions. These have similar results, but the newer, more psychometrically robust, revision37 yields slightly larger treatment effects and results in the follow-up effect for the total score moving into the range for conventional significance. SlowMo, although brief, at eight sessions, had a clinically worthwhile effect on delusions: using the new cut-off points provided by Freeman et al. 37 for the GPTS persecution, 38.5% of the SlowMo group (compared with 31.6% of the TAU group) no longer met the criteria for presence of a persecutory delusion at follow-up (increasing from 10% and 11%, respectively, at baseline) (see Appendix 1, Table 17). Furthermore, on observer-rated measures of paranoia (PSYRATS34 and SAPS36), the effect sizes were, in general, moderate and, for this reason, were greater than the rates reported in meta-analyses of longer courses of CBT for psychosis for delusions using these measures. 3,57 The clinically important target of distress associated with the delusion was reduced by the end of treatment with SlowMo compared with TAU, and sustained at follow-up.

SlowMo targets reasoning to improve paranoia. As intended, improvements were observed in belief flexibility and the possibility of being mistaken. 42 By contrast, JTC9 showed little evidence of change. On the tests of reasoning as a mechanism, the possibility of being mistaken mediated change in paranoia, explaining about 56% of the variance at follow-up. This is consistent with our earlier proof-of-concept study, which also found that the possibility of being mistaken mediated paranoia, but JTC did not mediate paranoia. 25 In the light of this and other meta-analytic evidence,10 we suggest that JTC (as assessed through the classic beads task9) may be best considered as trait like and relatively unresponsive to change over time, conferring vulnerability to persecutory beliefs, but with less evidence of a dynamic relationship with paranoia severity. This evolving understanding of reasoning biases and paranoia has resulted in our foregrounding of the promotion of ‘slow thinking’ and greater flexibility, with the aim of generating compensatory strategies for real-world fast thinking by encouraging the deliberative act of slowing down in the moment (see Ward and Garety15). Self-reported reductions in fast and slow thinking, as assessed by the Fast and Slow Thinking Questionnaire,44 suggest that the therapy may have changed awareness of and preferences for both unhelpful fast styles of thinking and useful slow styles of thinking, in keeping with the explicit and consistent SlowMo focus on building meta-cognitive awareness of thinking ‘habits’.

As Freeman58 has noted, persecutory delusions arise from a combination of causes, with each causal factor increasing the probability of such fears occurring. Another factor is worry. The Worry Intervention Trial28 demonstrated comparable changes in paranoia on PSYRATS delusions (Cohen’s d = 0.49) using a brief, six-session, cognitive–behavioural worry intervention and provided evidence that these changes were mediated through worry. 28 SlowMo showed a similar change in paranoia on the PSYRATS (Cohen’s d = 0.5) and, although not explicitly targeted in SlowMo, the observed changes in worry were also found to act as a mechanism of change in paranoia; paranoia improved as a consequence of reducing worry, with a similar proportion of the variance to that found for belief flexibility. This was not hypothesised. However, given that worry is a mechanism that clearly drives paranoia, which is described by participants as part of an emotional reaction to fast thinking, and SlowMo altered worry, we can conclude that worry reduction constitutes part of the treatment route for SlowMo.

We note that SlowMo shares features with worry reduction techniques. Both involve noticing one’s thoughts, consider approaches for decentring worrying thoughts and help with strategies to shift attention elsewhere, for example, in SlowMo, shifting from fast thoughts to alternative and safer (i.e. less worrying) thoughts. However, worry improved less in SlowMo than when it was directly targeted in the Worry Intervention Trial,28 as might be expected. Furthermore, we cannot tell from the current study how far worry and belief flexibility are independent routes to change, nor whether or not there might be other mechanisms for treatment effects, such as the parallel improvements in self-concepts and well-being that occurred.

Our original hypotheses, specified in our trial protocol, derived from a theory of change in which the primary process underpinning SlowMo was through reasoning. However, the evidence from this study suggests that there is the potential for other processes to also be involved in treatment effects. Indeed, our long-standing cognitive model of paranoia has proposed multifactorial causality, particularly highlighting both reasoning and emotional processes,59 and these findings are consistent with a multifactorial theory of change; future research should investigate the mechanisms to elucidate both the treatment effects of SlowMo and the causal mechanisms of paranoia. We intend to pursue these questions in future studies.

A limitation of the study is that the design did not control for the effects of time with a therapist. The choice of a TAU control was made because we wanted to test whether or not SlowMo confers benefits over and above standard care. In addition, we aimed to examine the mechanism of change and wished, therefore, to have a control condition which was, as far as possible, inert with respect to reasoning. Adjunctive treatments in TAU were closely monitored and were found to be similar across the groups: a few participants were given individual psychological therapy (albeit, as would be expected, slightly more in the TAU group). The types, dosage and frequency of antipsychotic and other medication were similar in each group.

Another limitation of the study is that when recruiting participants we focused on persecutory beliefs in the context of a clinical diagnosis of schizophrenia-spectrum psychosis and did not make an independent research diagnosis. The resulting clinical and demographic profile of participants appears to have been typical of a community sample of people with psychosis and long-standing delusions. The relatively high consent rate and low attrition rates indicate that our findings should be generalisable to this population; however, a fuller diagnostic and symptomatic assessment might have been more informative in terms of the generalisability of the findings to diagnostically selected participants.

Finally, we found that the effects of SlowMo are more consistent on persecutory delusions than ideas of reference. This was unanticipated. It may be that the persecutory beliefs, in improving, shifted down the hierarchy of paranoia60 to milder ideas of reference, but that therapy prevented such ideas and their experiential components from being actively elaborated into paranoid fears of intentional harm. Whether or not this was the case, we infer, however, that SlowMo should be developed to enhance work on referential ideas.

A central focus of developing SlowMo was to enhance the user experience of therapy to address implementation barriers. We used inclusive, human-centred design to create, to the best of our knowledge, the first blended digital therapy for paranoia, which sought to accommodate a diversity of needs. The impact of this on the service user’s experience is explored further in Chapter 3. We also worked with user research interviewers in a coproduction research model; this enabled us to evaluate the experience of participation in the trial and of therapy qualitatively, as reported in Chapter 4. We found that treatment effects were not moderated by baseline clinical or cognitive characteristics, or the presence of a carer. Thus, the SlowMo therapy design appears to be effective irrespective of these variations, something of crucial significance in relation to real-world implementation. Consistent with these findings, there was excellent uptake of face-to-face sessions and the mobile app. This suggests that the design achieved its aims of being trustworthy, enjoyable, memorable and easy to use. 7

A flexible approach to formulation aimed to ensure that the understanding of targeted processes (fast and slow thinking) was individualised and accessible. Out-of-clinic behavioural work modelled app use and promoted therapy generalisation. The mobile app enabled further personalisation of the therapy, using bubbles as visual metaphors to support learning, with step-by-step support to slow down in the moment, and immediate access to safer thoughts (that might otherwise be hindered by memory difficulties and threat-related arousal). The findings suggest that targeted CBT for psychosis, incorporating design and technology to improve people’s therapy experience and uptake, can facilitate a focus on the processes most likely to result in real-world change.

Conclusions

The SlowMo trial has demonstrated clinically worthwhile results, with consistent, sustained positive effects across a range of outcomes. These effects match or exceed those typically observed for standard CBT for psychosis, but were achieved in fewer sessions, which were accompanied by excellent engagement and retention, validating the therapy redesign. The results indicate that the treatment worked, in part, to help people to slow down their thinking, to be more flexible about their beliefs and to worry less, and was not moderated by baseline clinical severity, cognitive problems or the carer relationship. Further understanding of the mechanisms that mediate these improvements would be valuable. The trial results also argue for further implementation studies testing SlowMo’s real-world delivery within clinical pathways for persecutory delusions in a range of clinical settings.

Background

Method

Results

The impact of patient and public involvement

On the research

The positive impacts of PPI on the research study included the production of new recruitment leaflets and attendance at community team meetings to promote recruitment; the collection and analysis of interview data to explore service users’ experience of SlowMo therapy; and the co-production of the Plain English summary. The leaflet was produced by the team as additional patient-facing information to aid recruitment. It was also used by the PPI team as promotional material at team presentations, recovery college meetings and meetings of other such groups where service users were in attendance. For the qualitative interview study, which is reported in full in Chapter 4 of this report, the PPI team co-produced the focus group topic guide and conducted 22 qualitative interviews, all of which were led by either two PPI members or a PPI member supported by a research assistant. The data were analysed in two phases. In the first phase, the Sussex PPI team analysed a transcript collaboratively to produce a coding framework and held a series of meetings to reach consensus on initial themes. In the second phase, feedback was obtained through consultation with the London and Oxford PPI teams, and a further set of meetings led to a consensus on the final themes. In terms of study management, the PPI team co-produced regular summaries of PPI input to the steering committee and funding body. Dissemination outputs to date have included the co-produced Plain English summary, as well as a paper (currently in draft form) on the qualitative substudy of service users’ experiences of SlowMo therapy.

The major benefits of PPI on the trial were that the target recruitment was achieved with support from the PPI team; the qualitative substudy was co-produced and provided rich data concerning service users’ experiences of SlowMo therapy; and the Plain English summary of the results was co-produced: PPI members were provided with key method and results sections and were encouraged to write a lay summary in their own words. This was then drafted, shared and final feedback obtained before completion to produce the Plain English summary. A further impact of PPI input lay in the emergent outcomes that will be described in Wider impacts.

On the individual patient and public involvement members

There was a consistent core PPI group of five members throughout the 3-year study. PPI members worked well together and became more confident in their roles over time. Subjective qualitative feedback (see Boxes 1–3) revealed that PPI members felt that the PPI was well organised. Although PPI members found participation in the project daunting and challenging, as it involved a lot of travel, required them to overcome the hurdle of team presentations and was difficult to think on the spot, they felt well informed, well trained and encouraged, and were given time to understand and contribute such that everyone’s ideas were heard. PPI members reported impacts on their confidence, career aspirations, knowledge, insight and skills to support themselves in their roles. They felt that the work was varied, rewarding and empowering. There were no problems identified despite the difficult topics being discussed and the PPI input was valued, for example by service user participants, who appreciated being interviewed by service users who understood their experiences. The PPI members felt that there was a mutual benefit in helping to improve how ideas were executed and in providing authentic first-hand experience from someone at whom the therapy is aimed. However, not all PPI members provided subjective feedback; feedback was provided by PPI members who were more actively engaged.

Wider impacts

Importantly, in addition to the expected activities and resulting impacts, there were a variety of linked and ‘emergent’ activities and impacts. At a regional level, one of the PPI members produced an open letter reflecting their positive experiences of PPI membership and the importance of both PPI and research alongside front-line NHS work. This letter (see Appendix 2, Box 4) was used by the regional NHS PPI lead to promote and encourage more service users to take up PPI roles.

The PPI team developed the concept of the ‘SlowMo People’ website/page (Figure 4; see http://slowmotherapy.co.uk/slowmopeople/, accessed 26 May 2021) based on the Humans of New York website (www.humansofnewyork.com/). The website aimed to tell individual stories of how fast thinking can trip you up and how slowing down for a moment can be helpful. Drawing on both service users’ and researchers’ stories, the aim was to normalise the fast-thinking style while also presenting the real-life personal experiences of the impact of slowing down.

Finally, the Sussex PPI lead, team and therapist worked with one PPI member, the NHS communications team and a local newspaper to produce an article about the experience of paranoia and voice hearing, and the positive impact of receiving the SlowMo intervention. This was picked up nationally, resulting in a short film that was aired on prime-time television on the BBC’s The One Show in April 2018 (http://slowmotherapy.co.uk/news-2/, accessed 26 May 2021). The BBC’s The One Show has an audience of 5 million people. Feedback through Twitter suggested a major community value in providing a normalising portrayal and hopeful outcome for psychosis and voice hearing, for both general public and the people suffering with these experiences. The short film was also subsequently shown to people at the start of the therapy.

FIGURE 4.

The SlowMo People web page.

Discussion

Background

SlowMo therapy is, to the best of our knowledge, the first digital therapeutic for psychosis developed using inclusive human-centred design. 20,21 The design aimed to support adherence by improving the user experience of a targeted CBT for paranoia for the widest possible range of people. This chapter will describe the digital literacy of the therapy sample; the adherence to the SlowMo mobile app based on self-reported and system analytics; a survey evaluating the enjoyment, usefulness and ease of use of the SlowMo mobile app; and the technical issues related to the SlowMo therapy software and hardware.

Research questions

The research questions are as follows:

1. What is the digital literacy of the therapy sample and is this impacted by service users’ characteristics (i.e. gender, age, ethnicity and paranoia severity)?

2. Does the SlowMo mobile app have acceptable rates of self-reported and system analytics adherence, and are they impacted by service users’ characteristics (i.e. age, gender, ethnicity and paranoia severity)?

3. What are the self-reported rates of usefulness, enjoyment and usability for the SlowMo mobile app, and are they affected by service users’ characteristics (i.e. age, gender, ethnicity and paranoia severity)?

4. How prevalent are technical problems associated with use of the SlowMo web app and mobile app?

5. How prevalent are adverse events associated with use of the SlowMo web app and mobile app?

Methods and measures

Results

Digital literacy

Smartphone ownership and computer access in the SlowMo therapy group among those participants who attended at least one session, together with the frequency of use and confidence, are displayed in Table 13 by site and overall. This indicates that just over three-quarters of the sample owned a smartphone, which was consistent across all sites. For smartphone owners, the frequency of use was comparable in Sussex and Oxford and lower in London. A similar pattern was found for smartphone confidence. Computer access, frequency of use and confidence were the highest in Sussex, followed by Oxford, and then London. The impact of gender, age, ethnicity and paranoia severity on smartphone and computer ownership and on smartphone use and confidence is shown in Figures 5 and 6, respectively, with inferential statistics presented in Appendix 3, Table 22. There were significant age differences in smartphone literacy, with older people being less likely to report ownership and confidence in using a smartphone. Older people and women were also significantly less confident in using computers. Ethnicity had a significant impact on computer access and smartphone and computer confidence, with people from a black ethnic group reporting less access and less confidence than those from white and other ethnic groups. Paranoia severity did not have a significant relationship to digital literacy.

TABLE 13 - Smartphone and computer access, use and confidence in the SlowMo therapy group (N = 168)

n = 158, 93% completion.

n = 153, 91% completion.

Variable	Site			Overall
	Sussex	Oxford	London
Smartphone ownership reported, n (%)a	44 (77.2)	30 (76.9)	48 (77.4)	122 (77.2)
Computer access reported, n (%)b	42 (77.8)	26 (66.7)	34 (56.7)	102 (66.7)
Smartphone use, mean (SD)	63 (37)	61 (38)	57 (35)	60 (36)
Smartphone confidence, mean (SD)	65 (32)	62 (31)	55 (36)	60 (33)
Computer use, mean (SD)	51 (38)	46 (34)	43 (33)	47 (35)
Computer confidence, mean (SD)	63 (32)	57 (26)	50 (32)	56 (31)

FIGURE 5.

Smartphone ownership and computer access among people attending at least one therapy session (n = 168). (a) Smartphone ownership by gender, (b) computer access by gender, (c) smartphone ownership by age, (d) computer access by age, (e) smartphone ownership by ethnicity, (f) computer access by ethnicity, (g) smartphone ownership by paranoia severity and (h) computer access by paranoia severity.

FIGURE 6.

Smartphone and computer use and confidence among people attending at least one therapy session (n = 168). (a) Smartphone and computer use by gender, (b) smartphone and computer confidence by gender, (c) smartphone and computer use by age, (d) smartphone and computer confidence by age, (e) smartphone and computer frequency by ethnicity, (f) smartphone and computer confidence by ethnicity, (g) smartphone and computer use by paranoia severity and (h) smartphone and computer confidence by paranoia severity.

User experience scale for the SlowMo mobile app

The UES findings for each subscale and the total score are presented in Table 15. The UES was not offered to the participants who completed therapy at the beginning of the trial (n = 45). A further three participants were not eligible to complete the UES, as they declined any engagement with the SlowMo mobile app. For the remaining sample, the completion rate was 85%. UES ratings were comparable across all subscales, with the majority of people providing positive ratings for enjoyment, usability and usefulness. However, there was a large range of scores, suggesting that the mobile app was positively received by most but not all participants. Figure 7 shows the UES ratings in relation to gender, age, ethnicity and paranoia severity.

TABLE 15 - User experience scale ratings in therapy completers (N = 82)

UES variable	Site			Total
	Sussex	Oxford	London
Enjoyment (%)
Mean (SD)	72.08 (17.70)	73.32 (20.32)	75.19 (19.04)	73.75 (18.64)
Range	31–98	45–100	35–100	31–100
Usability (%)
Mean (SD)	73.71 (19.73)	74.32 (24.12)	73.86 (22.12)	73.89 (21.42)
Range	33–100	23–100	35–100	23–100
Usefulness (%)
Mean (SD)	76.32 (17.80)	79.10 (14.43)	77.44 (19.11)	77.35 (17.70)
Range	25–100	60–100	20–100	20–100
Total UES (%)
Mean (SD)	74.03 (16.84)	75.58 (17.61)	75.50 (17.44)	74.99 (17.06)
Range	30–99	46–100	42–100	30–100

FIGURE 7.

User experience survey subscales and total scores among therapy completers (n = 82). (a) Gender, (b) age, (c) ethnicity and (d) paranoia severity.

The UES ratings were compared with participant characteristics, as shown in Appendix 3, Table 25. There were significant differences depending on gender, with women reporting higher rates of enjoyment and usefulness; however, rates of usability were similar for male and female participants. The significant differences in smartphone confidence prior to therapy did not appear to affect the self-reported user experience, as there were no significant differences depending on age and ethnicity. There were also no differences in UES ratings in relation to paranoia severity.

Discussion

This chapter evaluated the user experience of the SlowMo mobile app. The data provide a validation of the inclusive, human-centred design of the SlowMo therapy, as excellent rates of self-reported and system analytics mobile app adherence were found. The a priori criterion for mobile app adherence was met by 80.7% of participants who completed all eight sessions and 26.1% of people used the mobile app at least once outside every session. The UES ratings further suggest that most people perceived the mobile app as easy to use, enjoyable and useful. Alongside the high rates of therapy session attendance and therapy fidelity reported in Chapter 1, the results suggest that the SlowMo design did enhance the user experience as intended, to support engagement and adherence. The ‘digital divide’ previously identified in psychosis research and evidenced in our digital literacy data did not appear to affect user experience, as age, ethnicity and paranoia severity did not influence self-reported adherence, system analytics adherence or UES findings. 107,108 The exception was that female participants were significantly more likely than male participants to be adherent to the mobile app and reported higher rates of usefulness and enjoyment, with comparable usability ratings. This is consistent with previous findings that women with psychosis are more likely to engage in digital therapeutics and suggests that development of SlowMo should focus on optimising the interface for men’s needs. 109 Unsurprisingly, people who reported being more confident and frequent users of smartphones prior to starting the therapy were more likely to be adherent to the mobile app. This insight emphasises the importance of digital literacy assessments so that individualised technical support can be provided; we plan to continue improving the SlowMo design to further enhance accessibility for those who are less familiar with technology.

The mobile app adherence rates were high, especially as mobile app use was encouraged only if it was in line with the person’s preferences, suggesting that this form of therapy was perceived as useful by participants. In contrast to other research investigating mobile apps for psychosis,96,98,130 the software did not provide regular prompts nor was use incentivised as part of the trial design. People were able to access some paper therapy resources, if they wished, for use outside sessions, and therapists reported that a blend of modalities was often valued. A further strength of the study is that we conducted multidimensional assessment of user experience using self-reported and objective measures, and specified adherence criteria a priori, in line with recommendations for assessing the user experience of digital therapeutics. 133,138 The findings reported here provide an initial validation of the SlowMo therapy design and we plan to conduct further analyses of digital usage of the mobile app. Important issues include granular examination of the functions used and the types of interactions, how usage varies over the course of therapy and how patterns of use relate to mental health outcomes. This work will help to elucidate whether reduced use reflects disengagement or e-attainment, and what constitutes a sufficient ‘dose’ of the mobile app for people to internalise the skill of slowing down, as well as potential detrimental patterns of use, such as excessive engagement. 111,115,139

Technical problems were assessed, consistent with reporting recommendations for digital therapeutic trials, and were infrequent. 135 They were mainly attributable to connectivity issues, emphasising the infrastructure challenges to scaling up digital therapeutics in the NHS. The technical problems with the SlowMo software were mostly because of issues in syncing the web app and mobile app data, and these issues reduced as the code was updated during the trial. The research tested a minimum viable product that had not yet been fully optimised. The trial context meant that therapists were willing and able to resolve technical issues. However, additional software development and maintenance will be required to minimise the need for technical support in the future.

A limitation of the work is that mobile app analytics were lost for 18 people in the therapy sample owing to a bug in the code; however, we do not anticipate that these analytic data would have differed from the rest of the sample. Another limitation is that we are in the process of developing an implementation strategy. This will be the focus for the next stage of our work and is critical given that most health technologies fail to be adopted, scaled up, spread and sustained, even where they are efficacious in RCTs. 140 Nonetheless, the tailoring of the SlowMo design to its specific target problem, a range of intended users, and the delivery context may support initial adoption, together with a strong value proposition to stakeholders that it has high rates of engagement and impact across a range of clinically meaningful outcomes. Further work will need to consider integration within existing care pathways and service design to support uptake. We intend to expand our inclusive, human-centred design participation beyond people with lived experience of psychosis, and intend to include a range of front-line therapists, service managers and commissioners. A health economic evaluation will be a necessary component of this research. Given the impact of SlowMo on a range of outcomes, we plan to build on this by incorporating other therapeutic targets and techniques. Our aim is to develop a modular digital therapy for psychosis, in line with the principles of agile science. 141 The SlowMo mobile app is currently user initiated, and some people may benefit from more responsive technology to deliver context-based interventions when they are needed. 142 We have already tested the feasibility of integrating wearable technology for stress monitoring into the SlowMo mobile app and intend to further explore this technology.

In conclusion, the findings suggest that the inclusive, human-centred design of SlowMo therapy supported the user experience of the intervention and resulted in excellent rates of adherence among a wide range of people. This comprehensive evaluation of the user experience of SlowMo therapy is in line with a recent coproduced call for digital therapeutic research to focus on how we can optimise existing interventions, the impact of psychosis on engagement, and whether or not digital therapies can improve reach and access for marginalised groups. 143 We further investigate the user experience of SlowMo therapy in the next chapter, with a coproduced qualitative study of the therapy experience. Together with the clinical efficacy and moderation results reported in Chapter 1, this work supports the further development of SlowMo therapy and testing in the NHS, with the ultimate aim to scale up, spread and sustain national and international implementation. Our approach underscores the need to focus on both effectiveness and user experience when developing digital therapeutics, and we strongly advocate adoption of this strategy to improve therapy outcomes for people with psychosis.

Introduction

I’m trying to do what the lady did, he showed me a video of this lady, she has the same problem as me and now when I was watching it she goes out and comes in and don’t let her neighbours worry her and I’m trying to do the same, I’m trying.

Paranoia, or fear of deliberate harm from others, is common and has a severe impact on mental health and quality of life,144,145 yet there are issues with both delivery and uptake of psychological therapies for paranoia and psychosis. 72 Large effect sizes have recently been found for the impact of brief causal-interventionist therapy approaches on paranoia that target sleep, worry and positive self-schema. 28,146–148 SlowMo therapy is a related causal-interventionist approach and is the first blended digital therapy for paranoia. It targets ‘fast-thinking’ habits or reasoning biases that are thought to underlie distressing fears of harm from others. 29 It is brief, with the potential to be delivered to large numbers of service users, and was designed in line with recommendations for improving implementation of digital therapies for psychosis. 129,131,149 The blended approach reduces the risk of therapist drift150 and the inclusive, human-centred design in the development of the intervention7 aimed to ensure that the therapy was usable, trustworthy, enjoyable, personalised and normalising, and offered flexible interpersonal support.

Service-user experiences of therapy are critical. Therapeutic alliance can have a causal impact on the effectiveness of treatments in psychosis,91 and alliance with online digital interventions may promote engagement. 151 Historically, therapeutic alliance is defined as the quality of the working relationship between the service user and the therapist in terms of shared goals and tasks, and a bond characterised by interest, warmth, empathy, authenticity, genuine concern, understanding and hope. 152–154 In blended therapies, this working relationship is extended to a ‘triangle of alliance’ between the service user, the clinician and the digital platform. 155,156 The traditional therapeutic alliance is potentially enhanced by the incorporation of technology, evidenced by greater commitment to goals and tasks. 157 The alliance is likely to vary across different platforms and levels of therapist involvement, and, in SlowMo therapy, this alliance is extended further to incorporate the service-user’s relationship with the mobile app.

However, evidence of user experiences of using a mobile app is only recently emerging. In a recent study,149 service users with psychosis reported that they felt a sense of autonomy and control while using an app to manage their symptoms. A proof-of-concept trial130 found high feasibility and acceptability as well as large treatment effect sizes for users of the Actissist app compared with a comparative symptom-tracking app. However, another app (FOCUS) offering cognitive interventions for psychosis found that, although usability rates were high, with 87% of users finding it easy to use and helpful for their symptoms, a small group of participants reported difficulties in engaging with the technology and expressed the need for more technical support. 137

The SlowMo trial, which we have reported in Chapter 1, found that the therapy had a positive impact on a range of mental health outcomes. In validating the aims of the design we have also demonstrated, in Chapters 1 and 3, excellent rates of adherence to the sessions and mobile app, and of self-reported ease of use, usefulness and enjoyment, that did not appear to be impacted by demographics; however, people who were less confident in using smartphones were less likely to make use of the mobile app. Given recommendations to employ multidimensional objective and subjective assessments of user experience133 to support implementation, this analysis will evaluate service users’ subjective qualitative experience of and alliance with the therapy, including the therapist, digital elements and their interaction. Qualitative feedback will thus be gathered to provide a detailed evaluation of the design, and be used to support further development of the intervention, in preparation for evaluation in an implementation trial context. There is much still to be understood about the implementation of digital interventions for psychosis and paranoia and, although progress has been made and implementation frameworks created,129 no study has so far evaluated a blended therapy with both digital input and therapist sessions.

The initial research questions related, therefore, to the uptake of the digital platform, therapy concepts and blended therapy approach, and how these were perceived and experienced by service users. The methods, data collection and analysis were coproduced with peer researchers. This approach enhances the rigour of qualitative methods because service users may be more forthcoming and open when interviewed by their peers. 66 Peer researcher involvement in the analysis may also lead to the identification of novel themes and critical reflections on the process. 79,158 The collaboration with peer researchers in the evaluation of the service user experiences of the blended SlowMo therapy thus adds methodological rigour to this approach and provides a rich subjective perspective.

Method

Results

Out of the 28 eligible participants who were approached in Sussex and London, five declined to participate and three were unreachable. One London participant did not consent to the audio-recording of their interview; therefore, field notes that were taken during their interview were included. Twenty-two participants took part, constituting 12.2% of the total SlowMo therapy sample. Their demographic details are provided in Table 16. Eighteen of the participants were male (82%), 18 were white British (82%), 19 were single (86%), 17 were unemployed (77%) and 14 lived alone (64%). Only three participants (14%) had received higher education beyond A Levels (Advanced Levels). The mean age for the whole sample was 44.9 years (range 20–64 years) and their mean GPTS part B score was 51.8 (range 29–79).

The qualitative sample demographics were similar to those of the main study reported in Chapter 1, in which participants were largely male (70%), of white ethnicity (70%), unemployed (80%) and living alone (58%), with a slightly lower average age of 42.6 years and a slightly higher GPTS part B score of 56.2 (SlowMo group).

Development and validation of theme structure

The final theme structure comprised six core themes and 20 subthemes. In terms of the six core themes, between phase 1 and phase 2, two themes remained unchanged (slowing things down, and improvements in paranoia and well-being). There was a slight change in wording of one theme, from ‘a new step on the recovery journey’ to ‘starting the SlowMo journey’, and a slight change of wording and focus of another, from ‘challenges in use of the SlowMo app’ to ‘approaches and challenges of technology’. The final two themes became more distinct: ‘feeling connected and understood’ became ‘value and learning from social connections’ and ‘drivers of progress’ was determined to reflect more specifically ‘the central role of the supportive therapist relationship’. Of the final 20 subthemes, six were identical from phase 1 to phase 2, five were new subthemes from phase 2, three changed wording slightly, three technical subthemes were expanded in focus, two learning subthemes were collapsed into one and another learning subtheme was expanded into two. All members of the qualitative team, including all PPI members, agreed the final structure (Figure 8).

FIGURE 8.

The structure of the qualitative experiences of SlowMo.

Starting the SlowMo journey

This theme captured the reasons for starting the SlowMo therapy, feelings before starting and the symptom barriers to engagement.

Reasons for starting the SlowMo therapy

Reasons for starting the SlowMo therapy included an opportunity offered and not sought, an opportunity for help, feeling stuck and a desire to take part in research:L17:S8:

It was [CPN] that got me to do it, she came to my house then she showed me this leaflet, she said, ‘You’ve got a 50/50 chance of doing it, would you like to do it?’. I said, ‘I’ll try, I’ll have a go’.

Oh mainly, ‘cause I wasn’t getting anywhere, my illness or whatever, has been on the same tablets for a long time. And I wasn’t getting anywhere so I thought I might try something.

Feelings before starting

Service users in all sites described feeling nervous and uncertain, but also open, willing and ready for new experiences:L21:O14:O13:

I was nervous, very nervous. I didn’t have no confidence, so I have now . . .

Apprehensive. Meeting with people I didn’t know.

I am eager to explore new opportunities as well, so that kind of made me go for it as well.

Voices and anxiety as barriers

Symptoms of voice hearing and anxiety were highlighted as specific barriers to starting SlowMo therapy:S10:

It was daunting at first, ‘cause it was the first time I’d actually spoken about them yeah. The way I felt with my voices. Basically it took me a good 10 minutes to listen to what she was saying because my voices were telling me not to listen and she was going to do this to me, she was going to do that to me yeah.

Central role of supportive therapist relationship

Importance of talking and being listened to

[what made the most difference?] Just being able to talk about my problems and focus on them and just come up with genuine ideas.

I found the talks we had, I found most invaluable. I thought that was amazing . . . actually having someone to talk to when we were going through it, I found amazing.

A positive therapist relationship

He made me really relaxed and any problems I had I could talk to him, which I don’t usually . . . So things came out that I haven’t told anybody.

Therapist as a supportive guide

The therapist’s approach to supporting both the therapy and the use of technology was described. The therapists were described as flexible, providing clear explanations, and their positive support promoted engagement and use of technology, while the computer content also supported the therapist:

I trusted her and felt comfortable with her. She explained everything very clearly . . . she was very sensitive towards my feelings, erm, when I was struggling. Erm, and with the computer she just helped me to see things more clearly.

I think if it was just the computer, then again it wouldn’t work; I think you need that interaction as well . . . I had one bad day where I just couldn’t focus and that was just really bad. But then I had someone who was really understanding about that and that made all the difference, and I didn’t feel, like, too pressured. I could learn at my own pace.

Never used a computer, but I learnt with [therapist].

I thought the computer was really there to support the, uhm, face-to-face talk. Uhm. It seemed to me that the therapy had been done so that it gave equal problems to the face-to-face therapy and computer therapy . . . I preferred in some ways more emphasis on the face to face perhaps . . . if it’s more psychological therapy which involves emotions . . . having a computer may sort of negate that, make it feel bland. Uhm. But in the way it’s done with [therapist], erm, it seemed to work quite well.

Slowing things down

This theme revealed how service users identified with the important central message from the SlowMo therapy. They found the concepts of slow and fast thinking to be helpful and valued the learning of this new skill, which importantly became more integrated into their thinking style over time with practice. However, some participants felt that progress through the sessions themselves was too slow and that the speed of delivery should be adapted to the participant. We aimed to address cognitive barriers to use; therefore, it is understandable that the intervention seemed insufficiently fast for more able participants. We have now integrated these insights into an updated product specification and will develop this ahead of an implementation research study:

It was very introductory kind of thing. I felt, maybe quite appropriately, it was quite slow actually to get to the juicy part, if you like. And [therapist] often would say I was often ahead of the project, ahead of the sessions, because I was thinking of things that were going to be introduced later on. So . . . for someone who is sort of getting to grips with it a bit quicker it could be condensed or you could introduce sort of the more advanced part of the earlier on.

Slow and fast thinking are relevant and helpful

I don’t worry so much, it’s my neighbours, they make me stressed and then I’ll say ‘No, I’ve got to slow down’. You have to because if not, if you carry on, you make yourself ill and you’ll land up in hospital.

I always insisted on going on . . . it might be a split-second decision . . . which is basically fast thinking. I was trained to always look out for the worst-case scenario . . . SlowMo slow thinking wasn’t difficult, but it was different . . . I found that using that where I live, all those idiots in the other blocks, if you think through possible other scenarios and then think ‘I don’t actually know those people and they don’t know me, so they can’t be talking about me’, whereas prior to SlowMo I would think ‘Why are they talking about me? What is going on?’ and that would stress me out really badly.

Oh the helpful bits were slowing up the thinking, thinking about other things that could be happening rather than just going to your first point.

Learning a new thinking skill set

It was very hard because me, I think quickly, but I slowed it down and I’ve learnt how to do that now.

Practice integrates the slow-thinking style

. . . my feelings are quite strong sometimes. And you have to keep on repeating about the other[thought], it could be something else. Sometimes you have to do it three or four times. It takes a while to get off what you are thinking cause the feelings are quite strong.

Instead of just believing and trusting in that fast-thinking conclusion that I have arrived at, there’s been more of an interaction on my part to counteract it with slow thinking . . . It was much closer to the end of therapy, I was quite actively engaging in slow thinking and quite often.

Value and learning from social connections

Vignettes and videos help to feel less isolated

Participants described how they related to the SlowMo vignettes and learnt that they were not alone:

I was hearing other people’s like feedback and . . . some of what they said sort of related to me a little bit . . . it was just helpful because some of the stuff they were saying sort of, it happened to me before . . . so, yeah it was, I just kind of relate to it, sort of.

Well just watching the videos and, erm, seeing like the people, I wasn’t alone, because young people like myself or younger than me or older than me does, err, have mental health, like I’m not the only one.

Learning and support through vignettes and peers

Participants described learning from and being inspired by the SlowMo vignettes and promotional videos, and liking the recommendations to learn from and be supported by others with the same lived experiences, although some people wanted a greater range of vignettes:

I’m trying to do what the lady did, he showed me a video of this lady, she has the same problem as me and now when I was watching it she goes out and comes in and don’t let her neighbours worry her and I’m trying to do the same, I’m trying.

[What made the most difference?] Erm I think the voices of real people thing. As soon as I left therapy every time, it’s stuck in my head you know. So if I get into that situation I try to rethink it the way that SlowMo taught me really, how to, how to do it.

Yeah, so what I said with the three people, it is a bit limited with what happened to them, and I think there is an awful lot more situations that people undergoing problems are confronted with and maybe you could actually put those into the software . . . like expanding it, make it sort of like a broader selection of situations and scenarios.

Approaches and challenges of technology

Personal relationship with the mobile app

Initially, people described that paranoia and self-consciousness were sometimes barriers to the use of the mobile app in public. Participants described viewing the mobile app as like a best friend, but also that they gradually moved from the mobile app to instilling responses in memory. However, the mobile app alone did not seem sufficient at the end of therapy and several participants expressed a desire for more sessions or even to take part again:

It’s difficult for me . . . yeah I always felt a bit of conscious somebody might be coming along and looking over your shoulder.

That’s right, that is my best friend.

I’m not sure if my condition has improved, but I don’t find I need to use it so much now.

I didn’t feel the need to get it [the mobile app] out of the pocket. Nothing to do with anybody else. But for me I just remembered it anyway.

Even though I don’t look at the phone, I just remember about slowing down.

It’s a shame that the therapy can’t carry on for longer. Once I stopped I felt there was something a bit missing. Which is why I am looking into taking up something now.

Use of technology to support positive outcomes

The mobile app was seen by participants as a tool to aid connectedness in daily life, the thought bubbles were seen as a guide and the computer scenarios and games were seen as learning tools:

The phone actually helps when you’re on the bus . . . if I start getting agitated about who’s looking at me and who’s not looking at me yeah, I just start playing with the bubble.

Every day if I go out, I always do what I need to do on it, like take my deep breaths and get me encouraged to go out . . . But that phone is always with me when I’m out. And if I stop, I use it as well.

Before I go out, check phone, then leave, pop the bubbles, slow down. I used it every day, not using it now. The tips helpful and personal message when have worries and message come up, worked as reminder.

Challenges to using technology

In some cases, the lack of interest in technology or the limits of the mobile app interface were a barrier to use and, as expected, technical issues with the minimum viable product were described. Recommended improvements included having larger fonts, animations in colour, a ‘check-in’ rating for how you’re feeling on the phone and written instructions on how to use the phone (onboarding), which is a priority for the next iteration:

. . . technology, and trying to keep up with it. I mean yesterday I tried to look up a few apps on the SlowMo and sometimes I was trying to find something, erm, something about err feeling insecure about something and I tried to find the answer that I gave myself, but I found that a little bit difficult.

I have bought things like iPods [Apple Inc., Cupertino, CA, USA] and Nintendo [Kyoto, Japan], but I know the main thing is the smartphone, but it’s too much technology for me. I can’t really be bothered to take the phone out with me.

. . . umm I did find when I wrote things for the bubbles, the type was a bit big and when the bubbles got smaller I couldn’t read everything it said.

. . . I had my own phone and I didn’t want to go around carrying two phones.

Cognitive demands of blended therapy as therapeutic or overwhelming

Some participants described the experience of a blended therapy approach as therapeutic, whereas others found it cognitively demanding:

You have got someone who is caring and understanding, but then you have got the visual and the video. And the things that might be passing through your mind . . . you need to sort of hit as many as possible to try and calm me down, so if you have got all of those stimulated by like looking and feeling, and then you are almost touching as well, it’s very engaging.

. . . just looking at the computer and, erm, and, also listening to [therapist], I mean, erm, it was quite, it was understanding what was on the computer with like all the cases they had, of the different [scenarios] . . . probably nearer to the end, my brain just totally, you know, shut down.

Improvements in paranoia and well-being

Participants at all sites reported multiple impacts on their mental health and well-being.

Decreases in paranoid thinking and in worry

Participants described a reduction in paranoia and worries and that the interventions supported people to have clearer thoughts:

Well the paranoia has dropped quite a bit, yeah. I’m not as extremely paranoid as I used to be.

I’m a worrier, but now I’m not a worrier as much as I was . . . it slowed me down quite a lot.

. . . you do think everybody is talking about you; I mean that was the big thing that I took from it. It just made me think ‘Hang on a minute, get a grip, it’s not like that at all’. That was the biggest thing that I took from it: just seeing things a bit more clearly.

Increased engagement with social life

I never used to go out, you see, and I go out on my scooter now with confidence. Before I wouldn’t; I always thought people were going to attack me and I don’t feel like that now.

I was not going out, not taking bus, at home all the time, walking. After 2/3 months, took the bus. Bike made me feel better and I start working.

I started up my art classes. I used to do a lot on my own, but now I go to other people for more structured art classes.

I suppose my social life has improved. It’s made me more comfortable around people I think. I have met more people and done more things I would say. Like just going to the movies, chilling or playing video games.

Increased confidence and perseverance

I am more confident, if I hear something I just brush it under the water; slow thinking for me, you know.

When I do my housework and they bellow at me ‘Stop doing, stop doing, we didn’t tell you to do it’, I just carry on doing it. I thought ‘No, I’m going to carry on doing it’.

Support with other mental health difficulties

Participants described support for a range of other mental health difficulties, including anxiety, stress, panic attacks, voice-hearing experiences and depression:

It’s teaching me to not sort of stress so much. Not, you know, not to get over anxious about stuff.

This is the first time in 5 years as I haven’t been in hospital with my depression and psychosis. So I think it’s really made a difference. Normally I am sort of in hospital 3 or 4 months a year.

Positive view of the future

Before SlowMo I was doomed, yeah, but now I’ve got a bit more positive outlook; yeah, I can actually live a bit of life.

I was like ‘I’m a weirdo, a nutter’, all that kind of thing, and then you realise that there is a lot of people in the same boat and there is a lot of nice people out there. It just gives you a more positive outlook.

Discussion

This chapter provided an in-depth exploration of the user experience of the SlowMo therapy design, generating insights for further development work and to support implementation. The six core themes captured the nerves and experiences of starting the SlowMo journey; the central role of the supportive therapist, who acted as a guide to accessing the therapy and the technology; and the relationships, approaches and challenges experienced in the use of the technology. Key features of the therapy from the participants’ perspectives were the concept of slowing things down, which was learnt and internalised through time and practice, and the value and learning achieved through social connections with the vignette characters that helped to normalise experiences and model coping strategies. As expected for a minimal viable product, technical difficulties were reported and recommendations were made for improvements. Positive impacts of the SlowMo therapy that were reported directly by participants included the impact on paranoia and other mental health difficulties, enhanced engagement with social and daily life, increased confidence and a positive outlook for the future.

Participants’ primary reasons for starting the SlowMo therapy were often a sense of feeling stuck and the opportunity to get help. Indeed, one of the peer researchers ‘stressed that people on the SlowMo trial had not generally been offered any other therapies before and it was important to bring out these issues’ (see Box 3) and some of the quotations make it clear that these issues had never been discussed before. As therapy progressed, participants expressed that the interactive multimedia feature of the vignettes made them feel connected, understood and less isolated in their experiences. Studies have also shown that participants enjoy digital platforms that are tailored, personalised and communicative. 149 Participants learned and retained their therapy skills better by observing the vignettes and several participants requested more characters dealing with different worries that often occur in paranoia. This experience of viewing others coping with similar experiences is an important factor that affects acceptability and engagement. 168 Participants recognised the importance of slowing down and focused on changing their thinking style by being more flexible in their thoughts and seeking more information, as discussed in Chapter 1. The thought bubbles helped participants to develop their skills to slow down their thinking and were an accessible resource that offered coping ideas and strategies (also discussed in Chapter 1).

A qualitative study on views of psychosis users reported that digital health interventions are seen as providing immediate and direct access to health care, reducing the pressure of waiting time. 169 A systematic review170 also found that ‘at-risk’ or first episode of psychosis individuals use the internet and mobile technologies for their mental health difficulties more frequently than those with longer-term psychosis, although this may have been related to users’ age and familiarity with technology. We found that participants were more likely to use the app earlier in the therapy, when paranoia was arguably more acute, and that the use of the app appeared to reduce as participants’ mental state improved and the thinking processes became internalised. This is consistent with the concept of e-attainment, as outlined in Chapter 3.

The presence of the therapist during the session was viewed as central to the process. The therapist was seen as critical in bridging the gap between the participant and the technical features of the mobile app, which in turn led to more favourable experiences of the technology. In the present study, the majority opinion was that the digital component of the therapy augmented the rich therapeutic relationship, in contrast to a stand-alone digital intervention. 171 This is supported by a recent systematic review168 that found that, when peer-to-peer interactions on mobile health (mHealth) apps were not moderated by clinicians or researchers, the retention rates slipped from very high (94.5%) to very low (14%), highlighting the role of the therapist in improving adherence. Importantly, however, in the current study, the participants also described a form of relationship that developed with the mobile app itself as an emerging experience. The mobile app reminded participants of the learning gained in therapy and was referred to as a ‘best friend’ in some cases.

Participants noted technical challenges, as expected given that the trial tested a minimal viable product of SlowMo therapy. 7 Technical recommendations have now been integrated into a product specification for an updated version of the SlowMo therapy, including improving the synching of content from the computer app to the mobile app, increasing font size, providing additional onboarding (written instructions) for use of the mobile app and coding the apps for cross-platform use. A study109 looking at factors affecting implementation of digital health interventions reported that interventions that are user friendly, adaptable to users’ needs and available on the participants’ own phone lead to better engagement. The participants who were recruited in previous qualitative studies by Aref-Adib et al. 171 and Bucci et al. 169 were mostly young, digital natives with a mean age of 28 years and 26 years, respectively. In the present study, the mean age of the participants was 45 years and, thus, a strength of this study was the broadly positive experience of blended therapy in this older age group. 109 Stigma surrounding mental health continued to be a barrier, especially for people suffering from paranoia, who, on occasion, felt self-conscious when using the mobile app in public. Although the SlowMo platform offers a normalising, supportive platform for participants, suspiciousness and pre-existing views towards technology might also affect participants’ usage. 109 Indeed, Chapter 3 has already shown that, although age, ethnicity and paranoia did not, in general, impact the use of the mobile app, pre-therapy confidence in technology did play a part.

Finally, participants reported improvements in paranoia, worry, confidence, distress, outlook and social life, mirroring the quantitative results from Chapter 1. These findings support the efficacy results of the SlowMo trial, which showed positive impacts of the therapy on paranoia and worry, alongside impacts on belief flexibility, fast and slow thinking, and other well-being, self-concept and quality-of-life outcomes. The therapy proved to have far-reaching impacts, as participants mentioned improvements in their social life and co-existing conditions in keeping with the broad pattern of improvements across secondary outcomes reported in Chapter 1. Participants reported increased self-confidence and better management of other stresses, which is consistent with the use of technology to promote a sense of autonomy and for mobile apps to offer real-time help. 169

To conclude, participants of the SlowMo therapy had a positive experience with the digital blended therapy. There were reductions in paranoia and improved overall well-being as a result of feeling understood, relating to vignettes, accepting and learning a new thinking style of slow thinking through interactive thought bubbles, and being supported by a therapist to reinforce their learning through the use of out-of-clinic sessions and the mobile app.

Conclusions

These findings offer valuable insights into participants’ experiences with SlowMo therapy. Importantly, many participants described feeling stuck prior to starting the therapy. They valued the central role of the supportive therapist, who guided them through the therapy and the use of the technology. Participants stated that key concepts of slow and fast thinking were helpful and that they valued learning from the social connections with the vignette characters. The personal relationship with the mobile app was experienced positively, and the integration of the slow-thinking style over time in a form of e-attainment was accompanied by improvements in paranoia, well-being and social integration, and reductions in worry, stress and depression. In this respect, the ‘triangle of alliance’ between the service user, the therapist and the digital platforms was highly effective, and there was a clear sense of a shared bond, goals and tasks to support improved paranoia and mental well-being. Important consideration was given to the less visible personal and social barriers that affect uptake of digital therapy, such as lack of instructions on how to use a mobile app, preference for using the mobile app on one’s own phone and the impact on usage pattern when symptoms become acute. This study, in combination with Chapter 3, provides critical information to support the development of the next iteration of the SlowMo blended therapy for future practice and implementation.

Contributions of authors

Philippa Garety (https://orcid.org/0000-0002-5637-1340) (Professor of Clinical Psychology) designed the trial, took main responsibility for the first draft of the study report, was the main lead for the trial, led the investigation in London, and provided input for the training and supervision in the therapy and digital platform.

Thomas Ward (https://orcid.org/0000-0002-7608-5755) (Research Clinical Psychologist, Trial Co-ordinator) provided input on the first draft of the study report, was the trial co-ordinator and therapy lead, and provided training and supervision in the therapy and digital platform.

Richard Emsley (https://orcid.org/0000-0002-1218-675X) (Professor of Medical Statistics and Trials Methodology) designed the trial, provided input on the first draft of the study report and carried out the statistical analysis.

Kathryn Greenwood (https://orcid.org/0000-0001-7899-8980) (Professor in Clinical Psychology) designed the trial, provided input on the first draft of the study report, was the PPI lead and led the investigation in Sussex.

Daniel Freeman (https://orcid.org/0000-0002-2541-2197) (Professor of Clinical Psychology) designed the trial and led the investigation in Oxford.

David Fowler (https://orcid.org/0000-0001-5806-2659) (Professor in Psychology) designed the trial and led the investigation in Sussex.

Elizabeth Kuipers (https://orcid.org/0000-0002-0041-7338) (Professor Emerita of Clinical Psychology) designed the trial, and provided input on the training and supervision in the therapy and digital platform.

Paul Bebbington (https://orcid.org/0000-0002-6030-7456) (Professor Emeritus of Psychiatry) designed the trial.

Graham Dunn (https://orcid.org/0000-0001-7091-3161) (Professor of Biostatistics) designed the trial.

Amy Hardy (https://orcid.org/0000-0002-6044-6093) (Research Clinical Psychologist) designed the trial, provided input on the first draft of the study report, was the digital lead, and provided training in and supervision on the therapy and digital platform.

All authors contributed to, read and approved the final report.

Graham Dunn died in 2019 before the study was completed. He was a leading statistician and trials methodologist, with particular expertise in mediation analysis. He worked with and influenced other members of the team over decades. His contribution to the design and conduct of the study was outstanding.

Publication

Garety P, Ward T, Emsley R, Greenwood K, Freeman D, Fowler D, et al. Effects of SlowMo, a blended digital therapy targeting reasoning, on paranoia among people with psychosis: a randomized clinical trial. [published online ahead of print 7 April 2021] JAMA Psychiatry 2021.

Data-sharing statement

The data sets generated and/or analysed during the current study will be available in anonymised form from the corresponding author (Philippa Garety) on reasonable request, subject to review, following the publication of results.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the MRC, NETSCC, the EME programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the EME programme or the Department of Health and Social Care.

User experience survey

We are interested in your experience of using the SlowMo mobile app. Your feedback is very important for helping us to improve the app, so please be as honest as possible.

Please rate each of the items below on a scale of 0 = ‘totally disagree’ to 10 = ‘totally agree’.

Rating

(0–10)

1. I enjoyed using the app.  _______

2. The app was too complicated to be helpful to me.  _______

3. The app was easy to use.   _______

4. The app was boring.  _______

5. The app helped me to manage my difficulties better.  _______

6. I would recommend the app to people with similar difficulties to me.  _______

7. I couldn’t get used to the app.  _______

8. The app felt relevant to me and my problems.  _______

9. I felt frustrated using the app.  _______

10. The information on the app was easy to understand.  _______

11. The app was fun.  ______

12. The app was not useful for managing my problems.  ______

1, 4*, 9*, 11 – enjoyment

2*, 3, 7, 11 – usability

5, 6, 8, 12* – usefulness/acceptability

* reverse score