Rapid evaluation of the Special Measures for Quality and challenged provider regimes: a mixed-methods study

Literature review using systematic methods

A rapid literature review of organisational failure in the public sector was conducted to guide our empirical research, particularly with respect to data analysis (see Chapter 3). Rapid-review methodology that uses a phased search approach was followed. 14 Rapid reviews follow a systematic review approach, but some steps are adapted to reduce the time required to complete the review (i.e. using large teams to review abstracts and full texts, and extract data; in lieu of dual screening and selection, a percentage of excluded articles are reviewed by a second reviewer and software can be used for data extraction and synthesis14).

Phase 1 of the review was based on a broad search of health services, business and management journals, and a review of the grey literature (e.g. think tank reports), to develop a theoretical understanding of the main characteristics of organisational failure and turnaround, and the types of interventions implemented to improve quality. This literature was used to develop a conceptual and theoretically informed framework that could be used to inform the phase 2 research questions (RQs), search strategy, inclusion criteria and interpretation of findings. We used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement15 to guide the reporting of the methods and findings. The review protocol was registered with PROSPERO (CRD42019131024). The full details of the review methodology are described in Chapter 3.

The review sought to answer the following RQs:

• Phase 1

  • How are ‘failing health-care organisations’ defined?

  • What are the theoretical approaches that have been used to explain organisational failure in and outside health care?

  • How is ‘organisational turnaround’ defined?

  • Which theoretical approaches have been used to study turnaround strategies (if any)?

• Phase 2 (covering health care, education and local government)

  • What are the main interventions used to improve quality?

  • Do the studies highlight any specific issues with implementation?

  • What are the interventions classified as ‘successful’?

  • Have any of these interventions been evaluated? If so, what is the impact and sustainability of improvements produced by these interventions?

  • What are the costs of these interventions?

Analysis of policy documents and interviews at a national level

To understand the nature of the SMQ/CP regimes and the programme theories that underpin them (i.e. the underlying assumptions and expectations about the purpose of the intervention and the anticipated impact), we conducted qualitative interviews with staff at a national level at the beginning of the evaluation. We also collected and reviewed reports and documents (n = 20) that would help us to understand the origins of the approach of the SMQ and CP regimes, the regulatory and policy context, and how these dual approaches had evolved over time (see Chapter 4). A small number of internal documents were shared with the team by NHS Improvement and interviewees early into the evaluation, which were incorporated for review.

Semistructured qualitative interviews were conducted with six staff at the national level. To identify relevant staff to invite for interview, the research team contacted representatives of key stakeholder groups – NHS Improvement, CQC and DHSC – to ask for recommendations for individuals involved in the SMQ/CP regimes. These individuals were then independently invited by e-mail to take part in a research interview. The purpose of these interviews was to better understand the nature of interventions deployed to support trusts and how the interventions were perceived by different stakeholders in relation to their programme theory/theories, and which interventions are viewed as being particularly effective, and under what conditions. The interview guide (see Appendix 1) covered three broad topic areas: (1) aims of the SMQ/CP regimes; (2) policy and interventions and (3) impact. The national interviews were professionally transcribed and analysed by a researcher (JL), who produced a thematic summary document for review by the wider project team early in the evaluation.

Multisite, mixed-method trust case studies

We conducted eight case studies (four ‘high level’ and four ‘in-depth’) using qualitative and quantitative approaches. We used a case study approach to explore the implementation of interventions in SMQ/CP trusts and reflect on any observed changes in processes and outcomes reported across specified time points (e.g. point of entry into or exit from SMQ). Case study research is common in management, business and organisational research and policy evaluations. Yin16 defined the case study as an ‘in-depth inquiry into a specific and complex phenomenon’. 16 Case studies typically employ a range of data collection methods, quantitative, qualitative or a mixture of both, to ‘construct narratives of past events, or accounts of specific cases’. 17

Sampling of case study trusts

Inclusion criteria:

• NHS trusts (ambulance, acute, mental health and/or community providers) placed in SMQ and/or CP regimes on or before 30 September 2018.

Exclusion criteria:

• trusts placed in SMQ and/or CP regimes (for the first time) after 30 September 2018

• trusts placed in Special Measures for Finance (SMF) only and never in SMQ/CP regimes.

To identify potential case study trusts, we conducted a preliminary analysis of data supplied by NHS Improvement on trusts that had entered SMQ since the regime began in July 2013, up to 30 September 2018. A total of 35 trusts entered SMQ; four trusts returned to SMQ (giving 39 episodes), 25 had exited SMQ and, as of 30 September 2018, there were 14 trusts in SMQ. The ‘watch list’ of CPs was initiated in July 2015. These trusts received interventions to prevent them entering SMQ. On 30 September 2018, there were 17 trusts on this list. Since July 2015, 44 trusts have been placed on this list, with 17 trusts leaving the CP regime because they entered SMQ and one trust leaving the list and subsequently returning. A total of 59 trusts entered SMQ or became a CP between July 2013 and September 2018. As of January 2019, there were 234 trusts in England, meaning that roughly one-quarter of trusts had experience with the SMQ or CP regime (Table 2).

TABLE 2 - Types of NHS trust entering the SMQ or CP regimes (July 2013 to September 2018)

Trust type	Trusts ever in the SMQ or CP regimes (n)	Trusts in SMQ (at September 2018) (n)	Trusts in the CP regime (at September 2018) (n)
Acute services only	33	7	9
Acute and community	18	4	7
Acute and mental health	1	1	0
Ambulance	2	1	0
Community and mental health	1	0	1
Mental health	4	1	0
Total	59	14	17

A total of 59 trusts met our inclusion criteria (see Table 2). We looked at the performance trajectories for these trusts and categorised them into four groups: prolonged poor performers, poor performers, shorter-term CPs and clear performance improvers (Table 3). Trusts were categorised based on the length of time spent in the SMQ/CP regimes and their progress over time, noting that some organisations were in the CP regime for only a short length of time and moved between categories. Therefore, although the categories are not fixed or absolute, our selected sites were in one of the categories at the time of sampling and met the criteria (e.g. re-entry into SMQ for a prolonged poor performer).

TABLE 3 - Descriptions of performance categories

Performance categories are neither exhaustive nor mutually exclusive.

Some trusts fit multiple categories.

Performance categorya	Performance category description	Trusts matching descriptionb
Prolonged poor performers	Trusts that have been in SMQ for ≥ 2 years since the introduction of the regime, including those trusts that re-enter SMQ after a period of exit	19
Poor performers	CP trusts that end up in SMQ	12
Shorter-term CPs	CPs that avoid entry into SMQ and have not previously been placed in SMQ. This may include trusts that merged with higher-performing providers. They are or were ‘challenged’ for < 2 years	20
Clear performance improvers	Trusts that have previously entered SMQ or CP but later achieved a good or outstanding overall CQC rating, without re-entry into either regime	9
Other trusts	Trusts that do not meet any of the other criteria (four trusts that were ‘challenged’ for a longer time, and one trust that left SMQ after a short period but has never been rated good or outstanding by CQC). These trusts were not sampled	5

Given that the overall objective of the case studies was to understand dynamics within trusts and their local contexts at different ends of the performance spectrum, we purposively sampled eight case study sites, with two sites from each performance category (Figure 1). We also aimed to recruit case study sites from a range of geographical locations and types of trusts. Of the two case studies within each performance trajectory, one was conducted ‘in depth’ and one at a ‘high level’.

FIGURE 1.

Purposive sampling model for case studies. Reproduced with permission from Fulop et al. 12 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits others to share and adapt this work for any purpose, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Copyright © 2023 Fulop et al.

Analysis of national indicators and workforce

This quantitative component of the evaluation explored relationships between being in a SMQ/CP regime and a set of performance and workforce indicators. For workforce, we were particularly interested in whether or not trusts found it more difficult to recruit and retain staff at different levels and evidence of the impact of interventions on the staff themselves. For the case study sites, these data were combined with trust inspection information from CQC. The workforce analysis was exploratory and was subject to the construction of a consistent and comparable workforce data set and sufficient sample sizes to establish any statistical links. One of our aims was to raise hypotheses that could be tested more robustly in future studies and reflected back to the case study sites for their qualitative insights.

Economic analysis

The economic analysis aimed to quantify the costs and benefits of different combinations of interventions used in SMQ/CP regimes from an NHS perspective using a cost–consequences analysis (CCA) approach (see Chapter 9). A CCA compares interventions in which the components of incremental costs (direct or indirect) and consequences (e.g. knowledge, behaviours and processes) are computed and listed, without aggregating these results into a cost-effectiveness ratio. 28 This approach enables one to look into process measures and qualitative findings in a quantitative manner, allowing for some insight as to how potential benefits compare with the cost of interventions.

A feasibility study for the economic analysis found that:

• A CCA was feasible, but it would be possible to evaluate different combinations of interventions only (i.e. it would not be possible to evaluate the benefits of each intervention individually). It would need to account for likely variation in the type and intensity of these interventions, for example the percentage of full-time equivalent time that the ID spends at the trust, different buddying models and varying receipt of funds spent in different ways. We will explore the impact of this variation on both costs and consequences.

• Costs could be measured using resource use and unit cost data collected during the multisite mixed-methods study.

• Consequences could be measured using qualitative data collected during the multisite mixed-methods study and/or combining it with quantitative data.

What was known?

• There is a limited understanding of whether or not and how improvement interventions aimed at improving the performance of health-care organisations are effective.

What this chapter adds

• Successful interventions included restructuring senior leadership teams, inspections (in schools) and internal reorganisation by external organisations.

• Most interventions were designed and implemented at an organisational level, without considering system context.

• Limited attention was paid to the potential negative consequences of the interventions and their costs.

Design

The review was based on the phased rapid review method proposed by Tricco et al. 14 and expanded the review of organisational failure published by Vaughn et al. 5 The rapid review method followed a systematic review approach, proposing adaptations to some of the steps to reduce the amount of time required to carry out the review (i.e. the use of large teams to review abstracts and full texts, and extract data; in lieu of dual screening and selection, a percentage of excluded articles is reviewed by a second reviewer, and software is used for data extraction and synthesis, as appropriate14).

The review included two phases. Phase 1 was based on a broad search of health services, business and management journals, and a review of the grey literature (e.g. think tank reports) to develop a theoretical understanding of the main characteristics of organisational failure and turnaround, and the types of interventions implemented to improve quality (for an example of this approach, see Ferlie et al. 37). This literature was used to develop a conceptual and theoretically informed framework (see Table 7). The framework was used to inform the phase 2 RQs, search strategy, inclusion criteria and interpretation of findings. We used the PRISMA statement15 to guide the reporting of the methods and findings. The review protocol was registered with PROSPERO (CRD42019131024).

Research questions

The review sought to answer the following RQs:

• Phase 1 (covering health services research, management and business studies)

  • How are ‘failing organisations’ defined?

  • What are the theoretical approaches that have been used to explain organisational failure?

  • How is ‘organisational turnaround’ defined?

  • Which theoretical approaches have been used to study turnaround strategies (if any)?

• Phase 2 (covering health care, education and local government)

  • What are the main interventions used to improve quality?

  • Do the studies highlight any specific issues with implementation?

  • What are the interventions classified as ‘successful’?

  • Have any of these interventions been evaluated? If so, what is the impact and sustainability of improvements produced by these interventions?

  • What are the costs of these interventions?

Phase 1

We used a phased search approach. 14 The first phase was broad, covering literature from the fields of health services research, management and business studies to identify overarching themes and definitions on regulation, performance and QI in health-care organisations and the public sector. Broad terms such as ‘organisational failure’, ‘organisational turnaround’, ‘special measures’ and ‘performance in organisations’ were used to identify initial relevant literature across the public sector. A second search targeted literature in the education sector. All other searches (3–5) focused on the health sector. Using a snowball technique, additional terms were found and inserted into a search strategy for five databases [MEDLINE, EMBASE™ (Elsevier, Amsterdam, the Netherlands), Cumulative Index to Nursing and Allied Health Literature (CINAHL) Plus, Web of Science™ (Clarivate Analytics, Philadelphia, PA, USA) and OpenGrey], creating longer and more complex search strategies (see Appendix 4). These databases were selected in consultation with a librarian who sought to identify the most relevant databases for the review topic.

Phase 1 focused on identifying the theoretical content from the literature on organisational failure and turnaround to develop a thematic framework to guide the review. We followed the approach for building thematic frameworks for reviews used by Ferlie et al. 37 Definitions for key concepts, such as ‘organisational failure/success’ and ‘turnaround’, were identified. Furthermore, we searched for the main theoretical frameworks used to explain these processes and synthesised their main characteristics. We sought to create a high-level overview of the different perspectives that have been used to explore failure, success and turnaround in organisations. The findings from phase 1 informed the RQs developed to guide phase 2 of the review.

Phase 2

Phase 1 thematic framework

The main components of the thematic framework are summarised in Table 7. The five waves of searches for phase 1 provided a working list of 56 relevant publications. Key examples of this literature are included in Table 7.

We found that three definitions of failure are common in the literature (as decline, crisis and below previously established performance level), but only one of these considers failures at a system level (i.e. beyond individual organisations and including multiple organisations) (see Table 7). Some authors argued that failure and success should not be considered discrete, opposite concepts, but should be understood as in a dialectical relationship (highlighting the contradictions and inherent tensions between components). The seven theoretical frameworks used to explain organisational success or failure (see Table 7) reproduce this focus on the organisation as the unit of analysis and neglect of system-level pressures, with the exception of two of the more recent ones (see Table 7, Section B – 6 and 7). Concepts of turnaround tend to privilege a linear conceptualisation of organisational recovery processes, with only one approach considering turnaround as a non-linear complex process (see Table 7, Section C– 3). Turnaround has also been explored as either an internal or an external approach, with limited discussion of the interaction between internal and external strategies.

The findings from phase 1 informed the RQs developed to guide phase 2 of the review. We sought to explore the interventions delivered in low-performing and high-performing organisations, identifying the underlying ideas that guided them, such as their conceptualisation of failure/success as an organisational or system-wide feature, the perception of turnaround as a linear or non-linear process, and the extent to which they considered the interactions between internal and external strategies to guide turnaround processes. As a result of the findings from this phase, and the consideration of success and failure in a dialectical relationship, we decided to develop a phase 2 that explored the experiences of both low-performing and high-performing organisations.

Phase 2 results

The initial search yielded 3607 published articles (Figure 2). These were screened based on their title and the type of the article, resulting in 1386 articles. These articles were further screened on the basis of their abstracts, which left 111 articles for full-text review. Full-text review of these articles led to 41 articles that met the inclusion criteria. One additional article was identified by reviewing the bibliography of the included articles, leading to 42 articles being included in the review. We excluded articles that focused on improvement in individual pupil outcomes (i.e. reading levels) and not general school performance, or articles that did not discuss specific interventions used for improvement.

FIGURE 2.

Study selection procedure.

Characteristics of the included studies

Seventeen of the studies took place in the USA, 20 were from the UK, one took place in the UK and the USA, one took place in Canada, two took place in Israel and one was a comparison across six European countries (Table 8). The publications were relatively recent, with most articles published post-2010. Study designs varied, but most studies were qualitative, followed by quantitative and mixed-methods designs.

Definitions of failure and turnaround

We examined definitions of failure and success in the articles in relation to the thematic framework that we developed in phase 1 (see Table 7). Failure/success appeared to be defined in most studies as ‘organisational performance that is persistently below or above some minimally acceptable level’45 (see Table 7, Section A – 3). Low-performing and high-performing organisations were defined as such in relation to nationally established ratings or indices (i.e. Audit Commission Comprehensive Performance Assessment ratings or Academic Performance Index). This definition distinguished between the minimum acceptable level of performance and performance that is ‘persistently’ below and above this acceptable level. The focus of this definition of failure/success tended to be on the organisation and was not applied to the wider system.

Some studies have tried to incorporate a ‘success/failure as a system property’ approach by considering the relationships between the provider organisation and other external organisations; however, even in these studies, consideration of system-level properties was limited. In most studies, failure was considered to be produced by limited or dysfunctional organisational learning (see Table 7, Section B – 5).

Preventing failure and producing improvements were dependent on changes in organisational culture. Some studies indicated that individual interventions aimed at QI were not effective if they did not address problems in organisational culture. 82

In relation to turnaround strategies, none of the 11 studies in the health-care sector described turnaround strategies imposed by external parties, yet these strategies were frequent in studies from education and local authorities. For instance, Jas69,78 argued that for turnaround to be effective in local authorities, it needed to be externally driven. In health care, most of the turnaround strategies were based on relational/mutual arrangements. Some studies framed turnaround under RRR (replacement, retrenchment and renewal), which we previously identified in phase 1 (including these three aspects of the intervention or only some of them). Replacement can refer to the replacement of executive members of a board, retrenchment is based on using stricter financial controls and focusing on performance targets, and renewal strategies could involve changing organisational culture and improving stakeholder engagement. 4 Most of the interventions that we analysed followed a renewal approach, with few examples of replacement. In local authorities, retrenchment (i.e. reduction of spending in particular areas) was seen as producing negative consequences.

Type of intervention

One of the aims of the review was to explore the types of interventions used to improve quality in low-performing and high-performing organisations. The types of interventions varied by sector, but we found overlap in a few of these. We were able to group the interventions into 10 main categories (Table 9):

1. financial incentives (including pay for performance schemes and grants)

2. external partnerships and sharing of practice

3. QI training

4. reorganisation at multiple levels, including senior leadership level, and the use of external interim managers

5. development of existing leadership and/or middle management

6. identification of organisational goals or priorities

7. use of routine data and establishment of performance standards (including dashboards)

8. standardising care practices

9. RRR (replacement, retrenchment and renewal)

10. interventions involving external inspections.

Features of ‘successful interventions’

Some of the articles described interventions that produced and maintained improvements in quality. The authors reflected on the features that made these interventions successful. These included the need to (1) consider the hospital-wide co-ordination of interventions, not limited to specific departments or services; (2) establish protected time for staff to implement the changes; (3) ensure staff engagement in the identification of problems and development of the interventions (to guarantee ownership); (4) develop strong relationships with other organisations (to share good practice); and (5) identify clear goals and targets to meet as a result of the intervention and use data to monitor progress.

Issues to consider in implementation

We identified a few trends across sectors in relation to lessons learned in the design and implementation of improvement interventions for low-performing and high-performing organisations. Our review confirmed the findings of previous reviews that have stated that improvement interventions are shaped by the organisational culture, with negative cultures framed by limited ownership, lack of collaboration, hierarchies and disconnected leadership. 5 Features of organisational cultures associated successful implementation of improvement interventions include a ‘can-do attitude’ and the desire to improve on the part of staff, as well as engaged leadership. Our review also revealed interesting reflections on the need to consider processes of implementation. For instance, in the case of school inspections, Ehren et al. 99 argued that, although these might be beneficial for schools, this was contingent on the content of the feedback and how the feedback was communicated to schools after inspections. The authors found that feedback was more effective if it included detailed information on performance expectations and a clear understanding of current teaching conditions. 99 Studies on the process of carrying out inspections in health care have indicated that inspections were more reliable if carried out by larger teams and if inspectors were allowed to have discussions and received appropriate training. 101

No effects, potential negative outcomes and unintended consequences

Some of the studies in schools highlighted the negative consequences of being labelled as a failing organisation; for instance, there were important implications for recruitment and retention of both staff and pupils, relationships with parents and the community, and links with the local authority. Chapman and Harris62 found that top-down reform that treats all schools as the same is unlikely to secure long-term improvement and change, as schools should be free to select the approaches to change that suit their particular needs. External pressures were also seen as negative, as in some cases they resulted in the ‘repackaging’ or ‘recycling’ of ideas and approaches (in the case of this study, restructuring plans) that did not support the meeting of organisational goals or contribute to learning. 91

The external versus internal debate was also present in studies focusing on school inspections and special measures, and some authors argue in favour of the use of school self-evaluations rather than external inspections. 98 Recent research on the use of inspections in health care has also shown that external inspections have no effects on performance. 100,102

Costs associated with the interventions

We found that limited attention was paid to the costs of the interventions or cost savings produced by the interventions. Furthermore, the studies did not explore opportunity costs involved in implementing the interventions, the use of the time and resources to make different changes, and the extent to which the changes could have been carried out without the intervention.

Implications for future research

The articles included in the review identified gaps in research. One proposal was to explore how turnaround strategies change through time, taking into consideration the historical context of organisations. Many studies captured only a snapshot of the intervention and organisational culture, missing the nuances of how change was negotiated. Another gap identified was the need to take into account whether improvement approaches were internally or externally driven interventions by regulatory bodies, for example. The role of external partnerships in creating and sustaining improvements is currently being explored in an ongoing review,103 yet additional work is required to identify the components of interventions that might respond better to internal drive compared with those that might benefit from external support (or a combination of both). In relation to this, some articles highlighted the need to make sure that studies of these types of intervention capture the experiences of staff members across all layers of the organisation, particularly front-line staff and lower management (as many studies have focused on changes taking place at senior leadership levels).

What was known?

• A review of research considering the impact of interventions on organisational turnaround in the health, education and local government sectors found that most improvement strategies define failure in relation to the inability of organisations to meet pre-established performance standards and that turnaround is commonly thought of as a linear process designed to fix problems and bring organisations up to the ‘appropriate’ level (see Chapter 3).

• Understanding of the nature of the SMQ/CP regimes and the programme theories that underpin them was limited.

What this chapter adds

• National stakeholders perceive the SMQ regime as a ‘support’ programme that aims to enable organisations to bring about improvements.

• The CP regime was viewed as a way to provide access to additional resources for struggling organisations. CP was not intended to be a long-term intervention.

• QI plans are a central element of SMQ/CP regimes and are intended to focus organisational leaders on quality indicators and patient outcomes (as opposed to financial stability only).

• National teams recognise that providers might be part of ‘challenged systems’ and that this needs consideration.

Background to Special Measures for Quality and challenged provider regimes

There is a history in public sector management of independent inspections, which are found in education from the 1990s and extending over time into local government and health and social care (see Chapter 3). The current incarnation of the health regulator, the CQC, reports to parliament via the DHSC and was founded following the Health and Social Care Act 2008. 121 From 2010 onwards, health and social care providers began to register with the CQC. Today, the regulator inspects adult social care (residential and community), NHS general practices, NHS trusts, dentists and independent providers. Registered providers pay a fee to the CQC, which accounts for the majority of the agency’s annual income.

In 2013, a number of high-profile reports highlighted major issues in standards of care within NHS hospitals. The first, the Francis Report,112 investigated in great depth the cultural and performance failings at Mid Staffordshire NHS Foundation Trust. This was followed by Sir Bruce Keogh’s review6 of 14 NHS hospitals with high mortality rates. The latter report drew attention to an absence of knowledge at the board and management level about the causes of high mortality, and raised concerns with regards to QI:

We did not see sufficient evidence to demonstrate that many Board and clinical leaders were effectively driving quality improvement. In a number of trusts, the capability of medical directors and/or directors of nursing was questioned by the review teams. Common concerns were:

1. poor articulation of the strategy for improving quality;

2. many trusts had findings from quality and safety reviews undertaken recently by internal and external parties but could not show a comprehensive and consistent approach to learning from these; and

3. a significant disconnect between what the clinical leadership said were the key risks and issues and what was actually happening in wards and departments around the hospitals.

NHS England, 20136 Contains public sector information licensed under the Open Government Licence v3.0

Shortly thereafter, Don Berwick and an independent group of experts in QI, patient safety and health organisations and systems were asked to review patient safety in the NHS. 113 The resulting publication, A Promise to Learn – A Commitment to Act,113 underscored the importance of regulation, yet pointed out that:

The current NHS regulatory system is bewildering in its complexity and prone to both overlaps of remit and gaps between different agencies. It should be simplified.

Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry, 2013113

The experts made 10 recommendations for the NHS to address patient safety, placing emphasis on culture change, quality assurance (QA) and continuous learning within health organisations to bring about long-term improvements in patient care. The report further suggested that the NHS regulatory system was ‘bewildering in its complexity and prone to both overlaps of remit and gaps between different agencies’ and that it should be ‘simplified’. 113 The report influenced the formation of the national Patient Safety Collaborative, a programme run by NHS England and delivered at the regional level by the Academic Health Sciences Networks.

A new CQC regulatory regime emerged following these high-profile reviews published in short succession, one intended to challenge the poor quality and complacency found within some parts of the NHS and drive a QI agenda nationally. The CQC inspections that are now a familiar feature of the health and social care system in England began in late 2013, in the wake of the Keogh and Francis investigations. 6,114 Closer working between arm’s-length health-care agencies (these organisations have a role in the processes or administrative functions of government, but operate outside of central government departments with varying levels of independence and distance from ministers122) and regulators was another important development during this period, especially between Monitor, the CQC and the NHS Trust Development Agency (TDA). A joint statement by these agencies and NHS England in May 2013123 outlined how the regulatory and assessment approach was to be delivered in future:

The Francis report makes a compelling case that the regulation of NHS trusts and foundation trusts should change in two key ways. Firstly, equal emphasis should be placed on addressing failures of quality as for failures of finance. And second, overlap and duplication in the roles of regulatory agencies must be removed.

In future, this division of roles will be simpler and clearer: the Care Quality Commission (CQC) will focus on assessing and reporting on quality; and Monitor and the NHS Trust Development Agency (TDA) will be responsible for using their enforcement powers to address quality problems. To free up time to care, the overall regulatory burden on providers will be radically reduced but, where there are failings in quality of care, there will be a stronger response.

Department of Health and Social Care124 Contains public sector information licensed under the Open Government Licence v3.0

The role of the Care Quality Commission and NHS Improvement in regulation

By June 2016, the CQC had inspected 136 NHS acute non-specialist trusts and 17 specialist trusts. 7 In this period, NHS trusts that had not obtained foundation trust status were supported by the NHS TDA, while Monitor had oversight for NHS foundation trusts. However, in 2016, NHS Improvement brought together these health agencies along with patient safety (including the national reporting and learning system) and intensive support and advancing change teams. The focus of NHS Improvement was to ensure that NHS trusts provided quality care, met the conditions of their license agreements and were financially sustainable. Although NHS Improvement and NHS England are in the process of co-joining, NHS Improvement still retains overarching responsibility for holding NHS providers to account and intervening if persistent quality challenges are identified by the CQC or through local intelligence gathered within health regions. NHS Improvement describes the organisation’s remit as ‘helping trusts to improve quality of care, finances and use of resources, operational performance, leadership and contribution to strategic change across local health systems’ (contains public sector information licensed under the Open Government Licence v3.0). 125

In practical terms, the regulatory system for health and social care in England has evolved a process of combining CQC ratings that interrogate patient safety and quality with the investigation of organisational leadership (the ‘well-led’ domain) and use of resources. The SMF regime has materialised in parallel and, although not the specific focus of this evaluation, the financial context of the NHS trusts studied for the case studies was taken into consideration. There are around 400 inspectors responsible for the regulation of hospitals, both NHS and independent providers. In an interview during the evaluation, the CQC was described as having an ‘influencing’ role within the health and social care sectors:

… the regulatory powers are there and we do use them, but we don’t see them as our main tool, we see our main tool is influencing interaction, and influencing is through meeting organisations, talking to providers, it is by inspectors producing a report, that describes all the things going on with providers and it is about producing reports on the wider sector.

CQC interviewee

The same respondent described the CQC as having a role of holding providers to account:

If we find a serious safety problem in a clinical service, you always, the next question is why has this been allowed to get to this stage? Why is it the CQC found this, rather than the trust knows it and is dealing with it itself?

CQC interviewee

The Special Measures for Quality regime

The challenged provider regime

Programme theory

In this section, we address the programme theories that guide the central (e.g. IDs, buddying and funding) and additional interventions (e.g. leadership changes) delivered to trusts in the SMQ/CP regimes. Stakeholders at NHS Improvement perceived the SMQ regime as a ‘support’ programme that enabled organisations to bring about improvements (correspondence and interviews). The intent was that SMQ would be ‘helpful’ and enable NHS providers to address the issues that had caused the poor CQC ratings, and that the QI funding would be targeted at problem areas. ‘Fundamental’ changes were expected to drive forward improvements, such as improving clinical leadership and engagement and working on the organisational culture and governance. IDs in particular were expected to proactively engage organisational leaders and develop an improvement strategy:

Special measures is meant to be about support from NHS Improvement, but also support from the wider system in helping them. So it’s meant to be a helpful regime, and it’s often portrayed as a kind of punishment that that isn’t the intention, but of course, it’s public, it’s reputational, I think it’s is often perceived as a punishment, but it’s not a punishment, it’s meant to be saying, ‘actually we just don’t think, we think you need extra help in order to move yourself out of this position’.

CQC interviewee

There was awareness that deep-seated issues typically underpinned problems in trusts’ performance and would take significant time and effort to address, especially in large and geographically dispersed providers:

… it’s leadership, governance, engagement and culture and so on. So I think the ones that really struggle have sometimes got, they’re sometimes big organisations usually more than one site and they may or may not be far apart but quite often they are, they will quite often be organisations that have kind of merged from other ones but frankly have never really merged they’ve never developed a unified culture, a sense of belonging to the overall organisation.

NHS Improvement interviewee

Interviews further suggested that if there were concerns among the regulators and arm’s-length bodies that organisational leaders were unable to address performance and quality concerns, and were not engaging staff well, such a situation could not be allowed to continue because system confidence in the provider was lacking.

The CP regime was viewed as a method for providing access to additional resources for struggling organisations and was not intended to be a long-term intervention:

It’s not an oh you’re on the naughty step this is genuinely an access to resource, help, budgeting, etc., in an aim to try and prevent them going into special measures and there’s quite a lot of examples of where we have spared them deteriorating and they’ve got better. Some we haven’t been able to stop them so that’s quite interesting in itself.

NHS Improvement interviewee

A problem with the CP intervention, however, was that it might be a ‘marginal decision’ as to whether or not to place an organisation in SMQ or CP. If an organisation was placed in the CP regime, the trust would not get the intense level of longer-term support that it might actually need:

I just wonder at times if it isn’t better to just go the whole hog and then you can probably bring about improvement faster.

NHS Improvement interviewee

For this reason, the SMQ programme was perceived by national respondents to be a deeper approach that ‘unearths a lot’ of problems and could expose local services in ‘turmoil’ and that had been problematic for years. However, there was recognition that providers might be part of ‘challenged systems’ and that this needed taking into consideration.

There were, therefore, indications that intensive support from SMQ was believed to expedite organisation-wide QIs, especially if problematic leadership and governance difficulties were promptly tackled. It was understood that instability and overreliance on interim senior leadership roles was unhelpful, although it often occurred in practice. SMQ and CP trusts were expected to produce a QI plan outlining their approach to improvement, suggesting that focusing the minds of organisational leaders on quality indicators and patient outcomes (as opposed to financial stability only) was central to the approach.

Interventions used by NHS Improvement to address performance and their perceived influence and impact

A team within NHS Improvement has responsibility for overseeing the SMQ and CP regimes and reports to the NHS Improvement medical director and the DHSC. This team works closely with a group of national IDs and has oversight of the programme budget that funds the support provided to trusts (see Chapter 6 for further information). The buddying intervention was supported by NHS Improvement’s ‘My Peer Improvement Budget’, whereas the other interventions, QI funding and IDs, were covered by the NHS Improvement SMQ and CP fund.

The interventions delivered to trusts in the SMQ/CP regimes varied from one trust to another and depended on their particular circumstances and needs. The interventions covered the appointment of an ID; a review of the trust’s leadership capability; access to funds for QI; a plan for improvement, including options for diagnostic work on assessing medical engagement; buddying with other trusts; and commissioning external expertise. As detailed throughout this report, these interventions were often delivered within a context of leadership changes at both board and senior management level.

There was said to be co-ordination, ‘open’ communication and ‘very regular contact’ between the senior leaders at the CQC and the senior leaders at NHS Improvement, and interaction at the regional level occurring between regional CQC and NHS IDs, in delivering support to NHS providers.

Perceived influence and impact of the Special Measures for Quality and challenged provider regimes

Special Measures for Quality was viewed in central government as a ‘useful model’, although admittedly one that could first ‘destabilise’ trusts and be very disruptive. It was acknowledged that there was often a great deal of media and political interest, which could put further strain on a provider, staff and those supporting the organisation:

Quite often there’ll be resignations above board level or the exec[utive] level, there’ll be interims quite possibly in place, there’ll be people who are very worried about the position and their own personal position in this. And you know, there have to be sometimes some quite challenging conversations about how the future might look.

NHS Improvement interviewee

There were unanswered questions about trusts that re-enter SMQ or are placed in the CP regime not long after exiting SMQ, and how this might be prevented. We were informed that work was under way at NHS Improvement and the DHSC to explore wider system dynamics, especially how troubled regional health economies might contribute to a trust’s position and prolonged poor performance, financial sustainability or quality issues.

Overall, there was a lack of systematic knowledge about why some trusts re-enter SMQ and why others successfully exit and sustain QIs to achieve an overall ‘good’ or ‘outstanding’ CQC rating.

A report published by the CQC in 2014,116 which reflected on the regime after its first year, noted the following:

No single factor accounts for the improvements that have been made or for the different pace of change at individual trusts. It is important to note that the trusts did not all start from the same baseline. The size of the task was larger for some trusts than for others, especially for those covering two or more locations that are widely separated geographically. In addition, some of these trusts were known to have been struggling to provide high quality care for several years.

© Care Quality Commission 2014. Special Measures: One Year On116

It was felt that the CP regime had prevented a number of trusts from avoiding entry into SMQ:

… that’s probably one of the more powerful things that was developed and of course that all goes on behind the scenes, it’s not visible, because they never get special measures, therefore, the issue is never brought into the public domain.

CQC interviewee

However, the boundary between CP and SMQ trusts could, in practice, be blurred:

There are organisations that end up on the challenge list, that arguably should be in special measures … NHSI [NHS Improvement] make that decision, not CQC … quite often you’ve got a new chief exec[utive] going in and you need to give them a bit of airtime to get on and deliver and putting them into special measures does create a much bigger wave of people coming to review and scrutinise and want action plans, challenge doesn’t have quite such an impact, it gives them a little bit of airtime, to get on and do things.

ID, NHS Improvement interviewee

What was known?

• Organisational failure and turnaround have been a focus of research in the health, education and local government sectors (see Chapter 3).

• Previous research has not addressed the characteristics of SMQ/CP trusts compared with those of other trusts.

What this chapter adds

• The SMQ/CP trusts have slightly older populations, with more attendees living in rural areas but fewer in deprived districts. A smaller proportion are foundation trusts. Characteristics, such as size and number of separate sites providing acute care, did not differ significantly from other trusts.

• National-level analysis found that sickness absence, staff vacancy rates, proportions of consultants and nurse–doctor ratio were not significantly different from national means at SMQ/CP trusts when they enter the regime.

• However, they had significantly worse staff survey results for six of the nine survey domains.

• Our case study trusts do not differ greatly from SMQ/CP trusts overall with these measures, but had significantly higher staff vacancy rates.

• Common underlying reasons for entry into SMQ/CP among our eight case studies included issues with leadership, organisational culture, governance, workforce, finance and estates.

Mixed-method case studies

Eight case study (four ‘high level’, four ‘in-depth’) trusts were purposively sampled from 59 trusts that entered SMQ or CP between July 2013, when the SMQ regime began, and September 2018. The performance trajectories of the 59 trusts were categorised into four groups: prolonged poor performers, poor performers, shorter-term CPs and clear performance improvers (see Table 3).

We sampled two sites from each performance category, with consideration given to geographical location and types of trust. The mixed-methods case studies used both qualitative and quantitative approaches (see Chapter 2). Interview, observational, documentary and quantitative data from the case studies were triangulated and analysed thematically and comparatively.

Analysis of national indicators

To investigate the characteristics of trusts that entered the SMQ/CP regime compared with those of other trusts, we considered organisational characteristics, such as the category of trust, size and numbers of sites, alongside indicators of the type of population they serve (e.g. age, rurality and deprivation). We also analysed relative performance when trusts enter SMQ or become challenged, such as against the 4-hour waiting time target for emergency departments (EDs), the 62-day cancer target and the staff survey. Among the cancer statistics, we also analysed skin cancers specifically in response to feedback from NHS England and NHS Improvement following sharing interim findings. A further group of indicators related to workforce, in particular the mix of staff employed at each trust, sickness absence and staff vacancy rates. A description of the chosen performance indicators is provided in Table 11.

Most of these measures were derived only for acute trusts owing to the nature of the service and the types of data they collect. In addition, the number of mental and community health providers and ambulance providers that have entered the SMQ/CP regimes is too small to enable meaningful comparisons with their peers. The indication of foundation trust status and staff survey results were the only indicators that used data from all types of trust. The rates of emergency caesarean sections were derived directly from patient-level HES data. The codes used to identify these are explained in the Appendix 8.

For organisational and demographic data, we compared outcomes for all trusts during the 2018/19 financial year. Demographic information came from HES and was derived from a count of all unique individuals recorded as attending ED or outpatients or admitted as an inpatient over the year. However, deprivation and rurality information were available for the inpatient data only. Differences between SMQ/CP trusts and all other acute trusts were analysed using standard two-sample t-tests for means or chi-squared tests for proportions.

We chose as our time point for performance measures the date that the trust first entered SMQ or CP, whichever was the earliest, effectively the time when concerns about quality were first officially noted since the SMQ regime started. The exceptions were staff vacancy rates and staff-mix variables (nurses and consultants) for which, because we had data from after the first trusts entered SMQ only, we chose the first time that a trust entered SMQ or CP on or after the earliest date in our data, which may have been the second time of entry.

Comparison with other trusts was not straightforward because trusts entered SMQ/CP at different times when the overall national rates may have changed. For example, two trusts may enter at different times with similar performance against the 4-hour ED target and because of changing national rates, one may be better than the national average on entry and the other significantly worse. Therefore, we calculated standardised differences between each trust’s value on entry and the national comparator using a z-scoring approach. 124 We adjusted for overdispersion with an additive hierarchical model. Where crude z-scores were skewed, we first applied a square root transformation to normalise the data before adjusting for overdispersion. Mean z-scores for the SMQ/CP sample were then compared with the baseline of zero, assuming standard normal distributions. Given that overdispersion is already accounted for in the summary hospital-level mortality indicator (SHMI) after using a log-transformation, we were able to generate z-scores from a combination of the SHMI values and the values of the two standard deviation control limits. Alongside analysing the characteristics of all SMQ/CP trusts, we also investigated the characteristics of the eight case study sites to observe how representative our selected group was of the larger population of trusts. Dates when trusts entered SMQ/CP were derived from a combination of data provided by NHS Improvement and information from the CQC website. Further details on these methods are described in the Appendix 5.

Analysis of the national staff survey

The NHS staff survey asks staff across the NHS in England about their experiences of working for their respective NHS organisations across a number of domains (Table 12). Annual survey data are available for the years 2014 or 2015 (depending on the theme) to 2018. Results cover the 230 NHS trusts that took part in the survey. Survey themes are summary scores derived from the responses to specific individual survey questions. Scores range from 0 (worst) to 10 (best). Theme results are weighted by occupational group. Details on the technical details of the survey and how theme scores are calculated can be found in the technical document published by NHS England. 138

The SMQ/CP trusts were compared with all other trusts based on the year that they entered the SMQ or CP regimes for the first time (irrespective of whether they entered as a CP or went straight into SMQ). Of 230 trusts included in the staff survey, historical data were available for 40 trusts joining SMQ/CP between 2014 and 2018 and 37 trusts joining between 2015 and 2018. The survey uses seven benchmarking groups based on trust type to allow for fair comparison between trusts.

To ascertain the relative position of a trust compared with all other trusts within a given benchmarking group, quartile ranks were derived for each theme and year based on a trust’s score and their benchmarking group. We then calculated the number of SMQ/CP trusts that were placed in the bottom quartile in the year that they joined and the year that they left SMQ/CP. The expected value was calculated as the total number of trusts that joined or left SMQ/CP over the survey period (and where historical data were available) multiplied by 0.25. Binomial confidence intervals were derived at the 95% confidence level.

Organisational characteristics and trends of all Special Measures for Quality/challenged provider trusts and the eight case study trusts

Since July 2013, when 11 trusts entered SMQ after the Keogh Review,6 a total of 62 trusts have been in either SMQ or CP, with several in both categories. The numbers in each category are shown in Table 13.

Of all 47 CP trusts, 18 (38%) were escalated to SMQ. Four trusts in SMQ subsequently re-entered SMQ at a later date.

In total, 54 out of the 62 trusts (87%) in SMQ or CP have been acute trusts, with the remainder comprising three ambulance trusts and five mental health or community providers. Ten (28%) trusts in SMQ have also been in SMF. A further six trusts have been in SMF only.

Taking a snapshot of all trusts in 2018/19, the characteristics of trusts that had once been in SMQ or CP compared with the rest are shown in Table 14.

When pooled together as a group, trusts in SMQ and CP have slightly older populations (mean age 44.1 vs. 43.9) with more attendees living in rural areas (22.7% vs. 17.2%) but fewer in deprived districts (20.6% vs. 23.7%). A smaller proportion are foundation trusts (50% vs. 70%). However, there is little difference in the size and number of separate sites providing acute care.

A breakdown by region is shown in Figure 3. SMQ/CP providers are more prevalent in the Midlands and East (33% of all trusts in the region) and South East (39%) than in other regions.

FIGURE 3.

NHS trusts by region (percentages are proportions ever in SMQ or CP): n = 242.

Performance characteristics

Figure 4 shows the outcomes for trusts against different national performance metrics when they first enter SMQ or enter CP (whichever is earlier).

FIGURE 4.

Outcomes for trusts when the first enter SMQ/CP. Mean z-scores compared with national value with 95% CIs (positive values reflect worse outcomes). Emergency caesarean sections data126 Copyright © [2019], NHS Digital. Re-used with permission of NHS Digital. All rights reserved.

When trusts enter SMQ/CP, mean performance against the 4-hour target has been significantly worse than the national rates, as are standardised mortality rates measured with the SHMI. However, given that mortality was one of the measures used to identify the Keogh trusts in 2013,6 this would have an influence on this result. Removing the Keogh trusts, the mortality rates were no longer significantly worse.

Workforce statistics

Figure 5 illustrates outcomes for our chosen workforce metrics when trusts enter SMQ/CP. For the sickness absence rates, these reflect the first times the trusts’ enter SMQ/CP. Given that vacancy statistics were not available before April 2017, and staff-mix indicators were not available before October 2013, these are for trusts that enter SMQ/CP on or after those dates only. If this is for a second time and the first entry is before the data are available, then the values at the time of second entry are chosen. For the SMQ/CP trusts, none of these values is statistically significant compared with national values.

FIGURE 5.

Workforce statistics for trusts when they first enter SMQ or CP. Mean z-scores compared with national value with 95% CIs (positive values reflect larger numbers).

Staff survey results

For each of the domains of the staff survey, Figure 6 illustrates the number of SMQ/CP trusts in the bottom quartile of results when they enter and compared with the expected number. For six out of the nine domains, results are worse than would be expected. The only domains without significant findings are those associated with quality of appraisals, quality of care and safe-environment violence.

FIGURE 6.

Numbers of SMQ/CP trusts in the bottom quartile within each staff survey domain when they first enter SMQ or CP (whichever is earlier) compared with expected values. Error bars are 95% CIs.

Features of trusts that re-enter (prolonged poor performers)

Four trusts in SMQ subsequently re-enter SMQ: three acute providers and one mental health/community provider. There are no consistent characteristics among the performance or workforce indicators that we have analysed for these trusts, although all are situated on the east coast and serve more rural populations than is typical. Specifically, for the three acute trusts, 46% of admitted patients were from rural areas (range 33–64%) compared with 23% from the SMQ/CP trusts as a whole.

A further five trusts re-entered as CPs after a previous period of being in CP or in SMQ. Of these, three are based in outer London and another is from the south east.

Characteristics of the eight case studies

A summary of the characteristics of the eight case study trusts is presented in Table 15. The case studies are all acute trusts, and no ambulance, mental health or solely community providers were included.

As a group, the eight case study trusts have similar characteristics to the SMQ/CP trusts in general (see Table 14). However, on entry to SMQ for the first time, they had a notably higher rate of ED 4-hour breaches than other SMQ or CP sites (see Figure 4), although the differences are not significant. Vacancy rates were significantly higher for the case study sites (see Figure 5), but data existed for only three of these at their time of entry to SMQ/CP, one of which had a particularly high rate.

Underlying reasons for entry into Special Measures for Quality or challenged provider regimes (eight case studies)

From our case study analysis using RAP sheets to summarise findings from interviews and documents, we identified a number of common underlying reasons for entry into SMQ or CP among our eight case studies (Table 16). These broadly related to issues concerning leadership, organisational culture, governance, workforce, finance and estates, and chime with findings from a number of studies of characteristics of poor-performing or failing organisations (see Chapter 3).

What was known?

• There is a growing body of literature on organisational turnaround from the health, education and local government sectors, with some examples of successful interventions reported to be restructuring of senior leadership teams, inspections (in schools) and internal reorganisation by external organisations (see Chapter 3).

• NHS Improvement supports SMQ/CP trusts through oversight and a range of interventions (see Chapter 4).

What this chapter adds

• The type, duration and perceived value of interventions provided by NHS Improvement vary between trusts.

• IDs were considered to be helpful when using a coaching style and offering tactical advice for interacting with the regulators. IDs were sometimes felt to bring additional demands and there were mixed views about the amount of time that IDs should spend in organisations.

• Buddying worked best when ‘buddy’ organisations had similar contexts. Buddies were most commonly used to learn about good practice in relation to specific problems.

• Funding was perceived as being mainly used to cover posts and external consultants and experts. Some participants felt that there was a risk that trusts might need to spend their way out of SMQ.

• New leadership teams were a key driver for change, bringing new ideas and approaches. Previous SMQ experience was seen as helpful.

• Other pieces of work viewed as interventions included ‘deep dives’ into data on specific topics/service areas carried out with NHS Improvement staff and responses to additional oversight and scrutiny from the regulators.

Improvement directors

Improvement directors were appointed by NHS Improvement when deemed necessary for a trust in a SMQ or CP regime. The normal cycle for an ID normally includes 1–3 months of intensive support, followed by 3 additional months of maintenance support (although this might vary by trust). Some IDs highlighted that a limit on the time that they were involved with organisations was required:

… personally I think you lose your effectiveness as an improvement director when you’ve been in an organisation for about 8 to 10 months.

Senior director, case study trust

However, this feeling was not always shared with trusts because some felt that IDs left when important work was still to be undertaken:

… our ID was withdrawn last summer, long before we had been re-inspected or let alone, come out of special measure, so I employed one of my former IDs to help us […] but the trust had to fund that itself.

Senior director, case study trust

Improvement directors can cover more than one trust at the same time and some trusts in a CP regime might not have access to an ID.

Improvement directors were used across most of the case studies included in the study, but there were mixed views in relation to their value. A few trusts viewed IDs in a negative light, indicating that they were not able to enact change, acted as a spy for the regulators or were seen as another source of demand:

Sometimes [the ID] did not feel like support, it felt like it was just another person coming in to ask, ‘have you done this?’.

Senior director, case study trust

However, in most cases, IDs were considered an asset, as capable of ‘making things happen’ (senior director, case study trust). The style used by IDs to engage with organisations (embedding themselves or maintaining distance), work with senior leadership teams and produce improvements varied. IDs deemed successful by trusts were those who used a coaching style with the senior leadership team, offered tactical advice on how to deal with the regulators, protected the senior leadership team from the regulators to ensure permanence, and gave staff confidence to make changes.

Buddying

Buddying, also referred to as peer improvement, can be arranged directly by NHS Improvement, by the ID or by the trust themselves. Buddying can be on a departmental level or trust wide. Buddying is often formalised through a memorandum of understanding. In the case of our study, we were able to identify cases of buddying in only a few of the trusts, and some of these had arranged their own buddies. The use of buddies also varied by trust, but most trusts used them to learn about good practices in relation to specific problems. The appropriateness of buddies was discussed frequently, and the first selection of buddies made by NHS Improvement was not always considered a good choice by trusts in SMQ:

They are a large metropolitan university teaching hospital and we are not and it just wasn’t compatible and wasn’t really working and the improvement work that they were doing with us was focused on things we didn’t need immediate help with.

Senior director, case study trust

The appropriateness of the matching was dependent on geographic location (with close distance seen as positive), trust type and size and experience with similar problems.

Funding

Trusts in SMQ and CP regimes had the opportunity to apply for and, if successful, access funds to help support improvement activities. Trusts labelled as CPs were able to apply for up to £200,000 and trusts in SMQ for up to £500,000. Trusts also received additional financial support when leaving SMQ (£100,000). Applications for the funds are reviewed internally at NHS Improvement and approved by the executive medical director:

… [the application] needs to be mapped to things that are going to make a difference and […] how it has impacted and made a difference.

NHS Improvement representative

Six of our eight case study trusts reported that they had access to the support funds provided by NHS Improvement. The funds were reported to be mainly used to cover the costs of external consultants and to fund posts. In some cases, trusts expressed concern that they would need to ‘spend their way out of special measures’ (senior director, case study trust). The funds were not considered to be enough to implement long-lasting QI (see Chapter 9, which includes an analysis of how funds were spent by five of the eight case study trusts).

Other relevant interventions

What was known?

• There is limited evidence on how the different NHS Improvement interventions are received and how effective the interventions are at delivering and sustaining performance improvement (see Chapter 3).

What this chapter adds

• Staff have a strong emotional response when the trust is placed in SMQ/CP.

• Entry into SMQ/CP was often thought to be necessary and ultimately can be viewed as a catalyst for change. However, some staff can view SMQ/CP as heavy-handed scrutiny or punishment.

• There is a view that local system-wide issues also need to be addressed for a trust to make and sustain improvements in quality and performance improvement over time.

• Trusts’ responses to being placed in SMQ/CP involved making improvements in the following eight domains: governance, accountability and leadership; service delivery; data monitoring and use of data; organisational culture and staff engagement; workforce; QI plan or strategy; QI interventions, methods or techniques; and estates and environment.

• We have identified the key characteristics of trusts that do and do not exit SMQ/CP.

Performance journeys of the eight case studies

A summary of each case study trust’s performance improvement journey is presented in Figure 7.

FIGURE 7.

Summary of the performance improvement journeys of the eight case study trusts (to October 2019).

Perspectives on Special Measures for Quality/challenged provider

Role of the system in Special Measures for Quality/challenged provider regimes

Responses to being placed in Special Measures for Quality/challenged provider regimes

Characteristics of trusts linked to clear improvement or remaining in Special Measures for Quality/challenged provider regimes

In this section, we provide a summary of characteristics of trusts that have improved and those that remain in SMQ/CP regimes. Table 19 provides an assessment of board maturity18 of four trusts that we studied in depth. This appears to indicate, as expected, that the highest board maturity was evident in the trust with the most sustained improvement. However, an analysis across all eight case studies indicates differing characteristics between the clear improvers and those that had not exited SMQ/CP. These are summarised in Tables 20 and 21.

What was known?

• There is limited previous analysis exploring the overall performance trajectories of trusts entering SMQ/CP regimes.

What this chapter adds

• Relative to national trends, entry to SMQ/CP regimes corresponded to positive changes in 4-hour waits in EDs, mortality and delayed transfers of care. Trends in sickness and absence improved after trusts left the regimes.

• There is some evidence that staff survey results improved over the period that trusts were in the regime.

• The regime appeared to have no influence on RTT times or cancer waiting times.

Length of time in each regime

Of the trusts that left the SMQ regime, the average time that was spent in the regime was 27 months, with a range from 5 to 49 months (Table 22). Half of the 36 trusts were in SMQ for more than 2 years and four re-entered having previously left. Of the 26 CP trusts that did not enter SMQ, just over half (54%) did not leave CP within 1 year. Half of the trusts in SMQ were previously challenged.

Impact on all SMQ/CP trusts

The results of our analysis of changes in trends of indicators before, during and after a period of SMQ/CP are shown in Table 23, which presents results for four different aspects of their trajectories:

• pre-intervention slope – how trends in the indicator compare with national trends before entry to SMQ/CP

• change due to intervention – a measure of how values of the indicator change after trusts enter SMQ/CP relative to national values

• change in slope during the intervention – a measure of how the trajectory changes while trusts are in SMQ/CP

• change in slope after the intervention – a measure of how the trajectory changes after a trust leaves the regime compared with what the trajectory was before it entered.

For three of these indicators (RTT time, delayed transfers of care and sickness absence), trends were significantly worse than the national trajectory before trusts entered the SMQ or CP regime for the first time. After entering the SMQ/CP regimes, trends significantly improved for ED waits and mortality, and there were significant improvements without changes in trend for delayed transfers of care. For mortality, trends also improved, even when excluding the initial 11 Keogh trusts that entered SMQ in July 2013.

There is evidence that trends in ED waits and sickness absence improved after the period in SMQ/CP. There is no evidence of trends in RTT times or cancer waits being affected by the regime. Moreover, in the case of the former, trends were worse than average before they enter. Conversely, trends in emergency caesarean section rates were improving for the SMQ/CP trusts when they entered the regime but then appeared to increase.

Proportions of consultants declined relative to national trends after trusts entered the quality regime. In addition, while in SMQ/CP, the doctor to nurse ratio increased, although this was not sustained.

The impact on the staff survey results is shown in Table 24. The number of trusts with survey results corresponding to both year of entry and year of exit is relatively small. However, allowing for general improvements in staff survey results over recent years, there are significant improvements in five of the nine domains.

Impact on case study sites

For the eight individual case study sites, outcomes against tested performance indicators are shown in Table 25. These show some evidence of individual trusts improving with respect to ED waiting, waiting times for skin cancer, delayed transfers of care, mortality and sickness absence. However, there are some trusts for which outcomes appear to have worsened once entering the regime. Notably, there is no evidence of any improvements with respect to RTT or 62-day waits for all cancers as a group.

What was known?

• Very few studies have addressed the costs and consequences of improvement interventions in health, education and local government settings (see Chapter 3).

• The relative costs of the NHS Improvement interventions and their consequences have not been previously evaluated.

What this chapter adds

• At the national level, during 2018–19, the mean funds spent on trusts was £388,259 (n = 19) for trusts under the SMQ regime and £173,625 for trusts under the CP regime (n = 23). An additional £100,000 was earmarked for each trust leaving the SMQ regime (n = 4). A mean of £61,134 was spent on peer improvement for all 42 trusts under SMQ/CP regimes.

• The largest components of NHS Improvement spending for our case studies were identified as interventions directed at ‘training on cultural change’ (33.6%), ‘workforce quality and safety’ (21.7%) and ‘governance and assurance’ (18.4%).

• Only 15.8% of SMQ trusts exited the regime within 24 months (and did not re-enter within the next 12 months) and 65.2% of CP trusts left the CP regime after 12 months (and did not re-enter within the next 12 months).

• The interventions delivered to trusts as part of the SMQ/CP regimes show a positive effect on staff-based measures (e.g. ‘support to staff from immediate managers’, ‘quality of staff’s appraisals’, ‘staff engagement’ and ‘staff’s experiences with safety’), while there were less improvements in the context of ‘promotion of staff’s health and well-being’, ‘staff’s satisfaction with quality of care’ or ‘organisation’s actions on quality, diversity and inclusion’.

• The findings on the effect of SMQ/CP regimes on financial stability are not clear-cut. Most trusts that exited SMQ (68.4% of the ‘exiting’ group) experienced the same financial stability before and after exiting, although this share was lower for the group still remaining in the regime (46.2% of the ‘remaining’ group). In fact, none of the trusts in this ‘remaining’ group experienced an increase in the financial stability, whereas about 10.5% of the ‘exiting’ group did.

Measuring costs

Measuring consequences

Two types of consequences are considered in this study: primary consequences (relating to the entry and exit of the trusts in the SMQ and CP regimes) and secondary consequences (relating to staff experiences, internal quality changes within the trusts and the trust’s financial stability). Given the aims of the regimes for trusts to exit SMQ within 24 months and CP within 12 months from the date of entry, the two primary consequences considered here are (1) exiting the SMQ regime within 24 months and not re-entering the SMQ or CP within the next 12 months, and (2) exiting the CP regime within 12 months and not re-entering CP or SMQ within the next 12 months.

The secondary consequences indicating staff experience and the internal quality changes after the intervention are derived from the NHS staff survey 2014–18138 and are organised under nine different themes (see Appendix 7). All themes are scored on a 0- to 10-point scale and reported here as mean scores. A higher score always indicates a more favourable result. Each theme includes between three and nine statements/questions.

In addition, a tenth secondary consequence on financial stability measures the financial performance of trusts on a scale from 1 to 4, where 1 reflects the strongest performance and 4 the weakest (based on the NHS Improvement – Quality Tracker). To determine if trusts under the SMQ/CP regimes became financially stable, we have chosen a threshold where those trusts that scored from 1 to 3 are considered to be financially stable and those scoring 4 are considered not to be. The choice of this threshold was based on the observed data, given that about 90% of the trusts under the SMQ/CP regimes had a score of 3 or 4.

Costs

Consequences

What was known?

• There is limited qualitative evidence regarding direct experiences of the SMQ/CP regimes and the factors felt to be most helpful for making improvements and for sustaining change.

What this chapter adds

• Lessons for health-care providers, staff and health system leaders suggested by participants to support sustainable change were prioritising staff, stable and mature leadership, building healthy relationships with the regulators, and consideration of improvement at a system level.

• Lessons for regulators and policy-makers suggested by participants to support sustainable change were to address the emotional cost of SMQ/CP, deliver trust-specific support, avoid duplicate reporting, recognise that longer timelines are needed to embed change and provide system-level support.

Insights for health-care providers, staff and health system leaders

Insights for health-care providers, staff and health system leaders were organised in the following themes:

• Staff need to be considered the top priority.

• Stable and mature leadership is required to make sustainable changes.

• Improvement depends on the capacity to develop healthy relationships with the regulators.

• Improvement needs to be considered at a system level through partnerships and collaboration with other organisations.

All of the trusts in the study had carried out some work on staff engagement or morale. Participants talked frequently about the importance of ‘putting staff first’ and ‘considering their needs’. A dominant assumption was that if staff were involved in identifying the changes that needed to be made, then they would experience greater ownership over them, participating in the implementation and sustainability of improvement. It was noted that both senior and front-line staff need protected time to undertake improvement work and cannot be expected to do this work on top of their usual workloads.

Interview participants also argued that trusts in SMQ or CP regimes needed to make sure that they directed their efforts at developing mature leadership teams who could remain in leadership roles throughout the implementation of the changes. Two main aspects of leadership were highlighted: (1) leadership needed to be mature and experienced enough to deal with the demands made by regulators in the light of the operational pressures prevalent in the own organisations (i.e. financial deficit, capacity issues, etc.), and (2) leaders needed to remain in the organisations or in leadership roles long enough to see the changes through. The first point entailed developing collaborative relationships with the regulators, but also knowing when to push back if demands were considered unhelpful or unrealistic. It was also related to the capacity of leadership teams to move beyond the point-by-point recommendations made by organisations, such as CQC and NHS Improvement, to the development of trust-wide improvement strategies.

The continuity of leadership teams was outlined as an issue in the light of local contexts heavily shaped by staff turnover. Some of the trusts included in our study had suffered a complete restructure of senior leadership teams and trust boards or had gone through multiple CEOs in a short amount of time. This was an issue recognised at the trust level, as well as at a more regional level by CCGs and NHS England and NHS Improvement regional representatives. There were discussions around the need to ‘protect CEOs’ to allow for this continuity and make sure planned changes were implemented. Related to this, trust leaders saw a benefit in developing collaborative and healthy relationships with the regulators by seeing them as a source of support or as ‘critical friends’.

Despite the fact that most NHS Improvement interventions had an organisational-level focus (with the exception of buddying), several trusts and CCG representatives recognised the importance of considering organisational failure and improvement at a system level. Some participants discussed the similarities in performance issues of trusts across their patch, and that improvement was not seen as sustainable until ‘systemic structural fault lines’ were resolved. Learning from other organisations was seen as beneficial, but a general belief was that organisational failure would not be resolved by buddying between two organisations. System-wide changes were needed and this required changing support structures and interventions to operate at a system level also.

Regulatory bodies

• Time is needed to implement and embed sustainable changes – 2–3 years not 1 year – and staff should be given ‘slack’ to develop and implement changes.

• Strategies to support improvement need to be more trust specific.

• Duplication of reporting requirements to different regulatory bodies should be reduced.

• Consideration should be given to the ‘fit’ of IDs with the context of the trust they work with and discussed with the trust involved.

• Consideration should be given to the provision of sustainable funds required to improve patients’ outcomes.

Trust leadership

• Stable leadership is needed once the new team is established because of the time it takes to make improvements, otherwise problems are perpetuated.

• Inclusion of people with previous experience with SMQ in senior leadership teams can help manage regulatory requirements and bring knowledge and confidence to enacting change.

Staff and culture

• Staff engagement and an organisational culture that supports learning are key to sustainable improvement.

Emotional costs and stigma

• Ways to mitigate the emotional cost and stigma of SMQ are needed; for example, the way trusts are informed they are in SMQ. The name of the regime could be changed to intensive support as opposed to ‘special measures’ and provide emotional support for leaders who require it.

Quality improvement strategies and capabilities

• Development of organisation-wide QI strategies and capabilities is important.

Local systems

• Poor organisational performance needs to be considered at both organisational and system levels (considering the factors that might be hindering improvement at the regional level).

• Trusts in SMQ/CP need support from other organisations in the system.

Patients and the public

• Engagement with patients and the public should be emphasised as an important part of the process of making improvements.

• Involvement from Healthwatch and CCGs can help to bring feedback from patients and the public and supports a focus on patient experience as part of QI.

Governance, accountability and leadership

• Address poor existing governance structures (e.g. lack of clear lines of reporting, poor clinical engagement).

• Financial investment (e.g. external advisors and reviews) has been effective in improving governance.

• Provide transparency, assurance, risk management and financial management with clear lines of accountability for quality and performance: ‘board to ward’.

• Senior teams need to encourage accountability for patient safety and quality at all levels and embed an open and listening improvement culture.

• It is important that staff understand why specific changes to processes are necessary and become engaged in governance meetings and structures.

• Effective leadership teams are stable, ‘visible, supportive and approachable’, work as a team and encourage learning and application of new knowledge.

• Effective clinical leadership is needed, through the MD and CN roles and/or reorganisation of governance pathways.

• Senior teams need to establish good working relationships with the regulators and facilitate positive collaborations with external partners and peers.

Service delivery

• Ensure compliance with national standards and CQC and NHS Improvement recommendations.

• Prioritise service-level improvements that will ensure patient and staff safety.

Data monitoring and use of data

• Ensure systematic use of data across the organisation for QI and QA.

• Focus on understanding what data are describing and triangulate standard metrics with other sources of information, including the concerns of staff and patients.

• Treat data as a prompt to investigate quality.

• Consider bespoke data collection for specific purposes.

Organisational culture and staff engagement

• Improvement relies on better staff morale and culture.

• How accepting leaders and staff are of the need to change varies – for effective change, leaders and staff need to accept and input into improvement plans.

• Positive engagement and investment in staff are essential:

  • Develop strategies for better communication/listening to staff concerns.

  • Address ‘bullying’ or ‘insular’ cultures.

  • Promote staff appraisals, celebrate staff success and widen opportunities for education and training.

  • Support growth of patient safety culture through staff training and development.

  • Close communication gaps between the senior executive team and front-line staff, for example listening events and increased senior and board presence on wards.

  • Encourage a culture of openness and transparency.

Workforce

• Address problems with staffing levels to ensure patient safety:

  • Staff appraisals.

  • Improved workforce planning.

  • Financial investment to increase staff numbers.

Quality improvement plan or strategy

• Leadership teams need to establish a vision for culture change and continuous improvement.

• Develop a deeper understanding of problem areas and causation.

• IDs playing a key role in developing the QI plan/strategy for the trust.

• Allow staff dedicated time to work on QI and do not expect them to undertake QI activities on top of their existing work.

Quality improvement interventions, methods or techniques

• Use of a formal ‘method’ or strategy for QI that is embedded throughout the organisation.

• Dedicated resources are needed for QI: trusts need to develop new internal capabilities.

• Receptive to new, external knowledge to improve performance.

Estates and equipment

• Strategies and financial investment are needed to improve working and patient environments to ensure safety, improve capacity and modernised services.

Facilitators of improvement

• Positive engagement with system peers and partners:

  • In some cases, providers were already working together, for example in integrated care systems.

• Whole local system response to quality issues.

• QA that is linked to strong governance and data collection processes across the system.

• Supportive OAG meetings involving local system peers and partners.

• Financial sustainability planning across the system.

Barriers to improvement

• It can take time and considerable effort to build/rebuild external relationships.

• System-wide issues that are difficult to address, for example workforce and financial pressures.

• OAG meetings can be a ‘ritual stoning’ for the trust.

Conclusions, suggestions for future research and lessons for future evaluations

High-quality interventions that successfully support struggling health-care organisations are essential and an issue that is an international concern. Through our evaluation, we have delivered a greater understanding of the programme theory, impact and staff views and experiences of the SMQ/CP regimes, with formative feedback shared with key stakeholders. Lessons for trusts on responding to interventions in SMQ/CP regimes, as well as lessons for the DHSC and NHS Improvement on how to support these, trusts have been proposed. These lessons are timely because the oversight model is currently being redesigned by NHS England and NHS Improvement.

Given both the rapid nature of this study and that the findings have raised further questions, we propose that future areas of research should include the following:

• Evaluation of the impact of the new NHS Improvement/NHS England operating framework,119 particularly with its emphasis on system improvement.

• Evaluate SMQ/CP at the local system level and expand the range of stakeholders external to the trust giving viewpoints on SMQ/CP, including local government and other health service providers.

• Study of trusts in SMF and an exploration of the link between SMQ and SMF.

• Further research to understand the impact of SMQ on the financial stability of trusts in SMQ/CP.

• Research to delineate the timescales required for sustainable improvement.

• Longitudinal empirical data collection to look at the sustainability of improvement over time and improve our understanding of why some trusts re-enter SMQ (e.g. follow-up case studies or additional observations through embedded research154).

• Consider what interventions are most helpful to trusts that remain in SMQ/CP for long periods.

• Further research on the early detection of trusts in difficulty as well as early intervention/prevention to determine which strategies are effective.

An exploration of how to engage with patients and the public when a trust goes into SMQ.

Reflections on the methods used in this study have led us to draw out the following lessons for future evaluations:

• Use mixed-methods approaches, combining quantitative and qualitative data, for example structured use of board reports (including papers) and CQC inspection reports can provide insights about trust’s trajectories, which are helpful and powerful if combined with quantitative indicators derived from nationally available routine data. Mixed-method case studies are, therefore, most valuable for insights that have depth and internal validity.

• Data collection for monitoring could be tailored to the concerns in each organisation and be used not only for internal monitoring (which happens to an extent already) but also to feed into the overall evaluation framework. It would mean using different measures for different trusts but would have the advantage of greater specificity. In the national analysis provided in this study, we have used the same metrics for all trusts, which has been important for making comparisons, although the metrics may not always be closely related to the actual improvements that a trust needs to make.

• If possible, conduct evaluations prospectively in ‘real time’ using sequential monitoring techniques for assessing whether or not is something is working while under the regime. Such techniques would also have value in identifying longer-term impact, that is whether or not any improvements are sustained after they leave, and pick up any deterioration against these indicators.

• When considering system-wide impacts, it will be extremely important to know the financial stability of the trusts under the SMQ/CP regime. We have looked at this by using a simple scale measure for such stability before, during and after the regime. However, it would be equally important to link this indicator to changes in direct/indirect costs and also to additional opportunity costs that are raised while a trust is in the SMQ or CP regime. It was not possible to make this link retrospectively, but this could be accounted for if all such indicators are to be part of routinely reported data.

General introduction and background

• To begin, could you tell me about your professional background? [PROMPT]

  • How long have you worked at DHSC/NHS Improvement?

  • How long have you been in this particular role?

• What are your main areas of responsibility? [PROMPT]

  • Do you have specific responsibilities in relation to NHS provider performance and/or quality?

• In terms of this organisation’s governance, what committees or teams do you report to and how often is this? [PROMPT]

  • Board.

  • Senior officials.

The Special Measures Regime: general (20 minutes)

We are looking at the special measures regime for quality and the impact of different types of interventions delivered within trusts by NHS Improvement.

• What do you see as the main aim of the Special Measures programme for Quality?

• Could you describe the typical process when NHS trusts enter Special Measures for Quality?

• Do you have a view on why some NHS trusts fall into the special measures regime for quality? [PROMPT]

  • Are there warning signs that you have observed in practice?

• When a trust has entered Special Measures for Quality, how does NHS Improvement work with other organisations such as the CQC to support that trust to improve? [PROMPT]

  • Do the CQC and NHS Improvement work jointly together to share information? If so, how does work in practice and what data or information is shared?

  • Do you collect specific data for trusts in Special Measures for Quality – beyond that which is already routinely collected in the NHS? If so, please explain.

  • Does NHS Improvement also engage local commissioners or other local health system leaders when a trust enters Special Measures for Quality?

• Why do you think some NHS trusts struggle to leave Special Measures and require external support for a longer time, for example over 18 months?

• Could you describe the process for identifying when a NHS trust is placed on the ‘challenged provider’ list?

• Do you have a view on why some NHS trusts end up on the ‘challenged provider’ list but avoid entering Special Measures?

Policy and interventions (NHS Improvement)

We would like to understand more about the specific quality support mechanisms or interventions issued by NHS Improvement to trusts that enter the Special Measures regime for Quality or are placed on the challenged provider list.

• What are your perceptions of the interventions used by NHS Improvement to improve quality in NHS trusts placed in Special Measures for Quality, such as:

  • appointment of an ID

  • buddying with another (higher performing) trust

  • access to funds to spend on quality improvement

  • senior executive and/or board leadership change.v

• What are the aims of each of these different mechanisms?

• What outcomes or impacts do you expect from each of these? [PROMPT]

  • How are these achieved locally?

• What is the process for choosing the types of support mechanisms or interventions that trusts receive? [PROMPT]

  • To what extent are trusts involved in these discussions?

  • Is the process different for challenged and special measures trusts?

• In your opinion, is it better to deliver some of these interventions or support mechanisms together – as a bundle of support for example? [PROMPT]

  • Do you have a sense of which intervention – or combination of interventions – work well in NHS organisations in Special Measures or challenged? Why is this?

• Do you know where the specific interventions used in the Special Measures regime originate from, such as the policies or frameworks that have encouraged their use over time?

• Do you observe any of the improvement interventions listed above to work better in certain types of trusts or contexts than others? [PROMPT]

  • ambulance or mental health trust or an acute trust

  • rural or urban areas

  • small or large trusts

  • different organisational cultures or health system contexts

• In your experience, how do NHS organisations receiving the NHS Improvement support interventions tend to respond?

  • Does their response change over time?

• What are the common difficulties or barriers that NHS organisations face when in Special Measures for Quality and implementing these support interventions?

• Has the implementation of these interventions ever led to any unanticipated negative outcomes in any NHS trusts?

• What types of actions do you think NHS leaders, CEOs and the board should undertake to exit Special Measures or leave the challenged provider list and sustain exit?

• Do you have any other comments about the role of CEOs and the board in supporting health-care organisations to promote quality improvement over time? [PROMPT]

  • For example, what organisational skills or capabilities they should seek to develop?

• Are there any other quality improvement interventions or NHS policies that we should be aware of in this study and support quality improvement in challenged NHS trusts or those in Special Measures?

• Is there anything you would like to add that we have not covered?

General introduction and background

• To begin, could you tell me about your professional background? [PROMPT]

  • How long have you worked at the CQC?

  • How long have you been in this particular role?

• What are your main areas of responsibility?

• In terms of this organisation’s governance, what committees or agencies do you report to and how often is this? [PROMPT]

  • Board.

  • NHS agencies or central government (e.g. NHS Improvement/NHS England/DHSC).

The Special Measures Regime: general

We are looking at the special measures regime for quality and the impact of different types of interventions delivered within trusts by NHS Improvement.

• What do you see as the main aim of the Special Measures regime for Quality?

• Could you describe the typical process that occurs when NHS trusts enter Special Measures for Quality, from a regulatory perspective? [PROMPT]

  • How is a decision reached about whether to place a trust in Special Measures for Quality? For example, if it receives an ‘inadequate’ rating.

  • What evidence or type of informs the decision?

• Once a trust enters Special Measures for Quality, how does the CQC work with other organisations in the health system to make improvements at that trust, for example:

  • How do the CQC and NHS Improvement work jointly?

  • What information is shared?

• How does the CQC monitor a trusts performance after they are placed in Special Measures for Quality? [PROMPT]

  • Do you collect specific types of data for trusts in Special Measures for Quality to track improvements?

  • Does a trust have to collect or send specific data to the CQC?

• Does the CQC follow up at intervals with the trust?

• Could you explain how the process of re-inspection of a trust in Special Measures works in practice?

• Does the CQC also have a role in identifying NHS trusts on the ‘challenged provider’ list? [PROMPT]

  • [If so]: how does the CQC work with NHS to address quality issues in challenged provider?

• Does the CQC have a role in decisions about whether, and when, a trust exits or stays in Special Measures for Quality?

  • Does the CQC have a role in decisions about whether and when a trust leaves the Challenged Provider ‘watch list’?

Specific interventions

• What are your perceptions of the interventions used by NHS Improvement to improve quality in NHS trusts placed in Special Measures for Quality, such as the following: [PROMPT]

  • Appointment of an ID.

  • Buddying with another (higher performing) trust.

  • Access to funds to spend on QI.

  • Senior executive and/or board leadership change.

  • [PROMPT]: Any others not listed?

• Do you know where the specific interventions used in the Special Measures regime originate from – such as which policies or frameworks have encouraged their use over time?

• What outcomes or impacts do you expect from each of these?

• Do you have a sense of which type of intervention – or combination of interventions – works well in NHS organisations with quality issues? And why?

• Do some quality improvement interventions work better in certain types of trusts or local health systems than in others? [PROMPT]

  • ambulance or mental health trust or an acute trust

  • rural or urban areas

  • different organisational cultures or health system contexts?

• In your experience, how do NHS organisations receiving the NHS Improvement support interventions respond initially?

  • Does their response change over time?

  • What are some of the barriers they face when implementing these interventions?

• In your view, do the interventions lead to sustainable quality improvement in trusts over time? [PROMPT]

  • In your view, how attributable are improvements in quality to the NHS Improvement interventions discussed above?

• Has the implementation of these interventions ever led to any unanticipated negative outcomes in any NHS trusts?

• What types of actions would you recommend trust CEOs and boards undertake to improve quality and exit Special Measures permanently?

• Do you have any other comments about the role of NHS system leaders, CEOs or trust boards in supporting health-care organisations to promote quality improvement over time? For example, the organisational skills or capabilities they should seek to develop.

• Is there anything you would like to add that we have not covered?

Introduction/warm-up

• What is your role in this organisation?

• How long have you worked here for?

• To confirm, were you working in this organisation when it was placed on the challenged provider list?

Special Measures for Quality/Challenged Provider: background

• Could you describe any specific quality issues that have been previously identified in this trust in CQC inspections?

• Have any other issues been identified in this trust by NHS Improvement or the Department of Health and Social Care?

• Was this during a visit from a regional team or central team?

• Do you think it was necessary for this organisation to be placed in SMQ/CP?

• What has been the short-term impact of being placed in SMQ/CP?

• How did staff respond to the news?

• Was there an impact on recruitment or workforce retention?

NHS Improvement interventions

• [Unprompted]: Has this organisation received support from NHS Improvement to make changes in performance and quality?

• [If yes, please describe]

• When did this begin?

• [Prompted]: Has this organisation received any of the following interventions from NHS Improvement?

  • appointment of an Improvement Director

  • appointment of a ‘buddying’ trust

  • funding dedicated to quality improvement activities.

• [For each of the above that is mentioned] Do you recall when this intervention began?

• [For each of the above that is mentioned] Do you recall for how long the intervention was received?

• What was your experience of the intervention?

• Do you think the intervention(s) supported this organisation to make changes? [If so] how?

• Have changes in senior leadership or governance been recommended by NHS Improvement, an ID or the regulator?

• Did an ID develop a quality improvement plan with this organisation? [If so]:

  • Was this plan helpful?

  • Were any recommendations unhelpful?

  • Were any recommendations particularly challenging to implement?

Deployment of resources to deliver QI and opportunity costs

• Have there been direct costs to this organisation of being placed on in SMQ/CP?

• Have there been direct costs to this organisation to implement the kinds of improvements recommended by NHS Improvement or the CQC? If so, could you provide some examples?

• [If not discussed above]: Has this organisation applied for QI funds from NHS Improvement to drive quality improvement locally?

• If so, how was this money allocated locally? What was it used for?

• Has this organisation changed how you allocate staff time or resources in the organisation?

• Do any new activities prompt a trade-off in terms of staff time or resources?

Quality improvement knowledge and organisational capabilities for QI

• Have you applied any quality improvement methodologies in this organisation? For example, Lean, Virginia Mason, PDSA cycles, IHI.

• Do you use any other management tools or information systems to monitor quality improvements in the organisation?

• Have you started to collect any new types of data to monitor changes in performance since being placed in SMQ/CP?

• Have you implemented any service innovations to support better outcomes, quality or patient experience since being placed in SMQ/CP?

• Has this organisation brought in external management consultants or external experts to support performance or quality improvement? If so, did this bring about any changes locally?

• Does the organisation engage with patients and the public around quality issues? If so, how does this work in practice?

Local health economy and stakeholders (outer context)

• Does this organisation have any important strategic or collaborative partnerships with other local providers or commissioners?

• If so, how does this arrangement work in practice?

Wrap up questions

• On reflection, what has been most helpful to this organisation to support any changes in quality and performance over time? [PROMPT]

  • funding

  • NHS Improvement interventions

  • leadership/governance changes

  • other.

• What advice would you give to other health-care organisations seeking to improve their performance and avoid entry into SMQ/CP?

• Anything else we should be aware of or that you would like to add?

Preliminaries

• Ensure participant has reviewed information sheet and signed consent form/sends digitally in case of telephone interview.

• Explain plans for feedback and dissemination.

• Seek permission to record.

Questions