Evaluation of a national surveillance system for mortality alerts: a mixed-methods study

Inconsistent relationship between mortality measures and quality of care

An often cited argument put forward by critics12,15,16 against the validity of mortality measures is that there is poor agreement between process measures and mortality outcomes. One widely quoted ‘systematic review’17 reviewed 36 studies and 51 processes of care and found a correlation between better quality of care and lower risk-adjusted mortality ‘in under half the relationships (26/51 51%) but the remainder showed no correlation (16/51 31%) or a paradoxical correlation (9/51 18%)’. 17 The authors concluded that the relationship between risk-adjusted mortality and quality of care was neither ‘consistent nor reliable’. Aside from an obvious point that 51% is not ‘under half’, the study undertakes no review of the quality of papers and makes no comment on sample size. Many of the studies in which no relationship was found were simply too small to detect an effect with any statistical significance. Additionally, it is entirely plausible that some process measures of care either are unrelated to mortality or have a relationship too small to detect without detracting from the validity of hospital mortality measures as an indicator of quality of care. Despite the weaknesses of the review, the paper has been cited many times in support of criticisms of the validity of mortality measures, even though there were a substantial number of studies that found a relationship. In our study, we have looked for relationships between some structure and process measures (agreed a priori in the study protocol) and alerting and non-alerting trusts.

Avoidable deaths

Although we have never made the claim that the MASS is a measure of avoidable deaths, it has been suggested that ‘avoidable deaths’ identified through case note reviews might be a more useful tool for exploring quality of care issues in relation to death; they have been proposed,14 and are being advocated,18 to be used routinely in NHS organisations. Case note reviews are by no means a gold standard. Studies trying to identify avoidable deaths find only low to moderate levels of agreement between reviewers19–21 with typical intraclass correlation coefficients in the range 0.4–0.6. Higher rates of reliability can be achieved by setting strict criteria for the reviewers as to what should be considered avoidable. However, the stricter the criteria, the more likely that important cases will be excluded. In one study,22 125 categories of avoidable death were listed, but there were still events that qualified as avoidable deaths that did not fall within them.

There is also some evidence that reviewers of case notes are less likely to classify avoidability among more sick and vulnerable patients. In a recent retrospective case note review of 1000 deaths, the observation was made that the fact that ‘patients were more likely to experience a problem in care if they were less functionally impaired, were elective admissions and had a longer life expectancy on admission was inconsistent with studies in other countries and might reflect a bias among reviewers towards discounting problems in the most frail, sick patients’. 23

The idea of avoidability as determined through a case note review is limited when considering the impact of quality of care on mortality. As an example, a study examining hospital admissions resulting from acute myocardial infarction (AMI)24 found that in-hospital mortality was lower with shorter door-to-needle times. Patients waiting > 45 minutes for fibrinolytic therapy had 37% higher odds of death than those waiting ≤ 30 minutes. There was no cut-off point at which delay becomes fatal, but on average a patient who waits longer for treatment is less likely to survive than one who is treated in < 30 minutes. It is rarely possible to say for any given patient that extra time waited contributed to their death. However, we know that, in aggregate, across a group of patients who wait longer, more of them will die and that some of them would have survived if treatment had been provided earlier.

Hogan et al. 14 attempted to relate estimates of avoidable deaths to HSMRs. Using only one reviewer per case, they looked at 34 hospital trusts, and reviewed 100 case notes in each. With an average of just over three ‘preventable’ deaths per hospital, with such small numbers, it is not surprising that they found no significant difference in the proportion of avoidable deaths across the trusts they examined. However, despite this, Hogan et al. 14 go on to examine the relationship of the tiny numbers of avoidable deaths in each trust with HSMRs. It is, again, no surprise, given the small numbers, that no significant relationship was found. In fact, one statistician commented that, given the study weaknesses, ‘the authors were trying to detect not just an implausibly large, but an impossibly large effect size’. 25 However, Hogan et al. 14 then make the bold assertion that ‘the absence of a significant association between hospital-wide standardised mortality ratios and avoidable death proportions suggests that neither metric is a helpful or informative indicator of the quality of a trust’. 14 As the same statistician commented, ‘robustly inferring the validity of hospital mortality indicators based on the findings of this study is impossible’. 25 To make any conclusion about the quality of other indicators based on case note reviews, particularly involving such small numbers, would seem to go way beyond the evidence in hand.

Weak signals

It is the low proportion of preventable deaths arising from these case note reviews that critics of mortality indicators use to argue that there are too few preventable deaths to account for the degree of variation in HSMRs. The extent to which any indicator is useful depends on the degree of ‘signal’ versus ‘noise’. With mortality measures, noise could be due to random variation or due to other factors such as unadjusted for case-mix or coding issues. The mortality alert surveillance system attempts to tackle the problem of random variation by setting high alarm thresholds, giving a very low false alarm rate as a result of simple random variation (probability of false alarm is < 0.001 over a 12-month period). Despite this high threshold, signals still arise.

The signal-to-noise ratio relates to the positive predictive value of an indicator. The true positive predictive value for HSMRs is unknown. There is some indication in the Keogh mortality review, where, of the 14 trusts inspected for quality of care on the grounds of high mortality rates by Sir Bruce Keogh in 2013,26 11 were regarded as being sufficiently poor to warrant being placed in special measures. This would equate to a positive predictive value of 0.79. This evidence is anecdotal and relates to HSMRs rather than to specific diagnosis and procedure mortality alerts. Part of the motivation to carry out this National Institute for Health Research-funded study was to go beyond anecdote and examine, more systematically, the relationship between mortality alerts and other external indicators, and to look at the findings and impact of our alerting system.

Case mix and coding

Case-mix adjustment based on administrative data has been criticised12,13,27 because for some variables it depends on the accuracy of clinical coding secondary diagnoses for comorbidity and palliative care. Some potentially valid theoretical concerns have been raised, particularly but not exclusively regarding case-mix adjustment, but little robust quantitative evidence has been presented as to the size of their effect. 28 The well-cited paper by Mohammed et al. ,27 which points to these methodological biases (specifically in HSMRs), uses a hypothetical example to illustrate the potential effect size. The paper focuses on at least two mechanisms that might contribute to this so-called ‘constant risk fallacy’: the first involves differential measurement error and the second involves inconsistent proxy measures of risk. In their analysis of real data, the authors argue that inconsistencies in risk of death for emergency admissions support their assertion. They suggest that large variations in the proportions of elective/non-elective patients with zero length of stay indicate that systematically different admission policies were being adopted across hospitals. Unfortunately, their data seemed to erroneously include day cases, which explains the low crude death rates and mean length of stay and affects the proportion of admissions that are emergencies. A paper published subsequently29 attempted to actually quantify the effect of coding and other concerns on the HSMR, including palliative care, short-stay observation patients, multiple admissions per patient and a failure to capture post-transfer or post-discharge deaths. The study concluded that, despite theoretical concerns about bias, HSMRs in most hospital trusts were in practice usually only modestly affected by the differences in the predictive models incorporated into this sensitivity analysis. There were a few notable exceptions, particularly relating to palliative care and timelier linkage with out-of-hospital deaths, although the paper also referred to improved coding guidelines that may address some of these issues.

Given the potential limitations to administrative data, the argument has been made for making better use of the increasing number of specialised sources of data, in particular those used to support national clinical audits. 13 Our group has carried out comparisons of administrative data with clinical audit data in the past, comparing risk prediction models and quality. One study compared the performance of risk prediction models for death in hospital based on administrative data with published results based on data derived from three clinical databases. 6 This concluded that routinely collected administrative data can be used to predict risk with similar discrimination to clinical databases, and that could be used to adjust for case mix for monitoring health-care performance. Two other papers looked at coverage and completeness of surgical procedures in colorectal surgery30 and vascular surgery5 in two clinical audit databases compared with hospital administrative data. They found between twice and four times as many cases recorded in the administrative data as in the audit data sets, suggesting significant under-recording. They also found that 39% of patients within the colorectal cancer database had missing data for risk factors used for risk adjustment. Although these papers were published in 2007 and 2008, more up-to-date reports from even well-established and reputable audits such as the Myocardial Ischaemia National Audit Project (MINAP) suggest that poor data quality, particularly on patient characteristics, renders the risk adjustment of mortality rates unreliable. 31 Given that the accuracy of administrative data appears to be improving and that clinical coding is now regarded as predominantly of good quality,32 the argument for abandoning it in favour of much more expensive, and often less complete, audit data seems rather less tenable.

Search strategy

Our search terms (within the title and abstract) were Hospital AND (Monitor* OR Surveillance) AND (Death OR Mortality). We adapted our search for MEDLINE, The Cochrane Library and Web of Science. Literature was also retrieved using article references and citation tracking.

Data collection, analysis and reporting

We screened initially on the title; a second screen was on the abstract and the final screen was on the full papers. One reviewer carried out screening, a second reviewer checked the final screen and both reviewers reached a consensus on which studies to include in the review.

Mortality monitoring methods

There are two types of methods reported in the literature for monitoring hospital mortality. The first uses a cross-sectional measurement of condition-specific standardised mortality ratios2 and HSMRs (all-cause mortality) in Canada,55 the Netherlands57 and Australia. 60 Using this method, assessment is achieved through benchmarking, whereby hospitals are assessed against set standards or using interhospital comparisons. The Australian Institute of Health and Welfare commissioned a report on measuring and reporting mortality in hospital patients to determine whether or not it was possible to produce accurate and valid indicators of in-hospital mortality using Australian administrative data. 58 Risk-adjusted modelling predicted or explained the variation in mortality rates to a similar extent as in international literature, and authors concluded that high or rising HSMRs could be used to signal a potential problem within a hospital.

The second method investigates cumulative measures over time using statistical process control charts. Steiner et al. ,71 in 2000, introduced the idea of using risk-adjusted CUSUM charts for monitoring mortality in patients undergoing cardiac surgery. Since then, there have been several international papers investigating the use of control charts to monitor the quality of hospital care in other settings;56,62,66,68 however, these studies were on clinical audit data and not on administrative hospital data. Bottle and Aylin35 published a paper in 2007 that investigated the use of statistical control charts for monitoring clinical performance, including in-hospital mortality. The methodology described was the basis for the national surveillance system for mortality alerts that is evaluated in this report. Pilcher et al. 61 investigated whether or not the use of a statistical process control chart could have detected the high death rate in the ICU at Bundaberg Base Hospital, around which there was considerable publicity because of perceived excess mortality between 2003 and 2005. The authors concluded that the methodology was able to detect fluctuations in relative risk of death in the Bundaberg ICU that were not apparent when standardised mortality ratios were used.

Validation

Two papers used the methodology to identify hospitals in which there were reported failings in quality as outliers. The paper by Aylin et al. 2 examined paediatric cardiac surgery and confirmed high mortality rates identified through surgical audits, and a large public inquiry, while Pilcher et al. 61 investigated the use of statistical process control chart methodology compared with previous reports of high death rates at an ICU, although it is not clear from their paper how the initially high rates were determined. Hogan et al. 14 attempted to use case note reviews as a gold standard. They reviewed 100 case notes of deceased patients in 34 hospital trusts. They found an average of just over three ‘preventable’ deaths per hospital. The authors then went on to examine the relationship of these small numbers of avoidable deaths in each trust with HSMRs. The study was woefully underpowered and no significant relationship was found. Despite this, the authors claimed that ‘the absence of a significant association between hospital-wide SMRs [standardised mortality ratios] and avoidable death proportions suggests that neither metric is a helpful or informative indicator of the quality of a trust’. 14

Evaluations

Many health-care systems internationally use routine administrative data to monitor quality indicators, including hospital mortality. Examples of these are the analytic toolkits, provided by commercial companies such as Dr Foster (UK), Telstra (Melbourne, VIC, Australia; www.telstra.com.au) and 3M (Maplewood, MN, USA; www.3m.com). However, we did not find published literature on evaluations of these toolkits.

Clinical audits (or databases) have been used for a number of years to monitor outcomes such as mortality in a range of settings and conditions, but particularly in surgery. 43,45 Audits involve data collection and analysis that is time-consuming and expensive to compile. However, we identified one evaluation of a study using clinical audit data in our literature search. The study was on the American College of Surgeons National Surgical Quality Improvement Program^® (ACS NSQIP^®). 64 ACS NSQIP collects data from participating hospitals and analyses them, and participating hospitals are provided with reports that permit risk-adjusted comparisons, including 30-day mortality, with a surgical quality standard. Cohen et al. 70 carried out an ‘evaluation’ of the surgical outcomes over time, and found a decreasing trend in mortality in 62% of participating hospitals and an average of 0.8% annual reduction in mortality rate. There are, however, limitations to this study. There are no control (non-ACS NSQIP participating) hospitals, and the reported trends could have been influenced by secular trends unrelated to ACS NSQIP. The authors do acknowledge this limitation, yet they still classify the article as an evaluation.

Acute bed occupancy data

The bed availability and occupancy data are published quarterly by NHS England and identify, for each NHS Health Care Provider, the number of bed-days available for patients to have treatment or care. 133 For our analysis, we focused on the percentage of beds occupied over a 3-month period (a quarter) in general and acute wards. We analysed data for the calendar years 2011 to 2013, using the fourth-quarter data from the financial year 2010/11 and excluding the fourth-quarter data from the financial year 2013/14.

Nurse-to-bed ratio

NHS Workforce Statistics are available from NHS Digital. 134 The monthly publication is an accurate summary of the validated data extracted from the NHS’s human resources and payroll system. The data include full-time equivalent figures for all NHS hospital staff groups working in England. We used the mean number of full-time equivalent qualified nursing staff (over a 3-month period) as the numerator and quarterly bed availability as the denominator for each NHS trust. We analysed data for the calendar years 2011 to 2013.

Trust financial data

Annual financial NHS foundation trust data are provided by Monitor135 and we downloaded data for financial years 2011/12–2013/14 from the UK government information website. 17 For NHS trust financial data, we put in a freedom of information request to the Department of Health (reference 1041857) for the same years. As the two data sets are compiled from difference sources, we could not directly combine the data sets. We therefore created a binary variable to determine whether or not a trust was in deficit in a given year. We used the value for ‘surplus before impairments’ to determine whether or not a foundation trust was in deficit.

General Medical Council National Training Survey

The National Training Survey is carried out by the General Medical Council. 136 The aim of the survey is to ensure that medical education and training is meeting the standards set to support high-quality medical care and patient safety across the UK. In 2012, 51,316 doctors in training completed the survey out of 54,035 who were eligible, giving a response rate of 95.0%. The survey data cover over 12 questions or indicators, within 100 specialities/departments, in all NHS trusts in England. Data are available at a trust level from 2012/13; we used 2012/13 and 2013/14 data. Mean percentage satisfaction scores were presented only if there were more than three trainee respondents for a specific specialty and question within a trust. We created a weighted trust score across all indicators and all departments for each trust by year. To weight the hospital score, we needed to estimate the number of trainees (N) responding to a question. From the provided mean and 95% confidence interval (CI), we could calculate the standard error of the mean (SE):

We were also provided with the standard deviation (SD), and using:

we could calculate the number of trainees as:

The weighted mean is the sum of the partial means, which was calculated as the mean multiplied by the number of trainees (for a trust/specialty/question) divided by the total number of trust trainees:

where k denotes a single question answered by trainees within a single specialty for each trust.

NHS Litigation Authority

The NHS Litigation Authority is a not-for-profit part of the NHS that handles negligence and other claims against the NHS in England. 137 The NHS Litigation Authority conducts a risk-management assessment within each hospital and awards a rating basis against a set of risk management standards. The highest risk management assessment a trust can achieve is a level 3. The NHS Litigation Authority has been publishing factsheets on risk management assessment levels since 2002/3. The factsheets include data on the trust, the date of the last assessment and the level achieved. We used data from 2011/12–2013/14 and dichotomized trusts’ risk assessment achievement into those that had a risk management assessment level 1 versus those with levels 2 or 3 for each quarter over the 3 years.

Myocardial Ischaemia National Audit Project

Myocardial Ischaemia National Audit Project is the national clinical audit of the management of heart attack. 138 MINAP captures information on all patients with a heart attack directly after treatment. All providers audited have common definitions of clinical important variables and common standards of good quality. In our analysis we focused on percutaneous coronary interventions (PCIs). National and international guidance recommend that in the emergency treatment of patients with ST-elevation myocardial infarction, a blockage in one of the heart’s major arteries, primary PCI should be performed within 90 minutes of arrival at the heart attack centre. Not all hospitals have the facility to perform primary PCI. We were unable to obtain information on which trusts provide PCI (independent of MINAP); we therefore included all trusts that provided primary PCI data in our analysis and assumed that the data were complete. We focused on the quality indicator ‘proportion of all patients who received primary PCI within 90 minutes from arrival at the heart attack centre’. We analysed data for years 2011/12 and 2012/13.

National Inpatient Survey

The NHS Patient Survey Programme is run by Picker Institute Europe on behalf of the CQC. 139 The views of patients about the care they have recently received from all NHS health-care providers in England have been systematically gathered since 2005. Eligible patients for the survey were aged ≥ 16 years, had at least one overnight stay and were in hospital between June and August of the collection year. The questions cover four specific areas on access and waiting; on safe, high-quality co-ordinated care; on better information and more choice; and on building closer relationships. All questions are collated to give an overall patient experience score. We used annual National Inpatient Survey data for years 2011 and 2012.

NHS Safety Thermometer

The NHS Safety Thermometer is a tool designed to support patient safety and improvement. The tool focuses on four harms for which there is clinical consensus that they are largely preventable through good-quality patient care. These harms are pressure ulcers, falls, urinary tract infections in patients with a catheter, and new venous thromboembolisms. The prevalence of patient harms is recorded within each health-care setting, with the aim of providing information for performance monitoring. 140 NHS Safety Thermometer data are supplied by NHS Digital and are available, as monthly reports, from April 2012. For our analysis, we included all harms within an acute hospital ward setting, using data for the financial years 2012/13–2013/14.

Summary hospital-level mortality indicator

Summary hospital-level mortality indicator data were supplied by NHS Digital. 141 These annual standardised mortality statistics are generated using HES provider spells linked to the Office for National Statistics mortality data. The SHMI is a ratio of the observed number of deaths [the total number of finished provider spells for the trust that resulted in a death either in hospital or within 30 days (inclusive) of discharge from the trust] to the expected number of deaths for a trust adjusting for patient case mix. The SHMI risk model adjusts for age, gender, admission method, year index, Charlson Comorbidity Index and diagnosis grouping. 33 We used data for the financial years 2011/12 to 2013/14. There were no SHMI data for specialist trusts such as Birmingham Women’s and Children’s NHS Foundation Trust.

Hospital-standardised mortality ratio

Hospital-standardised mortality ratio data were accessed from Dr Foster Hospital Guides. 34 These annual standardised mortality statistics were generated by Dr Foster Intelligence using HES provider spells. The HSMR is a ratio of the observed number of deaths (the total number of finished provider spells for the trust that resulted in a death) to the expected number of deaths for a trust calculated from a risk-adjusted model, multiplied by 100. The risk model adjusted for various factors as in Table 4. 25 Data were available at an annual trust level for the financial 2011/12 and 2012/13.

Descriptive analysis of Imperial College mortality alerts

We investigated trends in alerts and frequency of alerts for specific diagnoses and procedures from the Imperial College alerts generated between April 2007 and December 2014.

We classified hospital trusts according to the number of alerts that they received between 2007 and 2014: single alerting trusts received one alert and multiple alerting trusts received two or more alerts.

Descriptive analysis of Care Quality Commission investigations into the Imperial College mortality alerts

Copies of all of the alerts, including those not sent to trusts, are sent to the CQC, the regulator of health and social care in England. The CQC has been running its own mortality alerting system using HES. 36 On the basis of either alerting system, the CQC may write to a trust to request a trust-level investigation and response. 36

On receipt of a trust response, a panel of clinicians and analysts reviews the information received. At this stage the CQC may close the case (and request follow-up by its regional teams) or request further information. The process chart outlining the CQC’s follow-up of mortality alerts is shown in Figure 5.

FIGURE 5.

Process chart outlining the CQC’s follow-up of mortality alerts.

We systematically reviewed the documents held by the CQC for each mortality alert generated by Imperial College for a subset of total alerts between 2011 and 2013. This time period was chosen to enable us to investigate the CQC’s paper trail with the resources that we had available and, additionally, to allow time at the end of the period for investigations to be completed. The CQC documentation data were collected and compiled at the CQC site by one reviewer to protect the potentially sensitive information contained within the documents. Each alert leads to substantial communication between the CQC and the trusts, and the CQC checked the data collected before releasing them to Imperial College. Initially, the data collection comprised up to 121 pieces of data but, as we progressed with the data collection, we were able to limit the data to the 44 most relevant pieces from four document types during the collection process. Table 6 summarises the data and documents available at the CQC that related to mortality alert reviews.

Summary information was predominantly located in documents used in the CQC panel meeting ‘CQC assessment of response’. The findings from trust investigations, summarised by the CQC analysts, were used to classify the trust’s conclusion as to the reasons for the alert. When the CQC analysts noted that trusts had identified deficiencies, we included these findings in our data collection. The broad classifications of findings were (1) no improvement required, (2) case mix, (3) coding, (4) care and (5) care and coding. When CQC analysts had not summarised the findings from a trust, the reviewer classified the trust response using information taken directly from ‘trust response’ documents. If the trust referred to plans to improve care or coding, then these were used to classify findings. The classifications of findings were subjective, but all decisions were made by one reviewer. Trusts did not always assess the likelihood of the findings affecting the outcome (the alert). When there was a reference to an outcome potentially being affected, we recorded this.

We developed a data collection form in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA), which we piloted and refined during the data collection. Cases were eligible for inclusion if the case was opened by the CQC in response to an Imperial College mortality alert (i.e. not in response to a mortality alert generated internally at the CQC). We checked the quality of the data at various points in the data collection. As letter dates were predominantly sequential (i.e. trust responses should follow CQC data requests), dates were checked regularly to identify non-sequential letters. The final collated data set was shared with the CQC, which had an opportunity to comment.

We assessed trust findings across all diagnosis/procedure groups and focused on sepsis alerts. The number of AMI alerts between 2011 and 2013 was too small to assess separately.

Changes to coding following an alert

After receiving an alert letter, internal investigations are likely to occur, alongside an investigation by the CQC. We assume that an alerting trust will review its coding as an important mechanism in the investigation process. On the basis of these investigations, HES data submitted at the time of the alert may be recoded and resubmitted by the trust. We analysed coding changes for both AMI and sepsis alerts, comparing data contained within an alert letter (based on provisional monthly data) with data from the annual HES publication. Each alert letter indicates the trust and disease group triggering the alert; the defined period before the alert (covering 1 year); the number of admissions; the number of deaths (observed); and the relative risk of death over the defined period. We examined sent letters dated between April 2011 and December 2013 for AMI and sepsis, and extracted data from each alert letter. Using HES (annual publication), we extracted data covering the same date period and condition for those trusts that received mortality alerts. The risk adjustment model used to calculate relative risk was the same for both provisional and annual HES extracts. We described the coding changes between alert letter and annual HES data as a percentage change for admissions, deaths and relative risk.

Two calculations have been used to describe changes in the data:

1. absolute change = new data – old data

2. relative change = (new data – old data)/old data.

Controlling for false discovery rate

Our investigation of mortality alerts with other measures of quality involved the analysis of 11 outcomes. Multiple significance testing increases the probability of false-positive findings as a result of random chance alone. This is the false discovery rate (FDR). We controlled for the FDR using methodology described by Benjamini and Hochberg. 142

1. We ranked p-values (P) from each significance test from smallest to the largest: P₁ ≤, P₂ ≤ . . . ≤ P_m (where m is the total number of significant tests).

2. We denoted H(i) as the null hypothesis corresponding to P(i) (where i is the rank number).

3. We rejected H(i) if P(i) ≤imq∗ (where q* is 0.05, our set threshold for the p-value).

Statistical methods

We used an interrupted time series design. Interrupted time series is the strongest quasi-experimental approach for evaluating longitudinal effects of interventions. 143 The design estimates a trend before an intervention, the impact of an intervention and the change in trend after the intervention. The start of the intervention was taken as the month of alert and the month after a 9-month lag. The 9-month lag is the hypothesised minimum time for a hospital trust to receive the mortality alert letter, to investigate the alert and to bring about change within the hospital setting, and was in part supported by workstream 2 findings.

After a high cumulative mortality rate, resulting in a mortality alert, rates may fall as a result of the phenomenon of regression to the mean. 137 Our analysis needed to differentiate between an immediate fall in trust mortality relative risk resulting from regression to the mean and a genuine fall resulting from improvement of care or amendments to coding. To achieve this, our investigation employed three models, graphically displayed in Figure 6.

FIGURE 6.

Interrupted time series analysis models. (a) Two parameters: a, pre slope and b, post-slope factor; (b) three parameters: a, pre slope, b, level change, and c, post-slope factor; and (c) three parameters: a, pre slope, b, level change (after lag period), and c, post-slope factor.

Model 1 (broken stick model) measured two parameters: the increasing slope in relative risk before a mortality alert (measured as a monthly increase) and the change in slope after. We restricted the data to 12 months before and 9 months of the lag period (the time we assumed the trust was investigating and implementing improvements to care or amendments to coding).

This model estimates the log relative risk as:

Model 2 measured three parameters: the increasing slope in relative risk before the mortality alert, a level (step) change after the mortality alert and the subsequent change in slope. Again, we restricted the data to 12 months before and the 9 months after an alert.

This model estimates the log relative risk as:

Model 3 ignored trends in the lag phase and measured three parameters: the increasing slope in relative risk before the mortality alert, a level (step) change after a 9-month lag period and the slope following the lag period.

This model estimates the log relative risk as:

(Data are excluded during the lag period.)

A more detailed description of the models’ parameters is given in Appendix 3.

A 12-month pre-alert increase was calculated as:

A monthly relative risk determining the trend after alert was calculated as:

We hypothesised that the decline in relative risks would be steeper after a lag period (model 3), when trusts have instigated improvement to care or amendments to coding, than during the lag period (models 1 and 2), when any decline would be influenced by a regression to the mean.

We modelled the data using generalised estimating equations based on a Poisson distribution. This semiparametric modelling compensates for the correlation between repeated measures of relative risk from individual trusts over the study period. It also allows for distribution assumptions of the data to be relaxed. Similar methodology assessed against a randomised controlled trial found the results to be concordant. 144 Generalised estimating equations calculates population-averaged parameter estimates.

We investigated all mortality alerts, AMI alerts and sepsis alerts. We carried out sensitivity analyses with lag periods of 3, 6 and 12 months. Further sensitivity analyses investigated crude risk (number of deaths over number of admissions).

Statistical analysis was carried out using the SAS 9.2 software package (SAS Institute, Cary, NC, USA) and Stata version 13 (StataCorp, College Station, TX, USA).

Case study site selection

Using stratified and theoretical sampling methods, case study sites were drawn from trusts alerted in two disparate clinical conditions, presumed to require local institutional responses of varying scope: sepsis and AMI. The target sample was 12 English acute care trusts, six in each condition area. Sites were selected from review of Imperial College’s historic alert data.

Trusts were selected and approached to participate in the study according to two sampling strata: (1) receipt of an alert in either sepsis or AMI in the 2011–14 period (six sites in each category), and (2) receipt of single or multiple alerts (i.e. single vs. repeat alerting sites: three sites within each category for both sepsis and AMI).

Application of sampling criteria 1 yielded 22 candidate sites, comprising six AMI and 16 sepsis. Of the 16 sepsis sites, one trust no longer existed as a result of a merger and was excluded. Of the remainder, eight sites were selected based on examination of a number of additional criteria applied to maximise variation across the final included case studies. These included single or repeat alert status, geographical location, trust size, how recent the last alert was, total number of historic alert letters sent to the site, availability of relative risk data and statistical trend in relative risk, gleaned from fitting interrupted time series models. Regarding the last criterion, time series models were fitted to individual trusts’ monthly relative risk data using segmented regression analysis with interrupts representing alert letter dates. Knowledge of the robustness and direction of apparent changes in trend were used to maximise variation across included sites. The final set of 14 target sites that were contacted with an invitation to participate in the study comprised six AMI sites (three single and three repeat alerting sites) and eight AMI sites (two single and six repeat alerting sites) (Table 7).

Site recruitment

Recruitment and gaining approval for local data collection at the 14 identified sites involved contacting the medical director at each site with an invitation letter and details of the study (see Appendices 4 and 5) before seeking local research and development approval (Integrated Research Application System application) for the site visit itself. Once the research team received a letter of access, a preliminary telephone interview was conducted with the medical director or mortality lead, using a standard pro forma (see Appendix 6), to plan the site visit, including identifying key personnel for interview. Following initial contact, three sites declined to participate in the study, stating resource and personnel availability as the primary reasons.

The final research sample comprised four AMI sites (three multiple and one single alert site) and seven sepsis sites (two single and five multiple alert sites). The additional sampling characteristics of the final research sample are indicated in Table 6. The final sample was therefore achieved through a combination of stratified sampling, maximum variation sampling and, to a lesser degree, opportunistic availability.

Data collection

Workstream 2 qualitative case study data from the 11 included trusts were collected between February and December 2015, representing a total of 22 site visits and 72 separate face-to-face interviews with 73 informants. In addition, the research team undertook 11 preliminary telephone interviews with site leads.

For each site visit, a locally nominated site visit co-ordinator was contacted and key informants were agreed based on prior conversation with the medical director or mortality lead. These informants were to include (1) the chief executive officer (CEO), (2) the medical director, (3) the mortality lead (or relevant governance leads, as appropriate), (4) informatics and coding leads linked to mortality, (5) key leads and informants linked to mortality in the implicated clinical areas and (6) any other informants related to the trust’s programmes and structures relative to mortality reduction (e.g. patient safety, quality improvement and service development programmes). A schedule was developed for the site visit, giving individual interview timings (usually allowing at least 1 hour for each interview).

Before the site visit, invitation letters for participation (see Appendix 7), consent forms (see Appendix 8) and participant information forms (see Appendix 9) were circulated to all interviewees. Participants were assured of confidentiality and given the opportunity to ask questions at the start of each interview before they signed the consent form. Interviews lasted between 30 and 90 minutes (typically 50–60 minutes) and, with the participant’s consent, were recorded for later transcription. A small number (< 10) of participants were unavailable on the day of the site visit and these people were followed up with either subsequent visits or telephone interviews. One participant declined to have their interview recorded. Table 8 provides an overview of the site visit dates and the number of interviews conducted at each site.

Data collection followed standard practices for the conducting of semistructured qualitative research interviews. A comprehensive interview schedule was used as a guide during the interview (see Appendix 10), comprising 24 individual questions with prompts linked to the research questions and review of potential theoretical insights into institutional responses to mortality alerts. The interview started with broad introductory questions designed to frame the conversation and put respondents at ease. Subsequently, a chronological approach was taken to the discussion of the specific instances of alerts, starting with the context surrounding the first alert, the short-term response, longer-term developments following the alert(s) and finally the overall evaluation of mortality alerting from the respondent’s perspective. During the interview, care was taken not to lead the respondent, and a flexible approach was taken to the sequence and nature of topics explored in order to allow novel themes to emerge. Following initial responses to open questions, prompts were used to explore the themes raised in depth and to probe for relevance to mortality alerting mechanisms and processes. The site visit research team comprised two senior health service researchers with substantial experience in qualitative data collection and organisational research, and one junior researcher with expertise in medical sociology. The site visits and interviews were conducted with either one or two researchers present, and one or other of the senior researchers attended each site visit (95% of the interviews were conducted with a senior researcher present).

Data analysis

To fully explore the qualitative data related to institutional responses to mortality alerts at the case study sites, a number of parallel subanalyses were performed to address the main evaluative research questions, to explore themes linked to prior theory and to support the other areas of the study, such as the survey work. Figure 7 provides an overview of how the workstream 2 mixed-methods components and subanalyses inter-relate.

FIGURE 7.

Overview of the qualitative analysis process and main outputs from mixed-methods research, including input from multiple stakeholder perspectives into the interpretation of the cases and their theoretical significance, and the design of evaluative measures. DFI reps, Dr Foster Intelligence representatives.

The main qualitative analysis work centred on the interpretation of the 11 institutional case studies conducted during the course of the programme. It was supported by input from five researchers with backgrounds in psychology, patient safety, sociology and health services research. Once the researchers were familiar with the data set, coding followed standard practices in qualitative research, drawing on grounded theory and framework analysis as methodological frame and the realist evaluative position as epistemological stance. Inductive open-coding was initially undertaken to thematically explore the data set and allow coding categories to emerge from the analysis (i.e. grounded in the data). Hierarchical categorisation was undertaken to begin to organise themes. As the coding framework moved towards data saturation (with fewer novel categories emerging), the coding incorporated increasingly iterative elements and deductive reasoning (i.e. established templates and frames developed in one area of the data set were applied to others), drawing on the constant comparative approach to iterate key definitions and distinctions. The later stages of refinement of coding were subject to repeated review and discussion among the broader research team, review of prior theory and structure imposed through the primary research questions and aims. NVivo software was used to support the initial and hierarchical coding, with some elements of the later stages of iterative refinement undertaken in word processor tables (using Microsoft Word;^® Microsoft Corporation, Redmond, WA, USA).

Once iterative coding and category refinement had been initiated, the research team began to structure the data according to the key outputs required by the project. Adopting a realist evaluative stance, the aim of the analysis was to generate rich descriptions of how the alerts, as a form of feedback intervention, instigated interactions between local context, institutional behaviour and institutional outcomes (objectives 1 and 4). Initially, individual case narratives were produced that integrated testimony from informants within a single site in a historical narrative concerning the institutional response to the target alerts. A standardised template was used for the case narratives, including circumstances leading up to the first alert, short- and long-term responses and parallel evolution of the trust’s work on sepsis/AMI that was not directly attributable to the alert. Throughout the process, clinical, organisational and informatics perspectives were sought that had a focus on historical and factual content. A subsequent step was performing a thematic analysis of each case narrative in which the case was populated with salient quotations from the data set, as well as an interpretative researcher commentary, in preparation for comparative analysis.

Following the development of the individual case narratives, the analysis moved to an integrative mode in order to allow inference and interpretation to be drawn across the whole data set, and used a comparison of the cases to identify important areas of internal/external context and organisational behaviour relative to effective responses to signals in mortality data. Here, an evaluative perspective was gained through the reported perceptions and views of experienced informants concerning what worked in their organisations. The ‘effectiveness’ of specific characteristics was, therefore, inferred through reported links to mechanisms/processes and outcomes that were regarded by informants as representative of adaptive or capable organisational responses to signals in mortality data. The integrative analysis took two principal forms: (1) a cross-case comparative analysis in which case-based observations were combined and interpreted against the core sampling criteria (e.g. a comparison of sepsis and AMI alerting sites and a comparison of single and multiple alerting sites), and (2) the development and validation of an evaluative framework for institutional capacity to respond to signals in mortality data.

The evaluative framework analysis was based on the 73 interviews from the main qualitative study, with additional stakeholder engagement activity. The qualitative analysis approach used was deductive framework analysis so that potential evaluative dimensions for institutional behaviour could be described. In parallel to the interviews, workshops were conducted with Dr Foster Intelligence and the CQC to develop and refine understanding of the mortality alert process and to gain additional perspectives on the emergent findings: specifically, validation of the emergent framework and perspectives related to what constitutes mature and effective institutional arrangements for mortality governance and response to signals in mortality data. Qualitative data were analysed using the framework method, in which both inductive and deductive reasoning were combined to develop and iterate key themes and subthemes. Based on initial open-coding and the early-stage iterative refinement of categories, nine broad themes were identified to serve as a frame for further exploration in the data set. Multiple perspectives were incorporated throughout the analysis process through regular review and discussion across the research team, which consisted of two psychologists, a sociologist and a health-care manager with expertise in patient safety. The resulting thematic analysis was used to structure early versions of the survey instrument used within workstream 2 in order to ensure that response items and scales were both thematically balanced and theoretically grounded in the emerging perspectives on effective institutional responses.

To produce an integrated interpretation from the workstream 2 subanalyses, and to support practical outputs and guidance for future work in this area, the final stage in the analysis was to develop a model of MASSs as a complex intervention. A model framework was developed to accommodate key classes of variables (internal and external context, institutional preconditions, organisational behaviour, and intermediary and target outcomes). The overall approach to sense-making through the workstream 2 qualitative analysis was, therefore, to surface interactions between mechanisms (predominantly organisational behaviour), context (internal structure and external environment) and outcomes linked to institutional capacity for the reduction of avoidable mortality, in accordance with the realist position on complex interventions.

Design of the survey instrument

The survey instrument was developed based on insights from the qualitative case study data and with input from key stakeholders, including Dr Foster site co-ordinators, CQC mortality outlier programme leads, reviewers and analysts, and input from the research project strategic steering group, including feedback from the public and patient representatives. The analysis of the qualitative data was used to inform the development of a structured evaluative instrument. To do this, the core dimensions of the emergent evaluative framework were used to guide the formation of a number of candidate survey items. These items were iterated to address any issues associated with, for example, duplication and wording. This resulted in a number of iterations of the survey, which were continuously built on until the final survey instrument emerged.

The final survey contained both quantitative and free-text items. The quantitative items included categorical, ordinal and scale-based items, while the free-text items allowed for qualitative expansion and further detail. The survey was made up of seven sections, which are outlined in Table 9.

Validation of the survey instrument

A piloting process was undertaken in which a number of key individuals were given the opportunity to comment on and contribute to the development of the survey items and the way in which these items were presented and structured for respondents. A number of methods were employed to support the piloting process, including the cognitive walkthrough/think-aloud method with a local mortality lead, the circulation of the survey by e-mail to key stakeholders and a focused discussion during project advisory and steering group meetings.

Data collection

Survey data collection opened on 11 May 2016 and closed on 10 June 2016. Only one response was required from each invited trust, with the target respondent the trust mortality lead or medical director. An initial personalised contact letter and a paper-based copy of the survey were circulated to all trusts that had received an Imperial College mortality alert (120 trusts). Multiple personalised follow-up e-mails were sent out directly to medical directors, and an electronic version of the survey was made available on request. A total of 78 survey responses were received, which represents a 65% response rate.

Analysis

All survey responses were entered manually into SPSS version 22 (IBM Corporation, Armonk, NY, USA) in preparation for analysis. To ensure that all research questions could be answered, a number of secondary metrics were created based on external data, and these were incorporated into the master database:

• total number of Imperial College alerts sent (2007–16)

• total number of CQC alerts sent (2011–13)

• interval between information request date and first response date for most recent alert in CQC database

• interval between information request date and last response date for most recent alert in CQC database

• total length of investigation for most recent alert in CQC database

• HSMR data (2012)

• SHMI data (2015).

The following steps were then taken, in line with the research questions.

• A missing data analysis was conducted to ensure that there were no significant differences between invited trusts that responded and invited trusts that did not.

• Descriptive statistics were run to categorise and quantify organisational behavioural and contextual aspects of the response to alerts.

• Aggregated scales were computed for specific sections of the survey (including subscales of the evaluative framework).

• Inferential statistics were run to classify and quantify the relationships between type and frequency of alerting, key contextual and organisational behavioural characteristics and intermediate outcomes.

• A simple thematic approach was employed to analyse the free-text data. All data fragments were grouped and categorised in relation to the key research questions.

Overview

AMI-M.03 is a large acute trust with five hospital sites, serving a population of approximately 450,000. The trust has approximately 950 beds. The main site of AMI-M.03 hospital is a tertiary referral hospital. The trust was one of the 14 identified in the Keogh review. In recent years, it has had a new CEO and medical director, as well as a new lead for mortality.

Relative risk profile

Figure 17 displays the relative risk profile for site AMI-m-03 between January 2009 and January 2014. The green lines represent the date of an alert and the blue lines represent the date of the subsequent letter. Table 16 provides the summary.

FIGURE 17.

Time series displaying relative risk profile and alert (green)/letter (blue) dates for site AMI-m.03.

AMI-m.03 thematic analysis

AMI-m.03 received repeat alerts for AMI in September 2011 and April 2012. An analysis is generated from interviews with five members of the trust executive team, plus one specialty manager and one manager responsible for coding, and is informed by trust documentary sources.

Trust preconditions before the first alert

Trust responses to an acute myocardial infarction alert

Trust responses to the second acute myocardial infarction alert

Outcomes of acute myocardial infarction alerts

Parallel evolution of institutional focus on mortality reduction in acute myocardial infarction

In the analysis of data sources, it was difficult to attribute direct causality of the AMI alert. The mortality alerts sent from Dr Foster Intelligence did appear to have some impact on how mortality was organised in the trust, but it is also accepted that trust arrangements also developed in partial response to the Keogh Review and CQC inspections and occurred as natural progression of trust strategy.

Overview

The Sep-s.06 NHS trust is an acute trust serving a population of circa 255,000. It was established in 2014 following a reorganisation of NHS trusts in the area and runs two acute hospitals. The trust received a single alert for septicaemia (except in labour) in October 2012. An analysis is generated from interviews with five members of the trust executive team, one specialty manager and one manager responsible for coding, and is informed by trust documentary sources.

Relative risk profile

Figure 18 displays the relative risk profile for site Sep-s.06 between January 2009 and January 2014. The green lines represent the date of an alert and the blue lines represent the date of the subsequent letter. Table 17 provides the summary.

FIGURE 18.

Relative risk profile and alert (green)/letter (blue) dates for site Sep-s.06.

Sep-s.06 thematic analysis

Trust responses to sepsis alert

A universal approach

Respondents highlighted how the trust’s attention had focused on a universal approach to ‘insisting every doctor looks at their deaths’ (CEO Sep-s.06): ‘in medicine last year they were reviewing a random selection of cases, now we’re insisting every doctor looks at their deaths’. It was reported that this was implemented as a result of an external ‘dictat’ and that it had created an immense challenge for the trust. This was especially onerous in specialties focused on geriatric care. The medical director commented that this universal focus provided a broad understanding of trust mortality but did not provide depth. He also suggested that clinical judgement should dictate issues for which there appeared to be a problem and then investigate a random sample of case notes (Sep-s.06).

A forensic ‘big data’ approach: investigating mortality and targeting action

The alert appeared to have some part in stimulating the trust to apply a forensic approach to investigating mortality for sepsis. The engagement of an informal opinion leader in quality improvement initiatives around sepsis also appeared to have triggered a focus on mortality. As highlighted earlier, the use of the Sepsis 6 quality improvement initiative was ongoing before the alert, and after the alert it provided a detailed methodology for investigating the alert. 145 The alert was said to have provided a powerful trigger and urgency for focused action. Benchmarking HSMR trust performance data with those of other trusts was also considered useful.

Developing innovative tools to tackle sepsis mortality

During data collation for the case note review, respondents identified how the trust was starting to implement a ‘mortality pro forma’ that provided a standardised approach for reviewing case notes. It applied a grading (A–E) and identified the organisational level required for further detailed review. The forensic documenting of patient pathways also appeared to be a ‘powerful tool’ for nurses, as it provided evidence that they could use to persuade doctors that escalation of treatment for sepsis needed to take place.

Resourcing organisational capabilities enabling improvement in mortality

Analysis highlights how the trust investment in organisational capabilities such as IT infrastructure will support monitoring, awareness and education in addressing sepsis mortality. For example, escalating patient treatment early was identified as critical in limiting mortality for sepsis. Investing in an electronic awareness system for patient observation [modified early warning score (MEWS)] would enable greater awareness of patient deterioration and hence facilitate early treatment. Likewise, reference was made to the implementation of an ‘e-learning package’ aimed at facilitating awareness of sepsis.

Generally, it was perceived that the alert had encouraged the senior executive team to provide funding for staff to support investigation and problem-solving around sepsis.

Outcomes of sepsis alert

Trust outcomes of the alert are characterised as encouraging proactive engagement with the CQC: early intervention for ‘suspected sepsis’; increased trust confidence in delivering high-quality care and attention to increasing systemisation of care; and consultants taking a greater interest in their outcomes.

The trust-reported outcomes of the alert for sepsis highlight that the trust has adopted a proactive approach whereby internal review can highlight problems with mortality and the trust will engage proactively to notify the CQC. It was viewed that alerts were reducing as the trust was tackling problems in sepsis awareness. The CEO considered that the trust was reducing variation and increasing systemisation in care across the trust.

Parallel evolution of institutional focus on mortality reduction in sepsis

It was conceded that the improvement in mortality was not exclusively triggered by the alert but that other co-evolving strategies and cultural changes were also important. A matron identified that attention to investigating sepsis mortality ‘was already rolling’ and that the alert ‘gave us a bit of a spring board’. Clinical involvement and dispersed leadership was also recognised as a strategic direction for the trust and this was not specifically linked to the alert. The CEO also reported a strategic focus on integrating mortality review structures that linked to a cultural focus on patient safety and quality improvement organisational changes.

Trust respondents perceived that mortality had become part of the organisation’s guiding vision aimed at expounding core cultural values.

Comparison of trust preconditions

A comparison of trust preconditions gave rise to a number of common themes. Both Sep-s.06 and Sep-s.05 appeared to have experienced:

• problems with senior leadership

• fragmented structures and processes for mortality review and improvement

• poor engagement between clinicians and managers

• individual clinicians championing improvements in sepsis awareness and treatment.

Problems with senior leadership

Important preconditions in both trusts included the experience of problems with senior leadership engagement in mortality review. In Sep-s.05, it was evident that clinicians felt much distrust of and antipathy towards the CEO. All respondents referred to poor clinical engagement with senior leaders and disagreement over the implementation of the IT-enabled patient records system:

And the chief executive at the time he wouldn’t listen to clinicians. [XXXX], he’s the one that implemented. He wouldn’t listen to us. He’d shout.

Consultant in critical care, Sep-s.05

It was a very dictatorial style. It was very much ‘my way or no way’. It was very much not including of the clinicians that were on the shop floor and disregard for much of the clinicians’ thoughts.

Associate medical director, Sep-s.05

Fragmented structures and processes for mortality review and improvement

Respondents in both Sep-s.06 and Sep-s.05 reported that structures and processes supporting the investigation of mortality were fragmented and lacked both integration and substance across organisational levels (corporate, divisional and specialties). In Sep-s.06, at corporate level a mortality committee existed but strategic approaches to addressing mortality, such as a ‘mortality reduction plan’, were absent:

We didn’t have a mortality reduction plan that was owned across the trust and we didn’t have embedded mortality, morbidity, meetings across specialties.

CEO, Sep-s.06

In Sep-s.05, respondents highlighted how the ‘mortality steering group’ did not always meet because of multiple changes in staff and that the trust’s routine attention to monitoring mortality was ‘ad hoc’ and had, at times, broken down:

There was no mortality steering group, the medics didn’t particularly look at their mortality and the data . . . was meaningless at most meetings . . . it has been a bit ad hoc, a bit unstandardised, various different departments doing what they felt was appropriate/not appropriate. There was no line of sight from the organisation about who was doing mortality reviews and how many mortality reviews were being done, and the outcome of those mortality reviews.

Associate medical director, Sep-s.05

Poor engagement between clinicians and managers

In both trusts, interviewees remarked on poor engagement between clinicians and management. In Sep-s.06 managers appeared to display an administrative approach, whereby they functioned in ’silos’ and failed to share and communicate information:

The organisation two and a half years ago was quite administratively managed; I think actually the idea of a manager’s good day wasn’t to see a doctor but to sit in offices and talk to other managers, and then worry about why we were so overspent, why we weren’t hitting the A&E target.

CEO, Sep-s.06

Individual clinicians championing improvements in sepsis awareness and treatment

In both trusts there were reports of a long history of quality improvement work in sepsis. However, often this lacked strategic direction and was undertaken by committed individual clinicians. In Sep-s.06, for example, one nurse in the surgical department had attended a conference and learnt about quality improvement focused on the implementation of the ‘sepsis bundle six’ and had managed to galvanise her colleagues to volunteer their time. This led to improvement in mortality around sepsis by applying quality improvement research and implementation techniques:

We worked for about 18 months to 2 years retrospectively looking at how patients with sepsis were managed and we were mortified if I’m honest. But I don’t think it was something that had really hit the radar at that point and I think we’d focused so much on clinical conditions that actually the secondary conditions like sepsis had gone by the wayside. So our research, the research that we did with the retrospective studies was really quite shocking.

Matron for general surgery, Sep-s.06

Comparison of trust responses to a single alert for sepsis

The comparison of trust responses to a single alert focused on the following themes:

• regulator pressure coercing the trust response

• useful as a trigger for a constructive approach

• an integrated trust-wide response

• a forensic approach in reviewing mortality

• development of innovative tools

• involvement of frontline clinicians.

Regulator pressure coercing the trust response

In both trusts there were reports that the alert for sepsis was a trigger for action. Sep-s.06 emphasised how the CQC intervention had coerced trust leaders to ‘jump because you’ve got to’ and ‘jobs were on the line’ if the trust did not look good externally. In Sep-s.05, it was also conceded that CQC intervention had directed the trust to investigate the alert and then develop action plans.

Useful as a trigger for a constructive approach

Staff in Sep-s.06 reported that the alert was a trigger for a constructive approach. This constructive approach in responding to CQC inspection was demonstrated in how the ‘sepsis team’ engaged with the public and media at trust board meetings, where they talked openly about how they would tackle sepsis:

Well there’s two things, one is you’ve got to do it because the CQC tells you you’ve got to do these things, but the other thing is, given that we’ve got to look at the data anyway, let’s use it to focus in a constructive way on things that we want . . . so the sepsis team, for example, have come to the trust board . . . And they talked openly in the trust board with the media there about what they’d done to cut deaths . . .

Trust mortality lead, Sep-s.06

A forensic approach in reviewing mortality

In both trusts, the alert and the CQC follow-up had led to a forensic approach to reviewing mortality. In Sep-s.05 this was entirely focused on the deaths highlighted in the alert. However, in Sep-s.06, it appeared to kickstart a ‘big data’ approach that linked back and informed quality improvement linked to Sepsis 6.

The impact to the sepsis bundle which was in response, partly in response to that alert, although not totally in response to that alert. But the alert gave us an opportunity to look in much more detail at those patients and we could look at patients whether they were in the medical division, in surgery or wherever, because we pulled all the notes and did a huge amount of work when that alert came out. So, you know, we looked at around 98 deaths.

Chief nurse, Sep-s.06

In Sep-s.06, the forensic investigation involved a case note review and the triangulation of data across hospitals, divisions and specialties, before reporting back to the CQC. Clinicians also undertook a routine universal case note review of all deaths. Likewise, in Sep-s.06, HSMR performance data was benchmarked against other trusts.

Information technology-enabled mortality review

In applying a forensic analysis of trust mortality, Sep-s.06 also developed an IT-enabled mortality review process. This used an e-mortality pro forma that set out a standard grading system for reviewing deaths and that graded deaths depending on whether the review needed trust-wide attention or was confined to specialty level.

An integrated trust-wide response

Both trusts reported an integrated trust-wide process for responding to the alerts. This focused on, first, the alert being validated and then a central structure (a mortality committee in Sep-s.06 and the clinical governance team in Sep-s.05) co-ordinating a trust-wide investigation. In Sep-s.05, respondents highlighted how the head of clinical governance asked the clinical director in the division implicated in the alert to recruit a senior clinician and task them with undertaking a case note review. This senior clinician produced a formal report, detailing action points, that formed the basis of the report sent to the CQC. In Sep-s.06 more detailed structures and roles were reported, involving the participation of a mortality lead and connecting structures such as regular mortality and morbidity groups at divisional levels.

Now we’ve had a big focus on getting a culture of morbidity and mortality meetings at divisional and directorate level. We’ve set some standards for those meetings.

CEO, Sep-s.06

Those in Sep-s.06 also highlighted how working groups assembled for medical and surgical divisions worked together to explore mortality following the alert. The outcomes of the mortality review were also fed into the trust’s quality improvement strategy process.

So we started a working group and from there it was – it’s unusual in this trust for the two divisions to work together. Surgery do their thing and medicine do their thing and never the twain shall meet. But with the sepsis working group it’s very much a core group of people who have taken this forward.

Matron for general surgery, Sep-s.06

Trust-wide awareness of the alert

Respondents in Sep-s.05 differed in their levels of awareness of the alert. The head of clinical governance suggested that those in trust divisions were aware that they were outliers for mortality and had been proactive in identifying problems to the CQC. However, the sepsis lead reported that he had not been aware of the alert, although he did concede that internal alerting systems were improving.

In Sep-s.06, a universal approach to reviewing patient deaths was in place. This promoted trust-wide awareness of mortality, as every doctor was responsible for reviewing each of their patient deaths:

In medicine last year they were reviewing a random selection of cases, now we’re insisting every doctor looks at their deaths.

CEO, Sep-s.06

Clinical engagement and cultural ownership of sepsis improvement

Sep-s.06 reported trust-wide intensive clinical engagement in mortality review. Clinicians also appeared to develop ownership of trust mortality. Opinion leaders in this trust, operating as informal sepsis champions, encouraged ‘buy-in’ and were instrumental in developing innovations within specialties.

The response in Sep-s.06 to the alert also appeared to prompt and motivate clinicians’ engagement by appealing to their implicit problem-solving epistemology mentality:

The other is then, you know, throughout the trust otherwise, every clinician wants to know how they’re doing and if mortality’s one of those measures, which it is, then if there’s something wrong, it motivates them to find out why . . . They all equally are animated when you get an alert, but that’s good because it engages them in the process. And then it engages them in the reduction plan as well, so an overarching mortality reduction plan, they all want to do the best they possibly . . . we use it in that way.

Chief nurse, Sep-s.06

Similarly, clinical champions in Sep-s.06, operating as boundary spanners, supported mortality review and made sense of how clinical diagnosis for sepsis was coded and the implications of this for alerts.

In contrast, in Sep-s.05 there appeared to be less focus on clinical involvement in the mortality review and investigation processes linked to the alert. For example, the sepsis lead acknowledged that there was a problem with coding for sepsis but conceded that he had not had any involvement in trying to address this problem, as he had not been asked for his input. Likewise, universal case note review by clinicians of all deaths was not undertaken.

In summary, the trusts differed in their responses in so far as Sep-s.05 limited its response to directly addressing the specific deaths highlighted in the alert. Sep-s.06 appeared to follow a more robust and clinically involved process of case note review but also started to develop a culture that encouraged an awareness of mortality linked to sepsis and ongoing improvements in sepsis care.

Comparative analysis of outcomes arising from the alert for sepsis

A comparative analysis of outcomes generated by the alert suggests that Sep-s.05 generated solutions directly related to problems arising from investigating the alert. These related to training junior doctors in writing up case notes and providing greater clarity in diagnosis and comorbidities. Junior doctors were also trained on greater awareness of sepsis and the importance of early intervention. However, cultural changes that supported ongoing improvement in sepsis awareness and routine case note review of all deaths appeared limited, although it was noted that more recently the new trust leaders had started to address clinician disengagement.

In contrast, Sep-s.06 addressed problems arising from alert investigations and went further in enabling the development of a trust culture that supported the routine clinical case note review of all deaths, improved sepsis awareness, the use of innovative IT-enabled technology to systemise routine case note review and monitoring of mortality and deterioration of patient condition. Sep-s.06 trust outcomes of the alert are characterised as encouraging proactive engagement with the CQC, encouraging early intervention for ‘suspected sepsis’, increasing trust confidence in delivering high-quality care and increasing systemisation of care.

Comparative analysis of parallel development of processes for sepsis mortality

Respondents in both Sep-s.05 and Sep-s.06 conceded that other factors influenced the evolution of trust mortality processes and improvement in sepsis mortality. The important emergent themes included:

• regulatory pressures link to changes in trust leadership

• engagement with external bodies

• clinical engagement and dispersed leadership of mortality review and QI

• integrated structures and processes

• a culture supportive of mortality awareness

• the use of data analysis tools to monitor mortality.

Regulatory pressures link to changes in trust leadership

Extensive organisational turbulence experienced in Sep-s.05 at the time of the alert prevented trust-wide engagement in mortality review. Regulatory pressures associated with Monitor and CQC intervention led to substantive changes in trust leadership. The resulting changes in trust leadership appear to have influenced improved clinical involvement in mortality review.

Engagement with external bodies

In Sep-s.05, consultants in critical care highlighted how their specialty had engaged for more than 5 years with the Intensive Care National Audit and Research Centre (ICNARC), whereby a specialist audit clerk inputs data into the ICNARC database and each quarter the critical care department receives a nationally produced report. They noted that this educates them in improvement on critical care practices and develops them professionally:

It gives us the opportunity to build on that which improves our professional fulfilments as doctors . . . And improves the care for our patients . . . So I’ve been emphasising the use of this from an education point of view.

Consultant, critical care, Sep-s.05

Similarly, both trusts were engaged with external bodies in clinical auditing specialty performance and participation in quality improvement initiatives for sepsis, and this appears to have played a part in improving the awareness and treatment of sepsis.

Clinical engagement in mortality review and dispersed leadership of mortality review and quality improvement

In Sep-s.06, a matron identified that attention to investigating sepsis mortality ‘was already rolling’ and that the alert ‘gave us a bit of a spring board’. Clinical involvement and dispersed leadership was also recognised as a strategic direction for the trust, and this was not specifically linked to the alert:

I think our deliberate approach to developing the organisation was based around getting the people who do the work, who are clever people, involved in leading services really. And I mean that’s a value and I have; it’s a value that the nurse and medical director have and their executives. So whilst there was a national impetus it’s kind of a no-brainer and the really successful organisations that excel have a number of characteristics – one of which is clinical excellence – and you only get that by effective clinical leadership. So it’s been deliberate from our point of view . . . And so I’ve tried to change it from quite a centralised cliquey sort of way of leading into a much more open transparent distributed way where people – I’m trying to give clinical leaders and execs clear role responsibilities really and their goals.

CEO, Sep-s.06

Integrated structures and processes

In Sep-s.05 it was suggested that major changes in trust structures had occurred as a result of Monitor intervention and that this had led to changes in trust senior leadership who had acted ‘as a new broom’ providing an ‘overhaul of trust governance’. The head of governance highlighted this as follows:

So a complete overhaul of all governance throughout the organisation starting at the board level and going downwards. Now clearly that includes mortality in terms of divisional governance, in terms of corporate governance, but also in terms of clinical governance. So that’s the impact. It makes you start again, basically.

Head of governance, Sep-s.05

The new medical director suggested that assurance of high-quality care needed to be facilitated, so divisional directors sit on the mortality review group:

We have five clinical divisions and they’re all represented on the mortality alerts group along with a few other people. We have [XXXX] who’s one of the senior nurses responsible for quality within the organisation. We have a representative from our coding unit. And normally Dr Foster will send along one of their team to assist and steer us through the data at the mortality alerts group.

Medical director, Sep-s.05

The three new associate director roles were created for standards, patient safety and revalidation. In the trust hierarchy, these sit beneath the medical director. One associate medical director responsible for standards, clinical effectiveness and research has responsibility for mortality review.

A culture supportive of mortality awareness

Trust respondents in Sep-s.06 perceived that mortality had become part of the organisation’s guiding vision aimed at expounding core cultural values:

Well it’s one of our, it’s one of our key strategic aims to reduce mortality, it’s explicit in our vision. And so the vision has lots of things in it, but one of the critical success factors is delivering excellence in quality and patient experience and mortality is a key element of that, so it’s clearly explicit throughout the document that everybody sees.

Chief nurse, Sep-s.06

In Sep-s.06, the CEO also reported a strategic focus on integrating mortality review structures that linked to a cultural focus on patient safety and quality improvement organisational changes:

I think we’ve had huge pressures with the A&E target and our performance on that has been awful, but actually the one thing that we’ve kept shape on is actually the quality. And I’ve just been with the CQC just now and actually they’re saying ‘Your A&E department is great’ despite the fact the patients are waiting long periods of time. And so I think we kind of kept our shape because we saw quality front and centre, and actually for us on the efficiency equation quality and efficiency are different sides of the same coin.

CEO, Sep-s.06

Comparison of trust preconditions

Comparison of trust preconditions for receiving multiple alerts gave rise to the following common themes:

• catchment characteristics

• problems with coding mortality

• fragmented organisational structures

• financial and operational pressures

• trust involvement in sepsis improvement work

• leadership

• sentinel events.

Catchment characteristics

It was commonly reported that catchment characteristics had an impact on trust mortality, and all trusts identified that deprivation and lower life expectancy had an impact on trust mortality. In Sep-m.09, trust staff highlighted that this related to immigration centres, alcohol and psychiatric diseases, and questioned whether the mortality calculations for the trust had taken account of this:

We’ve got lots of urban poverty . . . We have a lot of alcohol and psychiatric disease, and they score heavily in the 15–45 age group. And unless you know that we’re number three in the country for this particular problem, it’s hard to understand why we seem to have so many deaths, most of which happen outside and we can do nothing about because they’re just brought in and die in 24 hours on the intensive care unit.

Mortality lead, Sep-m.09.

Two trusts (Sep-m.07 and Sep-m.11) identified that poor community service provision in their catchment had an impact on trust mortality. Both trusts reported that terminally ill patients were being sent to the trust to die; the trusts reported limited palliative care systems and a lack of suitable staff:

We have a lot of young people dying of cancer who tend to die of cancer in the hospital rather than in the community. And we didn’t have very well-developed and palliative care systems, we didn’t have our own palliative care consultants.

Former medical director, Sep-m.07

That is a challenge for us compared to some other organisations where community services may be better developed and people have a choice to die outside of hospital. There was also a view that some of our community provision, our care home provision, was not as robust in this area as it is in some others. So, again, the choices for people are more limited. Now, again, all of those things might impinge on your HSMR.

Medical director, Sep-m.11

Problems with coding mortality

Both Sep-m.09 and Sep-m.10 reported issues with coding quality. In Sep-m.09, this was linked to shortages of skilled coders, and in Sep-m.10 respondents reported a lack of standardisation in coding mortality, as two of the trust hospitals adopted a different methodology:

The coders are organised – we’ve got coders on both of our main hospital sites and they have different models in each trust, in each of the hospitals. So in [B] the coders go to the wards and pick up the notes on discharge or death and will scrutinise them there, will take them back to their office if need be but sometimes they just take the details they need there and then so leave the notes in the system. At [A], the notes flow to the coding department.

Deputy head of information, Sep-m.10

In Sep-m.09, the mortality lead drew attention to limitations in trust capabilities for coding. These were linked to reductions in staffing allocation for non-clinical staff. It was noted that quality in coding was essential, as coding errors had had an impact on the usefulness and validity of alerts:

So if you put rubbish in, you get rubbish out. I understand that bit. Our chief coder, in the last few months, has gone onto retirement and is due to come back, so I don’t have a relationship with the new people. But my understanding seems to be that they’re short of people. We’re having a halt on recruitment, because we don’t want to spend money on non-clinical staff, so I kind of see this as a threat.

Mortality lead, Sep-m.09

Fragmented organisational structures

In two trusts (Sep-m.07 and Sep-m.10) it was emphasised that the structures and processes for addressing and recording mortality were fragmented. In Sep-m.07, although a central co-ordinating team or mortality review group examined trust mortality, respondents reported that mortality investigation was fragmented across the trust, and it was noted that senior leadership was disconnected and had not ensured that learning around mortality review was shared across the trust:

I think a lot of our mortality was, at that time, when I first came into post was most probably a little bit disconnected between the top of the organisation which the board would have numbers presented as a HSMR type data as part of a performance matrix, and then the mortality reviews, mortality and morbidity meetings which would be held within departments, and very often associated with all of this, and I think to be fair to say that there would be a differential between how good some of those mortality reviews were.

Former medical director, Sep-m.07

Likewise, the deputy head of information in Sep-m.10, who had worked in the trust for 25 years, said that before 2010 there was little knowledge about alerts in the trust and little interest in mortality data:

I’m not sure that we even knew that there were Dr Foster CQC alerts . . . they certainly didn’t come my way so if they arrived I don’t know to whom and I don’t know what was done with them.

Deputy head of information, Sep-m.10

Financial and operational pressures

In Sep-m.11, concern was raised about the impact of restructuring on finances and staff redundancies. Some trusts highlighted problems in specific care pathways. For example, in Sep-m.09, the chief nursing officer remarked that the trust was ‘stressed’ by challenges in the emergency pathway but also confirmed that there were no financial pressures, as the trust was in financial balance:

I mean the organisation was in financial balance, there were some challenges around the emergency pathway in delivering the 4-hour standard, other performance metrics were all pretty much online, so I wouldn’t have said the organisation was, other than the emergency pathway, stressed.

Chief nursing officer, Sep-m.09

Trust involvement in sepsis improvement work

All trusts referred to participation in external improvement programmes [Safer Patients Initiative (SPI) and ‘Think Sepsis’] related to sepsis. In Sep-m.11, Sep-m.09 and Sep-m.07, attention was focused on how trust champions or sepsis leads had implemented the sepsis care bundle and had undertaken work to promote the IT-enabled early identification of sepsis and patient escalation processes. It was noted that both Sep-m.07 and Sep-m.09 had developed IT-enabled systems for logging patient conditions. This enabled the visibility of patient deterioration and timely intervention.

In Sep-m.09, the trust sepsis lead recounted that the trust had been active for many years in responding to national campaigns focused on sepsis awareness. The move to a new hospital site in 2006 was linked to the implementation of the Electronic, Physiological Patient Observation Track and Trigger System (EPOTTS) in 2007:

In 2006 we moved into this hospital and 2007–8 we started working with an electronic system for track and trigger. So it was Electronic, Physiological Patient Observation Track and Trigger System, EPOTTS for short.

Mortality lead, Sep-m.09

This previous work on sepsis improvement appeared to focus primarily on the emergency department (ED) and critical care, as most sepsis leads were consultants in these specialties. For this reason, staff in most trusts (Sep-m.09, Sep-m.11 and Sep-m.07) were generally aware of sepsis problems. The associate medical director in Sep-m.11 said:

The alerts . . . have never told us anything we didn’t know already . . .

Associate medical director, Sep-m.11

Leadership

Many trusts drew attention to the way in which opinion leaders had been instrumental in addressing trust mortality. The medical director in Sep-m.07 had taken an interest in analysing mortality. In other trusts (Sep-m.09), champions working in particular specialties had led on sepsis work. It was noted that attention to sepsis mortality, and to mortality in general, was not reported as a trust-wide activity.

Sentinel events

One trust, Sep-m.09, reported that sentinel events or major patient safety failures linked to sepsis had driven a focus on improving trust-wide mortality. Two sentinel events were referred to as follows:

I started in the April of 2009 and I became aware of the alerts when we’d had two very serious cases of sepsis, or a failure to recognise sepsis within our emergency department, so we had a young girl who was 14 years of age and we had a young mum, I think, in the twenties or thirties, I can’t quite remember the full details. Anyway, and both of those cases went through a root cause analysis review following a serious incident.

Mortality lead, Sep-m.09

This was the only trust in which sentinel events were described in the context of preconditions for mortality alerts.

Comparison of trust responses to the alerts for sepsis

A comparative analysis of the trusts’ responses to multiple alerts for sepsis gave rise to the following common themes:

• responses to first and subsequent alerts for sepsis

• a centralised approach

• clinical engagement in case note review

• a forensic approach

• the identification of problems

• an action-focused approach linked to improvements.

Responses to first and subsequent alerts for sepsis

Difficulties arose in discriminating distinct responses to the individual alerts. Only one respondent in Sep-m.07 was able to identify the trust’s response to the first alert. In the other trusts, respondents highlighted how the trust leadership had increasingly prioritised attention as the number of alerts received increased. In Sep-m.11, Sep-m.09 and Sep-m.07, a forensic approach involved validating the alert, undertaking case note review, identifying problems with coding and care and developing corrective processes to address these issues. Sep-m.11 appeared to move faster in addressing problems, whereas Sep-m.09 and Sep-m.07 stepped up action to improve sepsis early identification and treatment only after the second alert. In Sep-m.09 in particular, interest in sepsis mortality was linked to investigation of sentinel events or major patient safety incidents linked to failures in identifying sepsis. Sep-m.10 appeared to also prioritise trust actions in responding to the alert but was still unable to complete the circle of identifying problems and then moving on to implementing solutions. Overall, Sep-m.10 appeared less clinically engaged.

A centralised approach

In some trusts (Sep-m.07 and Sep-m.11), the response to the first alert took a centralised approach. In Sep-m.07, the process started with the CEO who received the alert passing it to the medical director, who co-ordinated the response. A review process was initiated of 60 case notes by the head of coding and the medical director. Discussion and review of some of the case notes was undertaken by the mortality group, comprising consultants from ED, critical care, surgery, medicine and palliative care and a clinical audit lead. From this investigation was undertaken of coding and reports produced of review findings and action plans for the trust board, quality committee and clinical governance committee. The medical director would also send a report to all consultants explaining the alert, the results of the review and the action plan:

They come into the chief exec who then passes it onto me and I do the response, so I get people to help with some of the mortality reviews, we would do a report, look at some of the coding issues, put what we were doing, including a board report on mortality, we would go back to board to the quality committee and it would get sent to the consultants in terms of . . . so that would be the formal route, governance route that it would be reported to . . . So you have the board and then the quality committee. It’s slightly different now how they do it. And then it would’ve been reported to all as part of – I would do a medical director’s report to the consultants to say, ‘We’ve had this alert come’, etc.

Former deputy medical director, consultant critical care, Sep-m.07

A forensic approach

All trusts highlighted an increasingly forensic approach on receipt of the alerts. In Sep-m.07, this process was perceived as proactive, as the former medical director said that the trust had already identified a problem with mortality for sepsis before the alert (2013) and had already started to review case notes before receiving the CQC letter. The medical director reported that he had been monitoring trust mortality over a long period and had a lot of experience of using Dr Foster tools and the trust’s CHKS system:

I believe that the ITU [Intensive Therapy Unit] one, we noticed our figures were going up and we asked them to come in before we actually triggered, and we’d actually had the review and the report back by the time the CQC had picked up, and our figures were coming back down again. So that would have been 2013.

Former deputy medical director, consultant critical care, Sep-m.07

Emphasis was placed on thoroughly investigating the alert rather than immediately blaming it on poor coding. It was noted that some trusts appeared to respond to alerts by adjusting their coding and ‘manipulating the data’. This was viewed as an ‘unemotionally intelligent thing to do’ and in this trust the approach was to be ‘honest’ with coding. The approach was explained as follows:

To start with, he [former medical director] didn’t want to dismiss any of this as a coding issue and we’re not in a place where we can do that, we certainly weren’t back 3 years ago, because blaming the coding was just an unemotionally intelligent thing to do. And when you look at some trusts, I think, who’ve like hired kind of people to help them with it they’ve laid themselves open a little bit, I think, haven’t they, to sort of manipulating the data. So I can honestly say that, although our coding might not be as good as it can be, I think there’s every reason to assume that it’s actually honest.

Blaming it on the coding is just a blind alley, isn’t it, I mean, it’s pointless in a way because you just . . . you want to be absolutely sure that it’s not quality of care before you actually turn to the coding issue otherwise it just looks dismissive and . . . Plus the fact you miss an opportunity to actually improve things.

Interim medical director, Sep-m.07

In Sep-m.11, senior leaders highlighted the forensic examination of case notes, an analysis of care pathways, triangulation of data, production of action plans and implementation of new and improved practices:

We used to pounce on the monthly data from Dr Foster and go through it in enormous detail and we knew where we were heading in every single diagnostic group and we used to map them or we used to trend . . .

Associate medical director, Sep-m.11

This forensic approach to the alert was viewed as including review of 50 case notes, analysis, production of action plans and ‘spending time making sure those actions are embedded’ (head of governance). Respondents also commented that the process was very time-consuming and highlighted that it had not identified anything more than had already been covered in SPI-inspired improvement work:

The sepsis one, particularly the big one that we did, it was huge. It was myself, [XXXX] who’s the consultant nurse and another one of the . . . other consultants and I think I just cleared my diary for something like 6 weeks . . . do the analysis of it, come up with something, write an action plan and then spend the time making sure that those actions are imbedded is massive for an organisation and when it doesn’t really tell you anything at the end of it that you didn’t already know, it’s kind of like, ‘OK!’ . . .

Head of governance, Sep-m.11

Sep-m.09 also followed a systematic and forensic process. A working party was set up to review case notes of deaths occurring during the designated period of the alert. The head of nursing described how the sentinel events and the alert were joint drivers of a working party forensic investigation. This involved, first, validation of the alert by checking the coding and then the governance department identifying the time scale for the review process and instructing that a mortality review process was undertaken by the service that the alert related to. There was a peer review within that service and a report was presented to the mortality review committee. This report was then sent for scrutiny by senior trust leaders in the patient safety committee.

In Sep-m.10, respondents emphasised that validating the alert was necessary and drew parallels with Mid Staffordshire where ‘they didn’t trust HSMR data’. Sep-m.10 respondents stressed that, in responding to the alert, they were not trying to dismantle the alert but they were ‘healthily sceptical of it’. The deputy head of information also stressed that validating the alert comprised the following stages:

Right, let’s make sure (a) we understand what it’s telling us, (b) that we agree with what it’s telling us.

Deputy head of information, Sep-m.10

It was suggested that the trust needed to ‘prove’ that the alert was worth investigating and that, invariably, the alert ‘numbers matched’:

I think we start from a position of wanting to prove to ourselves that it’s something worth investigating which is why my first task is can I match the numbers? So, if we match the numbers and we almost invariably do, it’s relatively unusual to have a significant discrepancy.

Deputy head of information, Sep-m.10

The nursing director reported that validation of the alert was considered useful, as it made the trust ‘concentrate’ and ‘focus its’ attention and ‘dig down’ to identify problems. Overall, the alert was ‘a slap in the face’ that made the trust question how it was delivering care:

I think they do because I think we are doing all of this work and then something will come and slap you on the face and you find that the infection rates and the caesarean sections have gone up, but why?

Nursing director, Sep-m.10

Only after completing this process did they move to an indepth investigation of deaths linked to the alert. Hence, it was reported:

We’ll have a look anyway because it’s a good thing to audit your patients and see what’s happened.

Deputy head of information, Sep-m.10

Unlike the other repeat alerted trusts this process, Sep-m.10 lacked continuity, as it was reported that, initially, this information was forwarded to either the medical director or the director of clinical strategy. The medical director reported this inconsistent approach as follows:

They didn’t come to me and indeed what we’ve done in the organisation is maybe streamline that a little bit in that they would previously have probably gone to the information department for the first instance and then maybe have got to me eventually but not necessarily consistently.

Medical director, Sep-m.10

Identifying problems in coding

All trusts identified problems with coding arising from poor case note write-up. However, in one trust (Sep-m.10) intensive attention to identifying coding problems occurred only after the second alert.

In Sep-m.07 and Sep-m.11, case note review identified errors in coding for sepsis linked to palliative care. This related to elderly patients admitted from nursing homes with ‘multiple bed sores who were basically coming to the end of their lives’. The deaths of these patients were not preventable even when they had been treated well. The former medical director described this as follows:

When I looked at it they were a very desperate group, very desperate, and we did a bit of a coding thing with the chief coder, and I think we had something like 63 cases. I did about half of them the 30-odd cases . . . I don’t think sepsis was always the first choice that should’ve been when we were looking back at the coding, and some of them were, like I said, they were 90-year-olds from nursing homes with multiple bed sores who were basically coming to the end of their lives, so whether they were – I’m not particularly interested in the mortality ratios or whatever, what I want to see is where is the preventability of the death.

Former medical director, Sep-m.07

Overall, it was reported that there had not been any errors in the care of these patients:

I can’t remember seeing anything where there was poor care . . . I think sometimes I wonder whether this would be coded right or wrongly, and I think coding is an issue.

Former medical director, Sep-m.07

Errors in coding were widely acknowledged in all trusts; for example, in Sep-m.10, the medical director suggested:

There is nothing wrong with our clinical processes or our clinical care but still we’re an outlier and what we’ve narrowed it down to is things like coding.

Medical director, Sep-m.10

In Sep-m.10, the recording of primary diagnosis was perceived as ambiguous, as were failures in recording comorbidities:

Is there something about the way we’re coding primary diagnosis or comorbidities that would set us differently from everyone else? The other thing we thought, we’re one of the first trusts into the sepsis six bundle, care bundle programme.

Deputy head of information, Sep-m.10

The medical director also suggested that recording of comorbidities was low and that this failure to identify and code comorbidities could result in higher mortality indicators:

So we know we are numbers that the number of comorbidities per patient in our organisation is low.

In Sep-m.10, it was also mooted that the system for ‘capturing coding’ was different in the two main hospital sites. For example, mortality coding was high in the hospital site where the coding staff did not engage directly with clinicians on the wards:

We know that the process we’ve got for capturing coding is not very good. We know that the process we’ve got for capturing coding is different in [A] than in [B]. We know that our coding process is clinically linked in [A] because the coders go to the wards and sit down with the clinical staff and go through the coding with the clinical staff in [A] but they don’t in [B].

Medical director, Sep-m.10

Despite this highlighting of poor coding, it was emphasised that it was important to thoroughly investigated the alert rather than immediately blame it on poor coding. It was also noted that some trusts appeared to respond to alerts by adjusting their coding and ‘manipulating the data’. In Sep-m.07, this approach was explained as follows:

Blaming it on the coding is just a blind alley, isn’t it, I mean, it’s pointless in a way because you just . . . you want to be absolutely sure that it’s not quality of care before you actually turn to the coding issue otherwise it just looks dismissive and . . . Plus the fact you miss an opportunity to actually improve things.

Interim medical director, Sep-m.07

In Sep-m.11, attention also focused on whether or not patients should have been sent to the trust at all, and also on the nature of their primary diagnosis, with a focus on where they had developed sepsis:

Should they have come in the first place? Did they develop sepsis in this organisation? Because the problem with a lot of these is the primary diagnosis and they’re coming in with them so actually then you’re asking a different set of questions like did we treat them in a timely and the appropriate manner rather than could we have avoided them getting it in the first place.

Head of governance, Sep-m.11

In Sep-m.09, alerts were seen as smoke signals and something to prompt investigation, but it was conceded that in a complex organisation this can mean that irrelevant things are investigated, which could distract attention away from more pressing issues:

What they do is they’re a smoke signal of various sorts . . . So these alerts are useful. They tell us to go and look. The problem is, you can be distracted by certain alerts and you go looking at things that don’t warrant attention . . . So the clarity of the signalling is a problem.

Clinical lead for mortality, Sep-m.09

In Sep-m.10, the CQC’s involvement ‘gave teeth’ to alerts, which spurred trust leadership to address hospital mortality and forced them to drill down and explore sepsis mortality:

I suspect when the CQC itself started to generate the alerts I think that was probably the trigger for us because . . . I think the CQC gave it the teeth that people like chief executives started to say, ‘What’s going on here? We’ve got an alert. Are we killing people? We must know more about this’.

Deputy head of information, Sep-m.10

In so doing, Sep-m.10 investigated by completing a ‘full mortality audit’ and then identified a host of problems associated with high mortality for sepsis. These included problems with the writing up of case notes identifying comorbidities, the recording of primary diagnosis and the coding of mortality.

We have narrowed it down to things like coding, which is obviously a natural port of call. Is there something about the way we’re coding primary diagnosis or comorbidities that would set us differently from everyone else?

Head of information, Sep-m.10

Likewise, by drilling down and exploring the alerts forensically, Sep-m.10 also identified problems with junior doctors completing death certificates:

If you actually look at this hospital, there’s four, five deaths a day in the two hospital sites. Why don’t the consultants do it? Because it would be more accurate, wouldn’t it?

Consultant sepsis lead, Sep-m.10

Analysis suggests that the Sep-m.10 was comparing its mortality indicators with those of other trusts and identifying how its particular care pathways may be linked to it being recorded as an outlier in terms of mortality. It was noted that all unscheduled care patients were ‘fired at’ two acute care units and were assessed there for a short period, which may not be the model used in other trusts:

Almost all of our unscheduled care patients go into these two ACUs [acute care units], one on each site, that is again making us a bit different from the average in the country because although acute medical wards are not unusual, deliberately firing all your patients that way and then keeping them a relatively short amount of time may not be exactly the same model of care or pathway of care that other places have.

Deputy head of information, Sep-m.10

In Sep-m.09, problems with coding were also identified, as it was clear that the coders did not know how to code sepsis, and so guidelines were written to address these:

The other thing that was found is that the coders weren’t really clear at all, in what they should put into what codes because it wasn’t clear. So we wrote some common sense guidance forms to what counts as what [in terms of sepsis] . . . I’m not entirely sure I’m right on this, but at the time there wasn’t a code for septicaemia which is a bug in the blood and there wasn’t a code for sepsis syndrome, yeah, with the management, of that, a lot a people would put as sepsis was coded at septicaemia and that proved it was septicaemia and so we made a very sensible, I can’t remember what it was, a way for coders to be clear of what counts as what.

Deputy chief medical officer, Sep-m.09

An action-focused approach: implementation of the Sepsis 6 bundle

Most trusts identified errors in care and mobilised actions to implement the ‘Sepsis 6′ care bundle. In Sep-m.07 the alerts were seen as making clinical staff aware of the problem and then mobilising the trust to act effectively, as follows:

‘Look we have a problem, this is what we’re going to do about it’. So that was useful to gain people’s attention.

Sepsis lead, Sep-m.07

In Sep-m.07 a direct response to the sepsis alert was the implementation of the Sepsis 6 care bundle. The interim medical director reported this as follows:

We did introduce things like . . . the ‘sepsis bundle’, I think it is, and then he’s been [inaudible] doing that, and so we have responded to the alerts positively.

Interim medical director, Sep-m.07

In Sep-m.11, the case note review process highlighted known problems around sepsis care and contributed to initiating the application of care quality programme that brought together care bundles (sepsis, AMI, pressure ulcers). It was reported that the trust had already been investigating and monitoring sepsis mortality and had recognised that there was a problem.

Given the fact that the focus was on immediate response to sepsis and A&E was obviously a focus of a lot of the work that we did and a lot of the enthusiasm actually, and that’s not perfect yet either but that was reasonable.

Associate medical director, Sep-m.11

It was noted that the alert stimulated further investigation of sepsis mortality in A&E.

We did a lot of work and had already done a lot of work on the 2012 alert prior to it arriving. So we were already had done the investigation prior to the alert arriving. Then with the alert arriving we went, ‘Oh yeah, we know we have a problem’. We have done some investigation but then obviously we went through the formal mortality review investigation that they wanted us, which was fine because we had done a lot of investigation.

Nurse consultant, sepsis, Sep-m.11

In Sep-m.10, work on the Sepsis 6 care bundle started in 2012, after the second alert arrived (sepsis lead). When data collection started, the trust found that only 20% of patients were receiving the bundle on admission. In response, it stepped up implementation of the bundle.

But it’s gone originally from about 20% up to 80%. So within severe sepsis, the definition of severe sepsis being organ dysfunction and unwell, high lactate, that kind of thing, we do feel we’ve made an improvement.

Sepsis lead, Sep-m.10

It was noted that clinicians working within the ED had undertaken considerable efforts to implement the Sepsis 6 bundle that had led to a significant increase in the use of the bundle, from 16% in April 2012 to 91% coverage.

Despite the implementation of the Sepsis 6 care bundle after the alert, respondents in Sep-m.10 were keen to attribute problems to coding errors rather to than errors in care. It was suggested that care was appropriate and the patients concerned were ‘very frail’, had many comorbidities and had been admitted from nursing homes. It was noted that this cohort of patients deteriorated very quickly and died. It was suggested that doctors had queried sepsis in the case notes and the patient had received antibiotics so, as the first entry in the notes referred to sepsis, the cause of patient death was recorded as sepsis. Nevertheless, respondents suggested that patient care had been appropriate and that in this situation death was inevitable. This was cited as an ‘acceptable death’.

A cultural approach focused on high-quality care

All trusts set out to use quality improvement methodology to improve care for sepsis. One trust (Sep-m.11) emphasised a response focused on developing a culture focused on high-quality care rather than on chasing alerts. It was felt that by ‘chasing’ and dealing with each alert individually, the longer-term culture change needed would be lost:

I think that is where it was important for us to develop the care of the acutely ill action plan because it became really clear to us that you could just be chasing alerts. And even in HSMR, I think HSMR has I think 56 conditions or whatever else that sit within that, I think SHMI [Summary Hospital-Level Mortality Indicator] I think is 130-odd, you can get into the fixation of which this is. And we’ve done some of that. but the reason why we talked about having a plan per se rather than just chasing the lights is because we know that some of it is more than just the indicators themselves; it’s about culture, it’s about looking at the whole pathway and, therefore, if you just chase the lights what you do is you miss some of the real longer-term stuff that you have to do that eventually will improve the situation around alerting.

CEO, Sep-m.11

Alert driving clinical engagement

Leaders in Sep-m.09 identified how the alerts were used as a ploy to promote clinician engagement in case note review. It was suggested that when clinicians responded to the alert they became fully engaged in discovering if there had been errors and, if so, why. It was posited that clinicians were often less engaged in this process. The head of nursing (Sep-m.09) suggested that clinicians were ‘driven by outputs’, in this case the alert, and so became fully engaged in investigating the alert. This was referred to as follows:

But normally they are a difficult bunch. And I remember they weren’t a problem in terms of engaging with the work, primarily because outputs are quite, they’re quite driven by outputs anyway, so there were two reasons, one they wanted to prove it wasn’t true, or two if there was a problem they’d need to go away and fix it. So, yeah, I mean they weren’t really challenging but I mean it’s natural for clinicians where they’ve got a potentially mortality outlier to initially review the data from the perspective that there might be something wrong with the data rather than necessarily what’s wrong with the care.

Head of nursing, Sep-m.09

In Sep-m.11, the alerts were perceived as gaining the attention of clinicians:

Did it [CQC alert] get clinicians’ attention?

Oh god yes.

In Sep-m.11, every death is reviewed and the coding validated by the consultant concerned with the death, thereby ensuring that the coding is accurate. If it is not correct then the consultant will record why and, if necessary, put this information in the patient’s notes too. In this way any changes to the notes are documented separately on a sheet held in the coding department.

In Sep-m.10, trust-wide clinician involvement in case note review was inconsistent. Sep-m.10 respondents reported that clinical involvement in case not review as onerous and dependent on volunteers. The deputy head of information reported that it had been hard to find willing volunteers among the consultants to help with this. It was noted that clinicians were not automatically informed by the organisation about the receipt of a mortality alert specifically:

Yes. So the reality is – I’ve been here since 2004–5, so throughout all of these alerts I’ve been here. The reality is as a clinician I’m somewhat removed from that, you know. It comes into this central kind of management suite and various people are involved.

Consultant, Sep-m.10

Comparative analysis of outcomes from repeat alerts for sepsis

A comparison of the reported outcomes from multiple alerts yielded the following common themes:

• implementation of Sepsis 6 care bundle

• coding of acute kidney injury

• increases in staffing of palliative care

• changes in clinical practice regarding the diagnosis of sepsis

• innovative sepsis training

• challenging practice and rewarding good clinical practice

• changes in coding guidelines

• changes in service design

• wider cultural change

• a focus on resourcing data analysis.

Most trusts (Sep-m.11, Sep-m.09 and Sep-m.07) suggested that the alerts had led to a greater impetus in implementing the Sepsis 6 care bundle. Some trusts (Sep-m.07 and Sep-m.09) highlighted changes in coding, and one trust (Sep-m.09) identified extensive improvements in care processes and innovative practices as evidence of greater sepsis awareness. One trust (Sep-m.11) focused on a broader cultural approach, concentrating on quality improvement rather than on ‘chasing the alerts’. In one trust in particular (Sep-m.10) there were reflections on how only limited changes in the management of sepsis had arisen from the alerts:

Well yes but I don’t think that they’re – they’re helpful as a lever but I don’t think they’re helpful in terms of actually telling us where our problem is because I think certainly on the basis of the mortality alerts we’ve seen they haven’t said, ‘Oh my god in sepsis we’ve got a systematic problem with our management of sepsis’ because for instance we thought maybe we have got a systematic problem because we take too long to get the antibiotics in. We’re dramatically better at that now and we know that across the whole organisation and in some areas even better than others and our mortality hasn’t dropped. Now that’s not to say the care isn’t better and I’m absolutely convinced the care is better and there is people probably who aren’t dying now who did die previously or whatever, but it’s such a crude measure mortality as an indicator of the quality of care that you’ve got to be doing something really bad or really good to get your figures looking fantastic . . .

Medical director, Sep-m.10

In considering coding changes arising from the alerts, the Sep-m.10 medical director also suggested that many trusts had changed their coding in response to the alerts. In this way they suggested that many trusts were ‘gaming’ with their coding in order to fix their HSMR.

Implementation of Sepsis 6 care bundle

Implementation of the Sepsis 6 care bundle was recognised to be a direct consequence of the alert and the CQC letter (Sep-m.07). In Sep-m.09, the alerts required a case note review, and the second alert led to an overhaul of the sepsis campaign:

The second one I know my colleague had 56 sets of notes to go through. Which he did. To come out with learning from it. And at the same time there’s been a big sepsis campaign so there were two, there’s a big sepsis campaign within the trust. Which had been redone a year or so ago.

Deputy chief medical officer, Sep-m.09

The alerts were considered to have helped the further implementation of the Sepsis 6 work:

But the Sepsis 6 and those, the figures and all that kind of thing, which are now done on a regular basis and presented in grand round-type fora and in terms of outputs on the audits and that and people joining in with it, so some of that was generated from the septicaemia alerts.

Deputy chief medical officer, Sep-m.09

In Sep-m.10, respondents generally conceded that the mortality alerts may have heightened the profile of the sepsis group across the organisation:

Now if we hadn’t had mortality alerts would we have a sepsis group now in the trust is an interesting question because we’ve had a mortality alert around a sepsis and we do have a sepsis group. So it’s really difficult to un-know what you know isn’t it and I think we probably would have a sepsis group but whether it would’ve gained such a high profile in the trust if we hadn’t had a mortality alert I don’t know.

Medical director, Sep-m.10

Coding of acute kidney injury

The alert identified errors in sepsis coding, led to an awareness of sepsis and led to changes in coding; hence in correcting the coding mortality for sepsis the coding of acute kidney injury increased:

People come in with acute injury sepsis, it’s chicken and egg isn’t it? I think when there was a big concentration on sepsis I think if you look at our actual numbers I think they fell a bit, and then the acute kidney injury went up a bit because I think people saw that we’d coded things as acute kidney injury rather than sepsis. So there are silly patterns like that that you see that goes up and that goes down. Nothing deliberate it’s just some of these people may be aware that you’re looking at sepsis or something and they’re picking up, ‘Well the acute kidney injuries is the major component here rather than the sepsis'.

Former deputy medical director, Sep-m.07

Increases in staffing of palliative care

Case note review identified high numbers of patients in the palliative care pathway; as a result palliative care consultants were recruited and the trust was able to cope better. Nurses also offered a 7-day service in palliative care:

Now there’s certain groups, our SHMI is fine, actually our HSMR is as expected now. But it’s taken a while to come back down to that, and I think there’s several reason for that, and one is that we’re able to cope more of the palliative care that is going on because we’ve got palliative care consultants now, and they also offer 7-day week service, not the consultant but the nurses there.

Former deputy medical director, Sep-m.07

Changes in clinical practice regarding diagnosis of sepsis

It was generally understood that the early administering of antibiotics in the treatment of sepsis was necessary. However, diagnosis by doctors was difficult and could often take a long time. Changes were made that allowed the treatment of sepsis to proceed if there was a suspicion of sepsis rather than a definitive diagnosis. This allowed the wider health team to have input in deciding on treatment. It was suggested that because nursing staff spent more time with the patients on the wards, their monitoring input could facilitate more timely intervention:

But we made a couple of fundamental changes internally . . . so when [XXXX] and I were reviewing it internally in the organisation, one of the clear focuses was more about diagnosis, so it was seen as a medical issue. What [XXXX] and I tried to do was move it away from being a diagnosis to more suspect sepsis to draw upon the wider health-care team. So we worked on the principle it could be sepsis and anybody could raise sepsis as an issue, and then you would go through the sepsis protocol to determine whether or not it was. Most of the people who give their hands on hour-by-hour care are the nursing staff, so diagnosis wouldn’t necessarily fit within their remit, so what we try to do is shift the emphasis away from very specific defined outputs to a much more broader set of physiological presentations that would alert people to say sepsis could be occurring here, you need extra help, so what we did, we’ve broadened the base of people who would raise the concern around sepsis based on a number of very broad physiological criteria.

Nursing director, Sep-m.09

Innovative sepsis awareness training

A training programme, ‘Power Training’, was introduced. This used social movement theory to spread knowledge of sepsis awareness. A concise programme was devised that covered the main learning points, which were distilled into five or six Microsoft Powerpoint^® (Microsoft Corporation, Redmond, WA, USA) slides. This was delivered in < 10 minutes. Those trained were then responsible for training 10 more staff. This led to 1000 staff being trained within 6 weeks. This ‘Power Training’ was referred to as follows:

So we did one thing we developed here which was actually quite innovative, is something that we call Power Training. We distilled down into no more than about five to six PowerPoint slides, the very, very key fundamentals, the information you’re trying to get across. You’re not allowed to go beyond 10 minutes. We set some, we tried to use social movement theory, so we tried to get, so we trained 10 people, those 10 people have got to go away and train another 10 people and those 10 people, so it was more of a catalyst to prompt so it was simple enough that I could give it to anybody as part of the clinical team and their responsibility was to go and find another 5 or 10 people and train them. So very rapidly over about a, I can’t remember, it was about a 6-week period, we trained nearly 1000 staff.

Nursing director, Sep-m.09

This programme was supported by more conventional marketing of sepsis awareness using posters, boards and leaflets:

At the same time as we did a lot of marketing so posters and boards and other types of training material went out and we made some fundamental changes in terms of how we documented sepsis to make people more aware of it.

Nursing director, Sep-m.09

Challenging clinical practice and rewarding good clinical practice

Senior clinicians checked that the environmental items facilitating the management of sepsis were in place. They also then verbally tested clinicians’ knowledge of sepsis training. This process was known as ‘Check and Challenge’:

So after about 3 months, yeah, we did something called Check and Challenge. So Check is, you go physically go the environment and check whether or not all the environmental things you expect to be in place to manage sepsis are in place, so pictures, boards, stickers, blood bottles, everything you would physically, so you know people could not come back to you and say, ‘Well, I didn’t know about it because there wasn’t a poster’, or people say, ‘Well there was no stickers available or there was no blood bottles’. So you physically check the environment . . . And then you challenge people’s knowledge on the power training, so you go, there was a, you take 5 or 10 people from a ward or environment, anybody who’s physically there, so on the labour ward, actually, the chief medical officer was there because he was a gynaecologist, and there were loads of doctors and there was a medical student and there were some midwives, so whoever’s there, you go through a set series of questions and you test out their knowledge as to how much they really understood what was in the programme.

Nursing director, Sep-m.09

Good performance in the ‘Check and Challenge’ process was rewarded in the form of an Outstanding Service Care Awards (OSCAs), and this often led to competition between wards:

We developed a sepsis award. So, we have something every year called the OSCAs, which is the Outstanding Service Care Awards . . . And it’s a way we get the very best of what happens in the organisation. We give them prizes and days out for people. And that year, because we’d had quite a challenge around sepsis, we had a sepsis award and it went to the maternity team. The maternity team were one of the people who had one of these sentinel events. But they trained, and they trained everybody in maternity, both community midwives and also inpatient midwives on sepsis, they go the best scores through the Check and Challenge process.

Nursing director, Sep-m.09

Changes in coding guidelines

From the alerts it became clear that the coders did not know how to code sepsis, and so guidelines were written:

The other thing that was found is that the coders weren’t really clear at all, in what they should put into what codes because it wasn’t clear. So we wrote some common sense guidance forms to what counts as what.

Deputy chief medical officer, Sep-m.09

Changes in service design

Changes in service design and the relationship between the ED and acute medicine were reported as an outcome of the alert and sentinel events. A holding area was created that allowing patients to be triaged from outside into this area, which enabled patients to be fast tracked. GPs sent patients to one area and ambulances to the other, thereby preventing patients waiting on trolleys in corridors and reducing patient waiting times.

The relationship between the acute medical unit and ED, and acute medicine, so two physical areas and arguably three teams, got sufficiently complicated that what they did was they created a new holding area. So they triaged patients from outside into this holding area, where they were seen as a fast-track arrangement, really. And the GPs were told to send the patients to either one or the other, and the ambulances to the other. So they got the waiting times more or less under control, but they’ve got more control of what goes on in each unit and less of the sort of lying in corridors scenario that we’re faced with.

Mortality lead, Sep-m.09

Wider cultural change

The alert for sepsis was recognised as a useful trigger and a focus for action on sepsis as well as for wider cultural change focusing on improvement in care pathways:

We talked about having a plan per se rather than just chasing the lights is because we know that some of it is more than just the indicators themselves; it’s about culture, it’s about looking at the whole pathway and, therefore, if you just chase the lights what you do is you miss some of the real longer term stuff that you have to do that eventually will improve the situation around alerting.

CEO, Sep-m.11

A focus on resourcing data analysis

It was noted that the alerts had resulted in investment in specialist analysis of trust mortality. In Sep-m.11, a specialist information manager was recruited and the number of coders increased:

Nowadays it’s not me doing that because we’ve got analysts who do all of that stuff . . . certainly we were a way ahead of the CQC letters when they came.

Associate medical director, Sep-m.11

Comparative analysis of parallel development of processes for sepsis mortality

It was conceded that the combined parallel evolution of trust structures, processes, culture and leadership contributed to improvements in sepsis mortality. Parallel evolution was reflected in the following common themes:

• external intervention by regulatory authorities

• routine use and triangulation of mortality indicators

• difficulties in resourcing clinicians and triangulating information in case note review and analysis

• developing a trust culture focused on shared learning to deliver high-quality care

• engagement with external organisations for knowledge and expertise around sepsis improvement

• differing clinical perception of care bundles leads to confusion and error (normative pressures)

• universal case note review

• integrated structures and processes supportive of sepsis mortality review

• addressing poor clinical awareness of sepsis and difficulties in coding

• persuading consultants to engage in mortality review

• IT-enabled forensic mortality review and patient monitoring

• senior leaders prioritising trust mortality reduction

• clinical champions improve timeliness in administering first doses of antibiotics.

External intervention by regulatory authorities

It was suggested that Sep-m.07 had implemented many strategies to address both sepsis mortality and mortality in general. This was driven by a host of regularity pressures resulting from the trust being identified as an outlier for SHMI and HSMR mortality indicators. These included the Keogh Review (February 2013), Special Measures (2013) and CQC inspection (2014).

Respondents reported that this external scrutiny directed action towards addressing mortality failures. For some, this scrutiny was a distraction from looking at the key problems concerning quality of access for patients:

When there’s so much external scrutiny on an organisation you tend to start throwing a bit of resource at it and it gets more attention. It becomes a distraction more than anything I think. I’m not a great believer in HSMR, SHMI and you say that’s because you’re at the opposite end of the scale. What I really believe people should be doing – people say we have the quality of access to health care in this country, well we don’t, and quite clearly we don’t. You go to different hospitals and you’ll have different levels of care because each hospital has different types of services, different departments. Your outcome will be different where you go. So if you have an acute MI [myocardial infarction] and you come into a hospital like [Sep-m.07] where we don’t do acute PCIs. You would need to be put in the back of an ambulance and sent somewhere, and you’re outcomes and results are going be worse than if you went to a hospital that has that thing on site.

Deputy medical director, Sep-m.07

However, it was conceded that that external scrutiny ‘had raised our game in terms of actually understanding our services’ (deputy medical director) and it was noted that improvements in care had resulted from this scrutiny, especially in terms of understanding and using data:

I think has actually been extremely good for us raising our game in terms of actually understanding our services and being able to present them and report them.

Deputy medical director, Sep-m.07

Routine use and triangulation of mortality information

Sep-m.07 was viewed as becoming ‘data rich’, using IT-enabled CHKS, HED and Dr Foster tools and accessing internal trust data via Meditech (electronic patient records), the mortality monitoring system, safety thermometers, ward dashboards and score cards:

And it covers all of our dashboards and thermometers and serious incidents and so on, to reassure them effectively. It’s to reassure our commissioners from a quality perspective. And I think as an organisation, I mean, we are very data-rich, and I think a lot of what we’re trying to do generally, particularly with the mortality statistics and reports, is to try and describe them to the appropriate people in context where they can be assured that there aren’t any serious problems.

Interim medical director, Sep-m.07

A focus on accessing triangulating and using mortality information also drove improvements in mortality. Many respondents identified how they accessed Dr Foster data to identify trust mortality in depth and understand how coding fed into mortality indicators:

You can go on the system and you can look. At the moment I know there’s some things, something called Subcutaneous skin legions, we’re sky high on HSMR, very small numbers of deaths, but when you go onto them they’re all bed sores for people who are from nursing homes who are bed bound who are coming to the end of their life. So you can go on and you can look, and the other thing that we’re code high on is things like VF [ventricular fibrillation] arrests.

Deputy medical director, Sep-m.07

A comparison of mortality indicators across trusts also was useful for identifying trust problems in coding; for example:

From the SHMI where you have the comparative one, and you can compare trusts and you can look at how mortality how palliative care coding has changed. You have to have local trust and then look at them from a CCG point of view to have a look at your own coding because I was thinking, ‘I wonder if we’re coding things differently’ and that’s how I got interested in this pneumonia one because I thought, ‘Blimmin heck we’re just not coding pneumonia’.

Deputy medical director, Sep-m.07

Developing a trust culture focused on shared learning to deliver high-quality care

In Sep-m.11, leaders report a key strategic focus in developing a trust culture supportive of high-quality care through shared learning and staff engagement in continuous review and improvement in care pathways.

In addressing mortality, those at Sep-m.11 did not make an assumption that mortality was a result of coding but rather ensured that all aspects, including care and coding, were considered:

The assumption is that this is not coding, this is a quality issue until proven otherwise. We have quite a high percentage of people who die in hospital as well.

Medical director, Sep-m.11

Despite espousing this cultural shift, it was noted that it was difficult for the trust to resource coding and pathway investigation. The coding department was described as stretched, with 4.5 whole-time equivalent posts vacant.

Sep-m.11 realised that it needed an organisational development strategy and a method for delivering culture change. Learning from other services, it adopted an approach involving four elements. Part of this involved developing coaching skills in the organisation so that different people could coach others on an improvement project: coaching circles. The view here is that by ‘chasing’ and dealing with each alert individually, the longer-term culture change that is needed will be lost:

I think that is where it was important for us to develop the care of the acutely ill action plan because it became really clear to us that you could just be chasing alerts. And even in HSMR, I think HSMR has I think 56 conditions or whatever else that sit within that, I think SHMI I think is 130-odd, you can get into the fixation of which this is. And we’ve done some of that. but the reason why we talked about having a plan per se rather than just chasing the lights is because we know that some of it is more than just the indicators themselves; it’s about culture, it’s about looking at the whole pathway and, therefore, if you just chase the lights what you do is you miss some of the real longer-term stuff that you have to do that eventually will improve the situation around alerting.

CEO, Sep-m.11

Knowledge of mortality was shared by the electronic sharing of case note reviews and coding knowledge was shared through the trust’s intranet. Hence all consultants can view data concerning all deaths reviewed:

It’s open access, you can search by consultant so you can have look at the reviews on patients under their care or you can look at what the reviewer’s found in the review. You can look at it by ward. There’s all sorts of ways you can cut the data . . . We said, ‘Well if we did them all upfront then actually we could pull through anything on COPD [chronic obstructive pulmonary disease]. Right let’s have a look at all the deaths that are reviewed on the patients that had a primary diagnosis of COPD or pneumonia’ and they just pull it and cohort it that way. So it’s much easier because the reviews have already been done.

Head of governance, Sep-m.11

Despite this trust’s drive for transparency and learning, it was suggested that the mortality and morbidity meetings were not always open learning environments, as junior doctors were scrutinised by senior doctors:

Perinatal people have always looked at theirs, but again it’s usually a case of a junior doctor presenting the case and the obstetrician sniping between one another to see what they do, it’s not a learning exercise particularly.

Associate medical director, Sep-m.11

Engagement with external organisations for knowledge and expertise around sepsis improvement

Sep-m.07 engaged in extensive external engagement that supported improvement in quality of care. Reviews focused on problems areas such as COPD, sepsis, ITU and mortality review. For example, reviews were undertaken by the Royal College of Surgery and local critical care networks. The trust called in experts on mortality statistics and reporting from academic institutions and experts in quality improvement specifically for sepsis:

So I’ve invited in external reviews, so over the time we’ve had one in respiratory looking particularly around COPD, we had an external review of the Royal College of Surgery by the Royal College of Surgery in [inaudible] and we also had – well the [XXXX] Critical Care Network came in and did a review of the ITU. All of these things were triggered by slightly different things, but it was an external assurance visit to really get an understanding of, ‘Is there a problem here that we should be highlighting’.

Former deputy medical director, Sep-m.07

In Sep-m.11, it was noted that ongoing sepsis improvement, stimulated by participation in the SPI national programme, had led to some improvement in mortality for sepsis and had also been a driver for action. A nurse consultant in the trust had been the sepsis lead for 5 or 6 years (so at approximately the time of the first alert), and the trust was also working on introducing the sepsis care bundle as part of its work for the SPI. Hence, the view was that the alert was not the sole driver of improvement for sepsis mortality:

There wasn’t anything major and I think that was quite evident. We already had a collaborative group looking at sepsis which was very active at the time and we’d already made some headway into our outcome targets around reducing mortality. We already had a step change so in a way, because we were looking back retrospectively, a way back, that’s when the data’s published, we were quite assured and obviously our local CCG were as well because we built that into the actual planning. Most of those were already in our driver diagram for that piece of work.

Sepsis lead, Sep-m.11

The continuing high HSMR prompted Sep-m.11 to review its coding, and it received an external review by a large consultancy company. Sep-m.10 also has an active programme of quality and safety improvement and has sent staff to the Institute for Healthcare Improvement international forum, with the chief nurse attending four times. This methodology was applied by junior doctors applying PDSA (Plan-Do-Study-Act) cycles and auditing implementation of sepsis bundles and antibiotic prescribing within ‘the golden hour’. Likewise, the trust also had plans to resort to further external engagement if the received more alerts.

Universal case note review

Routine case note review of all deaths was implemented across trusts; hence, when an alert was received there would be a review of the deaths contained in it.

You see we review all our deaths, so we know. So in terms of, you know, is there any excess work? No, there isn’t. So when the alert arrives, if an alert at that level arrives rather than just a red alert, if a CQC alert or an Imperial College alert which you copy to CQC, if that arrives then we look at it and we go, ‘What period does this cover? OK, this covers these 6 months’, and we know the numbers in those 6 months and over the last two occasions we’ve gone, ‘OK, well we’ve reviewed all the deaths in that period already’.

Chief medical officer, Sep-m.09

A report is produced every 2 weeks in Sep-m.09. This is discussed by the mortality committee, which meets every 2 weeks for 4 hours and scrutinises case note reviews, Dr Foster mortality data and other trust mortality indicators:

It’s been overall completely essentially because it used to be once a month for 1 hour. It’s now twice a month for 4 hours, so altogether we spend 4 hours talking about mortality in a meeting forum. We have a systematic way of approaching not just the HSMR but we look at our palliative care coding, we look at our comorbidity rates, we look at the alerts and we look at which alert needs investigating, we look at who’s going to investigate it, we look at the pathway, so if there’s an alert or high relative risk for say CCF, congestive cardiac failure, then we look at those number of notes, maybe 20 sets of notes, and we will work out whether there’s a problem pre hospital or post discharge or it is just in hospital care that’s causing the alert.

Deputy chief medical officer, Sep-m.09

The associate medical director of Sep-m.11 noted that the trust had been forensically examining mortality since 2006, so alerts were never a surprise. In December 2014, the trust implemented a two-tier review of mortality. Twenty people/teams are involved in this review process, which is two staged. Stage 1 examines key indicators, and, if any of these highlights a preventable death, a more detailed review (stage 2) is undertaken. If the first-level review finds any issues of concern, then a second-level and more in-depth review is undertaken; at this deeper level serious issues of preventability are highlighted but also any issues of quality of care are addressed.

So the current system, we have a two-level review and this has just been running since December where we have a ‘quick and dirty’, if you like, level 1 review where we look for some key indicators around preventability really. So that review is done by, it’s either a senior nurse or a senior doctor in the organisation who will look at them. If they have any concerns around either preventability but also quality of care issues, then they can refer them for a second-level review and a more detailed review until it’s then completed on a level 2 review.

General manager for governance, Sep-m.11

Integrated structures and processes supporting mortality review

Respondents in Sep-m.09 described the current review process, whereby mortality alerts come in to the CEO and are sent on to the chief medical officer (who refers it to the mortality lead and mortality review committee chairperson) and to the clinical effectiveness manager (who co-ordinates the reviews). There is also a parallel coding and case note review. The mortality review follows a detailed integrated and systematic process. The mortality review committee comprises the three assistant medical directors, the chief medical officer, the head of governance, the head of coding, the nursing director, the mortality lead, two mortality administrative staff, representatives from the divisions and specialties and the Dr Foster representative. It was noted that all clinical leads were expected to attend:

At the committee there’s the chief medical officer and her three deputy chief medical officers, deputy chief nurse, the director of governance, some other governance support, and then each specialty, there’s about five specialty-related mortality leads who come regularly, and every other specialty has a mortality lead who comes and presents either reports off investigations or a six-monthly update of where they’re at in their department. So it’s quite well, we could have more, more of a network around the one person trying to do it, but that’s taken several years to grow and kind of become embedded in the system.

Deputy chief medical officer, Sep-m.09

In Sep-m.09, mortality profiles are produced for each specialty from the Dr Foster data and these also include data on the time of day, day of week and staffing levels. These data are considered important as each specialty contributes in one way or another to the overall trust HSMR, and so the chief medical officer wants all specialties to work towards reducing mortality, not just those that have high levels:

. . . so we produce specialty mortality profiles for specialties, so we give them specialty-specific mortality data. So, on an ongoing basis we try to add definitions, etc., to all those charts, the relative risks, so if the clinician’s got any questions after looking at SMPs [Specialty Mortality Profiles] they can always just come back to us and if there are any blips we try to explain to them what these blips are, and can we just be statistical anomalies or not.

Clinical effectiveness manager, Sep-m.09

You can also cross-check them with other specialities. So for example, quite a lot of people die in the intensive care unit, so you get two points of opinion on that.

Mortality lead, Sep-m.09

In Sep-m.09, the CCG chairperson also now attends the mortality committee:

So if we feel that the referral was, or the patient came in a moribund state or, you know, what was going on in the community, how did we get to that stage? And also for palliative care in terms of getting people to their choice of place to die, the fast pathway, all of that, if it doesn’t work, or if somebody has a plan in place but they are referred in, in their last hours of their life, then that’s not good for the patient. So it’s not just about number, it’s about patient care.

Chief medical officer, Sep-m.09

The board Sep-m.09 receives a monthly report to the quality governance committee, which is a subcommittee of the board. There is then a 6-monthly detailed report to the board:

And one of the things we have done, that’s me and the director of quality, is we have made sure that the non-executive directors, we’ve sat down with them for an hour and a half, 2 hours, explaining everything because for someone who’s not of a medical background, understanding it to someone is not easy.

Chief medical officer, Sep-m.09

In Sep-m.11, there was concern that the alerts did not help to resolve clinical issues, so the mortality review process was considered a much better way of finding underlying clinical issues:

. . . my background beef with the HSMRs and how you can focus it down to diagnostic groups and say, ‘Oh my god we’ve got’ and which is why we’re trying to get mortality case note review that will argument as to what an HSMR means, but when you start dividing it down into specific diagnostic groups you can pop one up there and you hit that and then another pops and you hit that, and all you’ve done it slightly coded them differently and you think this is a game but did we gain anything from doing it and you can chase them around endlessly and we were doing that for years.

Associate medical director, Sep-m.11

Addressing poor clinical awareness of sepsis and difficulties in coding

In Sep-m.11, coding for sepsis on admission had been a difficult issue as not all doctors who admitted patients realised that they needed to write sepsis in the notes. As a result, the sepsis team introduced the use of a sticker on the notes to identify anyone complying with and being treated under the Sepsis 6 banner:

But if it’s not written down . . . So they have to see – and what we have been using I think is our sepsis boxes, so our stickers, they have taken that as a sepsis code.

Nurse consultant, Sep-m.11

Information technology-enabled forensic mortality review and patient monitoring

The entire mortality review process in Sep-m.09 is electronically enabled. When a patient dies, the IT-enabled mortality review process generates a primary e-review form that is sent to the consultant responsible for that patient. The consultant is expected to access the case notes and complete the form within 30 days. Follow-up e-mail reminders ensure that this happens. After reviewing the case notes, the consultant completes the e-mortality review form and grades the death on the scale A–E. A denotes a death that was entirely unavoidable, whereas E denotes a serious incident. Deaths in the grade E are tackled immediately by the serious incident group, with a full review undertaken and presented to the mortality review committee. If a death is graded B–D, then the case is examined in detail during the specialty-level mortality meetings. This grading was explained as follows:

And then that will be reported as an E death. It’ll go to the serious incident group, who will take action very much immediately because it’s the kind of thing that will end up in the newspapers if you don’t, but then it’ll also go in a slightly slower and more formal way to the MRC [Mortality Review Committee] group, who will then pretty commonly decide that a root cause analysis investigation is required, and then that follows that track, really. And then the outcome of the root cause analysis will be disseminated. We’ve just had a look at the last 27 E deaths and kind of bundled them together to date, and what we’re planning to do is come up with a presentation quite soon, probably next month when they grand round system comes back, to disseminate the findings in an educational way.

Clinical lead for mortality, Sep-m.09

In Sep-m.09, the clinical effectiveness manager oversees the mortality review process and ensures that local clinical guidelines are in place and comply with National Institute for Health and Care Excellence guidelines.

Continuous IT monitoring of patients in Sep-m.09, in real time, serves to communicate changes in patient condition and enables treatment to be escalated quickly. This was evidenced by clinicians regularly inputting into the NEWS (National Early Warning Score) and OPERA systems. The oversight of the NEWS system was conducted daily by the sepsis outreach team.

In Sep-m.10, it was hoped that the introduction of electronic patient records in the near future would further support improvements in coding and informatics:

So there’s an actual medication definition for septicaemia, which these patients perhaps didn’t fulfil, OK? But you have got them, if you like, labelled as having that and therefore it enters the mortality spike. So we have got concerns about that. We’re very hopeful that moving to an electronic patient record and our data collection and monitoring and assessment will be far superior in a year, 18 months, because we’ve got a huge IT project happening here called Smart Care, which is procuring a whole electronic patient record, etc., etc. So we’re hoping some of these issues will be flushed out with that.

Consultant oncologist, Sep-m.10

Senior leaders prioritise trust mortality reduction

In Sep-m.11, the analysis suggests that the trust board and the senior executive team appeared to prioritise mortality; those on the trust board were described as being very interested in mortality data and HSMR. They took an active interest and would ‘grill’ the executives; they ensured that mortality data were included in the monthly reports and every second month they received a report on the programme to improve care for the acutely ill patient.

Trust senior leaders in Sep-m.11 highlighted that they wanted more timely alerts, and they noted that the alerts arrived a long time after the trust became aware of the issue. They also suggested that they were trying to be more proactive by monitoring trends so that they could avoid the stage at which an alert would be received:

For me, if we are getting to an alert situation where statistical outlier, where we’ve failed in some way because we shouldn’t have got that far, so that’s why we are trying to be more proactive and pre-empt that. So if we see something that is moving or above average or is heading in the wrong direction then we will try and investigate that before we get an alert.

Medical director, Sep-m.11

In Sep-m.10, senior leaders visibly promoted quality improvement in their regular walkabouts and drop-in sessions. Every month senior nurses, including the chief nurse, completed clinical shifts followed by a debriefing session. A senior nurse also highlighted how trust leaders had resourced a ‘bottom-up approach’, enabling junior doctors’ engagement in implementing improvements in care. Hence, a ‘bright ideas’ group had been set up to mobilise these changes:

I think leadership. From right at the top, I think leadership. I think actually having the resource to actually focus it, being given permission, so for [safety lead]’s post to then go out and get all the junior doctors and be involved in that and, again, then getting the junior doctors to lead on it. So I think it’s definitely about the leadership and having the time to resource it.

Nursing and midwifery executive director, Sep-m.10

Similarly, ‘safety cafés’ were set up to promote clinical involvement in improving care. This was reported as follows:

A safety café is where you have different tables – you have to give them a cup of coffee – lots of tables, and you’ll have different speakers who will do mini presentations. So we get a lot of the consultants . . . lots of senior nurses, actually talking to them about actually new documentation in the group, so you can get a lot of people around in one go.

Nursing and midwifery executive director, Sep-m.10

Clinical champions improve timeliness in administering first doses of antibiotics

In Sep-m.10, a consultant oncologist reported on how clinical champions in the ED had implemented the ‘FAB 60’ project, which set out a quality standard that all first doses of antibiotics (FAB) should be given within 60 minutes of prescription:

Well, we approached it hospital wide and it’s been picked up very quickly, and successfully in ED and acute care areas, partly probably because I’ve had two very motivated consultants, from ED and acute medicine, who have been on our group, who have been very good at taking the message back and supporting that.

Consultant oncologist, Sep-m.10

This had moved on to implementation in wards and an auditing of this process to monitor the time to administration of the first dose of antibiotics:

Our next plan is to deliver a new quality standard in the hospital called FAB 60, OK? F-A-B, we’ve got some branding. There’s some branding with it [laughter] . . . And so first dose antibiotic . . . So all first doses of antibiotics, FAB, should be given within 60 minutes. And we have done a first-wave audit of every ward as to how many times that’s being delivered.

Consultant oncologist, Sep-m.10

In Sep-m.10 it was also reported that the scoring of FAB could be included in the future implementation of the electronic prescribing system, so a scoring could be set visibly within a ward ‘dashboard’ and hence prevent delays in the administration of antibiotics.

Comparison of trust preconditions

A comparative analysis of trust pre-conditions generated the following common themes:

• recognition of high mortality for AMI

• evidence of tracking mortality for AMI

• changes in senior leadership

• poor finances

• external engagement to support learning around AMI mortality

• catchment characteristics influencing high AMI mortality.

Recognition of high mortality for acute myocardial infarction

Respondents in both AMI-m.03 and AMI-m.04 recognised that the trust had high mortality levels for AMI. In AMI-m.03, the mortality lead noted that staff had known that the trust had a history of high levels of mortality before the first AMI alert. He suggested that this knowledge had provided a trigger for trust participation in PRISM. 23 This national study aimed to examine avoidable deaths, develop a standard approach for local mortality review and focus on the review of 1000 randomly selected case notes of patients who died either in hospital or within 30 days of discharge:

Well the trigger for that was an historical knowledge that there was an apparent mortality excess on the then measures, and as well as beginning to look in detail at various cases, it was felt that we should participate in the PRISM study as well, and that again didn’t really show any clear deficiencies in care, that we could attribute to historical apparent excess mortality.

Mortality lead, AMI-m.03

In AMI-m.04, data quality appeared to have been reasonable before the MASS alert, as the information project manager had been aware that the trust was going to receive an alert. He had informed senior members of staff and an investigation had been instigated:

. . . I think it was before – we were already investigating it before the CQC said that they were going to investigate it, I think. Yeah, because we already had that background data there.

Information project manager, AMI-m.04

Evidence of tracking mortality for acute myocardial infarction

The first alerts in AMI-m.02 arrived in 2008 and the trust was reported to be using the Dr Foster tools. However, the contract ended in around 2011; there was a gap before they implemented CHKS in 2012, and then in early 2015 they changed to the HED tool. AMI-m.03 was at an early stage of tracking mortality using a CHKS system, which provided information on RAMI. The trust began to track RAMI, SHMI, HSMR and crude mortality. In AMI-m.04, tracking of trust mortality appeared to be less efficient, as the trust was using a hybrid system of paper and electronic notes. This led to more time for coding and analysis as information was accessed on different systems.

Changes in senior leadership disrupt trust mortality review processes

AMI-m.02 had a difficult time for many years, with several changes in senior management. None of the interviewees was in post in 2008, when the first alert arrived, but some were able to comment on changes since. Respondents identified that the former medical director, who was in post before the first alert, had had limited involvement in clinical governance. The clinical outcomes manager also reported that the trust board members had changed many times and that this had limited their oversight of trust affairs:

We’ve had a very changeable board. The board we have in place now is very different to what we’ve had before. They’re much more involved in what we do.

Clinical outcomes manager, AMI-m.02

In AMI-m.04, it was reported that a change in CEO (in 2011) appeared to have been instrumental in changing how the trust approached mortality review. Hence, the trust had moved from a ‘command and control’ leadership style, which focused on ‘following the rules and meeting deadlines’ and producing only a superficial account of trust mortality, to empowering staff to engage in mortality review. The mortality lead reported this as:

. . . we’ve had a change of chief executive, probably 4 years ago now . . . but we had a very command and control attitude. There’s a palpable difference in atmosphere now . . . I think we would have, we’d have done everything by the rules. We would have produced a report; somebody would have cracked a whip to make sure that a report was produced by the deadline date. It might not have been a great report, it might not have really felt real.

Mortality lead, AMI-m.04

Fragmented approach to reviewing trust mortality

Respondents in AMI-m.04 identified that the trust dealt with mortality in silos. They suggested that mortality review lacked integration with clinical governance, safety and complaints structures. Similarly, the analysis of trust HSMR was not linked with either complaints data or safety data. Respondents described the trust as having a ‘tick-box’ governance system. Mortality review was confined to corporate level and was undertaken by the central mortality review group and the clinical audit department. Similarly, the coding of mortality lacked clinical oversight.

In AMI-m.03, it was reported that around 2008 the medical director had asked one of his colleague to help review case notes of deaths. This was regarded as an informal weekly activity aimed at gaining a sense of mortality problems in the trust. After normal working hours, a review was undertaken of deaths for that particular week. It was conceded that ‘we didn’t know what we were looking for’ (deputy medical director) and that the exercise involved finding ‘out the current episode of patients’ journey and see whether they were actually managed properly and they had written the right diagnosis on the death certificate’ (deputy medical director).

AMI-m.03 possessed a centralised senior team to oversee trust mortality. However, it was suggested that within divisions and some specialties mortality teams were often inactive and it was noted that no mortality meeting took place in acute medicine. The CEO (newly appointed) reported that she chaired the ‘mortality committee’ and that when she first had arrived at the trust it had operated poorly. She suggested the committee had been overwhelmed by ‘reams of information’ and, hence, this made it difficult to make sense of mortality in the trust. It is posited that this arose owing to the absence of a supporting specialist infrastructure.

Poor finances influencing resourcing of mortality review

In AMI-m.02, a reduction in finances appeared to have influenced a ‘culling’ of governance staff 4 years ago. It was recognised that more governance staff were needed to support mortality review, but this had not been remedied.

AMI-m.04 recorded a deficit of £17.4M in 2013/14 and had been expecting to end 2014/15 with a deficit of £2.9M. This later provoked an investigation by Monitor. The CEO highlighted that the trust was a private finance initiative hospital and that this led to an exorbitant mortgage payment:

It’s a PFI [private finance initiative] hospital, and because it’s a PFI hospital it’s always so – you know, most trusts do not have to pay a £20 million mortgage payment. And so the figures have always been quite tight in terms of finances, and particularly cash. Not just income and expenditure but the cash that underpins that. So we haven’t historically built up savings here, so we haven’t got big surpluses that we can live off while things are getting together . . . So it’s always been quite tight financially and quite close and up against it.

CEO, AMI-m.04

External engagement to support learning around acute myocardial infarction mortality

In AMI-m.04, efforts were also made to work with NCEPOD (National Confidential Enquiry into Patient Outcome and Death) and provide external awareness of the processes and issues surrounding hospital mortality. In AMI-m.04, trust governance was considered not up to standard as the new leadership had appointed Deloitte to undertake a governance review:

‘No, the governance of that’s not right.’ So I brought Deloitte’s in, in my first year . . . So we did a quality, governance review.

CEO, AMI-m.04

In AMI-m.02, cardiologists reported in to the national clinical audit (MINAP) and received reports back reviewing AMI mortality. It was suggested that this feedback was the main trigger for action concerning mortality.

AMI-m.03 participated in the PRISM study, which also involved training experienced hospital consultants to assess the preventability of adverse incidents and the effects of the adverse incidents on the subsequent clinical progress of the patients. It was reflected that participation in this study had indicated that the trust had no clear deficiencies in care:

Well the trigger for that was an historical knowledge that there was an apparent mortality excess on the then measures, and as well as beginning to look in detail at various cases, it was felt that we should participate in the PRISM study as well, and that again didn’t really show any clear deficiencies in care, that we could attribute to historical apparent excess mortality.

Mortality lead, AMI-m.03

Catchment characteristics influencing high acute myocardial infarction mortality

Respondents identified that trust catchment characteristics influenced high levels of AMI mortality. The catchment of AMI-m.02 was described as ‘very deprived’ and it was suggested that this limited progress in improving HSMR. The clinical outcomes manager reported this as follows:

The board at the time kept saying it had to be 90, it had to be – but I think – you know, this is quite a deprived area. We serve a big population . . . we’re one of the fourth biggest trusts in the country, you know, and we are in a fairly poor area. We do have – you know, our case mix is very, very different . . . I think the reaction in the past has been to justify where we are.

Clinical outcomes manager, AMI-m.02

The catchment of AMI-m.03 was described as having high incidence of alcoholism, end-of-life care issues and poor community care. The deputy medical director reported this as follows:

I know these are isolated cases but they make the biggest picture. These are the ones we picked up, so this was reported to the board meeting so the background came, lots of them with an alcohol problem and end-of-life care issues and then we realised the services in the community are not as good to manage these patients. Now I know every trust will say that but if you look at the profile and I’ve got the profile of [trust AMI-m.03] from 2008 and every annual one, I just find it really bad and I’ve been in [AMI-m.03] for 32 years [laughs]. I love it working here, but it is – the profile is very bad and it is evident as well when you go around.

Deputy medical director, AMI-m.03

In AMI-m.03, many respondents suggested that the risk adjustment weighting for these factors within mortality indicators was inaccurate:

We’ve got one of the most deprived electoral wards in the country, there is a huge rate of smoking, drug abuse, teenage pregnancy, all of those things, which I know the SHMI and HSMR say they correct for but I still can’t believe that they properly correct for all of those factors.

Medical director, AMI-m.03

AMI-m.04 was also noted as serving a population of 214,000, where deprivation is lower than average. A merger with a local community trust in 2011 also created disruption and pressure on existing structures and processes.

A comparative analysis identified the following divergent themes:

• trust configuration and function influencing high AMI mortality

• a closed trust culture

• senior leaders’ participation in mortality review

• staff shortages.

Trust configuration and function influencing high acute myocardial infarction mortality

In AMI-m.02, the configuration of services appeared to be linked to AMI mortality. Before 2014, cardiology was split between the two sites, meaning that the catheter laboratory was situated at hospital A and acute services were located at hospital B. This meant that there was a ‘split site mentality’ (divisional director for specialist medicine), which prevented an integrated approach to cardiology. Similarly, arrangements were in place for patients with suspected AMI to be taken directly to a central myocardial infarction centre rather than to this trust, indicating that most people who died as a result of AMI were inpatients and not patients arriving in the ED.

A closed trust culture

In AMI-m.03, the director of nursing reflected that at the time of the first alert the trust lacked transparency and this was due to a lack of good information:

We didn’t have the same I suppose transparency in terms of systems to see that that was maybe starting to be a cause for concern. So my recollection going back to 2011 and I think some of the conversations we had at the time and obviously we were a different exec team, was probably feeling a little bit frustrated that the alert would come but it didn’t feel we were able to have predicted that ourselves.

Director of nursing, AMI-m.03

Senior leaders’ informal participation in mortality review

Before the alert, senior leaders’ informal participation in mortality review was the sole type of mortality review undertaken in AMI-m.03. It was reported that around 2008 the medical director had asked one of his colleagues to help review case notes of deaths. This was regarded as informal weekly activity aimed at gaining a sense of mortality problems in the trust. After normal working hours, a review was undertaken of deaths during that particular week. The deputy medical director conceded that we didn’t know what we were ’looking for’ and that the exercise involved finding ‘out the current episode of patient’s journey and see whether they were actually managed properly and they had written the right diagnosis on the death certificate’ (deputy medical director).

Staff shortages

Staff shortages had been reported in AMI-m.02 before the alert. It was noted that reduced finances had influenced a ‘culling’ of governance staff. Although more governance staff were needed to support mortality review, the number of staff was described as not having changed much. The head of coding also highlighted difficulties in recruiting people who had specialist coding expertise:

In 2008 the coding department here was being managed by a contract manager due to difficulty to find [staff] . . .

Head of coding, AMI-m.04

Comparison of trust responses to multiple acute myocardial infarction alerts

A move towards greater trust-wide participation in mortality review

Respondents suggested that their trusts were moving towards greater trust-wide participation in mortality review. AMI-m.03 had set up departmental coding and comorbidity meetings wherein all consultants, under the instruction of the previous medical director, were to review deaths. These deaths were then discussed during the mortality and morbidity meetings:

The steps we took were set up mortality and morbidity meetings within the departments. We asked – and I’ve got evidence to give you if you wish is by that previous medical director to say, ‘All deaths must be looked at by the consultants so that they can – the majority of them we won’t have any problem but some of them, if there are any issues, must be discussed in the mortality – M and M meeting and you should be also in charge of coding because there was an issue of coding’.

Deputy head of information, AMI-m.03

AMI-m.04 was empowering staff to engage in mortality review. The mortality lead reported this as:

. . . we’ve had a change of chief executive, probably 4 years ago now . . . but we had a very command and control attitude. There’s a palpable difference in atmosphere now.

Mortality lead, AMI-m.04

In AMI-m.02 there were mortality and morbidity meetings at a divisional level, but few of these were robust and the actions generated by them were limited. The divisional director reflected on this:

All the specialties I ensure that they have mortality and morbidity meetings which are documented, represented clearly and are all in ledger folders so that I actually can see what actions are. Few of them are more – well, put it this way, few of them are more robust, you will actually can articulate what the actions are out of those, few of them are not so robust.

Divisional director, specialist medicine, AMI-m.02

The medical director also reflected on the inadequacy of mortality review processes:

The way data was used when the alerts first came to the trust was not very good. Meetings surrounding mortality did not have a clear strategy, and nor was there a clear focus on how to improve areas that were clearly signalling a problem.

Medical director, AMI-m.02

Trust engagement with external organisation for support in mortality review

Engagement with external bodies for support in mortality review was a feature of one trust’s response. Respondents in AMI-m.03 suggested that a direct consequence of the alert was the commissioning of the Advancing Quality Alliance (AQuA) to undertake a comprehensive review of the trust’s mortality rates. This review focused on examining a range of factors that could explain the current and historically high mortality rates at the trust. This review centred on:

• the implications of local population characteristics for the methodology used to calculate expected mortality rates

• the impact of provision of services and quality of care both within the trust itself and in the wider community

• the systems and processes that are employed within the trust.

The conclusions of AQuA focused on the following:

• recognition of the trust’s strong track record in clinical quality improvement and that the non-mortality related quality indicators did not give concern

• the trusts’ standardised mortality rates (SHMI, HSMR or RAMI) were likely to be adversely affected by the way clinicians and the coding team interact and that the ‘expected deaths’ element of the standardised rate calculation can be affected by coding processes (emphasising that this needs urgent attention)

• a call for linking mortality reduction to other patient safety initiatives

• the need to develop a specific mortality reduction plan that cascades to divisions and departments and which reports to the trust board

• improving clinical quality by adopting care bundles, reviewing staffing levels in high risk areas, and working with primary care to improve end-of-life care planning.

A focus on identifying errors in coding of acute myocardial infarction

In responding to an alert, trusts focused on identifying errors in the coding of AMI mortality. In AMI-m.04 the first alert went to the clinical lead for mortality. The head of coding described being asked to pull the notes and examine the coding. An independent coder completed a review of case notes. Once this had been done, a ‘clinical investigation report’ was produced using all available data, including MINAP data and Draper and Dash [www.draperanddash.com (accessed 17 October 2017)]. This review identified that the inexperienced coders were coding using the death certificate rather than the admitting diagnosis. As a result, they were assuming that anyone who died from a myocardial infarction had been admitted with a myocardial infarction, which was incorrect. The head of coding highlighted this:

Because there were various signs and symptoms throughout the patient, the coders had then assumed that the reason for the admission was the MI, so they were not following the coding standards quite correctly due to their inexperience.

Head of coding, AMI-m.04

The coding review highlighted two further problems: (1) junior doctors were filling in the death certificates and not fully reviewing the notes and (2) junior coders were coding these deaths using the death certificate:

. . . the ones that were quite inexperienced were picking out something, like, in the death certificate. In this instance MI would have been recorded as cause of death. Because there were various signs and symptoms throughout the patient, the coders had then assumed that the reason for the admission was the MI, so they were not following the coding standards quite correctly due to their inexperience.

Head of coding, AMI-m.04

In response to the first alert, it became practice in AMI-m.04 for only experienced coders to code deaths:

Following that first alert, the decision was made in the trust that actually only the experienced coders would code deceased notes across the board. So we looked at then changing our practice and we have our band four coders, which are either ACC [Accredited Clinical Coder] or very, very experienced.

Head of coding, AMI-m.04

A system was also introduced in AMI-m.04 whereby coders would ask clinicians to check the coding for AMI. This process was followed if the coders lacked confidence with coding:

As a general theme within the trust, we actually put in a process where – if the coders were actually not overly confident about what they’ve actually coded for that patient . . . we then instigated a process whereby we would send a clinician responsible for that patient just a list of what we’d actually coded for that patient with a covering letter just to explain that this was a complicated patient and the coders weren’t definitely sure exactly what the main diagnosis treated was and could they just advise.

Head of coding, AMI-m.04

In AMI-m.03, coding errors were also highlighted. AQuA recommendations had indicated that the ‘expected deaths’ element of the SHMI, HSMR or RAMI can be affected by coding processes. They emphasised that coding was likely to be adversely affected by the way in which clinicians and the coding team interacted. It was suggested that coding processes needed urgent attention.

A strategy for using data analysis to identify problems in acute myocardial infarction care pathways

In AMI-m.03, and to a lesser extent in AMI-m.04, attention started to be focused on gathering and analysing data to inform the development of AMI care pathways. In AMI-m.03, leadership appeared to develop a strategic focus on generating both quality data and analysis that could usefully inform decision-making. This involved internally tracking mortality, auditing patient pathways and engaging with other trusts to compare performance. AMI-m.03 began to look for other trusts to compare itself with:

. . . so when we then start the investigations we’ll start to then look at quality metrics that are out there around the services that we have, and we start to look at activity patterns and we start to look at the type of services we run compared to other organisations and the fact that you’re a tertiary cardiac centre what does that mean, so again we wouldn’t just compare ourselves to everybody with AMI we would then start to look at trusts to perform tertiary services because again if we’re only getting the – if half are activities from the most acutely ill patients across the whole region, and we’re just getting them for primary PCI then our patient group is going to be different to a general DGH [District General Hospital] that doesn’t need those heavy interventions. It all forms part of that picture.

Deputy director of information, AMI-m.03

AMI-m.03 began to track mortality using CHKS, which provided information on RAMI. It also began to track RAMI, SHMI, HSMR and crude mortality:

When we produced the dashboards initially we used four measures and the idea was to look for correlation, so we put in there SHMI, HSMR, RAMI and crude mortality all the time, so you could actually then start to track through and see the word discrepancies and that’s another area to look.

Deputy director of information, AMI-m.03

In completing an AMI pathway audit, AMI-m.03 located three problem areas: (1) compliance of AMI treatment was a major issue, (2) evidence of the success of the primary PCI integrated pathway was inconsistent and (3) the success of an in-house smoking cessation campaign was also inconsistent. Recommendations suggested that divisions should develop action plans addressing non-compliance and improvement, actions plans should be monitored by the clinical audit department and progress should be fed back to the mortality board. The mortality lead also highlighted how, in developing the AMI pathway, ‘mission critical points of care’ should be identified. They identified areas of care that was evidence based and, hence, demonstrate how accumulated clinical learning transformed the AMI pathway:

We had a long-standing pathway for the treatment of cardiac chest pain, and before putting together a pathway, a revised pathway, there was a baseline audit conducted. The pathway wasn’t set up in a way that would allow you to identify what we now call ‘mission critical points of care’. You can take pathways for anything, you can pull together every bit of guidance that has come from every professional body on how you should or shouldn’t care for it . . . And having done a couple of audits on existing pathways, we looked at the pathway and tried to define them all in terms of what we have called ‘mission critical points’, that is things that are clearly known to make a . . . an evidence-based difference to the outcome . . . on the management of cardiac chest pain, against the existing pathway, we looked at every conceivable component of the care. It generated a suggested action plan and a report, and the cardiologists came in to comment on that. So this has occurred actually subsequent to, not concurrently, with the CQC response.

Mortality lead, AMI-m.03

In AMI-m.04, the generation and use of data to inform decision-making had been transformed. For example, the information provided to the trust board in the quality report appeared to have ‘improved out of all recognition’ (CEO). These data sources were included in the quality report, which revealed ‘trends and connections‘. The CEO highlighted how the trust now focused on detailed exploration of trends and triangulation of data:

Now you can see data, you can see trends, you can see connections . . . I think we only used to look at HSMR alerts, infection control rates, you know, a few indicators, but now we’re really into detail around trends and analysis and looking at different things. So comparing ones from early 2012 and early 2014, there’s a huge difference in the way that the reports now work.

CEO, AMI-m.04

In contrast, changes in leadership in AMI-m.02 prevented any strategic action to invest in mortality analysis tools. When the first alerts arrived in 2008, the trust was using the Dr Foster tools, but the contract ended in around 2011 and there was a gap before they moved to CHKS at some point in 2012. Then, around late 2014 they changed to using the HED tool.

Summary of trust responses to the first acute myocardial infarction alert

Some trusts demonstrated an initial corporate focus, with senior leaders delegating to senior clinicians in case note reviews of AMI patient deaths (AMI-m.02 and AMI-m.03) and a review of patient AMI deaths by a mortality committee led by the CEO (AMI-m.03 and AMI-m.02). Two trusts (AMI-m.02 and AMI-m.03) also appeared more integrated their response and appeared to engage beyond the senior leadership level by instituting specialty mortality and morbidity meetings. However, it seemed that these linkages appeared ad hoc and largely unstructured in AMI-m.02, whereas in AMI-m.03 coding and comorbidity meetings and reviews of deaths occurred regularly in specialty morbidity meetings. Only one trust (AMI-m.02) integrated the MASS alert with input into the quality and safety committee. In AMI-m.04 an administrative focus dominated. Here, most concern was given to checking and validating the alert by comparing trust data and checking coding. A clinical review of case notes appeared fragmented and patchy across the trust. Some trusts resourced coding, and AMI-m.02 funded a new head of coding. Similarly, AMI-m.03 resourced an AMI pathway audit and redesign, explored activity patterns and explored CHKS data using several quality metrics for mortality. Similarly, AMI-m.03 was described as engaging with external bodies (AQuA) and benchmarking with other trusts to evaluate and improve mortality for AMI.

Comparison of responses to the second and subsequent alerts for acute myocardial infarction

A comparison of responses to the second and subsequent alerts for AMI suggested that some trusts were surprised to receive a second alert, as they perceived that they had addressed problems in coding AMI. This ‘shock’ was referred to as follows:

We’d put a lot of work into place in the first one but we still alerted, so it was looking at what we also then needed to further do to make sure, you know, the alerts don’t occur any more. It was a particularly memorable one because, as I said, we’d done lots of different bits of work with – primarily it was probably down to the coding, the initial alert, but then going forward we’d addressed what we thought was the coding issue but we then alerted again.

Head of coding, AMI-m.04

In AMI-m.02, the CQC label at the top of the alert letter was said to make clinicians take notice and do the required work without challenging:

. . . particularly if an alert has come in through the CQC. There is something about having – you know, that banner on the page already will make people sit up and take notice. So they will buckle down and look at the cases and write their reviews and their patient summaries, and give their thoughts on where we are with it.

Clinical outcomes manager, AMI-m.02

The second alert appeared to drive trusts to drill down and improve mortality review processes, coding and acute myocardial infarction pathways

The analysis characterises common trust responses to the second and subsequent alerts for AMI as follows:

• a forensic approach

• boundary spanners address coding problems by facilitating clinical oversight of coding for AMI

• joint clinical and coding involvement in case note review identifies errors in AMI diagnosis and problems in care pathways

• trust leaders mobilise the mortality committee as ‘a boundary-spanning structure’

• senior leadership prioritising AMI mortality by investing in knowledge production, knowledge sharing and analysis focused on AMI mortality improvement.

A forensic approach

In following a forensic approach, trusts attempted to ‘drill’ underneath and identify deep-seated problems with AMI mortality. On receipt of the alert in 2012, all cases in AMI-m.02 were reviewed and major issues were identified with coding of AMI:

With one of our MI alerts, and I think it might have been the earlier one, so maybe 2011, we looked at quite a big cohort of patients, about 70, and actually there were about 22 or 23 who actually had acute MI. So there were – so initially the focus would have been with the team. When we dug down into what the issues were, actually – obviously there are issues there, but it wasn’t so much the clinical team. It was more an issue of how we were coding, what we were writing.

Clinical outcomes manager, AMI-m.02

In undertaking this rigorous review of case notes, it emerged that only around one-third had been coded correctly for AMI. It was perceived that coding of case notes was the problem rather than the quality of care of patients with AMI.

The analysis in AMI-m.03 suggested that the trust’s response to the AMI second alert was also forensic and comprised three stages. The first was an administrative process whereby the information was checked to see if it was correct. The second stage focused on operational care patterns involving care pathways and other clinical-level processes. The third stage (described as the most difficult) was system redesign:

There are three stages to looking at improvement in terms of mortality so you’ve got that first stage in terms of the administrative process is making sure the information is correct is the first thing to do, and then you’ve got operational care patterns so pathway work, etc., etc., but the third one and the hardest one then is around system redesign.

Coding lead, AMI-m.03

The medical director also reported on how this staged process was integrated across the trust levels:

So we’ll have a stage 1 review where the parent consultant will go through the notes of each patient who’s died and make a judgement as to whether the care was in any way inadequate or the death potentially preventable or inevitable. If there is a feeling that the death was anything other than inevitable then we will go into a stage 2 review which will be done by a separate consultant where they will do a deeper dive on a fixed pro forma, and then on the back of that that finding is fed back into the departmental morbidity and mortality committee and the outputs from that are fed back to [mortality lead] so that they can be disseminated throughout the trust through the mortality committee but also through our clinical policy forum which is a monthly meeting of the heads of department and divisional directors.

Medical director

Boundary spanners address coding problems by facilitating clinical oversight of coding for acute myocardial infarction

Boundary spanners appeared to make sense of problems in coding AMI. They acted as sense-makers and sense-givers as they bridged both coding and clinical understanding of AMI. In AMI-m.02 the new head of coding acted as a boundary spanner, whereas in AMI-m.03 consultant cardiologists fulfilled this role.

In AMI-m.02, resourcing of coding was improved when a highly qualified head of coding was recruited in 2011. In attempting to bridge knowledge the head of coding started by speaking to every clinical director and clinical lead to ask for help with the data and to explain the issues relating to their recording in the notes and the national data. He set out to motivate clinicians by stressing that they should take ownership of their data (through case note write-up):

I met every clinical director or clinical lead and I asked them to help me help them with their data, and made it very clear, ‘This is not my data. This is your data. It’s going to be against your name. So if your patient is dying and then the primary diagnosis is a chest pain, and it’s going to show nationally’.

Head of coding, AMI-m.02

The head of coding recounted how clinical leads were very defensive about their coding, as they felt that it was correct. However, the new head of coding was able to demonstrate that they were making serious errors and developed a ‘double-coding policy’, whereby lead clinicians could authorise the head of coding to code their data.

The clinicians were described as being somewhat defensive at first, and some as being very aggressive:

Well, I’ve had a consultant storm in my office on the third day, one of the top consultants here, telling me how rubbish my work is. And I said, ‘I’m really sorry, Mr X, you know, but I didn’t know I could cause so much rubbish in 3 days. But please take a seat. Let’s go and let’s find out’ . . . ‘this is a fractured neck of femur, it’s all open, and that’s why the trust is not getting money’. And my next question was, ‘What kind of plate are you using? Is it three pin or six pin? What’s the size of the incision? Are you opening the fracture side?’ And they get gobsmacked. And I said, ‘Because . . .’ And this was not coming from the clinician, it was coming from coding knowledge . . . ‘It doesn’t show it in these fracture [notes] . . . are you doing this one or are you doing this one? Just clarify it to me. I’ll put it in a double coded policy. You sign it and I’ll code it. It’s your data, not my data.’ And that approach – but it was like that, you know. It turned out we were doing it right, actually.

Head of coding, AMI-m.02

In AMI-m.03, a consultant cardiologist was appointed mortality lead in 2012 and set out to improve coding by ensuring greater clinical involvement. For example, the role of consultants in coding was extended to authorising and validating the coding of their patients:

. . . the trusts have tried really hard to address this and I’m not aware that’s a major problem anymore by ensuring the consultants review all of the codes, so we get the notes or e-mails with, ‘Are you happy with this coding?’ and we say, ‘Yeah we’re happy or no please change this’ because some of the stuff you get put on is bizarre.

Consultant cardiologist, AMI-m.03

Joint clinical and coding involvement in case note review identifies errors in AMI diagnosis and problems in care pathways

In AMI-m.04, the clinical analysis of AMI coding highlighted that the number of patients admitted in the general medical intake and under the care of a general physician was high. It was suggested that these patients might have been admitted with a more pressing condition, such as stroke, and were treated in wards outside the cardiology department and without the oversight of a cardiologist:

Increased risk from myocardial infarction was not in patients under cardiology, it was in patients who were not under cardiology. And yet, we’ve got a direct admission policy that if you come in with ST elevation myocardial infarction you go straight to the cath lab and you have your PCI, and you’re under cardiology. So that immediately raised a bit of a question about, well, actually these clearly aren’t straight forward heart attack patients, that when you split the data between cardiology versus non-cardiology, the non-cardiology was where the excess death was. And actually with the benefit of hindsight, that was telling us quite an important piece of information that these weren’t straight forward heart attack patients.

Consultant cardiologist, AMI-m.04

Clinical case note review in AMI-m.04 also identified two types of myocardial infarction, one arising from the rupture of a coronary plaque and resulting in a heart attack, and another arising from another condition, such as sepsis, that leads to raised troponin levels and heart failure. It was suggested that, in the latter case, the root cause of the admission should be coded as sepsis and not as AMI:

So what we came out with at the end of that first review was an action plan that said we need to be better at coding, and there are type 1 or type 2, or is it type A and type B, myocardial infarctions. So there’s the myocardial infarction that happens because you rupture a coronary plaque and you have a heart attack. And then there’s the patient who’s septic who’s got profound hypotension, who might have got slightly abnormal coronary anatomy, they stop perfusing their heart very well and their troponin rises, but actually the root cause of the admission diagnosis is sepsis and it’s not a heart attack. And actually the journey that we’ve been on related to the alerts is that our acute medical team are very focused on the troponin because they set a bundle of tests. And actually the diagnosis is sometimes not that clear when you’re in the acute medical ward, and it only becomes clear when you move to the post-acute ward. But of course, when you move to the post-acute ward you’re in your second episode within the hospital spell, and Dr Foster only uses your first episode.

Mortality lead, AMI-m.03

In AMI-m.03, a consultant cardiologist was appointed and extended the role of consultants to validating and authorising coding. In 2012, a consultant cardiologist was appointed as the mortality lead in acute medicine and began to do work to improve cardiology services:

. . . so my whole appointment was actually partly in response presumably I wasn’t here at the time but maybe perhaps to the alerts from previously, and so when I started working one of the things that I started doing was restructuring how we did things, so we introduced extra pathways for people with chest pain, and we introduced a certain criteria where the patients just bypassed the medicine and came straight to the tertiary centre, so my understanding that from within 2 or 3 months I started off to with the Dr Foster stuff but the SHMI certainly dropped from over 100 to about a 100 and has continued to be low and so did the length of stay.

Consultant cardiologist, AMI-m.03

Trust leaders mobilise the mortality committee as ‘a boundary-spanning structure’

Across all trusts the centralised mortality committee co-ordinated with and linked to localised specialty mortality and morbidity groups. Notably, AMI-m.03 appeared to have made a lot of progress in using the mortality committee to enable knowledge sharing and analysis of AMI mortality.

In AMI-m.03, senior leadership drove mortality oversight by the mortality committee. The CEO chaired this committee and instructed that it met on a bimonthly basis. Likewise, trust leaders highlighted that they had prioritised an improved level of knowledge and understanding of AMI within this committee. Hence, a GP was seconded to inform the trust about improved primary care engagement and communication. It was perceived that this could inform trust clinicians in their diagnosis of AMI and link back to primary care delivery. Similarly, trust leaders ensured that consultants with a formal responsibility for monitoring and analysing AMI mortality attended reported on mortality review, analysis and action plans, and implemented actions in their specialties. Non-executive directors, information and coding specialists also provided input and support.

This strategic focus on improving knowledge generation and analysis of AMI mortality was reported:

. . . so are we getting the right information, what’s it telling us, where are these issues that we need to discuss, are there some new issues or not as the case may be, and then perhaps the most important thing is what we are seeing, hearing, discussion in this meeting going to drivers to materially change from the action plans that we’ve got, or do we believe that the action plans that go from workforce action plans perhaps to flow of patients through the building, action plans to pathways to equipment perhaps or whatever. Have we heard something today that really – have we got an action plans that’s addressing this or a piece of work, if not do we need one? If we’ve got an action plan is it the right actions, etc., etc.? That’s fundamentally and if we can keep to those three because you’ve got round the table public health, consultants, you’ve got GPs, you’ve got individual specialities and clearly usually we have, for example, one of the cardiologists you could obviously talk – we want a tertiary cardiologist service that there could be a 3-hour meeting on cardiology alone quite easily.

CEO, AMI-m.03

It was noted that the GPs were also active in the CCG as commissioners, so the dynamics of the mortality committee could be difficult at times:

. . . our new chief exec continues to chair the mortality committee, and we have representation there from non-exec directors as well, from the two CCGs and from public health, as well as the mortality reduction leads from each department in the trust, and that meets . . . It was meeting monthly but we’ve subsequently reduced that to bimonthly meetings.

Medical director, AMI-m.03

The CEO also conceded that managing this committee was challenging owing to the diverse vested interests of participants; however, he regarded it as useful and working well:

There is an element of having to strike a balance between, and manage the meeting around the job of the GPs and public health in the room and there is an element of contributing to the understanding and the solutions, but there’s an element of holding us to account and for want of a better term they are there as commissioners and the two GP reps that we’ve got are quite active in the CCG that they’re not just jobbing GPs if you like. So that’s a balance that you’ve got to, and occasionally that has dynamics that need managing, but I think overall it works pretty well.

CEO, AMI-m.03

Senior leadership prioritising acute myocardial infarction mortality by investing in knowledge production, knowledge sharing and analysis focused on acute myocardial infarction mortality improvement

In AMI-m.03, the AMI alert was instrumental in driving leadership to prioritise AMI mortality reduction by accessing and improving the content of knowledge concerning AMI mortality and increasing the frequency of leadership oversight of AMI mortality. It was perceived that in drawing from richer information sources the trust could improve the analysis that would inform trust decision-making.

The strategic resourcing of mortality analysis and review was accomplished by investment in a new HEDs data analysis system and more clinical staff recruited specifically to support mortality review, action plans and the implementation of improved practice:

Well I would say that this trust, compared to the last trust I was, this trust has a much more sophisticated, robust data set, weekly data that I get . . . that you can go to individual desks, etc., etc. [In my last trust] we developed a lot of info but we didn’t get anywhere near the mortality stuff that we do [here].

Coding lead, AMI-m.03

Summary of trust responses to acute myocardial infarction alerts

After repeated alerts, trusts’ responses to AMI MASS alerts generally became more urgent, and structures (mortality committee, service level mortality and morbidity groups) for investigating AMI mortality at corporate and service level become more robust (AMI-m.03, AMI-m.02 and AMI-m.04).

Senior leaders were more visibly engaging with responses to AMI MASS alerts (AMI-m.03). Consultants became more involved in investigation of AMI mortality and become responsible for coding patient deaths (AMI-m.03). Attention also focused on forensic identification and correction of errors in coding (AMI-m.03, AMI-m.04 and AMI-m.02). In AMI-m.02 a key champion (head of coding) acted as a boundary spanner, working across organisational levels and departments. Likewise, in AMI-m.04, a lead consultant cardiologist acted as a boundary spanner and identified two key issues: (1) common instances of wrong coding of AMI and (2) and that patients not referred to cardiology were more likely to suffer an AMI. This highlighted difficulties in escalating care and led to the development of real-time monitoring and an improved scoring system for deteriorating patients. Coding was internally and externally audited (AMI-m.04) and there was increased engagement with specialties to address problems with coding case notes (AMI-m.02 and AMI-m.04). An analysis across trusts focused on triangulating data from operational care pathways (AMI-m.02 and AMI-m.03) and MINAP data and Draper and Dash (AMI-m.04), and working closely with Dr Foster representatives (AMI-m.04).

Comparative analysis of outcomes from repeat alerts for acute myocardial infarction

The comparative analysis of trust outcomes from repeat alerts for AMI yielded the following common themes:

• improvements in coding for AMI

• identifying errors in diagnosing AMI

• development of guidance for recognising AMI

• changes made to the AMI pathway

• tracking mortality and triangulating data

• resourcing more cardiologists

• promoting external engagement for knowledge acquisition.

Improvements in coding for acute myocardial infarction

Improvements in coding AMI were linked to the alert. More coding staff were resourced (AMI-m.02 and AMI-m.04). Clinical oversight and engagement has increased (AMI-m.02, AMI-m.03 and AMI-m.04). In AMI-m.04 coding of deaths was performed by a specialist coder who worked with the mortuary and with the clinicians to ensure that the codes were correct. They also assisted with mortality reviews:

For deceased patients, a change we put into place a few years ago is that the deceased coder actually goes down to the mortuary and reviews the notes in the mortuary, because what we were finding was sometimes notes were going for post mortems and we didn’t quite get the notes in time, or we didn’t have full access to all the episode history.

Head of coding, AMI-m.04

Identifying errors in diagnosing acute myocardial infarction

A forensic clinical case note review identified that myocardial infarction was being wrongly diagnosed. AMI-m.03’s specific target alerts allowed it to understand specific issues around AMI and chronic lung disease. It was noted that chronic lung disease could be mistaken for AMI:

I don’t know how good it is in terms of picking out avoidable mortality, because the frequency with which errors of omission and commission clearly contribute to a death is very low . . . One of the ones on acute myocardial infarction for example was the fact that in the absence of a diagnosis with the first 24 hours, if the traponin level, the heart enzyme that goes up if the heart has been damaged from a myocardial infarction, if the traponin level is elevated a bit, the working diagnosis will be a myocardial infarction. But in the elderly with chronic lung disease, with chronic renal disease, the traponin can be up a little bit anyway.

Mortality lead, AMI-m.03

Development of guidance for recognising acute myocardial infarction

AMI-m.03 developed an AMI pathway to provide guidance to clinical staff working in the A&E department on how to treat a patient with a suspected acute myocardial infarction based on best evidence and agreed local practice.

Changes made to the acute myocardial infarction pathway

An analysis of coding identified that AMI was occurring in medical wards and not in cardiology. It was understood that AMI was occurring in patients who had been admitted with other conditions and that in medical wards oversight by a cardiologist was limited. For example, in AMI-m.04, all patients under general medicine with suspected AMI would have a cardiologist review:

So the second alert, obviously we had a joint coding and clinical review, and it did highlight some areas where the processes we’d actually put into place probably weren’t working as well as they should . . . A cardiologist was involved, and he actually said, ‘Well no, that’s not quite right’. So what we then did was fine tune that process, and if we have any patients now that are under general medicine that don’t have a cardiologist input, we will send those to the cardiologist for a second review.

Head of coding, AMI-m.02

In AMI-m.03, consultant cardiologists had been instrumental changing the patients’ pathway across the area, acting as a tertiary centre:

So primary PCI is lots of data trials that show that basically instead of doing that if you just bring these people directly to a centre like us where we take them immediately to the cath lab and open up the artery physically and put a stent if necessary or just balloon and most of the time I put stents that’s better.

Consultant cardiologist, AMI-m.03

Focuses trust on researching mortality and improving care

In AMI-m.04, the head of information reported that the alerts had focused everyone’s attention and led to much internal research around improvement, which had improved, for example, palliative care arrangements:

I think they’re very effective. They certainly focus everyone’s attention. And it does mean that serious internal research is done as to why and what can be done to improve. And certainly in relation to palliative care as well, we’ve been – the local hospice, a clinician from there regularly attends the mortality group meeting as well, so it’s about getting better care for those patients that are going to die, because it’s an inevitable fact, isn’t it? It’s getting people to die in the right place and more appropriate place, comfortable place for them. I think that’s one of the positives that’s come from it.

Head of information, AMI-m.04

Comparative analysis of parallel development of processes for acute myocardial infarction mortality

A range of parallel contextual influences appeared to have an impact on changes in how AMI mortality was approached at trusts. Common emergent themes were:

• external regulatory pressures drive mortality oversight (AMI-m.02 and AMI-m.03)

• new trust leaders drive improvement (AMI-m.02, AMI-m.03 and AMI-m.04)

• development of trust culture to improve clinician involvement and harness data (AMI-m.02, AMI-m.03 and AMI-m.04)

• trust integrated structures and processes (AMI-m.02, AMI-m.03 and AMI-m.04)

• resourcing of IT infrastructure (AMI-m.02 and AMI-m.04) monitoring

• external engagement (AMI-m.04, AMI-m.02 and AMI-m.03) – trust involvement in safety-related improvement campaigns.

External regulatory pressures drive acute myocardial infarction mortality oversight

Care Quality Commission intervention and special measures were reported as forcing AMI-m.02 to address governance arrangements, A&E targets and finance, and set out priority areas for action. The CQC had visited again and reported some improvement but the trust remained in special measures at the end of data collection. It was noted that trust mortality levels were not covered specifically in the CQC review.

In AMI-m.03, the Keogh review and CQC inspections drove the trust to address improvement in mortality focused on the development of structures and processes to broadly address all trust mortality and, in some cases, AMI mortality specifically. Although the trust was one of the 14 identified in the Keogh review, it was not put into special measures after the review because it was deemed to be addressing the issues required to reduce mortality, and doing this well. Despite the trust having an internal process for reviewing mortality, the Keogh and CQC reviews found that this was not being applied uniformly across the trust, which had to be addressed:

[The CQC said] . . . ‘the process you’ve got for mortality reviews is a really good one but actually it’s not really robustly applied everywhere and you should be doing something about that’ so obviously we’ve been auditing that since.

CEO, AMI-m.03

The Keogh review suggested that the speed of any recognisable service change was unsatisfactory and that the pace of implementing improvements in patient care was at an unacceptable level. The review raised concerns about equipment safety checks, nurse staffing levels, incident review systems, shortage of medical staff, insufficient time for teaching, professional development and governance in specialties and inconsistent implementation of infection control. The Keogh review specifically criticised the trust’s AMI pathway as too long and complex.

The overarching message delivered to the trust was that it was attempting to implement too much and was, therefore, not doing anything effectively, which was borne out in the interviews:

One of the challenges back to us from the Keogh team was actually we were trying to do too much, so trying to grapple mortality reduction and safer care, the message that came back to us was, ‘You’ve got too many initiatives running and actually you need to narrow that down and focus’.

Chief nurse, AMI-m.03

In AMI-m.04, the Monitor intervention arising from poor financial management also led to regulatory oversight of trust activities and to a dramatic change in trust leadership.

New trust leadership drive improvements to improve quality of care and reduce mortality

Changes in senior leadership were common in all trusts. In AMI-m.04, they arose from the Monitor intervention. In AMI-m.02 and AMI-m.03 they arose after CQC and Keogh review investigations. New leaders were instrumental in prioritising the overhaul and integration of trust structures and processes to enable mortality review. A strategic focus was placed on mortality awareness and improved data quality across all divisions and specialties.

In AMI-m.02, the new management team gave managing avoidable mortality a new priority in the trust:

I just think it’s been given some clout and people realise how important it is now, and that because someone’s died that’s not actually the end of the story and, you know, that can impact on how we treat the next patient through the door, as opposed to, well, they’re dead so that’s kind of done . . . And it’s got some backing now, people that are interested in it. It’s kind of just growing and taking on a life of its own.

Clinical outcomes manager, AMI-m.02

In AMI-m.03, the CEO had been in post for 2 years, arriving just after the Keogh review, and the medical director had been in post for 3 years. The trust had also appointed a mortality lead 3 years previously who was a retired critical care consultant. All were appointed at the time of, or just after, the second AMI alert. It was suggested that these leaders brought unique skills and experience to tackling AMI mortality improvement. Similarly, it is possible that this new leadership team was in a position to start anew, unhindered by past decisions made around AMI mortality.

AMI-m.04 had a change of CEO in 2011 and a new medical director in 2014, and these changes were described as making a difference:

Soon after he [medical director] started one of the things he said was, ‘I don’t understand why we don’t put mortality first on our patient safety agenda. You know, isn’t it the most important that could possibly happen to anyone that comes into this hospital that they die? And so it needs to be top of the list’. And actually even this week, the chief exec, has sent out an e-mail . . . about the financial position that we’re in, a difficult financial position and Monitor helping us, but actually very strongly stating quality of care and outcomes being top of the list important priorities.

Mortality lead, AMI-m.04

Developing trust culture supportive of improving care, clinician involvement and harnessing data

In AMI-m.02, the focus was on developing a culture supportive of improving care and clinician involvement and harnessing data.

The clinical outcomes manager went on to say that as a result of this they had recognised how important it was for all clinicians to be involved in reviewing their patients’ notes and any death, and for this to be part of everyday working rather than a one-off event when an alert arrived:

So with our kind of day to day – trying to make this part of everyday business, really, mortality, as opposed to something that happens once a month that might tie in with a governance meeting. So everyone scurries around, doing a bit of M&M [morbidity and mortality review] and might present it at a governance meeting. It’s got to become part of everyday.

Clinical outcomes manager, AMI-m.02

The plan was also to specifically request that consultants review every one of their patients’ deaths, rather than asking junior doctors:

Your junior doctor is assessing the death of a patient under your care and passing judgement on the care your team delivered, are you comfortable with that?

Medical director, AMI-m.02

AMI-m.04 attempted to harness information to improve trust-wide understanding of mortality. This involved developing a narrative around mortality, the use of a mortality review process tool and developing an electronic database of mortality information. The idea of a narrative around mortality is deemed essential for understanding the entire process of mortality in the trust. The mortality report was said to be around 20 pages long but it was seen as too long for the general consultant population to read so work is now under way to condense it to four pages.

The mortality group met monthly and the information specialist provided a full report:

. . . there’s a monthly meeting, mortality group meeting, and I prepare a report for that that names any upcoming alerts and also links to things like the CUSUM charts that we review in the meeting to see if there’s anything coming up as well. And our Dr Foster rep is there as well normally.

Information specialist, AMI-m.04

The trust designed its own mortality review process and tool with a database to capture the information. Each department had the opportunity to amend the database to suit its own needs but there was a core set of data that everyone was required to complete. Clinical issues were added to the core data set to raise awareness of issues when clinicians were conducting mortality reviews:

. . . we’ve designed a pro forma for mortality reviews internally, we use the global trigger tool as our original template, we then embedded that review tool in a database that is a bespoke database.

Mortality lead, AMI-m.04

Part of the reason for this review and the database was to counter the fact that Dr Foster data were out of date by the time they arrived:

But one of the frustrations about Dr Foster is it’s always in arrears, it’s always such a long time ago that people have forgotten the cases, and learning should really be as soon after the event as possible when everyone in the team remembers what happened. As opposed to going back and looking at cases that some of which were 6 months old, and half the trainees have moved on.

Mortality lead, AMI-m.04

Resourcing information technology infrastructure

AMI-m.02 had a clinical outcomes manager with responsibility for the clinical harm review process and reviewing hospital mortality, and who worked closely with the medical director. A new system of mortality review was introduced. The previous system did not work as the death list came out 2 months in arrears. Now deaths were reviewed at the time when the death notification was completed. The clinical outcomes manager met with the associate medical directors to go through all of the deaths using the Hogan score23 as a trigger for avoidable deaths. From this, any information was fed back to the clinicians and to the mortality assurance group. The interviewees described the aim to be ahead of the curve in terms of knowing what is going on in the trust before an alert arrives.

Within the previous 6 months the trust had bought HED from Birmingham. Before this they had used CHKS. The reason for the change was that the clinicians wanted data that would help them to be more proactive:

I was really delighted when I came here that we’d bought HED because we’d used HED in my previous organisation when mortality was why that trust went into the Keogh special measures group up in [trust region]. So when we brought HED up in there, it just transformed the understanding of what the issues were because the richness of data was suddenly clinically useful.

Medical director, AMI-m.02

HQUIP also send data about mortality for those specialties contributing to national clinical audits.

AMI-m.03 uses the HED tools, and the lead in informatics produces a ‘heat map’ from the data to show which areas need to be focused on:

We also have a heat map which [XXXX] generates from our data which shows us areas of attention, and we are focusing on not so much the black squares but the big squares, so the areas where we’re seeing significant numbers of people with maybe a slightly elevated SHMI compared with very small numbers with a very significantly elevated SHMI . . .

Medical director, AMI-m.03

There has also been a focus on reducing harm from falls, pressure ulcers, MRSA and so on, and the medical director reflected that this overall focus on reducing harm had helped with the reduction in AMI mortality levels.

A focus on monitoring mortality

AMI-m.04 worked to understand the data better, as well as how alerts are generated:

We had a mortality group meeting last week, and we monitor things, and the other thing we did was we said, ‘Why didn’t we know this was about to happen again?’ Because we’d been monitoring the CUSUM chart, and actually what it turns out, I think, is that the CQC monitors at a different threshold. So you can set your threshold at 99% or 95% threshold on the CUSUM chart, and the CQC lowers their sensitivity, if you’ve had an alert they will look at a lower threshold for a future alert, whereas we were still looking a 99%, the CQC were looking at 95%.

Mortality lead, AMI-m.04

The trust now monitors deaths by day of the week and can show that there is no difference in the rate during the week and at weekends. Their HSMR is now around 90 and the SHMI is one of the best (top three) in the country. This was attributed to the work on sepsis and the triangulation of data to spot issues (gastroenterology was given as an example here).

There is a monthly conference call with the Dr Foster representative during which the current data are reviewed and any probable alerts are discussed so that staff can be proactive in spotting problems.

The coders sit with the clinicians when mortality reviews are undertaken and a joint mortality review is conducted:

Previously it had been a clinical review and a coding review and it wasn’t actually joined up. But since – not just as a result of that but since then we do a joint coding and clinical review, where myself and the coder go along with the clinical review and then we discuss the patients and look and make any changes or awareness actually at that review. So it’s more joined up and more fluent now.

Head of coding, AMI-m.04

The coding team has verified its coding of myocardial infarctions with the National Classification Centre for Clinical Coding:

So we’ve really sort of done a lot of work on making sure our coding met national standards, as I said, and with more experienced coders actually coding it and now the same coder coding, we’ve got the consistency in our coding.

Head of coding, AMI-m.04

The information specialist provided the medical director and the mortality lead with a spreadsheet detailing all of the patients who have died in the hospital. The information team also provides the clinical review team for mortality alerts with information about patients who have survived, in order to give a more complete picture.

Trust involvement in safety-related improvement campaigns

In AMI-m.04, the mortality lead’s engagement with NCEPOD allowed the trust to access useful information about how other trusts tackled mortality. It highlighted trust-wide approaches to reviewing deaths, frequency of meetings and techniques used. The head of mortality remarked on this:

It tells you, you know, by department and by hospital how many hospitals have got a trust-wide approach, or departments in hospitals what their approach is, frequency of meetings, a whole load of core data about what people are currently doing, there’s no narrative.

Head of mortality, AMI-m.04

The head of coding also identified how the trust audited practices against national standards:

The National Classification Centre for Clinical Coding, they advise us on coding issues. So that was part of that as well, because it was felt that, you know, the sequencing of coding from a clinical point of view wasn’t quite accurate, so we then confirmed that with our national standards and actually we were coding it to national standards. So we’ve really sort of done a lot of work on making sure our coding met national standards, as I said, and with more experienced coders actually coding it and now the same coder coding, we’ve got the consistency in our coding.

Head of coding, AMI-m.04

In AMI-m.02, coders were trained by the London Clinical Coding Academy, with two undergoing specific training to code mortality. By 2013 the coding was graded level 3 in the Information Governance national rating. This meant that they had around 95% accuracy across all coding, which is the top banding.

Summary of parallel evolution

The parallel evolution of AMI mortality was addressed by the real-time monitoring of AMI deaths (AMI-m.04), liaison with external networks for knowledge acquisition (MINAP, Dr Foster; AMI-m.04), resourcing cardiology by increasing staff (AMI-m.03), increasing staff involved in coding (AMI-m.02), investing in IT infrastructure (HED and training, mortality review database) (AMI-m.02, AMI-m.03), triangulating all data (AMI-m.04, AMI-m.03), developing a culture supportive of learning and problem-solving (e.g. CEO blogs, tweets, grand rounds; AMI-m.02) and developing a narrative around mortality (AMI-m.04).

Comparison of trust preconditions

Comparison of trust preconditions focused on:

• problems with senior leadership

• fragmented structures and processes for mortality review and improvement

• poor engagement between clinicians and managers

• improvement work led by informal mortality champions

• extreme regulatory pressures (CQC, Keogh review, quality accounts audit) creating organisational instability

• learning gained from trust engagement with external organisations

• trust culture characterised as ‘closed’, lack of awareness of mortality and limited staff engagement in addressing mortality.

Problems with senior leadership

Sep-s.06, Sep-s.05 and AMI-s.01 were reported as experiencing problems with senior leadership. In Sep-s.06, it appeared that senior leaders were disengaged and not prioritising mortality review. The matron recalled how:

In the beginning, being entirely honest with you, in the beginning I don’t think anybody was really that interested in what we were doing and why.

Matron for general surgery, Sep-s.06

Interest was limited to using HSMR statistics for insertion into trust documents:

I think before it was potentially just looking at the numbers and looking at ‘What’s the HSMR?’ and probably looking at it, I think as a lot of trusts probably did, when the hospital guide was due to be published and seeing what we were.

Head of quality, safety and compliance, Sep-s.06

In AMI-s.01, it was reported that trust leaders adopted a punitive and bullying approach as a senior clinician had been disciplined for engaging with the media and divulging severe problems with high mortality:

So because we knew this particular person had been disciplined because they had whistle-blown, it kind of made you think I’m not going to do that. So it was that kind of thing. If you stepped out of line or you made too much fuss that may be wasn’t the thing to do because it wasn’t going to get you anywhere.

Head of clinical audit, AMI-s.01

Leadership was also described as ‘command and control’. The newly appointed CEO suggested that staff failed to report incidents and also posited that this may have arisen because staff were fearful of the punitive consequences of reporting:

Nobody was engaged, it was very ‘command and control’, nobody – there were no incidents, nobody reported anything, so there were no incidents in the organisation, none. And there weren’t systems and processes for people to alert others and actually they wouldn’t alert others because they knew or were aware of those consequences.

CEO, AMI-s.01

Fragmented structures and processes for mortality review and improvement

In AMI-s.01, a fragmented and token approach was also suggested. Respondents highlighted that the trust leadership had failed to develop a strategic priority for investigating and tackling AMI mortality. Overall, they reported a fragmented and token approach that had failed to engage across divisions and specialties. It was suggested that mortality alerts went to the medical director who then sent them to the relevant consultants. The alerts were discussed at medical meetings but often the individual consultants argued that their care was good and then nothing else happened:

We didn’t really act on anything. We would get the information in, we’d see the report but nothing really happened. We would just note it, it would be minuted and then we would move on to the next thing.

Head of clinical audit, AMI-s.01

Poor engagement between clinicians and managers

In all trusts remarks were made about poor engagement between clinicians and management.

In Sep-s.06 managers appeared to display an administrative approach, whereby they functioned in silos and failed to share and communicate information. In AMI-s.01, limited clinical engagement in addressing mortality was reported. It was suggested that when senior consultants were made aware of mortality issues they attempted to shift the blame and argue about problems and solutions:

There was very much, ‘It’s not me, it must be someone else. So don’t ask me to improve my service because I’m alright and it must be somebody else that’s the problem’ . . . what would happen is the consultants would end up arguing over the problems so there didn’t seem to be a solution; it was all quite frustrating . . .

Head of clinical audit, AMI-s.01

Limited clinical engagement in addressing mortality was generally reported. The CEO highlighted this as ‘there was lack of engagement of clinicians’ and that this led to a ‘lack of noise in the system’.

Before receiving the alert, a lone clinician acting as a mortality consultant had reviewed case notes, but this had not generated any learning outcomes other than identifying that clinical coding was poor, and solutions had not been identified or implemented to correct this.

Improvement work led by informal mortality champions

Both Sep-s.06 and Sep-s.05 reported a long history of quality improvement work in sepsis and in AMI-s.01 a lone clinician was implementing care bundles. Across all trusts, this kind of improvement work often lacked strategic direction and was undertaken by committed individual clinicians.

AMI-s.01 was participating with AQuA, a NHS care quality improvement organisation, and implementing care bundles:

We did all of the [AQuA] programme which was AMI, heart failure, pneumonia, total hips and knees initially, stroke came online a few years ago . . .

Head of clinical effectiveness, AMI-s.01

However, this work was limited to a single clinician, who described this work as a ‘one-man band’.

Experience of severe operational pressures

Before receipt of the alert, Sep-s.05 in particular was experiencing much instability. This included a merger with the local community trust as well as major operational and financial pressures.

Sep-s.05 respondents emphasised how severe operational pressures had had an impact on trust mortality. This related to how, in winter, the crude mortality rate had doubled, there had been failures in identifying deteriorating patients and a failure to escalate patients, patient records were poorly written, the treatment of end-of-life patients was difficult and the discharge of patients was problematic as social care provision was poor. Hence, patients remaining in the hospital awaiting discharge were succumbing to hospital-acquired pneumonia:

Essentially it was the winter mortality rate. So our winter mortality rate, crude mortality had doubled, and essentially I then got the review.

Mortality lead, Sep-s.05

Sep-s.05 also reported staff shortages. In 2011, 60 redundancies were announced and a further 80 positions were made vacant. A critical care consultant referred to extensive staff losses:

Well we had clinicians leaving. You know we had some very good medical consultants leave and some very good surgical consultants leave. Only one anaesthetic consultant leave, probably not as a direct impact of that. But we did have . . . Certainly, we had clinicians leaving . . . Well the main problem we’ve got is probably the same as the lack of nursing staff on the wards is one of the main issues. So I think they haven’t got time to do it. And then the junior doctors seem to be too reluctant to call the seniors as well, is the other problem.

Critical care consultant, Sep-s.05

Extreme regulatory pressures (Care Quality Commission, Keogh review, quality accounts audit) creating organisational instability

Sep-s.05 and AMI-s.01 were experiencing extreme financial pressures. Sep-s.05 had overspent considerably on the implementation of an IT-enabled patient records system. The associate medical director reported this:

So probably around 2012. We were struggling for money. The chief exec was particularly – his managerial style was a particularly difficult one to deal with. We then got EPR which was electronic patient record which cost us £60 million which we have difficulty funding. So I think the downfall of our chief exec and our board was due to financial issues.

Associate medical director, Sep-s.05

These financial problems eventually led to Sep-s.05 being in breach of its financial obligations and to a Monitor intervention in 2013.

In AMI-s.01, financial pressure was seen as increasing, as the trust’s financial position had varied since 2010/11 from a surplus of £1.15M to a deficit of £4.77M. It was suggested that this had affected the resourcing of staff and prevented trust staff from attending meetings with regard to governance and mortality investigation. These intense financial pressures were also experienced by Sep-s.05, as the failed implementation of an IT-enabled patient records system had incurred costs of £60M and led to substantial staff redundancies.

AMI-s.01 was exposed to a host of regulatory pressures that sapped the energies of and demoralised staff. These pressures were described as emanating from the Quality accounts, Keogh review, CQC inspections and multiple alerts (Dr Foster):

I think the trust has been constantly under scrutiny for many years. I think the Keogh was the most high profile . . . But certainly when that report came out the morale within the trust obviously was quite poor because you’ve got nurses that are working very hard, doctors that are working very hard, and you’ve got really good areas and pockets of good practice. To be tarred with that brush it’s quite demoralising.

Head of clinical effectiveness, AMI-s.01

It was noted that the Keogh Review had identified real problems and that this pressure had caused much dissent in the trust executive team:

Keogh got it spot on and I will be very frank and honest because I’ve been with the Keogh team . . . on day 1 of the Keogh review the chief executive’s office is directly above me and on day 1 of the Keogh review the Keogh lead teams are sitting here, and the chief executive is calling and shouting at our medical director.

Head of clinical governance, AMI-s.01

In contrast, Sep-s.06 appeared to experience some regulatory pressure in CQC oversight of alerts but this pressure appeared to have had less of an impact than in either Sep-s.05 and AMI-s.01.

Learning gained from trust engagement with external organisations

It was generally accepted that AMI-s.01 was engaging with external bodies but not sharing knowledge and awareness of problems regarding mortality. For example, it was participating in the national audits for cardiac care. However, it was suggested that these audits were limited to particular questions and did not review overall care:

If, you’ve got a national audit . . . you are looking at specifics. So somebody would go through the case notes and say, ‘Has the patient had an echo within 24 hours or 2 weeks of discharge? Have we done an ECG? What was the initial ECG? Did the patient have a cardiac arrest? Yes, they had a cardiac arrest’. But you wouldn’t be doing that thorough review of the notes.

Consultant cardiologist, AMI-s.01

Clinicians had championed sepsis improvement in both Sep-s.06 and Sep-s.05 over many years and accessed knowledge and learning by participating in national programmes related to the Sepsis 6 care bundle. However, most activity was largely confined to critical care and ED departments.

Trust culture characterised as ‘closed’, lack of awareness of mortality and limited staff engagement in addressing mortality

In AMI-s.01 respondents generally reported that, before the AMI alert, the trust was ‘closed’ and lacked transparency. This was articulated by the director of quality and governance:

Having been able to compare it to other organisations, it was [limited] in its approach to transparency and openness. That inevitably had – and permeated through the whole way in which openness and transparency were seen.

Director of quality and governance, AMI-s.01

This echoed the closed culture reported in Sep-s.05 in which information was not shared, mortality review was ad hoc, and board meetings, before 2013, took place in private. This differed from the open culture reported in Sep-s.06.

Comparison of trust responses with a single alert for sepsis and acute myocardial infarction

Comparison of trust responses with a single alert for sepsis and AMI focused on the following themes:

• a forensic approach to reviewing mortality

• an integrated trust-wide response

• clinical engagement and cultural ownership of sepsis improvement

• a universal and timely response

• a case note review identifying errors in coding.

A forensic approach in reviewing mortality

In all trusts, the alert and CQC follow-up had led to a forensic approach in reviewing mortality. In Sep-s.05 this was entirely focused on the deaths highlighted in the alert. In Sep-s.06, forensic investigation involved case note review and the triangulation of data across hospitals, divisions and specialties:

We would initiate an investigation and investigation would inevitably be a case note review of all the cases which is, you know, usually the most I’ve had is I think about 120 . . . You would go through all the notes and . . . it might be a condition where they’re all in one speciality, usually they’re across several specialities which is helpful so you can give them each a bunch, they look at them, they would get the data back to us, we would formulate it into a, you know, into a report, we would develop an action plan based on that and we would send that back to the CQC.

Medical director, Sep-s.06

In AMI-s.01, staff employed a forensic approach to responding to the AMI alert and CQC inspection. This comprised investigating, analysing problems and triangulating data. Attention focused on the trust’s critical care model and AMI pathway and on producing an action plan:

For cardiology obviously we had an alert and we also had a CQC visit which looked at critical care, and the model of critical care we had running here was very odd in that we have an upstairs intensive care unit which looked after level 3 – so the sickest ones – and surgical high-dependency patients and we had a medical health and [inaudible] downstairs which the critical care people were not involved with, which was run by physicians as a medical HDU [high-dependency unit].

Medical director, AMI-s.01

In AMI-s.01, respondents accepted that a problem with AMI mortality had been identified and they followed a forensic approach and produced a detailed action plan:

Seeing that alert, that acute myocardial infarction is not being dealt with correctly and in addition to this we have been asked to produce what patients’ plans were at [AMI-s.01] hospital with the chest pain, so we had to provide a detailed response.

Medical director, AMI-s.01

A clinician working with the mortality committee worked with another clinician to investigate and question all documentation that related to the AMI Alert. This included admitting condition, comorbidities and interventions, highlighted as follows:

I’ll usually sit down with a consultant and you’ll got through the actual documentation looking at obviously admitting condition, comorbidities, looking at interventions, were they appropriate, because at the end of the day these are patients who are a problem because they died, so we work through them, we’re just looking, has the right intervention occurred, have we missed anything.

Resus officer, AMI-s.01

If problems are identified in the case notes, then a mortality pro forma document is completed and this is forwarded to the division involved for further investigation:

But if there are issues they escalate, so the investigating officers, so the consultant and the nurse usually will then fill out the pro forma, and that will be escalated to the division where the issue has been raised and they need to take it further.

Resus officer, AMI-s.01

A similarly forensic approach was adopted in Sep-s.06, involving the mortality committee co-ordinating the trust’s case note review. An electric pro forma for mortality was completed and the case notes were graded A–E, thereby standardising the mortality review process, identifying comorbidities and directing the extent and level of case note review.

In AMI-s.01, the AMI alert was investigated by exploring the AMI pathway and generating an action plan to address problems. For example, problems were identified in failures of examination of patients by a cardiologist in wards outside cardiology; difficulties in the speed of identification of AMI using the existing patient at risk score (PARS) observation system; and incorrect coding of AMI. For example, a consultant reported how an action plan addressed the way in which infrastructure limitations had resulted in AMI patients being distributed to wards across the trust that did not have a cardiologist on site:

For cardiology obviously we had an alert and we also had a CQC visit which looked at critical care, and the model of critical care we had running here was very odd in that we have an upstairs intensive care unit which looked after level 3 – so the sickest ones – and surgical high-dependency patients and we had a medical health and [inaudible] downstairs which the critical care people were not involved with, which was run by physicians as a medical HDU [high-dependency unit].

Medical director, AMI-s.01

A forensic examination of the case notes in AMI-s.01 also identified problems with the speed of identification of AMI and early intervention in wards outside critical care and cardiology. Respondents highlighted that, in response, an action plan had been put forward, whereby every patient with suspected AMI was to be assessed by a consultant cardiologist:

So that we introduced, and also we showed that every patient that comes in with non-stemi is seen by a consultant cardiologist, he decides what is the best route in discussion with the patient and other colleagues if colleagues are involved.

Consultant cardiologist, AMI-s.01

An integrated trust-wide response

All trusts reported an integrated trust-wide process for responding to alerts. This focused on validating the alert and then a central structure (in Sep-s.06 a mortality committee, and in Sep-s.05 the clinical governance team) co-ordinating a trust-wide investigation. In AMI-s.01, the response to the AMI alert was characterised as integrated across corporate, divisional and speciality levels. The alert was sent to the executive team and then forwarded to the clinical governance unit. This unit comprised subunits that focused on patient safety, quality improvement, clinical audit and effectiveness, clinical governance, litigation, compliance, PALS (patient advice and liaison service), assurance and governance. Trust mortality leads demonstrated a key focus on improving data quality in mortality review. The clinical effectiveness subunit played an important role in generating supporting background data related to the alert. These data included information on the divisions affected, patient pathways, staff involved and history of alerts. This information fed into corporate teams, such as the mortality committee and coding teams, and was forwarded to the divisional mortality teams. At the divisional level the investigation involved case note review, review of coding, a patient pathway analysis and recommendations for improvements in care or coding that were forwarded to the mortality committee.

Clinical engagement and cultural ownership of sepsis improvement

Sep-s.06 reported trust-wide intensive clinical engagement in mortality review. Clinicians also appeared to develop ownership of mortality. Opinion leaders in this trust operated as informal sepsis champions, encouraged ‘buy-in’ and were instrumental in developing innovations within specialties. Clinical champions in Sep-s.06 and AMI-s.01 operated as boundary spanners, supported mortality review and made sense of how clinical diagnosis for sepsis was coded and the implications of this for Imperial College alerts. In AMI-s.01, a data analysis identified clinicians operating as boundary spanners, bridging knowledge areas and cognitive frames. A consultant cardiologist working with coders in the case note review was able to ‘bring the coders into the reality of medicine’ and helped the clinician make sense of how they worked:

There is a revolutionary drive in the trust running to try to bring the coders into the reality of medicine . . . every day, there are actually at least three notes, there’s one which are done, and the two are waiting. These are, the coders think that what’s in the note is not correct, so it’s sent to me as a lead for cardiology, I look into it, I follow the national guidance . . . I have been to, I’ve been brought into the coders’ mentality, how do they think, how do they work, so I can understand when I am right and the manager how to help them getting a resolution out of the confusion about specific cases.

Consultant cardiologist, AMI-s.01

Likewise, a senior nurse in critical care in AMI-s.01 identified how she had supported mortality investigation and acted as ‘the link between nursing and medics’:

I’d fit in ICU and initially I had a bit of resistance from both the nurses and the doctors, it was because the nurses saw me as, ‘Oh, who does she think she is?’ and I’m open with them, I know all that, and then the doctors were a bit like dubious, but then we had an external review 6 months from [external trust] and they felt that I needed to be on the unit full-time and since I’ve helped with the mortality, got the documentation, and I’m the link between nursing and medics, they’re looking to employ two more!

Senior nurse, AMI-s.01

In contrast, Sep-s.05 appears less focused on clinical involvement in mortality review and investigation processes linked to the alert. For example, the sepsis lead acknowledged that there was a problem with coding for sepsis but conceded that he had no involvement in trying to address this problem, as he had not been asked for input. Likewise, universal case note review by clinicians of all deaths was not reported.

AMI-s.01 respondents reported that, in responding to the AMI alert, there was increased involvement of clinicians in mortality review with the mortality committee and divisional mortality teams. The medical director highlighted that it was important for clinicians to volunteer to participate in these teams. He suggested that it was deemed important so as to ‘keep the process clean’, legitimate and free of scrutiny, although it was recognised that clinicians’ time and energy were limited:

I did it on a volunteer basis and that’s part of the struggle is that people are giving up their time and energy to do this, and it’s still done in that way . . . but it was done on a volunteering basis because that was the only way I could keep it clean. And, I wanted it to be, as much as I could, bomb proof in terms of critique that people might make of the process.

Medical director, AMI-s.01

A universal and timely response

In AMI-s.01, most staff reported that the trust response to the AMI alert was to have a review of case notes for every death. Hence, the head of clinical efficiency and effectiveness said:

We systematically reviewed every single death in patient death from February by panel.

Head of clinical efficiency and effectiveness, AMI-s.01

The CEO highlighted how these investigations fed into organisational changes:

The team then undertook a retrospective review of all deaths in the previous 2 years, all complaints, incidents, inquests, etc., and mapped the care delivery problems and the required actions onto an organisational development plan.

CEO, AMI-s.01

Importantly, clinicians involved in a mortality alert were excluded from the case note review process and attention focused on timely review of case notes. An internal target was set of a 14-day review period, which was managed directly by the clinical governance unit and ensured that clinicians recorded a clear memory of the patients involved.

Sep-s.06 also responded by instigating the universal case note review of all deaths by doctors. In [region], this was viewed as onerous, especially in specialties such as geriatrics that had many deaths.

Case note review identifying errors in coding

Across all trusts receiving a single alert, case note review identified errors in coding. For sepsis, this related to junior staff unable to identify a diagnosis early and writing up case notes in an ambiguous way, without highlighting the comorbidities and primary condition. For AMI, errors in coding arose from misdiagnosis and poor case note write-up. Problems were also identified in junior doctors’ ability to identify myocardial infarction.

Comparative analysis of outcomes arising from the alert for sepsis and acute myocardial infarction

A comparative analysis of outcomes generated by the alert suggests that Sep-s.05 and AMI-s.01 generated solutions directly related to problems arising from the investigation of the alert. These related to training junior doctors in writing up case notes and providing greater clarity in diagnosis and comorbidities.

AMI-s.01 appeared to have reported more outcomes related to the alert for AMI itself. These included improvements in escalation of deteriorating patients, and moving from a PARS system to a NEWS system thereby facilitating nursing engagement:

With the escalation process, yeah. That is there and it’s very clearly identified in the deteriorating patient policy now that if you don’t get it from Dr X then you go to Dr Y, and if all else fails you get the consultant. Where I think previously, yes, I think a junior nurse would probably not have gone straight to a consultant.

Head of clinical efficiency and effectiveness, AMI-s.01

In AMI-s.01, it was viewed that implementation of the MEWS system had had an impact on the number of cardiac arrests:

The number of cardiac arrests that has been influenced, has been, I’m absolutely convinced, has been influenced by the implementation of MEWS and early warning scores full stop. But I think the biggest reason why your crash calls are falling off is DNACPR [do not attempt cardiopulmonary resuscitation]. But that’s only a personal opinion.

Resus officer, head of clinical efficiency and effectiveness, AMI-s.01

AMI-s.01 was also reported as effective in changing patient pathways as work had been done to improve the use of the ‘do not attempt cardiopulmonary resuscitation’ policy. This involved introducing ‘ceilings of care’ into the patient case notes. Similarly, a reported outcome of the AMI alert was investment in trust infrastructure to redesign critical care and acute coronary care, with a ‘pacing room’ for the treatment of patients with confirmed AMI. Respondents reported how a mortality review arising from the AMI alert had identified that AMI mortality was high in wards outside critical care and cardiology. It was considered that non-specialist doctors were incorrectly diagnosing AMI in these wards, and in other cases not treating patients with AMI in a timely manner. Thus, changes were made to the AMI pathways allowing cardiology consultants to treat patients in the ‘pacing room’ and to diagnose their condition accurately in wards outside cardiology and critical care. A new protocol also set out that all patients with suspected AMI throughout the trust were to be assessed by a cardiologist.

A further outcome of the AMI alert was to include GP engagement in the mortality review process. The mortality pro forma was also changed to allow clinicians to identify whether or not the AMI had occurred outside the hospital.

Comparative analysis of parallel development of processes for sepsis and acute myocardial infarction mortality

Respondents in Sep-s.05, Sep-s.06 and AMI-s.01 conceded that other factors influenced the evolution of the mortality process. In Sep-s.05, extensive organisational turbulence experienced at the time of the alert prevented trust-wide engagement in mortality review. Regulatory pressures associated with Monitor and CQC intervention led to substantive changes in trust leadership. These resulting changes appeared to have influenced improved clinical involvement in mortality review. Engagement with external bodies was reported as influencing awareness of both sepsis and AMI. Clinical auditing of specialty performance and participation in quality improvement initiatives in both Sep-s.05 and Sep-s.06 appeared to have played a part in improving the awareness and treatment of sepsis. In AMI-s.01, engagement with AQuA and PRISM accessed learning about mortality review.

For AMI-s.01 regulatory pressures generated from the Keogh review were described as ‘giving teeth’ and urgency to trust-wide changes. A culture emerged that encouraged engagement, introduced structures supportive of learning, linked patient safety and QI ongoing processes, and fed into AMI awareness and mortality improvement.

Key themes: presence of key committees, roles and processes for effective mortality governance; effective representation on mortality committees (both internal and external)

The role of the centralised mortality committee was emphasised. It was thought to be important to properly define the committee’s purpose and function, including leadership, membership and output:

And instead of the mortality committee being as small as it was, the chief executive began to take the chair, a little bit more direction on who should be represented at it, what sort of reports they should be producing . . .

Trust mortality lead

In terms of output from the centralised mortality committee, effective communication and dissemination to individual specialties was stressed. For example, this was sometimes achieved by inviting mortality chairpersons from individual departments to sit on the centralised committee and feed back, as well as conducting departmental-level mortality meetings:

We put a mortality chair for each and every department to collect the response for a whole mortality review, and make sure that minimum quarterly every department has a mortality meeting, and aim for 100% mortality reviews. Then once we set up that process then it became much easier for us to invite all mortality chairs to one mortality group meeting.

Trust mortality lead

Simultaneously, there was a strong need for intercommittee communication to ensure that key messages from mortality review reached the right people and to support broader organisational learning:

If there will be any cross-directed learning it will go to the governance committee. If there would be any trust-level issues, then the issues will be forwarded to HAC.

Trust mortality lead

Key themes: accuracy of mortality coding; investment in coding capability; provision of coding input to mortality review

Ensuring the quality of coding supported the effective use of the alert system and prevented false or misleading signals in the data:

. . . and I think getting the data quality right, along with palliative care, is what solves the coding, it solves the alert system. I don’t think there’s anything you can do in the alert system particularly, because if you send rubbish to Dr Foster it’ll come back as contaminated alerts.

Trust consultant anaesthetist

When we would look at the volume and compare to the volume with Dr Foster there was a big difference. And we were not alerting in Dr Foster but were alerting with the CQC.

Trust mortality lead

The recruitment of high-quality coders who were both experienced and permanent was also viewed to be a challenge that required rectifying:

So creating or hiring and recruiting, or training, more senior coders and putting them at the forefront of working with clinicians, I think it was a decision that was seen as promising at least.

Head of coding and data quality

What we’re lacking now is we’re lacking resources of senior coders. So what we need to do – and actually we’ve gone through this. What we need to do is more of this. So we know what we need to do and we have seen some results, but then we need a little bit more. And the clinical coding department hasn’t got the people, not because we haven’t got the money. We have the money.

Head of coding and data quality

In line with this view, coding-specific action plans were thought to be important to consider, and potentially execute, when a mortality alert was received:

So we put in place an action plan that was around improved coding and education.

Trust mortality lead

The other thing that was found is that the coders weren’t really clear at all, in what they should put into what codes because it wasn’t clear. So we wrote some common sense guidance forms to what counts as what.

Trust deputy chief medical officer

Key themes: effective organisational use of mortality data; analytic and predictive capacity; ability to detect and respond to signals and to drill down to identify priority areas based on available data

Proactive, as opposed to reactive, use of information was thought to be the key to anticipating potential alerts and acting on quality of care issues before they reached a pivotal stage. Not all participating organisations had been able to achieve this at the time of the interviews but it was often mentioned as something to strive for:

I wouldn’t say we definitely know they’re coming but they’re not always a surprise, because we know we’re getting close to a limit where we expect to be alerted.

Trust information project manager

And partly because actually we want to start collecting the data prospectively, rather than having a retrospective flag from Dr Foster . . .

Trust mortality lead

So if we can see that there’s going to be an alert because we’ve got the information, we would actually prospectively tell the CQC we were about to alert on it and we’re at this and we’re investigating and then just give them a plan of action.

Trust mortality lead

The availability of mortality data from various sources supported organisations in effective monitoring:

I think at [name of trust] we’ve got quite a rich amount of clinical data available online and that, in combination with some of the reporting we do, kind of helps you to get to the nub of the issues reasonably easily.

Trust information manager

The specificity of these available data was emphasised as being important:

And also being able to drill down to your specialty quickly would – I think that would make it more helpful.

Trust clinical lead for heart failure

Yeah, so we know the date range and we go and do our own analysis. Because in a sense what we can do very quickly is a much more detailed analysis down to very specifics that drive in that performance, and it isn’t there. We look at what it is, we look at the date range and we go and do our own analysis.

Trust head of coding

Alongside the availability of data, it was viewed to be of equal importance that an organisation has staff members, including clinicians, with the appropriate levels of analytical expertise to make optimal use of them:

I think there’s a lot of data there. I’d say the key thing is having sufficient expertise in terms of capacity and the knowledge to analyse what’s there. I’d say that’s probably – for us that would be the key challenge.

Trust information manager

And I would say it might be great to a statistician but if you want something that’s practically going to support you in a hospital, they’re trying to understand clinical care, outcomes and mortality, I think you need something a bit more user friendly, easy to understand, etc.

Trust information manager

The use of benchmarking and triangulation with other data sets was seen as useful for maximising the use of information:

If you look at our global HSMR it’s come down, we’ve improved in the last couple of years, certainly, and we rank ourselves, we present our data, we give the current year, the current month and then we do a 12-month rolling presentation of our mortality data. Partly 12-month rolling so that if we’re always improving we never overstate how much we’re improving, because it’s an improvement plan so you don’t want to rest on your laurels and say we’re really good, you want to make yourselves look as bad as you possibly can so that you’re always trying to improve more.

Trust mortality lead

I’d look on more sources of information like I go onto the head system, I go onto SHMI, I go onto HSMR and look at specific things, and analyse it.

Trust mortality lead

In terms of data presentation, some organisations had produced a mortality score card to explore mortality data more effectively:

I’m now managing to produce, you know, a proper mortality score card on the back of that, so that we can – you know, just that we can keep a snapshot of where we’re going, and what we’re seeing there, ties with what I’m looking at on head from our kind of data.

Trust mortality lead

Interviewees discussed the ways in which they identified triggers in mortality data for further investigation and/or improvement action. For example, one interviewee talked about the use of CIs to detect relative risk:

So originally we set up a monthly mortality report that we just looked at the list of top 90 diagnoses, or whatever it was, numerically the number of patients in each grouping, what was the relative risk with confidence intervals, and we would investigate something if the confidence intervals didn’t come to 100.

Trust mortality lead

It was emphasised that trends could be effectively used to both trigger and monitor improvement actions:

So we were able to then drill down to where the areas of concern were, show the information to the clinicians and say, ‘OK guys, we’re going to have to do something with this alert which is why we’re in trouble’. And they then started doing a really targeted piece of work on a cohort of patients. And it wasn’t data manipulation; it was improvement in systems and processes that actually took the organisation out of mortality risk picture, if that makes sense.

Trust medical director

Key themes: robustness of process; frequency of mortality review; ability to identify avoidable mortality; information technology support; reporting

Some organisations reported using a tool or defined system for comprehensive primary and secondary case note review:

Well, firstly there is a second-stage review so there’s a screening tool which is being used and if that highlights any areas then there’s a second-stage review. So probably about 10% of them end up in a second-stage review.

Trust medical director

So we put in a system whereby everyone gets reviewed, that is the aspiration. And that if on the basis of information available relating to the first 24 hours of admission, that includes previous history, if on the basis of that, survival was either thought probable or was expected, those cases then get a detailed review.

Trust medical director

The use of a systematic approach to reviewing was important to ensure standardisation and shared understanding of the process. This applied to both the process of case note review and the approach to classifying outcomes from it:

So previously it has been a bit ad hoc, a bit unstandardised, various different departments doing what they felt was appropriate/not appropriate. There was no line of sight from the organisation about who was doing mortality reviews and how many mortality reviews were being done, and the outcome of those mortality reviews.

Trust mortality lead

So one of the things that we’ve done with these secondary reviews, when we can get them back, is we have the outcomes of the findings classified under themes. So there are 11 themes that we’ve got . . . And then we try and see if we’ve got recurrent themes across places and so on.

Trust consultant anaesthetist

To ensure that the mortality review process was being executed sufficiently, quality monitoring processes were also employed. These included peer review, cross-checking across specialties, the development of guidelines and training, and reviews of random cases:

But actually one of the other principles for me is peer review. So one of the reasons I identified something when I did a review of acute and unspecified renal failure is because I’m not involved in that service. And it’s like doing a serious incident investigation, you get someone outside the department to do it because they see it with fresh eyes, they’re not constrained by the baggage of what happens in the department.

Trust mortality lead

The level of available support from IT and electronic systems was also thought to influence the effectiveness of mortality review. Interviewees associated better systems with better practice in this area:

Having gone to the NCEPOD review meeting 2 or 3 months ago on the way in which trusts engage with mortality reviews, it is clear that those who do it with least pain and with greatest efficiency have IT fully on board and the process electronic . . .

Trust mortality lead

Key themes: capacity to translate learning from alerts into local actions for improvement and implement them using appropriate methodology; reporting and oversight of quality improvement projects to ensure they deliver

It was thought important to have a systematic and thorough evidence-based approach to developing improvement plans based on mortality alerts:

And actually if you’re going to have – you either do old style mortality reviews for the sake of it, or you make it part of your quality improvement process. And to be part of kind of a QI process, you have to learn from what you’re looking at.

Trust chief nurse

Once improvement plans had been executed, it was necessary to monitor and evaluate the impact that they had on quality of care. In some cases, actions required revision when they did not achieve their goals owing to either poor implementation or general ineffectiveness:

But also making sure that the clinicians are properly supported in terms of the QI methodology so that we can chart the things that we need to do, we can measure ourselves effectively against those and then we can complete the cycle in terms of what else needs to happen.

Trust CEO

Key themes: attitude towards avoidability of mortality; willingness to investigate alerts as ‘valid’ signals; degree of open discussion on sensitive issues; sense of shared responsibility and accountability for avoidable mortality

The general approach taken to responding to mortality alerts was thought to be reflective of a broader organisational culture. Some organisations found it easier to exploit learning opportunities regardless of the underlying causes of an alert:

It may be that we are truly terrible or it may just be a little smouldering fire. You can’t ignore an alert, can you?

Trust consultant anaesthetist

. . . you could always do something, and there’s always learning, there is always something else that you could have thought of and done.

Trust CEO

This type of positive learning culture was often reflected in the integration of coding and clinical review and the avoidance of gaming through developing an open culture:

Absolutely not, the rule that I go by is that you have to have a coding and clinical review in parallel, and that every set of case notes that you open there is something that you can learn from them.

Trust mortality lead

Like we said, we’ve gone down the ‘let’s look at the quality’ and I’m aware that a lot of my other colleagues have gone down the ‘let’s look at the data’, maybe looking at the quality as well. We’ve assured ourselves that we are providing care to our patients and now we’re beginning to look at the data side of things as well and that aligns.

Trust mortality lead

A lot of organisations have come to problems because they assume this is down to coding and not down to quality of care. We have been very clear that we don’t make an assumption that this is a coding issue. So that’s the first thing I would say.

Trust medical director

We wanted to create a climate of openness and transparency and again part of the effort and space in the first instance is you build that by getting people to work together.

Trust CEO

It also required an organisation to attach sufficient significance to mortality as an indicator and increase and define its profile alongside competing priorities:

So with our kind of day to day – trying to make this part of everyday business, really, mortality, as opposed to something that happens once a month that might tie in with a governance meeting. So everyone scurries around, doing a bit of M&M [morbidity and mortality review], and might present it at a governance meeting. It’s got to become part of everyday.

Trust mortality lead

Interviewees expressed a need for organisations to devote greater time and resources to increasing clinicians’ engagement with mortality and ability to interpret and understand related data:

And I think the feeling was that generally the best trusts were those who were clinically led rather than managerially led, so the feeling was that there’s ambition in the trust to do the best for the local patients. So keen to provide a good service and to get everyone involved in doing that.

Trust mortality lead

An absence of clinical engagement could result in defensiveness from the front line, which was perceived as a barrier to effectively responding to mortality alerts:

And yes, it’s always difficult sometimes with clinicians to get them to believe and trust in the data. But once they do they normally become very sort of keen and interested in it.

Trust information project manager

So again, the power of data that becomes clinically relevant is the thing that engages clinicians to then being able to understand why they have a responsibility to engage. That’s the problem, it’s the engagement of the clinicians to the issues of mortality . . . we are failing them generally in connecting them to the number that comes in to what it means to them as a clinician.

Trust medical director

To achieve this, it was thought to be important for an organisation to engage in effective job/role planning and guarantee protected time to be devoted to mortality:

I think some of them would love to take it on but at the moment it’s proving difficult to get a handover because they’re so busy that they can’t free up the time to come and sit with me and go through it.

Trust mortality lead

Key themes: strategic prioritisation of reducing avoidable mortality; senior presence on mortality review committees and action groups set up to address issues raised by mortality alerts; sponsorship of actions

Focused and engaged senior leadership who explicitly assign priority and resource to mortality alerts were thought to be vital to supporting the process:

I believe in people leading by example, and quality is a really big – and safety is a really big issue to the board now, whereas before I think they were seen as totally removed from it.

Trust mortality lead

It’s been a long time coming, and it’s coming because of the engagement of a very different type of medical director.

Clinical outcomes manager

And I think part of my responsibility is also to, with the medical director, nurse director, set the tone for the organisation around how seriously this is taken, the amount of time that is taken and part of my responsibility is also around providing some of the resource to support if we take the QI methodology and how we’re using that, so means of improving care to ensure the resource is there to support that methodology as well.

Trust CEO

Alongside high-level sponsorship, there was a need for senior leads to play an active role in the practical processes and to provide support for those who were responsible for overseeing mortality:

I feel like I’ve got a voice and I feel like I’ve got back-up now. I’ve got somewhere to escalate to so when things don’t go as they planned and when I’ve hit that brick wall I can escalate to [XXXX], I can escalate to [XXXX] and it will be done, and that is the change and it’s quite . . . yeah, it’s quite refreshing.

Trust mortality lead

Now the chief executive has put out a very clear summary of what he expects of those divisional and departmental representatives.

Trust medical director

The chief medical officer introduced a completed revision in the last 3 years around the mortality reviews . . . so we have much more openness and transparency in relation to how we deal with mortality alerts, mortality review, there is an upward track directly to board, so mortality data is reviewed monthly at board.

Trust mortality lead

Key themes: frontline clinical and broad professional involvement in mortality review, investigation and action in response to alerts; clinical input into mortality review and coding

The integration between different professional groups was highlighted as key to a holistic understanding of and approach to mortality. Interviewees often referred to the integration of clinicians and coders as a key example of effective interprofessional collaboration:

It was dialogue, sitting together and working together, absolutely. And then we have consultants that are very receptive and they actually want to improve their data.

Head of coding

What we did do after the AMIs was I took a coding manager to lots of governance meetings with me . . . so we would try to get a case from that area and to show what their [clinicians’] perception of what happened was and what a coder read from the notes.

Trust medical director

However, even within professional groups, the integration of different levels of seniority was viewed as necessary, and achieving this required specific attention:

And we’ve struggled really to get the consultant body as a whole to take the leadership on that and tell them to engage with their juniors saying, ‘I expect the “bundle” to be done’. That’s a lot better now than it was though.

Trust mortality lead

I have a personal concern that because trainees may present in a public forum where their seniors are present, there may be concern about saying what they might truly think about care, which is why you need really consultant oversight to support them in their feelings, but I think most of the time, the issues do come out.

Trust Mortality Lead

Interviewees felt that information about mortality alerts required appropriate tailoring to its target audience(s) when it was to be disseminated across the organisation:

That’s designed by department across the hospital so that each department has the opportunity to adapt the database to make specific issues, or ask specific questions.

Trust consultant in respiratory and intensive care medicine/head of mortality

I think you’ve got to look at it in who is it designed for because I don’t think it’s actually designed to go straight down to our clinicians and to the ground floor.

Trust medical director

Multiple methods were employed to achieve effective dissemination, including presentations at grand rounds and other smaller forums, newsletters and magazines, and the sharing of key reports:

I often do the grand rounds and in my grand rounds I will put all the CQC alerts, what the trust had received, and what are the learning points.

Trust mortality lead

There is a new hospital magazine that includes lessons learnt, and I have a column in there, it has only been up for a couple of months where I will select three things from mortality review meetings that are of relevance trust wide.

Trust mortality lead

So we’ve got comms on board as well and that’s a huge part of managing sepsis, I think. And rather than me trying to do it on my laptop and trying to get messages out there and develop the comms myself . . . they’re designing the posters and they’re designing the comms and it’s on the intranet and it’s on the updated screen savers and there’s posters which change on a biweekly basis.

Trust sepsis lead

Key themes: availability of guidance, participation in national campaigns; pressure from public and regulatory bodies including the media

The association of mortality alerts with the CQC was thought to influence the ways in which organisations responded to them:

I think in any NHS trust, the CQC attached to something, my gut instinct is it makes people take notice of it. It just adds a little clout to it, I think. I don’t know. That’s my perception of it. I don’t think it’s a bad thing that they’re attached to it.

Trust mortality lead

And our chief exec’s never very impressed if it’s a CQC one. So there’s that element to it.

Trust service development/coding manager

Well there’s two things, one is you’ve got to do it because the CQC tells you you’ve got to do these things.

Trust mortality lead

The influence of public accountability and local reputation was also deemed to be important:

Yes, because in the press, the press were quite easy to knock us and . . . And we’re not perfect all the time but when listening to that, very, very powerful.

Trust mortality lead

. . . there was the exposure in the media around mortality and . . . I think that actually had a positive impact and one’s got to be careful around some of this because I think it’s very easy to jump to the wrong conclusions, but nevertheless, I think it galvanised a level of interest that perhaps it would have been difficult to generate otherwise.

Trust CEO

More specifically, the launch of broader clinical programmes encouraged organisations to engage in the mortality alert improvement process. For example, the Sepsis 6 national campaign was thought to have had a wide effect on professional practice:

And so we reckon we’re saving potentially 10 lives a month with the Sepsis 6 delivery. So there’s some workstreams, so changes in working practice that have been put in place in clinical practice that are probably contributing. And if I wanted to identify one that was the biggest hitter, it would be sepsis.

Trust consultant in respiratory and intensive care medicine/head of mortality

Organisational structure for mortality governance (survey section 2 free-text analysis)

Out of the 78 trusts that responded to the survey, only nine did not have dedicated trust-level mortality group or committee. These nine, however, had a group or body that was responsible for mortality reviews and responses to alerts. These groups varied from a specialty-level group (e.g. the cardiology department for AMI alerts) to a patient safety panel.

All of the trusts had someone chairing the group, whether that group was a dedicated mortality group or one of the nine discussed above. The majority of these chairpersons were senior clinical staff; over half (56%) were medical directors and around one-third (27%) were a deputy or associate medical director. Only a very small number of trusts (n = 5) had a dedicated mortality lead chairing the group.

Local investigation and mortality review process (survey sections 4 and 5 free-text analysis)

Trusts tended to use a standardised process for mortality reviews but there was variation in these processes between trusts. A significant number used external methodologies such as NCEPOD’s classificatory system or a variation of PRISM’s methodology.

Following case note reviews, trusts would formally report their results to some form of mortality meeting or group within the trust. These groups tended to be either mortality and morbidity meetings or a mortality committee. A small number would report to patient safety committees.

There was a higher level of variation in how the results of case note reviews were disseminated across the trust. A large number of trusts did this through some form of mortality review group. However, answers varied from disseminating results through global e-mails to newsletters. More descriptive answers to this question involved listing specific groups within the organisation.

When trusts first received a mortality alert there was a degree of inconsistency in the initial action undertaken across trusts. They tended to review the coding or the clinical notes but they also undertook other measures; for example, trusts reported that they either exclusively reviewed coding or did so with a number of other actions.

When a mortality review is undertaken, part of this process is case note review. The relevant specialty lead associated with the alert condition is predominantly the person responsible for the case note reviews when a mortality review is undertaken. A very small number of these are coupled with either clinical coders or the mortality lead at the trust.

Following an alert, a trust will generally develop certain actions in order to address it. The groups largely responsible for ensuring that these actions are put into place are dedicated mortality groups such as a mortality assurance group or mortality review committee. However, for a small number of trusts this was the sole responsibility of the medical director.

In your view, what are the most important factors that influence the quality of your trust’s responses to externally-generated mortality alerts?

There was a wide range of responses regarding what were considered the most important factors influencing the trust’s responses to the alerts. These factors were seen as negative or positive influences and were generated by factors related to either the alerting process itself (such as methodological issues of the alerts) or internal factors related to the trust’s internal standards of health care (such as patient safety concerns).

One of the most prevalent responses related to issues regarding time. Trusts viewed the alerts as too retrospective in their ability to provide an indication of what was occurring in the trust due the alert-to-letter time lag. Trusts also viewed the time constraint on responding to the CQC as a factor affecting the quality of response.

Another factor was in the ability of trusts to determine whether or not the alert possessed accurate data. This related to a number of factors, but was primarily due the issue of coding accuracy:

Too many of the alerts, when the case notes are pulled turn out to be coding error. We seem to struggle to correct this, but repeated coding based alerts destroy the credibility of the process.

The factor of coding accuracy was therefore said to undermine the entire process of the alerting system. This related to resource issues around to the quality of coding staff.

Case note reviews were said to have an impact on the quality of the response in terms of data validation:

I think our responses have been thorough and high quality but often there is little to report because case note reviews have demonstrated that deaths identified as being excessive or potentially avoidable by Dr Foster have been confirmed as being expected and unavoidable on case note review.

Alongside the issue of recruiting adequate coding staff, clinical engagement with the process was a factor that influenced the quality of trust responses. If clinicians engaged with the process, there was likely to be a high-quality response:

High-level clinical engagement to external alerts leads to a good out outcome.

The quality of response was also related to the ownership of the alerts.

Alongside issues related to data validity and clinical engagement, an important factor was the trust board and senior clinical leadership.

How effective do you think externally generated mortality alerts are for driving improvement in your trust and why?

Responses varied from ineffective to moderately effective to extremely effective. Many of the responses suggested that the alerts were extremely effective in allowing trusts to review problems that arise with data, and therefore improve data quality. However, there was little to suggest that the alerts spoke to any clinical improvement.

The alerts could be a method of focusing the trust board’s priorities in terms of resources:

Extremely effective – trust exec and board more focussed and more likely to identify/support resources required.

Because the alerts were generated by the CQC and carried its authority, they could also be used to influence the staff within the trust, not just the board and senior clinical staff:

It is useful to some extent. The fact that the alert is issued by CQC provides some weight to it and provides a lever in getting clinicians to discuss the alert and to develop action plans. This is useful as it is an avenue to raise the profile of mortality reviews and quality of care reviews with the alerts can sometimes be less specific and local clinical ownership can be difficult to identify and this makes it difficult to drive improvement as a result of the alert. In addition, alerts are not always caused by a deterioration in quality.

Here we can see an indirect result of the mortality alerting surveillance system’s effects. Owing simply to the authority that is carried by the CQC, the alert can make clinical teams take notice and become more aware of mortality and the required review process.

Although the trusts once viewed the alerts as having significance, the priority that mortality now had meant that they were less effective. They were also seen as a driver for reviews and allowing an interrogation of data. This tended to flag up areas other than a need for clinical improvement. They were also effective at showing the difference in coding and data quality, and mortality improvements can be seen.

What practical changes would increase the value of a mortality monitoring and alerting service to your organisation and the NHS?

Regarding the practical changes that would increase the value of a mortality monitoring and alerting service to the organisation and the NHS, these responses, in the main, related to the alerting process or the data inherent with the alerts. Alongside this, responses suggested that a system owned by the NHS and standardised across the health service would be appropriate and increase its value.

The sensitivity of the system would also increase its value. Coding accuracy and its implications for data validity were again issues, as was the time lag of the alerts. Specific conditions that had an affect related to the alerting system’s insensitivity to paediatric or neonatal mortality.

Data accuracy and validity, again, was important to respondents here and would affect any practical changes.

There were also suggestions about the role of the CQC. These tended to suggest that the CQC induces negative behaviour in the sense of making the responses to the alerts untrue and superficial, with one respondent describing the alert response as ‘a data management beauty pageant’ and ‘that the requirement to report the responses to the CQC is highly counterproductive and induces defensive behaviours’.

One issue that also tended to occur was the insensitivity of the risk-adjusted model. One respondent argued that the poor benchmarking of the risk-adjusted model related to a non-standardised system within the NHS:

If the monitoring/alerting system was NHS funded would save money and ensure all trusts are using the same risk adjustment model for data comparison. Increase in patient safety (non-mortality) alerts.

The idea of a national standardised alerting system was an important point raised by the respondents, for example sharing the quality of trusts’ use of the system and creating a national system of best practice responses to alerts. The use of mortality was seen as needing to change as it was viewed as too narrow an indicator of quality.

Is there a dedicated trust-level lead for mortality reduction in your trust?

No significant difference was found between trusts that did and trusts that did not have a trust-level lead for mortality reduction.

Are there specific mortality leads appointed in the following areas in your trust?

No significant difference was found between trusts that did and trusts that did not have specific mortality leads appointed at the individual division level.

No significant difference was found between trusts that did and trusts that did not have specific mortality leads appointed at the clinical directorate level.

No significant difference was found between trusts that did and trusts that did not have specific mortality leads appointed at the clinical specialty level.

Does your trust have a dedicated trust-level mortality group or committee in place currently?

No significant difference was found between trusts that did and did not have a dedicated trust-level mortality group or committee in place currently.

Does the mortality group report to the trust board?

No significant difference was found between trusts that did and did not have a mortality group that reported directly to the trust board.

Are specialty-level mortality data reviewed by the trust board as part of the organisation’s key performance indicators?

There is a significant difference between trusts that reported that specialty-level mortality data are reviewed by the trust board as part of the organisation’s key performance indicators [t(74) = 2.65, p = 0.01]. Trusts in which specialty-level mortality data are reviewed by the trust board reported significantly higher institutional capacity to respond to signals in mortality data (mean 6.28, SD 0.67) than those in which they are not (mean 5.77, SD 0.88). Figure 36 compares the mean institutional capacity to respond to signals in mortality data between trusts that report the review of specialty-level mortality data as part of key performance indicators and trusts that do not.

FIGURE 36.

Bar chart showing rated institutional capacity by presence of specialty-level trust board mortality data review.

Are case notes reviewed by someone independent of those responsible for the care of the patient?

No significant difference was found between trusts that did and trusts that did not report that case notes are reviewed by someone independent of those responsible for the care of the patient.

How are reviews of death instigated in your trust?

No significant difference was found between trusts that review deaths in response to both external alerts and alerts from their internal systems and trusts that routinely review all deaths in addition to carrying out reviews in response to alerts.

Descriptive analysis of the Imperial College mortality alerts

Our analysis found that there has been a fall in the number of mortality alerts over the study period; however, there has been a corresponding fall in national mortality rates. 152,153 This in itself would not lead to a reduction in the number of alerts, as they take into account national trends, but the reduction in number might reflect a reduction in variation between hospital trusts.

Analysis of Care Quality Commission investigations into the Imperial College mortality alerts

The CQC’s purpose is to maintain and improve standards of care in NHS trusts and it uses the mortality alerting system as part of its risk assessment profiles. We systematically reviewed all evidence held within the CQC, including its communications with trusts, resulting in full data capture during the study period. However, the reviewer of CQC information was not blinded to the aims and objectives of the study question, and although efforts were taken to standardise the data collection, bias may still have been introduced. Our review found that most trusts were able to identify areas that could be improved within their care pathways, and implemented action plans to address these issues. Action plans were created in 64% (99/154) of CQC cases after the initial investigation but 77% (118/154) by the time the cases were closed. This suggests that the CQC may be more likely to pursue the case until an action plan is created. We are unable to associate the creation of action plans with Imperial College mortality alerts, as we cannot disentangle the effect of the mortality alert letter from the input of the CQC. The creation of action plans to improve care in ‘outlier’ trusts is highlighted in the CQC annual reports. 36,38 In addition, we noted that there was variation in the clinical audit standards used by the trusts in their responses. For example, some trusts identified potential causes for the increased mortality, whereas others consistently failed to do this.

Changes to coding following an alert

We found that, following an AMI or sepsis alert, the majority of trusts made some changes to their data, possibly as a consequence of coding reviews. On average, the number of admissions and deaths with these diagnoses fell in alerting trusts following our reanalysis. However, we had no control group (non-alerting trusts) to investigate. The CQC mandates yearly audits of individual trust data quality, which will trigger an audit of coding within the trust;153 as a result, clinical coding is predominantly good quality. 32 However, such auditing could also explain the changes in coding that we found. Gaming seems to be an unlikely explanation, as there was actually an increase in the relative risks following sepsis changes in the data.

Investigation of mortality alerts and other measures of quality

Ours was an observation study that does not allow causality to be established, but it did allow us to investigate associations. Mortality alerts were associated with other indicators of quality such as patient satisfaction, overcrowding (% acute bed occupancy), staff resources (nurse-to-bed ratio) and trust deficit. We carried out significance tests to a significance level of 0.05, used cluster-robust standard errors and controlled for FDR. We hypothesised that mortality alerts were associated with our measures of quality and found associations in some of the indicators. AMI and sepsis analyses were underpowered owing to the small number of alerts. We matched time points, some of which were annual and others were monthly or quarterly. When outcomes cover wide time periods, this may affect the sensitivity of the analysis. Missing data were difficult to determine. In the absence of a consistent authoritative list of trusts in existence in each year of analysis, we compiled our own comprehensive list of acute hospital trusts (and their provider codes) from a range of sources, including the mortality alerts database, updating the status depending on trust closures and mergers. However, quality measures were not always complete; for example, measures of hospital mortality measurements did not include specialist NHS trusts. We assume that MINAP data are complete but do not have a comprehensive list of NHS trusts that provide PCI treatment. Most of our quality indicators are objective, which limits bias; however, NHS thermometer data on harms are self-reported. This could introduce bias, as trusts with poor care may also have poor reporting.

Analysing the association of an alert with trends in subsequent risk of death

Interrupted time series is a strong quasi-experimental study design (one of the strongest after randomised controlled trials). There are some factors that need to be noted. After extreme measurements subsequent measurements are likely to fall. This is termed regression to the mean. The Imperial College alerts are triggered by a sustained increase in relative mortality risk with a threshold set at a high level to ensure an estimated false alarm rate of 0.1% over a 12-month period of monitoring. It is, therefore, unlikely that the sustained high relative mortality risks are due to chance and that the subsequent falls are due to chance. However, our three models all found that trust-level relative risks of mortality fell after an alert and the majority of the decrease took place within the lag period, the period we hypothesised that it would take for the trust to be notified by an alert letter and implement changes to reduce mortality. Given these results, it is difficult to determine the true effect of the surveillance system, but these findings could indicate that trusts are monitoring their own mortality statistics and taking action before the alert letter. We found that the unadjusted risk of mortality fell after an alert to a similar extent as relative risk. Given that it is unlikely that case mix will drastically alter, over time, in a typical acute unit, we would expect effect sizes to be similar in adjusted and unadjusted models. This finding suggests that changes in coding after an alert contribute minimally to the fall.

Qualitative case-based analysis

Workstream 2 aimed to employ mixed methods to investigate the mechanisms by which mortality alerts were assimilated at the local trust level and influenced organisational behaviour through investigation of 11 in-depth case studies (objectives 1 and 4). Using a realist framework and theoretical insights from institutional theory, the analysis sought to describe interactions between the alerts (as feedback intervention), institutional context, organisational behaviour and outcomes. Using a cross-case analysis, we sought to achieve comparative perspectives on the effects of alerts in two diagnoses (sepsis and AMI) and the effects of single versus repeat alerts within these diagnoses in order to enrich our understanding of potential intervention mechanisms (objectives 5 and 6).

An integrative analysis across the reported case studies focused on a comparison of AMI-alerted and sepsis-alerted trusts, and of single- and repeat-alerted trusts, in accordance with the original research objectives, in order to identify common and divergent themes. The resulting analysis developed a deep insight into the ways in which trusts responded to AMI and sepsis alerts, and the institutional contextual and behavioural processes that support or regulate the response. The key findings from this analysis are summarised as follows and discussed subsequently.

• Alerts are considered useful in providing focus for trust intervention in addressing mortality.

• Case note review was the common response to alerts across all 11 sites.

• A forensic approach that completes the circle of problem identification, problem analysis, triangulation of data, identification of solutions and implementation of new ways of working.

• Use of innovative techniques to improve connectivity and knowledge sharing across trusts focused on tackling mortality (boundary spanners, boundary objects and electronic information sharing).

• Degree of clinical involvement in responding to alerts for sepsis and AMI.

• Senior clinicians identifying problems in the coding of sepsis and AMI diagnosis.

In addition to the outcomes and behaviours that were attributable to the mortality alert, as stimulus or intervention, a comparative qualitative analysis of cases revealed a range of institutional processes centred on the implicated mortality area that were not directly attributable to the alert as intervention, and hence were classified as parallel evolution of the trust’s focus on managing avoidable mortality within the area:

• changes in trust leaders and sentinel events

• an emphasis on building a trust culture focused on improving quality of care

• engagement with external organisations to provide knowledge and learning about mortality

• the development of integrated structures linking trust organisational levels (corporate, division and specialty).

The analysis suggests that the alerts for both AMI and sepsis did instigate institutional responses across all the case study sites. These responses were characterised as follows: the alert served as a trigger for action; a universal case note review response; a forensic approach to identifying problems, identifying solutions and implementing actions; the development of some innovative techniques in developing sepsis and AMI awareness; senior clinical involvement in case note review, coding of deaths and diagnosis identification. It was reported that these responses resulted in the following outcomes: case note review and coding improvements, specific changes in patient pathways, changes in diagnosis of sepsis and AMI, training of clinical staff in case note write-up and coding, greater transparency in patient deterioration and resourcing of some infrastructure changes. The results suggested that trusts receiving repeat alerts for sepsis and AMI appeared to demonstrate a slightly different approach and response from those trusts receiving a single alert for AMI and sepsis, but that the exact nature of variations in response were context-dependent. In some trusts, this difference was a slower movement or reaction to the alert than that in trusts that had received previous alerts in the same area. It is noted that this slow movement was often boosted by intervention from the regulator (CQC, Keogh review and Monitor), and it was this combination of interventions, information and regulation that appeared to link with a turnaround in how mortality was tackled. In many trusts, this coincided with a complete overhaul of the executive team and the development of a new regime for addressing mortality as a result of external pressure and scrutiny.

The comparative analysis of case studies not only revealed patterns of institutional behaviour that were directly attributable to the alert, but emphasised the important role that local contextual circumstances and parallel developmental processes and strategies, instigated by the trust in relation to sepsis and AMI mortality reduction, played in interacting with the response to alerts and regulatory follow-up. In contrast with directly attributable institutional responses, the parallel evolution of trust culture and processes was recognised across all trusts and often made it difficult for the research team to attribute causality in how the alerts influenced trust responses. This finding illustrates the complex nature of institutional behaviour around mortality governance and the role of mortality surveillance and alerting in that process. In those trusts that demonstrated mature mortality monitoring and reporting processes, the receipt of a mortality alert was often predicted from local review of dashboard indicators and use of subscription-based statistical tool packages. Investigation into the causes of abnormal trends in risk-adjusted mortality was often already under way at the time of the receipt of the alert letter, greatly facilitating the speed with which the institution could respond to the CQC requests for follow-up or engage with the Imperial College alerts team to seek clarification. As a result of previous scrutiny in response to above-expected mortality, several trusts had completely overhauled their organisational governance structures for mortality reduction and these substantial changes in organisational structure and behaviour were not directly attributable to the effect of a single alert or string of alerts within any particular area. Interestingly, several mortality and executive leads reported that the trust had been pursuing a long-term mortality-reduction agenda when the alert letter was received and they were subsequently able to use that alert letter to bolster local support, renew institutional commitment or engage dissenting staff groups in support of the programme. The information that the alert represented was, therefore, not as important as its symbolic representation of external scrutiny and the need for continuous remedial effort.

Taking the findings from the case-based analysis concerning the extent of locally-reported behaviour that we were able to attribute directly to the effects of the alerts, our experience in this study is relevant to understanding the extent of evaluation that is possible for a complex, institutional-level intervention such as the mortality alerting system. Our findings are more broadly relevant to any attempt to evaluate the impact of a complex informatics intervention, such as surveillance and feedback. First, for any such intervention, a large component of the active ingredients are context-specific and locally-determined. Information may be fed to a system (with or without guidance for how to act on it), but the system changes that arise (organisational structure and behaviour; quality improvement actions and processes) are not specified as part of the intervention, mainly because the alert is akin to a smoke signal, with the exact origin of the fire unknown. Second, in the absence of clear causality, the contribution of any one component of the system (i.e. the alert) to variance in the outcome of interest (mortality reduction) becomes difficult to ascertain without the benefit of the type of experimental controls that it is difficult to achieve when the units of analysis are themselves complex organisational systems. Finally, the opportunity for evaluation lies in identification and dissemination of what are judged to be adaptive intermediary processes and mechanisms, rather than the dose–response relationship with outcome.

This study has described the nuanced behavioural responses of institutions to intelligence on potentially avoidable mortality. A key message from our work is that it is as important to understand the local preconditions for effective responses – how maturity in institutional mortality monitoring, review and action processes can be achieved – as it is to seek the design of an optimum analytic toolset, set of metrics or mode of alerting. An important developmental step towards enhancing local capability to generate, understand and respond to signals in mortality data is, therefore, to provide guidance and evidence to promote institutional responses that are in the best interests of patient safety. We believe that this current study is an important step in that direction.

To present a more complete discussion and interpretation of the higher-level themes that emerged from the comparative case study analysis, the following sections provide meta-synthesis and review, drawing on knowledge gained across the separate subanalyses.

Key themes from integrated case analysis: institutional behaviour attributable to the alert as intervention

Parallel evolution of the trust’s focus on avoidable mortality (not attributed to the effects of the alert)

National cross-sectional survey

The national cross-sectional survey employed scales developed through the qualitative analysis of evaluative dimensions for institutional responses to mortality alerts, along with free-text items designed to capture elaborative information from respondents. The survey study sought to analyse variance in organisational structural and behavioural characteristics relative to mortality governance, to identify the main drivers of effective institutional responses and to capture mortality leads’ views of the current adequacy of the MASS.

The survey data suggested that 86% of responding trusts had a dedicated trust-level lead for mortality reduction and 92% had a dedicated trust-level mortality group or committee in place. These are high proportions and fall in line with recent policy development geared towards improving mortality monitoring and review processes within trusts. Over half of the mortality committees were chaired by a senior clinical member, who was the trust medical director in 56% of cases.

In addition to a high proportion of dedicated mortality groups, the majority of trusts (97%) subscribed to one or more commercially available data analytic tools to support mortality monitoring. Over 90% of trusts reported routinely using one or other of the common trust-level mortality indicators, HSMR and SHMI. Although trust-level investment in mortality monitoring and analytics was strong, the practice of review of granular mortality data at board level was more variable, with only 37% of trusts reporting that specialty-level mortality data were reviewed by the board as part of the organisation’s key performance indicators. The implication is that although trusts are clearly investing in informatics capabilities around mortality monitoring, analytics and reporting, they may lag behind in maximising the local use of these systems as a means of internal regulation of mortality and as a means of achieving granular prioritisation of areas for remedial action.

In terms of the intensity of local mortality review, on average 60% of deaths were reviewed for any given period by responding trusts, with 69% of responding trusts reporting that they routinely reviewed all deaths. Clearly, some variation in practice exists for the extent of mortality review undertaken and the specific methodologies used. Testimonies from the case study site visits suggested that, for some trusts, instigating mortality review processes to respond to a specific query was a challenging process, involving changes to roles and responsibilities, staffing time allocations, organisational and reporting structures, and information technologies. When reliable mortality review processes were in place and the process did not need to be configured in an ad hoc manner in response to a CQC request for local investigation, invariably interviewees reported a more effective response.

An examination of the responses to individual survey items within the institutional capacity to respond to signals in mortality data scale revealed stronger and weaker areas of practice. Across the sample, high agreement was recorded for the trust priority assigned to mortality reduction and senior leadership support for mortality monitoring. The weakest areas reported concerned accuracy of coding, the quality of specialty-level mortality data and understanding trends in specialty-level mortality data. The only item to achieve a negative response, on average, was the provision of protected time in people’s job plans for mortality-related work.

During the case study analysis, it became clear that the recipient’s perception of the validity of the alert as a function of data quality, coding accuracy and adequacy of the risk adjustment was an important moderator for the institutional response. The perception of the recipient in this case might be a collective institutional sense of identity or justice, or the judgement of one or more powerful individuals within the organisation who provided an accepted ‘local interpretation’ of the information the alert contained, which was then accepted by colleagues or used to shape the form of the initial response. During stakeholder consultation with the CQC, for example, it became apparent that often the CQC’s role was to attempt to focus the alerted trust on making an investigative response, following the rejection of an initial attempt by the organisation to dismiss the alert as non-valid or unreliable. The perception of data quality, coding and statistical accuracy of the analytics underpinning the alerts, as measured by our survey, is therefore a key area for development, given the relatively low maturity reported in this area.

On average, respondents felt that senior commitment to a mortality reduction agenda was a strong feature of current trust activities in this area, but that there was room for improvement in data accuracy and comprehension. Having dedicated time to work on mortality was a commonly reported issue and echoed opinions expressed consistently during site visits. Experience during the case study site visits additionally highlighted the complexity involved in making sense of locally available data and analytics related to mortality. Often, individuals with a specific interest in the area would dedicate significant amounts of unofficial time in studying and monitoring available data outputs in order to achieve an understanding of key metrics and analyses in an attempt to guide the trust’s work.

An important contribution of the current study lies in the formalisation of an evaluative framework for institutional capability to respond to signals in mortality data, based on extensive empirical qualitative work and the development, refinement and testing of associated scale measures. A qualitative framework analysis of testimony across the 11 case study sites elicited the following nine evaluative dimensions for understanding variance in institutional capability to respond to signals in mortality data.

1. Organisational structure for mortality governance. Key themes: presence of key committees, roles and processes for effective mortality governance; effective representation on mortality committees (both internal and external).

2. Coding. Key themes: accuracy of mortality coding; investment in coding capability; provision of coding input to mortality review.

3. Use of information, monitoring and reporting. Key themes: effective organisational use of mortality data; analytic and predictive capacity; ability to detect and respond to signals and to drill down to identify priority areas based on available data.

4. Local investigation and mortality review process. Key themes: robustness of process; frequency of mortality review; ability to identify avoidable mortality; IT support; reporting.

5. Local improvement mechanisms. Key themes: capacity to translate learning from alerts into local actions for improvement and implement them using appropriate methodology; reporting and oversight of quality improvement projects to ensure they deliver.

6. Organisational culture. Key themes: attitude towards avoidability of mortality; willingness to investigate alerts as ‘valid’ signals; degree of open discussion on sensitive issues; sense of shared responsibility and accountability for avoidable mortality.

7. Senior leadership and sponsorship. Key themes: strategic prioritisation of reducing avoidable mortality; senior presence on mortality review committees and action groups set up to address issues raised by mortality alerts; sponsorship of actions.

8. Interprofessional collaboration. Key themes: frontline clinical and broad professional involvement in mortality review, investigation and action in response to alerts; clinical input into mortality review and coding.

9. External environment. Key themes: availability of guidance; participation in national campaigns; pressure from public and regulatory bodies including the media.

An interesting feature of the evaluative dimensions for institutional capability to respond to signals in mortality data is that only two of the nine dimensions are directly related to informatics. The remainder of the dimensions address organisational structure and other aspects of internal and external context from a sociotechnical perspective. The implication of the breadth of dimensions found to be relevant is that although the validity of information content in the signal and its comprehension and interpretation on the part of the receiving trust are important, ultimately the success of the system as a holistic intervention is dependent on a range of additional local factors concerning how the signal is translated into actions/behaviour and how robustly those actions are implemented. Local action in response to signals in mortality data is, therefore, governed by the adequacy of responsible organisational structures and processes, local investigative mechanisms, the local culture and climate for interprofessional collaboration and openness to feedback on quality of care, strategic prioritisation for reducing avoidable mortality and external support.

In the final version of the survey instrument, an aggregated scale was developed to test the latent construct: institutional capability to respond to signals in mortality data, comprising 21 individual items derived from the qualitative definitions of the evaluative dimensions. The 21 items corresponded to seven of the evaluative dimensions (with the remaining two, culture and external factors, tested elsewhere in the survey instrument). The seven subscales tested were organisational structure for mortality governance, coding, use of information monitoring and reporting, local investigation and mortality review process, local improvement mechanisms, senior leadership and sponsorship, and interprofessional collaboration.

A correlational analysis of the evaluative dimensions revealed several interesting findings. First, the overall institutional capability scale achieved moderate positive correlations with three other variables tested within the survey: the length of time that current arrangements for mortality have been in place, the percentage of specialties within the trust that reliably review all deaths and the overall percentage of deaths that are reviewed by the trust in any given period. The last variable approached a large correlation (r = 0.406; p > 0.0001), demonstrating a particularly robust observed association. Clearly, the development of a trust’s capacity for response occurs over a period of time, and the rapid deployment of measures for sustained management of avoidable mortality may not be immediately effective. The observed association between overall institutional capability to respond to mortality alerts and the capacity of a trust to maintain comprehensive mortality review processes in all clinical areas is suggestive of the importance of having reliable and continuous institutional mortality review processes in place, in order for signals of potential patient safety vulnerabilities to be addressed in a timely and effective manner. The direction of causal inference here becomes important. Both the notion that establishing comprehensive trust-wide mortality review leads to alignment of other organisational areas for effective mortality governance and the converse, that investment in conducive organisational structure and context is a prerequisite for effective mortality review processes, are viable. The results from our qualitative analyses would favour the latter explanation, as the dependency of effective mortality review processes on broad cross-specialty engagement, interprofessional collaboration (e.g. coders working with clinicians) and effectively chaired review committees supported by capable data analytics was demonstrated repeatedly during our site visits.

Regarding the specific subscales employed to test the evaluative framework dimensions within the institutional capability scale, it is interesting to note that the percentage of deaths reviewed in any given period was positively associated with the capacity for local investigation and mortality review processes, robustness of improvement mechanisms and senior leadership support. The implication is that comprehensive internal mortality monitoring requires effective local action mechanisms and sustained senior-level priority within a health-care organisation. The qualitative case study results demonstrate the potential value of institutional investment in comprehensive, clinician-led, ongoing mortality review, if queries raised by alerts representing potential escalating risk to patient safety are to be resolved quickly and effectively.

When respondents were asked to evaluate the current arrangements for mortality alerting and surveillance directly, responses overall and for the vast majority of individual evaluative items were positive. The highest level of agreement was achieved for statements concerning willingness to invest staff and resources in responding to alerts and the role of mortality alerting in regulatory processes and public assurance. Overall, respondents agreed that continuous alerting and surveillance focused trusts on avoidable mortality in a constructive way, a viewpoint echoed by the majority of senior staff interviewed during site visits, who acknowledged that, even in the absence of a clear clinical service implication, the institutional process of investigating statistically abnormal mortality within an area had additional benefits in terms of the lessons learnt for improving coding accuracy and focusing staff and units on patient safety and quality of care. The survey responses additionally indicated that, taken as a whole, respondents felt to a modest degree that mortality alerts led to improvements in local review processes, data monitoring and reporting and multiprofessional engagement in mortality reduction, all sentiments that were expressed to the research team during site visits. The only area that received a slightly negative response, on average, was the perception that mortality alerts represented valid and reliable signals of problems in care delivery.

It is particularly interesting to note that respondents held positive views of the value of mortality surveillance and alerting despite lacking surety in the validity and reliability of the alerts as signals of problems in care. This was a somewhat surprising finding but, as previously mentioned, testimonies from key informants during the case study site visits would support the notion that the alerting system served multiple parallel functions and supported varied intermediary outcomes for the receiving institution, such as focusing attention, improving documentation or securing investment in improved informatics. The notion was expressed on multiple occasions that a diligent response to a mortality alert was synonymous with professionalism in clinical care and symbolic of the trust’s commitment to mortality reduction and high standards of patient care. When mortality alerts are found to be attributable to underlying data quality issues rather than to problems in clinical care, the actions that trusts take to improve coding and data accuracy should improve the reliability of future signals and reduce false positives.

When considering the barriers to reduction of avoidable mortality, respondents indicated that resource availability was a primary concern, followed by coding accuracy, risk adjustment adequacy and specificity and recency of mortality data. Other issues also received moderate attention, such as the availability of knowledge of and expertise in how to respond effectively to the alert. These findings are in accordance with existing research on the characteristics of effective feedback from quality indicators and medical registries, which holds that feedback must be continuous, relevant, timely and accompanied by guidance on remedial actions if it is to be maximally effective in improving outcomes.

Regarding the reported drivers for local institutional responses to mortality alerts, on average, respondents identified CQC involvement in mortality alerting and involvement in national programmes and campaigns in related areas as moderately strong motivating factors, followed by external reporting on mortality alerts received by the trust. Clearly, the external regulatory and public environment is an important consideration for trust leads when it comes to mortality, particularly being seen to respond to regulatory scrutiny and being able to present a strong track record to the public. It is encouraging that trust involvement in national campaigns, such as recent sepsis reduction programmes, is additionally a strong motivator in the level of resources a trust is willing to invest in responding to signals that may herald problems with care.

Qualitative study

In qualitative research, the opportunities afforded for sense-making and interpretation in the pursuit of depth of useful understanding of the mechanisms governing a phenomenon in one or more context-specific cases are favoured over traditionally experimental notions of generalisability and repeatability. Given the socially situated nature of the testimony provided by case study informants, along with the subjectively constructed nature of their narratives, our qualitative investigation of the institutional response to mortality alerts was, inevitably, possessed of subjectivity and bias in various forms. Subjectivity was inherent in the recall of our participants, the social context for data collection (interviews) and the preconceptions and perspectives of the researchers involved in the analysis and interpretation. The sensitive nature of the topic area may well have led certain respondents to adjust their accounts to present the institutional track record in this area in a positive light, or to omit other important details that they felt might have led to negative construction. The institutional responses to signals in mortality data were themselves often long-term institutional programmes involving multiple stakeholders, multiple analytic steps, service change and negotiations with external entities. As such, the institutional experience is often constructed at the individual level, with or without the benefit of hindsight, and events are subject to reinterpretation and post hoc reconstruction. As a consequence of complexity and subjectivity, a key issue experienced by the research team concerned the ability to link the organisational behaviour around local mortality governance described by informants, to the effects of receiving an alert (or the specific alert in question). Essentially, it was often difficult to make a causal attribution, for a reported behaviour, to the mortality alert, effectively framing the latter as stimulus or intervention. This effect was compounded when the time since the original alert instance was longer and spanned what were sometimes multiple changes in trust leadership and structure.

The preconceptions and position of the researcher(s) undertaking the inquiry may influence the course and outcomes of that inquiry, leading to subjectivity in interpretations of data. This is an acknowledged limitation of all qualitative research, and in the present study we took a variety of steps to maximise objectivity and usefulness in our interpretations.

An important consideration when appraising the research reported here concerns the critical distance of the qualitative research team and the opportunity for bias in interpretation of the data afforded by the researcher’s position as academics working at Imperial College London, the host organisation for the Dr Foster mortality alerting system. Despite this connection, the research team, under the guidance of its independently chaired steering committee, designed as part of the peer-reviewed funding application process a systematic and evaluation-driven approach to data collection and analysis that employed best practices in qualitative research. Furthermore, the qualitative research team responsible for visiting sites and collecting data were from a different department within Imperial College from the Dr Foster academic unit (surgery and cancer as opposed to public health) and, therefore, had limited experience of the alerting system and few prior conceptions concerning the value of such a system, other than a genuine desire to capture and describe service-level perceptions of this type of surveillance. The qualitative research team was therefore well placed to elicit balanced and systematic evaluative data. The resulting data, as demonstrated through the detailed case narratives and thematic analysis, contains both positive and negative viewpoints expressed by informants and that were subsequently synthesised into categories within the analytic process.

In the qualitative data collection activity, it was additionally important to be clear about which perspective the visiting research team represented, so as to not bias responses as a result of perceived demand characteristics. During the course of the study it was made clear to each interview respondent at the start of the interview that the interview team was independent of the alert-issuing academic unit at Imperial College, of the Dr Foster corporate entity and of the CQC itself, although it was pointed out that all of these organisations were key stakeholders of the results of the research. The interviewees were provided the opportunity to ask questions concerning this aspect and the standard information provided before they consented to the interview.

At the level of the individual case study sites, bias may have occurred in the presentation of key roles to provide information concerning institutional perspectives on mortality. This may also have led to the misrepresentation or omission of key perspectives, including that of the clinical front line, on the local institutional arrangements. The research team attempted to address these limitations in a number of ways through the design of sampling and recruitment at each site. Following site recruitment, the researchers held preliminary telephone meetings with each site lead (usually the mortality lead or medical director) to obtain information concerning whom to interview during the visit, in order to obtain the most comprehensive and balanced view of the site context. Sampling was also driven by a quota of defined roles that were linked to the research aims. Individuals’ availability for interview on the day was an issue in only a small minority of cases. This approach ensured that a diagonal cross-section of the organisation was sampled (i.e. key functional areas at different levels). We sought predominantly senior roles, as these were the levels at which the response to mortality alerts was overseen and these individuals sat on or chaired the committees that had the responsibility for mortality review and action planning. Frontline representation was also sought, however, in the form of the clinical leads responsible for the implicated clinical areas. Conversations with these individuals demonstrated sensitivity to frontline perspectives on mortality and mortality governance.

Throughout the study, many steps were taken to ensure that the data collection and analysis proceeded as systematically as possible (e.g. standardised data collection, systematic approach to sampling) and, in accordance with best practice in qualitative work, interpretations were iterated extensively and subject to peer review and reinterpretation by multiple researchers and advisors who had varying perspectives and expertise. Further objectivity was introduced through a phased approach to data analysis in which qualitative data was first processed on a case-by-case basis before comparative and integrative approaches were employed to establish robust themes across cases and surface critical differences. Despite these efforts, generalisation from the causal mechanisms and processes reported here should be undertaken only with the usual caveats that apply to case study research. Although many aspects of intervention–context–outcome interactions reported here for the institutional response to mortality alerts are tied to the potentially idiosyncratic features and histories of the organisations visited during the course of the study, we believe that there are valid and useful generalisable lessons to be learnt from the experiences of our 11 case study sites at many levels.

Survey study

Regarding the survey measure that was employed in the cross-sectional study, a considerable strength of the design was the extensive qualitative analysis that underpinned development of the items used within the measure itself, including the steps taken to gain broader stakeholder input into the development of the evaluative framework and the refinement and validation of the measurement instrument itself. Although internal consistency as indicated by examination of scale reliability metrics was deemed acceptable for a new measurement tool in a novel area of research, the achieved sample size was prohibitive of further psychometric analysis and exploratory factor analysis is advisable to support any further refinement and application of the measure.

Aside from the reliability and validity of the measurement tool, the usual caveats and threats to external validity for generalisation of survey research findings apply. The 65% response rate achieved, although good for an unincentivised national survey of this type, still omitted 35% of trusts approached to participate. Although variance was observed in both the responses to individual scale items in the survey and the qualitative views expressed, the possibility of the operation of response bias (i.e. systematic rather than random missing data) cannot be ruled out. It might be the case, for example, that trust leads who were aware of less capable local institutional arrangements for responding to mortality alerts, or who held negative views concerning the current arrangements for mortality surveillance, were less inclined to respond. The nature of the approach to sampling, targeting medical directors and mortality leads as representatives of the institutional stance on managing avoidable mortality, also had strengths and weaknesses. Our qualitative research suggested that only people in these particular roles who possessed the requisite oversight and experience to comment on the mechanics of local organisational arrangements for mortality governance. However, the possibility exists that bias entered the survey responses owing to a desire on the part of senior leads to present their organisation in a positive light. It was notable that respondents rated several leadership dimensions within the survey scales as some of the strongest aspects of the local system, and this result should, therefore, be interpreted with explicit knowledge of the sampling framework employed.

In the survey analysis, the novel nature of the research focus and associated lack of prior research to support clear causal hypotheses necessitated an exploratory analytic frame, which led to a larger number of statistical tests and an associated inflated family-wise error rate. Although the results from correlational analysis of the scale items are interpretable, possess face validity and represent interesting insight into the research questions, any specific inferences should be subject to further investigation and replication.

Finally, the timing of the survey study should be acknowledged in the context of broader NHS England policy concerning local mortality review and mortality governance. Consultation with trusts on the development of national guidance and compulsory practices for local mortality governance had taken place in early 2016, concluding some 2 months before the survey administration. Respondents, therefore, may have been primed to make aspirational responses or may have undertaken rapid recent development in local mortality governance processes in anticipation of the implementation of new guidance. Both eventualities would render aspects of the survey response unrepresentative of the state of local institutional capability 6 months previously, that is, at the time of the site visits. The research team were aware of this risk and steps were taken to accommodate this within the definition of the survey measures, which were adjusted to refer specifically to the previous 12-month period, rather than reflecting more recent developments.

A trust culture characterised as ‘closed’, lacking awareness of mortality and limited staff engagement in addressing of trust mortality

Respondents generally reported that before the AMI alert the trust was ‘closed’ and lacked transparency. Trust participants reported varying degrees of awareness regarding mortality. In one specialty clinical staff reported worry concerning mortality rates, whereas in other areas there was no awareness. However, clinical staff did appear to hold a general view that trust mortality had been a problem for many years. A punitive and bullying approach was described, as a senior clinician had been disciplined for engaging with the media about severe problems around high mortality, and this led to others being unwilling to speak up.

Limited learning gained from trust engagement with external organisations

It was generally accepted that the trust was engaging with external bodies but not sharing its knowledge and awareness of problems regarding mortality. For example, the trust was participating with a NHS care quality improvement organisation and implementing care bundles as far as possible. However, this work was limited to a single clinician, who described this work as a ‘one-man band’. Staff described how the knowledge gained through this external engagement was either not shared or not followed up.

Limited in clinical engagement in addressing mortality: senior clinicians denying responsibility for acute myocardial infarction mortality

Limited clinical engagement in addressing mortality was generally reported. It was suggested that when senior consultants were made aware of mortality issues they attempted to shift the blame and argue about problems and solutions with each other.

Extreme regulatory pressures (Care Quality Commission, Keogh Review, Quality Accounts Audit) create organisational instability and fatigue but also provide a new and improved way forward

It was conceded that the trust was exposed to a host of regulatory pressures that sapped its energies and demoralised staff. The CQC inspections were said to have laid bare how the trust was failing. It was posited that Imperial College alerts were less coercive when not followed up by a CQC inspection. However, in themselves these alerts (for multiple conditions) were seen as useful but still created organisational turbulence and fatigue, as time was limited to address each one. Respondents highlighted how the Keogh investigation had allowed the trust ‘to wipe the slate clean’.

Financial pressures limit resources

The trust’s financial position has varied since 2010/11 between a surplus of £1.15M and a deficit of £4.77M. It was suggested that this had affected the resourcing of staff and prevented the trust from attending meetings about governance and mortality investigation.

A fragmented and token approach to mortality that fails to engage trust divisions and specialities

Respondents suggested that the trust’s leadership had failed to develop a strategic priority for investigating and tackling AMI mortality. Overall, they reported a fragmented and token approach that had failed to engage across divisions and specialties.

An integrated approach linking trust levels and ensuring generation of high-quality data and improved ways of working

Trust response to the AMI alert was characterised now as integrated across corporate, divisional and specialty levels. The CEO highlighted how mortality investigations now feed into organisational changes. At divisional level investigation involves case note review, review of coding, patient pathway analysis and recommendations for improvements in care or coding. This information is all forwarded to the mortality committee for discussion, decision and action.

A universal and timely approach

An internal target is set for a 14-day post-death review period. Importantly, clinicians involved in a mortality alert are excluded from the case note review process. This is managed directly by the clinical governance unit.

A forensic approach: reviewing case notes and acute myocardial infarction pathways

It was reported that staff employed a forensic approach in responding to the alert and CQC inspection. This comprised investigating, analysing problems and triangulating data. Attention focused on the trust’s critical care model and AMI pathway and producing an action plan. A clinician on the mortality committee worked with another clinician to investigate and question all documentation that related to the AMI alert. This includes admitting condition, comorbidities and interventions.

Improved clinical engagement in responding to acute myocardial infarction alert

In responding to the alert respondents reported increased involvement of clinicians in mortality review with the mortality committee and divisional mortality teams. The medical director highlighted how it was important for clinician to volunteer in participating in these teams.

Clinicians as boundary spanners

Data analysis identified clinicians operating as boundary spanners, bridging knowledge areas and cognitive frames. A consultant cardiologist working with coders in case note review was able to ‘bring the coders into the reality of medicine’ and helped the clinician make sense of how they worked.

Changes in patient escalation processes

Changes were made in the early warning scoring system for escalating deteriorating patients.

Changes in patient pathways

Work was also done to improve the use of the ‘do not attempt cardio-pulmonary resuscitation’ policy. This involved introducing ‘ceilings of care’ being written into the patient case notes, which was reported as an outcome of the alert investigation.

Engagement with primary care

Difficulties were also identified regarding AMI originating in primary care. Hence, a key outcome of the alert was to include GP engagement in the mortality review process. The mortality pro forma was also changed to allow clinicians to identify whether the AMI had occurred outside the hospital.

Investment in infrastructure

A reported outcome of the alert was investment in trust infrastructure to redesign critical care and acute coronary care, with a ‘pacing room’ for treatment of patients with confirmed AMI. Respondents reported how mortality review arising from the alert had identified that AMI mortality was high in wards outside critical care and cardiology. Thus, changes were made in the AMI pathways.

Coding changes

Coding completely changed since the alert, with clinicians now much more involved. All cardiac-related deaths in the hospital were seen by the lead consultant cardiologist to review and validate the coding.

Training of staff

In addressing errors in case note write-up, incorrect diagnosis and early intervention for AMI treatment, trust staff reported much investment in training and wider quality improvement.

Leadership priority for culture change focused on quality improvement and patient safety

Trust leaders described how chasing the alerts was not productive. Instead, a broader priority to improve culture by focusing on quality and patient safety was more effective.

Changes in trust culture influencing mortality

Cultural change appeared to have stimulated how the trust approached mortality. Trust priorities were now driven by a cultural focus on patient safety and quality improvement. Frontline clinicians reported the culture as ‘more open’ and encouraging of staff engagement as ‘people feel that they can speak up more’.

Engagement with external organisations in benchmarking care and accessing knowledge and practices to improve mortality

Respondents identified how engagement with national quality improvement mortality programmes, national clinical audits and other trusts had supported changes both in mortality investigations of alerts and improved care. Engagement with other trusts was also reported as influencing changes in the new critical care model. The culture was described as now being more open and welcoming of new ideas from outside the trust.

Changes in structures and processes linked to improved connectivity between learning and improvement

The head of clinical audit described how structures and processes were very different since the Keogh review. They described as clinical governance integrating and providing reporting connectivity between learning from patient safety and complaints processes to ‘feeding back’ to frontline staff: ‘Nothing is the same . . . It’s all completely different’.

Investment in organisational capabilities (IT infrastructure: electronic incident reporting, mortality pro forma and electronic patient record and MEDICUS)

It was suggested that investment in IT infrastructure had improved data quality, enabling trust feedback and leading to improved knowledge for decision-making. The trust had recently implemented a new electronic incident reporting system, which had dramatically improved the reporting of incidents. The medical director referred to the use of an electronic database containing an audit of patient mortality from AMI.

Universal and timely review of all deaths

Respondents confirmed that a retrospective review of all deaths in the previous 2 years, including all complaints, incidents and inquests, was instituted by the senior leadership team. Care delivery problems were identified and the required actions were linked to an organisational development programme.

Catchment characteristics limit improvement in HSMR

The board at the time of the alerts in 2011 had wanted the HSMR down to 90 from 115 but they considered that the trust catchment described as ‘very deprived’ limited the progress they could make in improving HSMR.

Changes in leadership and financial difficulties create disruption in mortality review

The trust has had a difficult time for many years with several changes in senior management. None of the interviewees was in post in 2008 when the first alert arrived; however, some were able to comment on changes since. Respondents identified that the former medical directors in post before the first alert were not very involved in clinical governance. The clinical outcomes manager reported that the board members had changed many times and as a result they had limited involvement in trust affairs. Changes in ‘upper management’ were perceived by interviewees as ‘a little bit of a hindrance’ and a ‘break in the continuity of the mortality review group’. Financial problems resulted in a ‘culling’ governance and coding staff 4 years previously.

Discontinuity in tools used to support mortality review

When the first alerts arrived in 2008 the trust was using the Dr Foster tools, but the contract was ended in around 2011 and there was a gap before they went with CHKS in 2012. This has changed again and now the trust uses the HED tool.

Cardiology/acute myocardial infarction services

Before 2014, cardiology was split between the two sites, meaning that the catheter laboratory was situated at hospital A and acute services were located at hospital B. This was described as resulting in a ‘split-site mentality’ that prevented an integrated approach to cardiology.

For around 5 years there have been arrangements for patients with AMI to be taken directly to a central myocardial infarction centre rather than to local hospitals. This would suggest that the admission diagnosis of AMI at this trust was not clear-cut when such patients were collected by the ambulance service.

The cardiologists reported in to the national clinical audit (MINAP) and received reports back. These were the main triggers for action.

Lack of trust-wide involvement in trust response

At the time of the first alerts, these would arrive to the CEO and be given to the director of clinical governance, who would ask the clinical outcomes manager to get the notes and select a clinician to ask to review these and write a summary. The results would then go to the quality and safety committee, which reported to the board. It was unclear how long the mortality review committee had been in place, but it was described as having no clear focus or strategy. The head of coding identified how the mortality review group had stopped a couple of times owing to senior management changes.

It was suggested that changes in leadership had led to a lack of strategic direction in addressing mortality. This failure is demonstrated by changes and time lapses in purchasing new mortality analysis IT tools.

Universal case note review

When the AMI alert arrived in 2012 all cases were reviewed and major issues were identified with the coding of AMI. In undertaking this rigorous review of case notes it emerged that only around one-third had been coded correctly for AMI. It was perceived that coding of case notes was the problem, rather than quality of care for patients with AMI.

Poor structures and processes for mortality review limit long-term learning

There appeared to be no long-term learning from the target alerts. The medical director reported that when he was appointed < 1 year previously, the mortality review systems were limited. He explained that he did not receive the mortality alerts, that there was confusion around the process and that he did not have confidence that the institutional response was adequate. The way the data were presented was also said to be very poor, lacking ‘rhythm’. A divisional director also conceded that although there were mortality and morbidity meetings at divisional level, few were robust and the actions generated as a result of them were limited.

The medical director articulated that the plan now is to get the data down to divisional level and then to service level and to have ownership of the data by the clinicians. Work is under way to increase use of the HED tool and provide training in the use of it.

Newly recruited of head of coding motivates trust-wide clinical ownership and improvement in coding

When appointed to the role in 2011, the head of coding started by speaking to every clinical director and clinical lead to ask for help with the data and to explain the issues relating to their recording in the notes and the national data. They set out to motivate clinicians by stressing that that they should take ownership of their data (through case note write-up).

It was noted that clinical leads, at first, were very defensive with regard to their coding, as they felt that their coding was correct. However, the new head of coding was able to demonstrate that they were making serious errors and developed a ‘double-coding policy’. The clinicians were described as being somewhat defensive at first, with some being very aggressive. From this work, new systems were developed, including a validation programme whereby each year every specialty randomly selected 50 sets of notes and went through them with the coder to pick out issues and develop learning points to disseminate within their clinical department.

Care Quality Commission association with the alert triggering greater urgency

Having the CQC label at the top of the alert letter was said to make clinicians take notice and do the required work without challenge. The trust had a CQC review in 2013 and was put into special measures. The review looked at their governance arrangement, A&E targets and finance (large overspend) but did not specifically address AMI. The CQC visited again subsequently and reported improvement but the trust remained in special measures.

Overview

This is a large acute trust with seven hospitals sites formed from a merger 5 years ago and serving a population of around 220,000. The trust recorded a deficit of approximately £18M in 2013/14 and was expecting to end 2014/15 with a £3M deficit. The trust received three alerts for AMI mortality between 2011 and 2013 (Table 23).

Relative risk profile

Figure 39 displays the relative risk profile for AMI-m-03 between January 2008 and January 2014. The green lines represent the date of an alert and the blue lines the date of the subsequent letter.

FIGURE 39.

Relative risk profile and alert (green)/letter (blue) dates for AMI-m.04.

AMI-m.04 thematic analysis

Analysis is generated from interviews with two senior executive leaders, three consultant leads, one head of coding, one information lead and one deputy director of governance.

Overview

Sep-s.05 is an acute NHS trust serving a population of approximately 260,000. It has one site, Sep-s.05 general hospital. The trust has approximately 4000 employees and 100 consultants. It had severe financial problems from 2013 onwards and is still at a high risk with regard to finances. In 2013, Monitor began regulation after identifying some governance issues. The trust was seen to be in breach of its provider licence around three issues relating to its electronic patient records, the effectiveness of the board of directors, and the trust’s financial planning. In 2015 a CQC inspection said the trust ‘requires improvement’ (Table 24).