Design and evaluation of an interactive quality dashboard for national clinical audit data: a realist evaluation

National clinical audits

In the UK, the Healthcare Quality Improvement Partnership (HQIP) centrally develops and manages a programme of NCAs each year through the National Clinical Audit and Patient Outcomes Programme (NCAPOP). The number of NCAs included within the NCAPOP varies year to year. At the time of writing (November 2020), there were 61 NCAs included in the NCAPOP. The NHS standard contract requires all health-care organisations that provide NHS services to participate in the NCAPOP NCAs and gives commissioners the power to impose penalties on NHS trusts that fail to participate. 1 Since 2012, trusts have been responsible for funding a proportion of NCAPOP costs through subscription funding, whereby trusts pay an annual amount to HQIP (and this amount has remained at £10,000 since 2016). In addition, there are 82 independent NCAs, not part of the NCAPOP, funded either through subscription, by a charity or professional body or by NHS England (London, UK). Each year, the NHS England Quality Accounts list advises trusts which NCAs they should prioritise for participation.

Each NCA focuses on a particular clinical area or condition and aims to produce a publicly available report annually. The objective of NCAs is to systematically measure the quality of care delivered by clinical teams and health-care organisations and to stimulate quality improvement (QI). 2 Although there is evidence of positive impacts of NCAs,3–5 and previous research suggests that clinicians consider NCA annual reports important for identifying QI opportunities,6 there is variation within and between trusts in the extent to which they engage with NCA data. 6,7 A number of NCAs provide trusts with online access to more recent data and allow users to download data for local analysis. For example, the National Hip Fracture Database provides live data on how hospitals are meeting key performance indicators, plus interactive online charts, dashboards and benchmarking summaries. However, with a recognised shortage of data analysis skills within the NHS,8,9 the use of NCA data poses challenges for some trusts. 7 Consequently, NCA data are substantially underutilised and the potential for NCA data to inform QI is not being realised fully.

Dashboards

Dashboards use data visualisation techniques to provide information to individuals, services or organisations in either a paper or a computer-based format. 10 Use of such techniques is thought to improve data comprehension11 and reduce cognitive load,12 resulting in more efficient and effective decision-making. 13 A distinction can be made between clinical dashboards and quality dashboards. Clinical dashboards may provide data at the level of the patient, the clinician (showing all patients they are caring for and comparing them with their peers and national benchmarks) or allow the user to move between viewing information at both of these levels,14 with the aim of informing decisions about, and thereby improving, patient care. 15 By contrast, quality dashboards show performance at the ward or organisational level (and ideally will provide feedback to be used at both levels16,17) to inform operational decision-making and QI efforts. 18 Visualisations provided by quality dashboards can inform QI by supporting the identification of previously unnoticed patterns in the data. 19

Health-care providers, in the UK and internationally, are increasingly using dashboards to measure care quality and as the basis for QI initiatives. Lord Darzi’s next-stage review20 and the health informatics review,21 which were both published in 2008, recommended greater use of dashboards by NHS organisations. Two distinct strands of thinking permeated these reviews and subsequent policies and guidance on dashboards. One strand of thinking emphasised the need for summary real-time information for use by clinicians in their clinical work. The former NHS Connecting for Health agency encouraged these developments through its clinical dashboard project. 22–24 This was followed by an emphasis on the need for integrated real-time information on care quality, which could be interrogated by clinical teams for the purpose of learning. 25 The other strand of thinking focused on the need for trust boards and regulators to have summary performance information. NHS foundation trusts already published performance dashboards, required by Monitor, but otherwise trusts did not use dashboards for their own internal reporting, or to report to the Care Quality Commission (CQC) or (as they were then) primary care trusts. Developments in this area were prompted by two major reports in 2013, that is, the second Francis report on substandard care provided at Mid Staffordshire NHS Foundation Trust26 and the post-Francis Keogh review of 14 NHS trusts with persistently high mortality rates. 8

An empirical study of quality dashboard use in acute trusts in England was published in 2018. 16 A survey of 15 trusts found that most trusts had some form of quality dashboard in place for reporting measures, such as those included in the NHS Safety Thermometer. 27 However, there was significant variation in dashboard sophistication, the term being used to refer both to information technology (IT) systems and to the outputs from these IT systems, typically in the form of printed reports. The majority of dashboards still depended on resource-intensive manual collation of information from a number of systems by central performance management teams, with retrospective reports then being circulated to wards, directorates and trust boards. However, the study also revealed a clear ‘direction of travel’,16 with a desire for real-time quality dashboards.

Despite the enthusiasm for quality dashboards, most empirical literature has focused on clinical dashboards, with the consequence that little is known about how and in what contexts computer-based quality dashboards drive improvement. A review of studies of the use of computer-based quality and clinical dashboards published between 1996 and 2012 included 11 studies that evaluated their impact on care quality. 10 Only one dashboard met the definition of a quality dashboard,28 with the other 10 dashboards being clinical dashboards. Chapter 3 provides an update to this review, focusing on quality dashboards.

Phase 4: multisite case study

In March 2020, installation of QualDash v2.1 was in progress, with the intention that observations would resume once it had been installed in all sites. However, at this time, all on-site research in hospitals was suspended in response to the COVID-19 pandemic and, therefore, installation of QualDash v2.1 and observations were discontinued. This meant that 148.5 hours of observations were undertaken, rather than the planned 384 hours (see Sampling for further details). It also meant that we were unable to undertake teacher–learner cycle interviews with staff (see Data collection). We notified the National Institute for Health and Care Research (NIHR) of these challenges in our progress report submitted in March 2020.

Phase 4: interrupted time series study

We had intended to consider a range of measures (see Analysis). However, by January 2020, limited QualDash use within our study sites meant that QualDash use could not be hypothesised to have had an impact on these measures. Therefore, we agreed with our Study Steering Committee (SSC) in January 2020 that, if use did not increase, it was not appropriate to use those measures, instead focusing on impact on data quality, for which there was some qualitative evidence. We had planned to undertake further adoption activities following installation of QualDash v2.1, but we were unable to do so because on-site research in hospitals was suspended. Consequently, use did not increase and analysis considered measures of data quality. We notified NIHR of this change in our progress report submitted in March 2020. An additional change was that we had intended to undertake a controlled interrupted time series (ITS) study. By the time that data were to be received for analysis, only three sites had issued research and development (R&D) approval, with delays as a result of prioritising COVID-19 studies and restarting studies that had stopped because of the COVID-19 pandemic. Of the three control sites that had issued R&D approval, only one submitted data, with the failure of the other sites to do so presumably because of service pressures at that time. Given that data for only one control site were received and the matching was not relevant to the measures used in the final analysis, control data were not used, with time providing the control instead (see Sampling). Service pressures meant that site D did not submit data and, therefore, analysis was limited to sites A, B, C and E. We notified NIHR of potential challenges of obtaining data during the COVID-19 pandemic in our progress report submitted in March 2020.

Phase 5: assessment of wider applicability

Part of our plan for phase 5 of the study had been to assess the extent to which QualDash is suitable for different NCAs and the extent to which QualDash meets the needs of commissioners, through three focus groups. However, given the international focus on COVID-19 at the time, and a request from a region-wide Gold Command, we sought, instead, to understand the requirements of such a body (which included commissioners) and how QualDash could be revised to support their work, using interviews for this purpose (see Assessing the wider applicability of QualDash). At this time, we also received a request from one of our sites for us to adapt QualDash to monitor cardiology activity during COVID-19, which we did (see Chapter 9, Adapting QualDash to support daily monitoring). We notified NIHR of this additional work in our progress report submitted in March 2020, which was positively received.

Interview study

Workshop with national clinical audit suppliers

Co-design workshops

Observation of meetings

The phase 1 interviews identified a range of meetings, at different levels of trusts, at which quality and safety are discussed and QualDash could potentially be used. The research team considered, and the SSC agreed, that it would be useful to observe these meetings to provide further insight into how data were used to inform the design of QualDash. Therefore, meeting observations were added to phase 2.

Think-aloud technique

Heuristic evaluation

Sample

One focus group was held at each site and sought to involve intended QualDash users. Potential participants were provided with an information sheet about the focus group following their participation in either the phase 1 interviews or, for those at site A, following their participation in the co-design workshops. Potential participants were contacted by e-mail to see if they were willing to participate and, if so, to identify a suitable date. In addition, where contacts were identified, IT staff were invited to attend. Details of participants in each focus group are provided in Table 4. Focus groups were conducted between February and April 2019.

Data collection

Each focus group was facilitated by two researchers, who presented a study summary and provided a demonstration of QualDash. Post demonstration, the adoption strategies and associated activities that were identified and prioritised in phase 1 were presented to the group. Participants were encouraged to discuss why they thought a particular activity would work (or not), for which groups of staff within their trust and how the activity should be delivered. Each focus group was audio-recorded and researchers also made notes during the discussion. Each session was scheduled to last for 90 minutes and the discussion lasted between 41 minutes and 1 hour 9 minutes, with an average (mean) duration of 50 minutes.

Analysis

Audio-recordings were transcribed verbatim, checked for accuracy, anonymised and uploaded into NVivo for indexing. Codes used for indexing transcripts were devised by three researchers (NA, LM and RR), with the intention of categorising data related to assumptions about adoption, dashboard functionality and current/future NCA data use. Close reading of transcripts, indexing and a review of researcher notes were used to identify consistencies and variations across sites regarding participants’ theories about activities that would support adoption. For each site, the researchers summarised discussion of each strategy, including how it should be delivered and why participants felt that it might work to support QualDash uptake and use. These documents became a blueprint for adoption within each site.

Focus group analysis was also used to further inform the QualDash programme theory. The programme theory developed in phase 2 hypothesised potential QualDash impacts. In their original form, these hypotheses assumed that QualDash would be adopted successfully. The adoption focus groups analysis enabled theorisation around how and why use and uptake would be successful, which was subsequently added to the QualDash programme theory.

Multisite case study

Interrupted time series study

Trial feasibility and design

Phase 5 assessed the feasibility of a cluster RCT of QualDash, using the following three progression criteria: (1) QualDash used by ≥ 50% of intended users, (2) NCA data completeness improves or remains the same and (3) participants perceive QualDash to be useful and intend to continue using it after the study. Criteria 1 and 3 are concerned with the acceptability and uptake of the intervention and, therefore, have implications for recruitment to a trial. The second criterion is concerned with ensuring that the intervention does not have unintended negative consequences, which would affect both the success of the intervention (as QualDash will be less useful if data completeness is reduced) and the feasibility of outcome assessment. The third criterion is also concerned with participants’ perceptions of impact of QualDash on care.

As these progression criteria were met (although there were limitations in the data used for assessing whether or not the criteria were met), we drew on our findings from the multisite case study to determine which NCAs should be included in a trial. Having identified PICANet as a suitable NCA, we worked with the Leeds Institute of Clinical Trials Research to design an appropriate trial.

Assessing the wider applicability of QualDash

Part of our plan for phase 5 of the study had been to assess the extent to which QualDash is suitable for different NCAs and the extent to which QualDash meets the needs of commissioners, through three focus groups. However, given the international focus on COVID-19 at the time and a request from a region-wide Gold Command, we sought, instead, to understand the requirements for such a body (which included commissioners) and how QualDash could be revised to support their work, using interviews for this purpose.

Study characteristics

The 27 studies of quality dashboards were from Australia (n = 185), Canada (n = 386–88), Italy (n = 189), Malta (n = 190), the Netherlands (n = 391–93), Nigeria (n = 194), the UK (n = 427,28,95,96) and the USA (n = 1397–109). The dashboards were used in the hospital setting, including maternity and outpatient settings (n = 2227,85–91,93,95,97–108), mental health care (n = 128), primary care (n = 394,96,109) and care homes (n = 192). The dashboards incorporated a wide range of metrics, covering topics such as medication (n = 695,97–101), nutritional care (n = 192), care quality for specific conditions (n = 589,91,104,107,108) or at specific points in the patient pathway (n = 1105), organisation-wide care quality (n = 327,85,109), organisational performance (e.g. admission and discharge rates) (n = 285,90) and documentation and data quality (n = 685,96,102,103,105,106), with some dashboards covering more than one of these areas. A variety of study designs were used, including a cluster RCT (n = 1105), a controlled before-and-after study (n = 1101), before-and-after studies without controls (n = 885,89,94,99,100,102–104), ITS analyses (n = 286,97), qualitative studies with data collected through interviews and/or focus groups (n = 627,87,88,90,91,106) and surveys (n = 592,96,107–109).

Effect on outcomes and care processes

Thirteen studies85,86,94,95,97–105 provided quantitative data on the impact of quality dashboards, nine85,94,95,97–99,102–104 of which showed positive impacts. The other four studies86,100,101,105 showed a mixed impact, achieving improvement in some measures but not in others. Six95,97–101 of the 13 studies were concerned with medication, four95,97–99 of which showed positive impacts. Coleman et al. 95 evaluated a dashboard that showed ward performance levels of overdue medication doses that were available to view by clinical and managerial staff. Alongside introduction of the dashboard, weekly e-mails based on directorate-level information were sent to divisional directors and managers, with escalation to executive level if unacceptable thresholds were reached. Missed antibiotic doses decreased significantly [0.60 percentage points, 95% confidence interval (CI) 0.26 to 0.95 percentage points; p = 0.001], as did missed non-antibiotic doses (0.41 percentage points, 95% CI 0.11 to 0.70 percentage points; p = 0.007). 95

Lin et al. 97 evaluated an opioid safety initiative dashboard across 141 veterans’ health administration facilities. Although there was already a decreasing trend in high-dosage opioid prescribing, the dashboard was associated with an additional decrease of 331 (95% CI –378 to –284) patients per month receiving opioids of > 100 morphine equivalents, a decrease of 164 (95% CI –186 to –142) patients per month receiving opioids of > 200 morphine equivalents and a decrease of 781 (95% CI –969 to –593) patients per month receiving concurrent benzodiazepines. A champion at each facility was responsible for reviewing the dashboard, identifying prescribing variation and taking action to promote patient safety.

Michtalik et al. 98 evaluated a combined clinical and quality dashboard that displayed venous thromboembolism prophylaxis rates at individual and group levels. Monthly venous thromboembolism prophylaxis compliance rates were 86% (95% CI 85% to 88%) at baseline and 90% (95% CI 88% to 93%) during the dashboard period, representing a significant improvement in compliance (p = 0.01). At monthly meetings, the dashboard, group results and trends were discussed.

Simpao et al. 99 report on the development and evaluation of a dashboard displaying drug–drug interaction alerts and over-ride rates within their hospital electronic health records (EHRs). Dashboard use enabled the hospital’s Clinical Decision Support Committee, made up of clinicians and pharmacy representatives, to identify a series of alerts for deactivation. For pharmacists, the median alert rate prior to first deactivation was 58.74 [interquartile range (IQR) 54.98–60.48] alerts per 100 orders and 25.11 (IQR 23.45–26.57) alerts per 100 orders following the deactivations (p < 0.001). For clinicians, the baseline median alert rate was 19.73 (IQR 18.66–20.24) alerts per 100 orders and 15.11 (IQR 14.44–15.49) alerts per 100 orders following the deactivations (p < 0.001). Pharmacists’ over-ride rates for drug–drug interaction alerts not modified in the system decreased from a median of 93.06 (IQR 91.96–94.33) over-rides per 100 alerts to 85.68 (IQR 84.29–87.15) over-rides per 100 alerts (p < 0.001). The medication serious safety rate decreased during the study period, and no serious safety events were reported in association with deactivated alert rules.

Two studies100,101 concerned with medication showed mixed effects. Trinh et al. 100 evaluated a dashboard used by pharmacy managers and administrators that displayed adherence rates to antimicrobial and enoxaparin guidelines. Individualised performance feedback was provided to the pharmacists based on trends identified by the dashboard. The overall adherence rate increased significantly post intervention (90.1% vs. 91.9%; p = 0.040). The rate of documentation for non-adherent orders increased significantly post intervention (57.1% vs. 68.5%; p = 0.019). The percentage of pharmacists with at least 90% overall adherence increased post intervention, but was not statistically significant (52.9% vs. 70.6%; p = 0.103) and time to verification was similar before and after the study intervention.

Graber et al. 101 evaluated an interactive web-based antimicrobial dashboard across US Veterans Affairs medical centres. Intervention sites demonstrated a 2.1% decrease (95% CI –5.7% to 1.6%) in total antimicrobial use compared with a 2.5% increase (95% CI 0.8% to 4.1%) at non-intervention sites (absolute difference 4.6%; p = 0.025). Anti-pseudomonal antimicrobial use decreased by 3.4% (95% CI –8.2% to 1.7%) at intervention sites compared with a 3.6% increase (95% CI 0.8% to 6.5%) at non-intervention sites (absolute difference 7.0%; p = 0.018). Antimethicillin-resistant Staphylococcus aureus antimicrobial use decreased by 11.3% (95% CI –16.0% to 6.3%) at intervention sites compared with a 6.6% decrease (95% CI –9.1% to –3.9%) at non-intervention sites, but this was not statistically significant (absolute difference 4.7%; p = 0.092). Dashboard use was supported by a range of strategies, including a kick-off lecture to medical staff to promote antimicrobial stewardship, monthly learning collaborative calls with antimicrobial stewards and sharing of ‘lessons learned’ among stewards regarding effective use of information gleaned from the dashboard.

Five studies85,94,102,103,105 looked at the impact of quality dashboards on documentation and all found a positive impact. Clark et al. 85 evaluated a dashboard that provided real-time display of clinical, performance and patient-flow parameters at organisation and ward level. Although statistical significance was not reported in this before-and-after study, there was an increase in percentage of patients with estimated date of discharge recorded within 12 hours of admission (82% vs. 90%), a management plan documented and communicated to the patient within 24 hours of admission (78% vs. 100%), criteria for discharge parameters documented and developed by multidisciplinary team within 24 hours of admission (76% vs. 92%), a discharge plan communicated to patient 24 hours before discharge (48% vs. 86%) and a pharmacy script written 24 hours before discharge (62% vs. 84%).

Etamesor et al. 94 evaluated a dashboard displaying key routine immunisation indicators by local government area or health facility used by primary health workers in Nigeria. In this before-and-after study, report completeness increased (53% vs. 81%; statistical significance was not reported). The authors94 also state that report timeliness increased, although details are not provided.

Hagaman et al. 102 evaluated a dashboard showing compliance with a preoperative documentation standard. In this before-and-after study, there were two intervention periods. In phase 1, preoperative evaluation clinic (PEC) staff were given training on standard and monthly feedback via e-mail. In phase 2, the dashboard was introduced. PEC managers were given dashboard access. In addition, the metric was included in quarterly performance evaluations for PEC clinicians. Baseline adherence by PEC clinicians was 0%, increasing to 44.7% in phase 1 and to 91.5% in phase 2 (p < 0.0001). Partial adherence by PEC clinicians increased from 0.1% at baseline to 64.8% in phase 1 and to 98.2% in phase 2 (p < 0.001). Adherence by non-PEC clinicians who did not have access to the dashboard increased from 0.20% at baseline to 2.4% in phase 1 and to 12.8% in phase 2 (p < 0.0001). Partial adherence by non-PEC clinicians increased from 5.8% at baseline to 17.6% in phase 1 and to 36.0% in phase 2 (p < 0.001).

Patel et al. 105 evaluated the impact of a team-based dashboard and weekly in-person review of team data on medication reconciliation, a high-quality after-visit summary, a timely discharge summary and a composite discharge metric (i.e. a Discharge Mix Index). Completion of the Discharge Mix Index was achieved for 79.3% of patients in intervention group compared with 63.2% of patients in the control group (p < 0.0001). However, in the washout period, when in-person reviews were removed, there was no significant difference in performance between the two groups.

Dolan et al. 103 evaluated the impact of an anaesthesia opioid documentation discrepancy dashboard. In this before-and-after study, the dashboard was the last intervention in a series of QI interventions introduced. Prior to dashboard introduction, the rate of controlled substance documentation discrepancies was 1.14%. Following dashboard introduction, the rate reduced to 0.87% (i.e. a reduction of 24%; statistical significance was not reported).

One study104 provided quantitative data on the impact of condition-specific quality dashboards. Banerjee et al. 104 evaluated a heart failure dashboard that displayed aggregate data for all patients with a primary diagnosis of heart failure on discharge to support an ongoing QI initiative. Implementation of the QI initiative led to a reduction in 30-day index hospital all-cause heart failure admission rates from 18.2% at baseline to 14%. This was further reduced to 10.1% after dashboard introduction (p-value for trend = 0.0001).

Finally, one study86 provided quantitative data on the impact of a quality dashboard in maternity care. Weiss et al. 86 evaluated the Ontario maternal newborn dashboard in an ITS study. The results were mixed. There were statistically significant absolute decreases in rates of episiotomy [a decrease of 1.5 (95% CI 0.64 to 2.39) per 100 women], induction for postdates in women who were < 41 weeks at delivery [a decrease of 11.7 (95% CI 7.4 to 16.0) per 100 women] and repeat caesarean delivery in low-risk women performed before 39 weeks [a decrease of 10.4 (95% CI 9.3 to 11.5) per 100 women]. There was also an absolute increase in rate of appropriately timed group B streptococcus screening [an increase of 2.8 (95% CI 2.2 to 3.5) per 100 women]. However, rates of unsatisfactory newborn screening blood samples or formula supplementation at discharge were not significantly affected. There was also varying success between sites in achieving practice change.

Theories of audit and feedback

Quality dashboards are a form of audit and feedback. With origins in psychology, audit and feedback theories tend to focus on how audit and feedback interventions are intended to change the reasoning and behaviour of individuals and, therefore, operate at the individual or ‘micro’ level. Audit and feedback theories are premised on the idea that clinicians have an intrinsic motivation to improve and that, if presented with evidence that there is a discrepancy between their own performance and that of a target or standard, they will make efforts to improve their performance. Therefore, these theories focus on how characteristics of the audit and feedback intervention itself might trigger an individual’s intrinsic desire to improve. For example, contextual feedback intervention theory (CFIT) suggests that the standard must be perceived as desirable and achievable, and feedback about the discrepancy between their performance and the standard must be perceived as accurate. 110 CFIT contends that such feedback is more likely to change behaviour if it is timely, frequent, cognitively simple (e.g. presented graphically), unambiguous and provides concrete suggestions of how to improve performance. Hysong et al. ’s111 model of actionable feedback suggests that clinicians are more likely to respond to audit data if the data are perceived to be timely and non-punitive and if clinicians receive feedback about their own individual performance rather than aggregated data about overall performance of the organisation. If clinicians are able to customise how they view the data, then this leads to active engagement in sense-making, further increasing the likelihood that the data will be acted on. Clinical performance feedback intervention theory (CP-FIT) is a more recent addition to theories of audit and feedback, and goes beyond previous theories by drawing on both psychological and sociological theories to consider both individual characteristics and organisation or team characteristics that influence response to feedback. 112 Organisation or team characteristics considered to influence response to feedback are resource, competing priorities, leadership support, champions, teamwork, intraorganisational networks and extraorganisational networks. The three propositions of CP-FIT are (1) health-care professionals and organisations have a finite capacity to engage with and respond to feedback and, therefore, interventions that require less work, supply additive resource or are considered worthwhile enough to justify investment are more effective; (2) health-care professionals and organisations have strong beliefs regarding how patient care should be provided, which influence their interactions with feedback, such that those that align with and enhance those aspects are most effective; and (3) feedback interventions that successfully and directly support clinical behaviours for individual behaviours are most effective.

Although the audit and feedback literature is largely concerned with feedback to individual clinicians,113 the majority of NCAs provide feedback at the clinical team or provider level. Studies of national audits suggest that audit and feedback will have an impact only if (1) clinical teams are given adequate time to engage with and use the data7 and (2) clinical teams perceive that they have the power to make changes,114 emphasising the importance of contextual factors at the organisational or ‘meso’ level. At this level, theories of benchmarking can provide useful insight. Benchmarking refers to the identification, sharing and implementation of practices at team or organisational level that lead to excellent performance. 115 Van Helden and Tilemma’s116 model of benchmarking hypothesises that an organisation’s response to benchmarking is determined by different types of institutional pressures exerted on them by government, professional groups, interest groups and the general public. An organisation’s response pattern, which can vary from passive compliance to proactive manipulation of pressures, depends on its willingness and ability to conform to institutional pressures, which, in turn, is shaped by which stakeholders exert pressure and how, stakeholders’ reasons for exerting pressure, what the pressures are and the context in which the pressure is exerted.

Van Helden and Tilemma’s116 model of benchmarking draws on institutional theory and resource dependency theory, both of which emphasise the importance of contextual factors at the ‘macro’ level. Institutional theory focuses on problems associated with traditional top-down regulation and emergence of regulation through networks of actors. An underlying thread in institutional theory is the concept of legitimacy, with the institutional environment creating not only legal structures but also social and mental structures that actors must work within if they are to be accepted as legitimate actors. 117 At this level, we can also consider theories of ‘new professionalism’ that draw on Foucauldian concepts of disciplinary power and governmentality. 118,119 Governmentality, which refers to forms of self-surveillance that ensure that performance meets expectations, negating the need for managerial surveillance,74 has been found to be relevant to understanding QI in the NHS. For example, Martin et al. 119 revealed the importance of clinical leaders in championing quality and safety, and regular meetings that bring clinicians together to discuss quality and safety constructively. This theoretical lens would suggest that professional community is important to the success of NCAs, with social obligations among members of the community offering opportunities for critical reflection and improvement. 34,119

Theories of technology adoption

Successful introduction of technology involves interactions between individual clinicians and their work environment until the technology becomes embedded (i.e. routinely incorporated into everyday work) and integrated (i.e. sustained over time) into routine practice, which is a process known as ‘normalisation’. 120 A number of theories seek to explicate the contextual factors that influence this process. At the micro level, it is necessary to consider the relationship between the technology and the work practice. 121 For example, Fit between Individuals, Task and Technology is based on the idea that adoption of HIT depends on fit between user attributes (e.g. computer anxiety, motivation), technology attributes (e.g. usability, functionality, performance) and work practice attributes (e.g. organisation, task complexity). The TAM, which is widely used in HIT research, draws particular attention to perceived usefulness and perceived ease of use as contextual factors that influence a user’s attitude towards a technology and, consequently, their intention to use it. 77 In considering user attributes, attention should be given to the user’s ability to interpret different forms of data visualisation, with studies suggesting an individual’s ability to understand information presented in dashboards is often determined by their level of graph literacy. 122,123 For example, Gaissmaier et al. 122 found that individuals with low graph literacy understood numerical information better if it was presented as numbers, whereas individuals with high graph literacy had better comprehension when the same information was presented in a graphical format. 122

At the organisation (meso) level, it is necessary to consider the process through which the technology was developed and introduced. For example, normalisation process theory (NPT) suggests that, for successful integration to occur, the following four key constructs need to be considered: (1) coherence (i.e. sense-making, where individuals make sense of the new technology and how it differs from existing practice); (2) cognitive participation (i.e. the process of engaging individuals with the introduction of the technology); (3) collective action (i.e. how the work processes are adapted and altered to make the intervention happen); and (4) reflexive monitoring (i.e. the formal and informal appraisal of the benefits and costs of the intervention). 120,124,125 This suggests that it is more likely to become embedded into practice if clinical teams have been able to ‘make sense’ of a quality dashboard, have been engaged in the adoption process, have been able to adapt their work processes and are able to identify potential benefits to its introduction.

At the policy (macro) level, we can, again, turn to the class of theories discussed above that focus on governance arrangements. A quality dashboard could potentially increase access to NCA data, enabling greater scrutiny of clinicians’ practices by managers and of trust performance by commissioners. 118,119 If perceived this way, clinicians’ views regarding the appropriateness and legitimacy of this could serve to justify non-participation. 74 Therefore, power relations between clinical teams, quality subcommittees, trust boards and CCGs form an important macro context, shaping how quality dashboards are perceived and used and subsequent impacts. A second class of theories relevant at this level are historical institutional approaches. 43,45 Historical institutional approaches emphasise the extent to which current HITs are rarely self-contained, but are linked to one another in ‘e-infrastructures’. 44 Quality dashboards, and NCAs more generally, depend on NHS e-infrastructures that have evolved over time, such as the IT infrastructure that allows trusts to upload large data sets to regulators and other bodies, and the working practices that have evolved with them.

Collection of national clinical audit data

Interviews revealed variation between sites in the resources available for recording NCA data, particularly in terms of audit support staff and technology (Table 6). This variation had a number of impacts. In some sites, where NCA data were stored in local databases and staff had the necessary skills and time, reports and visualisations of the data were generated, increasing accessibility of the data for other staff members. This generation of reports and visualisations was reported in the cardiology departments at sites A, B and C and in the PICUs at sites A and C. However, this was not an option in sites that did not have these resources. Variation in resources also had an impact on timeliness of data collection. In some sites, data would be entered with only a day’s lag, whereas in one site it was reported that, although the site submits MINAP data regularly, these data relate to past patients and it can take them 6 months to obtain this information. Consequently the timeliness of data collection affected staff confidence of the data being accurate reflection of service performance. For example, at the PICU in site A, trust in the audit co-ordinator’s skills and experience meant that clinical colleagues and service managers were confident that their PICANet data were an accurate measure of service performance (more so than data stored in other trust systems).

Interviewees reported little change over time in the IT systems used for recording data, despite that some changes would have been desirable, such as the addition of fields as new fields were added to the NCA. Whether or not such changes were made depended on a mixture of whether or not the database was in house and, if in house, the extent to which the trust’s IT department supported that database and the extent to which the trust’s unit or service has the range of skills necessary to make the required changes themselves. For example, the database used by the PICU in site A was developed over 10 years ago by a junior doctor and was not maintained by the trust, nor did the PICU staff have the skills to make these changes. Consequently, new fields added to PICANet could not be recorded in this database. In contrast, in the cardiology department at site B, the information analyst had the skills to make revisions to the structure of the database and would do so when NICOR introduced a new field.

Staff changes were, in some cases, reported to affect the data collection process. For example, in site B, one clinician suggested that the quality of their PICANet data was poor, and attributed this to high turnover of audit support staff in the unit. Some sites also described a reduction in resources over time to support data collection. In site B, MINAP data collection and entry used to be carried out at assessment by nurses, but the nurses had been reassigned. Subsequently, data entry was carried out by a non-clinical information analyst and another team member, gathering data from patient notes. The non-clinical information analyst and the other team member were unable to submit data for all eligible patients (reporting having around 66% case ascertainment) and entered data for mandatory fields only, owing to lack of time. In site D, MINAP data collection used to be carried out contemporaneously by chest pain nurses at the bedside, using laptops connected wirelessly to the database; however, the nurses no longer carried out this role and, instead, data tended to be entered retrospectively from the notes, often after patients had been discharged.

The work involved in NCA data collection became more fully apparent during the phase 4 observations. Double data entry and use of paper data collection forms were common practices. For example, in the cardiology department at site A, each day the nurse specialist would print a ‘day sheet’ to identify which patients needed to be entered and then completed a MINAP data collection form for each patient, based on the latest MINAP data set. The nurse specialist gathered information from patients’ paper medical notes that they collected from the wards and the trust’s EHR to populate the form and then entered the data in the cardiology database. When the nurse specialist had completed data input for the month, they informed the trust’s IT department and it uploaded the data to NICOR, normally at the end of each month. Once uploaded, NICOR sent an error report to the trust IT department, which was passed to the nurse specialist for resolution. The nurse specialist checked and cross-referenced anomalies using patient notes, the local database and the trust’s EHR. It could take some time to resolve problems and some errors could not be resolved. For example, if ambulance staff had omitted to record the time at which patients arrived in the hospital, door-to-angiography times could not be calculated. Similarly, PICANet data collection in site A began in the PICU when patients were admitted to the ward. Junior doctors and nurses entered the data available to them on a paper form. The audit co-ordinator compared information in the forms with information in the admissions book and handover sheet, before entering the data on the database. On the ward, there was a large whiteboard on which staff recorded outstanding ‘jobs’ to be carried out for each patient. The audit co-ordinator wrote any queries about the forms on the board so that staff were aware of them and resolved them. The audit co-ordinator also obtained data and cross-referenced information against other trust systems, such as the hospital’s in-house EHR and patient administration system. This need to cross-check data across multiple systems was a feature of NCA data collection across sites. Further details of the work involved in NCA data collection are reported elsewhere. 141

Access and use of national clinical audit data by clinical teams

Interviewees at the clinical team level discussed three main channels through which they accessed NCA data: (1) the national report produced by the supplier, usually on an annual basis but in some cases less frequently; (2) data requests or online through the supplier’s website (a service offered by both MINAP and PICANet); and (3) local databases, if available (as described above).

Interviewees across all sites reported a designated NCA lead, typically a consultant, and received a copy of the national report. When discussing the value of national reports, data quality was raised frequently, with key issues being timeliness and validity. A key constraint in use of national reports for QI was that interviewees often perceived the data to be out of date and so could not be used to support an argument for change. For example, data in the September 2019 MINAP report were at least 18 months old on publication. Timely data, reflecting current performance, were considered essential if they were to inform QI.

Interviewees reported that, for clinical teams to engage with NCA data for QI, it was important that they trusted the quality of the data. The confidence that sites had in their own data depended on the resources available to them to support data collection, which varied, as described above. However, even when sites did trust their own data, they did not necessarily have confidence in the data submitted by other trusts nationally. Inconsistent or inaccurate coding was reported to have a negative impact on interviewees’ trust in MINAP data and on their ability to make meaningful comparisons with other trusts. Owing to inconsistent coding, one interviewee likened such assessments to ‘comparing apples and oranges’ (cardiologist, site E). A cardiologist at site D considered this a consequence of the expansion of data collected for MINAP, leading to it becoming ‘a tick box exercise of collecting data for the sake of collecting data with no effective feedback’ (cardiologist, site D).

Although more timely data could be accessed via requests to the supplier or the supplier website, in practice, interviewees reported challenges using these channels of access, particularly the time taken to receive data from the supplier. Equally, clinicians reported constraints on their time to log onto supplier websites to review metrics. However, some sites did make use of the supplier website. For example, the PICU at site A was accustomed to accessing PICANet data through the website because although they had their own local database, the website provided additional information. Specifically, the clinical lead reported monitoring the PICU’s SMR using the website, which provides a funnel plot visualisation.

Despite perceived limitations to accessing data from suppliers, the value of national comparator data was clearly recognised, something that clinical teams did not have when relying on data within their local databases. National comparisons were seen to offer opportunities to identify services performing well, which could act as examples of good practice from which clinical teams could learn. Making such comparisons was important to all sites, but perhaps especially within the PICUs, which, in contrast to cardiology services, did not have evidence-based standards against which to assess their performance and, therefore, felt the need to benchmark themselves against other units to assess their performance. (Although there are quality standards for PICUs, the standards relate to the organisation of care, covering areas such as staffing levels, facilities and equipment, the presence of guidelines and protocols, and the provision of information to families,142 in contrast to the specific patient-level and often time-sensitive targets set within cardiology.) National comparator data were also seen as important for those trusts wanting to attract patient referrals, potentially stimulating practice change if the service was not performing well compared with peers.

Sites that had local databases and audit support staff with the skills and time to generate visualisations of NCA data more frequently engaged with those data. Regular use of NCA data was reported in both the cardiology departments and the PICUs at sites A and C. For example, in the cardiology department at site A, the nurse specialist described receiving requests for reports on MINAP data from multiple sources, including both clinicians and managers. The nurse specialist reported that they ran monthly PCI clinical governance meetings, lasting around 3–4 hours, which were attended by cardiologists from the site and referring trusts. MINAP data were reviewed at these meetings, as well as deaths and complications for individual patients. These and other regularly requested ad hoc reports were produced by the IT department at the nurse specialist’s request, using MINAP data in the local database and generated via Structured Query Language (SQL) queries. There was little use of generic reports from the NICOR website, which the nurse specialist attributed to staff’s confidence in their own data. Examples of in-house ad hoc reports included activity levels within the catheterisation laboratories, numbers of procedures and door-to-angiography and call-to-balloon times. Where reports were wanted that could not be generated from the local database, the nurse specialist produced them themselves using Excel. For example, the nurse specialist generated a regular report on delays in treatment times and how many related to issues outside the service’s control.

In site C, the PCI project assistant used MINAP data to produce reports. Historically, these data were reviewed quarterly in clinical governance meetings, which were attended by other sites, and during which the different site data were compared. Although at the time of interviewing these meetings had been cancelled for some time because of organisational constraints, the project assistant continued to produce the reports for clinical staff to review separately (e.g. when requested as evidence to inform performance reviews). The acute chest pain nurses described how MINAP data had been used to support an improvement initiative, where they had noticed delays occurring in patients accessing the catheterisation laboratory within the target time, owing to the length of time that it took to receive an ECG beforehand. To address this, a new process was put in place to ensure that all patients presenting with chest pain were given an immediate ECG.

In the cardiology department at site B, a cardiologist reported that they tended to use BCIS data in their monthly morbidity and mortality meetings, which were attended by consultants and junior staff, because BCIS proactively sent, on a monthly basis, Microsoft PowerPoint^® (Microsoft Corporation, Redmond, WA, USA) slides displaying data in an attractive and concise way. This tension between MINAP and BCIS came up across sites. Although MINAP considers the pathway of care and captures data at the clinical team and organisational level, BCIS captures individual operator data. Interviewees noted that because these data reflected specifically on the standards of care provided by individuals, those individuals took ownership of these data and were more motivated to monitor these data routinely to ensure that they themselves were meeting required standards and that these data accurately reflected their performance. Similar comments were made by interviewees who participated in the British Association of Urological Surgeons audit, which also collects data at individual operator level. In considering differences between MINAP and PICANet, interviews suggested that PICANet appeared to generate greater levels of engagement among medical staff than MINAP, except among those doctors who used MINAP for research, rather than QI.

In the cardiology department at site D, clinical staff did not describe using their third-party database to produce reports. Instead, staff’s main interaction with MINAP data was via the national report. When that report was published, the MINAP lead reported that they presented the results and action plans at a monthly clinical audit meeting, which was attended by the clinical audit facilitator, consultants and nurses. In this site, constraints on resources, including staff being able to enter data in a timely way and limited clinician time to analyse and interact those data, presented challenges to more frequent data use. In addition, the MINAP lead reported a lack of support from the trust for QI, reducing motivation for engaging with MINAP data.

At site E, the chest pain nurse responsible for uploading MINAP data reported that they did not typically download their data from the NICOR website, unless they received a query from one of the doctors. Therefore, MINAP data were used in this service but were used to address specific queries as part of internal audit or reviews, rather than for routine monitoring.

In the PICU at site A, the PICANet clinical lead described the service as ‘obsessed’ with data. In addition to accessing data from the supplier website, the audit co-ordinator generated reports from their local database for review at monthly clinical governance meetings. The reports contained bar charts showing month-by-month data for performance measures, including those data reported as part of PICANet (e.g. readmissions within 48 hours of discharge, mortality and unplanned extubations), enabling PICU staff to monitor variation in the number of incidents over time.

At the PICU at site C, the database manager used an Excel spreadsheet to produce graphs for monthly multidisciplinary business meetings that were attended by the directorate manager, doctors, nurses, allied health professionals (e.g. pharmacists) and support staff. The metrics that were monitored overlapped with those at site A, although the database manager highlighted that extracting data to evidence unit activity according to Healthcare Resource Group (HRG) codes was also a priority, in preparation for Payment by Results (i.e. a system under which commissioners would pay health-care providers for each patient seen or treated, taking into account complexity of needs).

In terms of motivation for engaging with NCA data for QI, clinical teams with the necessary resources saw it as part of their role in caring for patients, ensuring that they provided safe and effective care. However, as indicated above, financial incentives also played a role.

In those sites that used their local NCA data, we identified two key ways in which clinical teams assessed their performance: (1) assessment of performance against evidence-based standards known to improve patient outcomes, as described above by the PCI project assistant at site C, and (2) monitoring trends over time. This type of monitoring was particularly important for clinical areas for which there were no evidence-based standards against which to compare service performance, as is the case for PICUs. Consequently, we found that clinical teams would use PICANet data to monitor trends over time, as described above. A paediatrician at site A described how monitoring PICANet data in this way led to a QI initiative. Through monitoring data, the site noticed an increase in the number of unplanned extubations and introduced training for nursing staff in taping and checking the tube position on a chest X-ray.

Although the example of QI above points specifically to an issue related to nursing practice, there was generally less engagement with PICANet data among nursing staff. For example, when asked about quality markers, a site A PICU nurse responded that they would tend to consider ‘many things, yeah, for example, central line infections, looking at those, ventilator-associated infections, problems with sedation and weaning’. In the PICU in site C, the nurses who were interviewed described robust processes for monitoring service quality, including ‘quality care rounds’ that audited aspects of care, such as infection control and the patient environment. A ‘harm-free care’ initiative was also used to assess patient safety, which included reviewing the number of inpatient falls, and ‘patient experience’ was captured using a patient care tracker. Consequently, nursing staff had access to other sources of data that provided information on nursing-sensitive care quality indicators. These data were said to overlap with only one PICANet metric considered a potential harm (i.e. unplanned extubations). By contrast, nursing staff in cardiology units were more likely to perceive MINAP metrics as relevant to the care that they delivered.

The above analysis points to resources in terms of local databases, audit support staff and the presence or absence of evidence-based guidelines as contextual factors that influenced the extent of NCA data use and how NCA data were used. Interviewees also described ways in which NCA data use varied over time and highlighted contextual factors that shaped those variations. One source of variation was national priorities. For example, historically, the collection and use of MINAP data had been supported by the political context, with the introduction of the NSF for coronary heart disease being motivated, in part, by what was considered an unacceptable variation in outcomes,143 leading to introduction of regional initiatives, such as regular regional meetings at which practice was shared. Clinicians described how MINAP had an impact on practice during this time. For example, when primary angioplasty was introduced, clinicians were able to see the number of patients treated and improvements in patient outcomes. Other clinicians reported how MINAP supported improvements in secondary prevention and treatment. However, the priority accorded to MINAP diminished once the original changes that it was designed to promote had been adopted widely across trusts. The changing political context appeared to have an impact on resources and infrastructure for collection and engagement with NCA data (e.g. reduction in resources to support data collection for MINAP at site D and the absence of clinical governance meetings in the cardiology department at site C), alongside an end to the regional networks more generally.

Engagement was also influenced by national targets that changed over time. MINAP includes a metric that is part of the BPT, which aims to improve quality by reducing unexplained variation using financial incentives, whereas PICANet includes measures that are part of Commissioning for Quality and Innovation (CQUIN). Introduced in 2009, the CQUIN system makes a proportion of a provider’s income conditional on demonstrating improvements in specified areas of patient care. When discussing with clinicians which metrics they prioritised, some included BPT and CQUIN measures.

Engagement was also found to vary over time depending on a trust’s prior performance in relation to a particular audit. In the PICU in site B, PICANet data access was via the supplier website. A consultant at this site explained that they did not routinely monitor their service performance using the PICANet website because this PICU’s mortality rates tended to be well within, if not below, the expected rates. This PICU was relatively small and clinicians felt that they would know immediately if the mortality rate changed. Therefore, the consultant no longer monitored the SMR as closely as they once did, and their main interaction with PICANet data was through the national report. This was in contrast to the PICU in site A, which, having previously been identified as an outlier, routinely used the PICANet website to monitor their SMR, as described above.

Access and use of national clinical audit data by divisional managers

Clinical teams in the five sites sat within divisions or clinical service units (CSUs), which, in several trusts, were headed by a ‘triumvirate’ that comprised a senior doctor, a nurse and a manager. At this level, there was responsibility for overseeing compliance with NCAs. Processes for this varied between trusts.

A clinical audit manager in site E described how site E confirmed which NCAs its CSU would be participating in. In response to the numerous mandatory and voluntary audits that a CSU might participate in, systems were put in place to (1) ensure that NCA data were submitted to the audit supplier and (2) monitor compliance and audit outputs. This clinical audit manager made sure that appropriate individuals within the CSU were aware of the publication of the NCA report and monitored their response to the report. A cardiologist in site E explained their process of review and dissemination of the NCA report:

As the [MINAP] lead I will summarise what I think are the main points and what I think the main changes that we might recommend from it. So, they don’t have to trawl through 100 pages or whatever. They’ll get the full version and then I will usually just send an e-mail saying: these are the main points, I think, from this and then the comment is usually: can you please just give us your clinical review of it and . . . what are the take-home messages and what changes, potentially might be needed as a consequence? . . . So, I send it back to [the trust’s clinical audit manager], to [a cardiac assessment nurse, whose role involves co-ordinating and validating MINAP data collection] and then it goes to . . . they have a clinical effectiveness board, a group of people who monitor all of these audits to make sure that the hospital is performing as it should be, in these big important areas because it is obviously publicly available documents and they want to make sure the trust is being portrayed well.

Site E, cardiologist

Summaries, like the one described above, aimed to disseminate learning from the NCA and assure those in governance roles that the service was responding appropriately to the audit and that the trust would not receive negative publicity as a result of any concerns raised in the NCA report and lose public (and commissioner) confidence. This trust had been identified in the Keogh8 review of trusts with persistently high mortality rates and, although this was not discussed in interviews at this site, this may have provided added motivation for ensuring that such processes were in place.

In site A, there were similar monitoring structures and the corporate quality governance team provided a standard template for clinical staff to summarise their response to NCA reports, recording areas where the service was performing well, any areas where the trust was an outlier and actions taken to resolve such issues.

Across the five sites, in addition to such monitoring processes, there were regular meetings at this level that considered NCA data (see Appendix 4, Table 16). These meetings tended to maintain general oversight of national NCA reports, checking that the reports had been responded to and considering any resultant issues, including business cases for expenditure for QI, which might be based on NCA data. It did not appear that NCA data were used in detail in these meetings, unless a particularly significant issue was identified that required higher-level input and, possibly, escalation to organisational levels.

Access and use of national clinical audit data by trust boards and subcommittees

Above the level of divisions or CSUs in trusts was the organisational level, where boards that governed trusts and their subcommittees were located. Trust board priorities were reported to be heavily influenced by CQC. A chief operating officer at site D provided insight into some service performance metrics that they reviewed:

Our dashboard has got our national performance metrics on it, so that’ll be A&E [accident and emergency], cancer waits, 18-week waits, they’re aligned to the CQC values of safe effective caring, responsive, so there are key metrics under all of the CQC requirements, so our performance dashboard is measured against those, so well led, so that would have the safe staffing, the attendance, the mandatory training, sickness profile of staff, which is why they all align.

Board members reported that they did not receive detailed reports on NCA data. Detailed analysis of this kind was generally delegated to clinical teams or (for high-risk areas) board subcommittees. The only routine way in which board members described encountering NCA data and reports was during their annual consideration of their trust’s quality accounts. However, there was provision, by exception, for boards to review performance in specific clinical areas in which they had been identified as outliers. Even then, boards did not review data in detail, but focused on approving action plans. Interviewees were clear that it was not possible to approve all recommendations arising from NCA reports, owing to financial constraints and, where recommendations related to employing new staff, to difficulties in recruiting people to some roles.

The remits of board subcommittees varied among sites, although there was always at least one remit with a specific role to assure care quality. Appendix 4, Table 17, summarises the meetings at this level across the five sites. Board subcommittees did not review NCA data in the same level of detail as clinical teams; however, board subcommittees might be involved in more in-depth reviews where particular risks were identified (e.g. when a unit appeared as an ‘outlier’ in a NCA report). Where such issues were raised, the relevant subcommittees may establish a QI project and ask for a report and action plan from the clinical area concerned. In site E, for example, a divisional clinical effectiveness committee reported action plans to address issues arising from NCA national reports to a trust-level effectiveness committee, which reported, in turn, to the trust board. If outliers were identified that represented a risk to the trust, then these were also reported to the trust’s medical director. In site A – a large trust – a quality management group with nine subcommittees reported to a quality assurance committee, which reported directly to the trust board. One of the quality assurance committee’s subcommittees (i.e. the safety and outcomes group) reviewed a quarterly audit report that showed trust-wide progress in submitting data and responding to NCA reports. NCA data were reviewed at different levels within this structure in site A, depending on the severity and sensitivity of the issue raised and the level of associated risk.

Interviewees explained that quality and safety subcommittees were supported in their work by NCA data that were as contemporaneous as possible, reflecting the current care quality context, and, ideally, that were no more than 1–3 months old. The work of quality and safety subcommittees was constrained by older data, especially when data were over 1 year old. The clinical audit manager in site E, for example, talked about the benefits for quality monitoring and improvement when NCAs, such as the National Hip Fracture Database, offered real-time access to data, which recalibrated each time users submitted new information.

By contrast, a quality subcommittee member in site D described problems associated with being unable to access recent NCA data when the subcommittee had needed to follow up action taken by a particular clinical area in which the mortality rate was out of line. In this case, NCA data were available from the supplier only every 6 months, which represented a significant constraint. The interviewee concluded that, given patient turnover, even 3-month-old data could constrain effectiveness of both clinical- and organisational-level QI processes.

Another point raised by several interviewees at this level illustrates the limits in confidence between different levels of trusts, relating to lack of timely data, which was seen to have an impact on their ability to engage with clinical teams about quality issues. Interviewees reported that clinicians seemed sometimes to use timeliness as a reason to dismiss data that put their service in a poor light, while disregarding the fact that data were old when they reflected well on their service.

There were concerns about NCAs that asked for what staff at this level considered too much data, without due regard to the resource implications for trusts or the utility of those data at either the clinical team or the organisational level. Particularly frustrating was when NCA suppliers significantly changed or added to their metrics, which generated a need for corresponding changes to local and third-party data storage systems. These changes could have unwelcome financial implications. Third-party suppliers, for example, might charge trusts to upgrade their software to reflect the changes and additional staffing might be needed for data input. Consequently, some staff called for more co-ordination of NCAs nationally and for more consultation with trusts about their resource implications. A trust board member in site A also described a lack of confidence at the organisational level in certain standards in some NCA reports, referring to them as ‘gold-plated’ (i.e. being regarded as unrealistic and unaffordable). This perceived lack of legitimacy meant that there was some reluctance among board members to approve expenditure arising from clinicians’ recommendations on NCA reports, especially when such expenditure was not connected directly to patient care.

Access and use of national clinical audit data by commissioners

As we were able to interview participants from only three CCGs (associated with sites A, B and E), the following account is necessarily limited and practice may vary in other CCGs. However, CCGs do include QI in their remit. For example, at the CCG associated with site A, one of their strategic objectives was ‘to transform care and drive continuous improvement in quality and safety,’ whereas the interviewee at the CCG associated with site E described how they worked with general practices to improve quality in primary care.

An interviewee in the CCG associated with site A described how issues of quality were considered from the contracting stage. When commissioning a service, the relevant quality manager would be involved in the process, developing, with the procurement team, questions relating to quality, which those bidding to provide the service would have to answer. Once a contract was established with a health-care provider for a particular service, the requirements relating to care quality would be written into that contract and the progress in meeting those requirements would be monitored, achieved, in part, through regular meetings with the provider. In the three CCGs interviewed, regular meetings with providers (including trusts) to monitor quality were described, using a range of metrics. In site A, for example, a clinical quality review group met bimonthly, with a remit to gain assurance that the care commissioned was of high quality, with a focus on clinical effectiveness. Meetings of the group were chaired by the CCG’s director of nursing and quality or by a senior trust representative, and the meetings were attended by senior clinicians and managers from the trust and CCG. The group received a quality report from the provider at each meeting. The report set out compliance with contractual performance requirements, which might relate to issues such as numbers of pressure ulcers, health-care-associated infections, falls, complaints, the Friends and Family Test and CQUINs, as well as compliance with national standards, such as National Institute for Health and Care Excellence guidance. In terms of NCAs, an interviewee at site A reported that CCGs were mainly interested in participation rates, rather than actual results. If there were concerns about care quality, then these concerns would be escalated to the contract management board and the CCG’s quality and safety committee (i.e. a statutory subcommittee of the CCG governing body). The quality and safety committee used an integrated quality and performance report, accompanied by an integrated quality and performance report dashboard. The printed report included a table of quality and safety performance measures, showing the number of incidences for each trust in the period and in the year to date, using red, amber and green colour-coding. In both interviews undertaken at the CCG for site A, interviewees emphasised the importance of the integrated quality and performance report data for their work. A similar (monthly) quality monitoring meeting took place in site B, to which providers submitted reports.

It was clear that commissioners used data extensively within quality monitoring and QI processes, but made relatively little use of NCA data. This was particularly the case for PICANet, as children’s paediatric intensive care is commissioned directly by NHS England specialist commissioning and not through CCGs. However, commissioners were always looking for new data sources to inform their work. Generally, CCGs welcomed data sources that could help them to assure and improve the quality of services that they commissioned, and used NCA data where this was the case. For example, an NCA mentioned by two of the CCGs was the National Diabetes Audit, which is used to monitor primary care performance. Within this, CCGs appeared to use data from only NCA national reports, which, as noted above, were often over 1 year out of date. Commissioners recognised that the need for more recent data was dependent on how data were going to be used. Commissioners acknowledged that there was little that they needed to measure in real time (other than issues such as accident and emergency waiting times) and, therefore, in some cases, year-old data were acceptable. However, when monitoring and improving current care, data that did not reflect the current care context represented a constraint. Data that were as close to ‘live’ as possible – for example, refreshed monthly – supported these processes. Finally, like interviewees from the other groups, commissioners talked about problems in trusting data perceived to be out of date.

Visualisation requirements

Interviewees described wanting a dashboard that, on first access, showed static visualisations of key performance metrics, displaying data for all patients. As to which metrics should be displayed, we asked interviewees about which metrics they considered to be important indicators of care quality, which linked to those identified in the mechanisms underpinning NCA data use in the NCA programme theory. For MINAP, prioritised metrics related to the delivery of evidence-based recommendations. For PICANet, emergency readmission within 48 hours and unplanned extubation were repeatedly described as important indicators of care quality; however, SMR was also considered to be a key metric (as part of the mechanism labelled ‘reputation’) and the one of greatest concern at organisational levels within trusts. Interviewees also prioritised metrics linked to financial incentives for both NCAs. Key metrics that were described for the two NCAs, and the reasons given, are summarised in Table 8. In line with current practice at the time, interviewees wanted QualDash to allow them to monitor their performance against particular targets/benchmarks and against national averages. However, interviewees also wanted to compare themselves with trusts that they considered to be similar to them in size and/or case mix.

Although key metrics should be shown when first logging in, interviewees also wanted access to metrics from other areas of the organisation to understand how their performance interacted with, and was affected by, those areas. For example, if a PICU received a large number of emergency readmissions from the HDU, then they might also want to look at metrics for the HDU. This was part of a broader desire for the visualisation of patient pathways to see metrics presented chronologically in terms of where they occurred within the pathway.

In looking at the questions that interviewees described wanting to ask of NCA data, we identified 124 tasks. The majority of these tasks involved only two variables, suggesting that simple visualisation techniques that users were already familiar with, such as bar graphs and pie charts, should be used, rather than developing novel visualisation techniques.

Interaction requirements

A key interaction requirement was that QualDash allowed users to quickly access, ‘at the touch of a button’, frequently reviewed data, which is in relation to the requirement described above that key metrics should be visible when users first access QualDash.

In discussing the choice of metrics to be displayed, interviewees talked about wanting a customisable dashboard, enabling them to select what metrics were displayed. Such an approach would enable users to select metrics that they perceived to be important signifiers of care quality.

Interviewees also described wanting to be able to customise the time period over which a particular metric was displayed. For example, for metrics for which users might expect to see fluctuations over a year, then it would be necessary to view data for several years. Interviewees also wanted to be able to select certain groups of patients (e.g. based on age or condition). For example, with MINAP, for some metrics, interviewees wanted to be able to look at data for non-STEMI patients only, whereas for other metrics, interviewees wanted to see data for STEMI patients only.

Selecting certain groups of patients to look at was part of the ability, also required by interviewees, to ‘drill down’ into the data to understand the reasons behind the numbers. Having assessed performance, a quality dashboard should support clinical teams to identify causes for metrics that raised concern. This might go as far as looking at details of individual patients.

Some interviewees within clinical teams talked about wanting a dashboard that could support simultaneous interaction by multiple users so as to be able to drill down into the data and discussed these within a meeting. These interviewees felt that, through this, QualDash could enhance the ability of teams to engage flexibly and in depth with NCA data.

Data quality

Data timeliness was considered to be a key requirement. For some interviewees, this meant that data should ideally be in ‘real time’. However, other interviewees considered that, for data to be used for QI, the most recent data should be no more than 3 months old. It was also felt to be important to notify users when incomplete data were displayed to ensure that users did not make erroneous assumptions based on incomplete data.

Reporting

Interviewees talked about wanting to generate reports that were customisable to their specific requirements that could also be edited at a later date (i.e. without having to start again if they needed to change something). Interviewees wanted to be able to cut and paste information from QualDash into Word and Excel to create their own reports. Given that the most common use of NCA data was to produce a summary of national reports to provide to other levels within the organisation, interviewees also wanted QualDash to produce reports that summarised the national report and compared the service’s performance with those results.

Notifications

There was a desire that QualDash made users aware of areas for improvement and of concern. This was both at the clinical team level, where clinical teams wanted to identify, seek to understand and resolve problems in advance of the national report, and at the higher levels of the organisation, where there was a focus on identifying outliers. A quality dashboard could support use of such data at organisational levels by enabling easy identification of a clinical area’s outlier status within an audit. One suggestion for achieving this was use of the red-amber-green rating system. This system was favoured by several interviewees because it was used frequently within health-care organisations and enabled users to identify potential problems at a glance. However, several interviewees also pointed out limitations with such an approach, most notably that it did not capture nuances within data, such as when a standard has not been achieved for sound clinical reasons. It was also felt that it would be useful if QualDash issued alerts (e.g. triggering an e-mail when performance drifted out of the normal range).

Co-design workshop 1

In the first workshop, we not only gave participants task cards based on the tasks identified through the phase 1 interviews and the workshop with NCA suppliers, but also allowed participants to write their own tasks. The next activity involved identifying the order in which users would want to use the data to answer certain questions, which can be referred to as ‘task sequences’. Figure 3 is an example of the output from this activity. Figure 3 shows the series of visualisations that one group of participants considered to be necessary for answering questions about call-to-balloon time, first looking at the overall achievement of the target, then seeing the achievement of the target over several months, and last looking at the breakdown of admissions by whether they were direct or indirect.

FIGURE 3.

Example output from co-design workshop 1, activity 2.

This activity revealed several key findings with implications for the design of QualDash:

• For each metric, there are what we can refer to as ‘entry-point tasks’. These ‘entry-point tasks’ are primary tasks that a user will want to undertake in relation to the metric, which involve monitoring a small number of measures over time.

• Investigation of further detail of a metric involves one or more of three subtasks: (1) breaking down the measure(s) for patient subcategories, (2) linking with other metric-related measures and (3) expanding in time to include different temporal granularities.

• Individual metrics have independent task sequences (i.e. what a user will want to explore after the entry-point tasks will vary according to the metric).

Although we intentionally involved staff from only one site in the co-design activities, it was reassuring to note that these findings reflected broader points made by participants in the interviews.

Prototype development

Following on from this first co-design workshop, we began to develop a QualDash prototype, seeking feedback from key clinical collaborators and audit support staff through this process. Details of the discussions during these meetings and their implications for the design of QualDash are reported elsewhere. 144

QualDash was constructed as a web-enabled dashboard generation engine, dynamically generated and rendered from a configuration file. QualDash was designed to address the key constraints of the use of NCA data captured in our NCA programme theory, while also incorporating requirements elicited from the phase 1 interviews and the learning about task sequences gathered from the first co-design workshop:

• QualDash provided immediate visualisations of key metrics to create a ‘level playing field’ between sites that did and did not have the resources of a local database and/or audit support staff able to produce visualisations. A QualCard was generated for each key metric, providing a quick view of all key metrics on accessing QualDash. It was not possible to produce visualisations of all metrics that participants identified as important. Specifically, data about unplanned extubation were not currently captured in the PICANet data, but, instead, unplanned extubation was inferred by detecting sequences of intubation followed by no intubation and then intubation again. QualDash included an R script that performed pre-processing steps, including calculating derived fields that were not readily available in the NCA data; however, it would have been computationally expensive and infeasible to run data validation procedures if we were to adapt the R script to detect the sequence used for inferring unplanned extubation.

• The QualCards could be expanded, providing three customisable visualisations (i.e. one for categories, one for quantities and one showing 3 years’ worth of data) to support tasks associated with the key metric. The initial version of the prototype was developed for PICANet. An expanded PICANet QualCard included in the prototype is shown in Figure 4 and all other QualCards included in the prototype are shown in Appendix 6, Figures 20–23. Users could customise the measures shown in the categories and quantities visualisations via graphical user interface buttons. The times subview used a small multiples view, with small line charts in each multiple linking the same time unit across years. A button enabled users to toggle between this alternative and a traditional multiseries line chart. QualDash was also designed in such a way that it would be easy to make customisations to QualCards, including creating new QualCards and adapting QualCards for a different NCA.

• To support users to drill down into the data easily, moving the cursor over a segment of the pie chart in the categories subview highlighted the distribution of the corresponding category over time in the main measures view. Similarly, moving the cursor over one or more month(s) in the bar chart caused the pie chart to respond and display the distribution of categories within the selected cohort.

• The grey area that outlined a QualCard acted as a handle that can be used to drag and reposition the card on the grid or expand it (when double-clicked).

• Given concerns with data timeliness, QualDash sought to improve access to timely data, providing users with a means to visualise data that they collected for the NCAs, without having to wait for data to be returned to them from the NCAs. Therefore, QualDash would sit on servers within each site, giving users control over how often data were uploaded. This was different from what was anticipated at the beginning of the study. Originally it was thought that QualDash would be hosted at the University of Leeds, with data received from NCA suppliers and users having web-based access to QualDash. A compromise of locating QualDash on local servers was that users would not have access to data from other trusts against which to compare their performance. This was considered to be a reasonable compromise, given that this was not included as one of the prioritised requirements at the end of phase 1 (see Box 2).

FIGURE 4.

Expanded PICANet mortality QualCard in QualDash prototype (using simulated data).

Feedback from the adoption focus groups

Feedback was gathered from the adoption focus groups, which, apart from at site A, were the first time that participants had seen QualDash. There were also a series of e-mail exchanges with clinicians following the adoption focus groups to clarify their comments. Although participants were positive about QualDash, a cardiologist in site D highlighted that the MINAP measures would need to be adapted for the cardiology service. In particular, the QualCard ‘call-to-balloon’ was not considered to be useful for site D, as the site typically did not provide the primary PCI treatment to which the metric relates.

Following on from this, effort was invested in clarifying what measures participating DGHs (i.e. sites D and E) wanted displayed on the MINAP dashboard through discussion with cardiologists at these sites. Discussions confirmed that most measures displayed were of interest to the service, but ‘call-to-balloon’ could be removed. The QualCards confirmed for installation of QualDash v1.0 are presented in Table 9.

Champion

In discussing the attributes of a ‘champion’, the research team initially expressed the role (based on phase 1 analysis) as someone who would encourage use of and disseminate information about QualDash. However, participants suggested that the role could encompass teaching other staff how to use it and how to overcome problems to support and maintain use, acting as ‘super users’. To undertake this role, champions would need to be trained to use QualDash and feel confident in doing so. In terms of delivery to impact adoption, participants discussed that a clinical champion was key, with some staff indicating that a ‘senior consultant’ would have the most impact. The rationale was that a senior clinical champion would have the authority, in comparison with IT or audit support staff, to lead the use of QualDash (e.g. put QualDash on the agenda of clinical governance meetings and then lead its use within meetings). Furthermore, a senior clinical champion was thought to have influence over the behaviour of their colleagues (i.e. their use of QualDash ‘lends weight’ to its benefits as a tool to support data use and would encourage others’ uptake).

It was agreed, based on focus group discussion, that at least one clinical champion from each participating cardiology department and PICU would be identified. To support a clinical champion, it was suggested that there could be two or more champions to support QualDash use within a clinical area (e.g. a clinician and an audit support staff member). Although staff did not see the role of champion as a large task, having more than one champion was viewed to be a way to share the load.

Awareness

Participants were also supportive of the use of an e-bulletin to raise awareness of QualDash, how QualDash could be accessed and how QualDash could benefit the service. It was suggested that responses to the e-bulletin (whether or not people read it) would depend on who sent it, suggesting that influence of the clinical champion was, again, key. However, it was noted that some staff would not read it regardless of who sent it. In terms of timing of e-mails, participants discussed sending them a couple of weeks in advance of installation of QualDash and again when QualDash was live as a prompt/reminder. Participants also advocated providing demonstrations of QualDash at appropriate meetings to disseminate information to stakeholders. For example, at site C, it was suggested that a presentation was given at the quarterly audit for the clinical effectiveness meeting. This meeting was protected time for clinicians and viewed as an efficient way to deliver information to a number of clinical staff at one time. Other suggestions were clinical governance meetings, multidisciplinary team meetings and sisters’ meetings.

Training

As in phase 1, participants suggested that QualDash should be intuitive to use and, therefore, require minimal training, a requirement operationalised in the design of the dashboard. In addition, participants supported the idea of online training on the basis that it would be difficult to find a time when clinicians could come together for face-to-face training. People could undertake online training at a time convenient for them and return to the session if they needed a refresher. Online training was also considered to be useful resource for new staff. Participants explained that any training materials offered should be in the form of quick reference tools, such as crib sheets, that could be easily accessed at the convenience of busy staff. However, a more fundamental point was noted during one focus group:

I think the most important thing when you’re introducing anything to clinicians is that it works. If you introduce it and it only works 80% of the time or 90% of the time, it’s a bit crap.

You lost interest.

The clinicians will tar it with a dodgy brush and they’re going to slag it off, and nobody will use it . . .

Based on lived experience of similar initiatives, participants highlighted that initial experiences of using QualDash needed to be positive (i.e. it had to work as intended when accessed by staff) to maintain their interest in and use of the technology.

Support and monitoring

There was variation among participants in the level of enthusiasm for the idea of feedback. We had anticipated that it would be the research team feeding back use and impact to the service, but the idea of clinicians giving examples of questions that they had and were able to answer using QualDash was proposed, tying in with their capacity to influence others. It was suggested that feedback could work to maintain enthusiasm by letting people know that QualDash use was making a difference. Providing feedback via meetings routinely held within the services was suggested to fit in with existing structures and processes.

Similarly, participants varied in their views of the benefits of an online forum. It was suggested that, in addition to leaving a question for response in a forum, users could share how they have used QualDash, linking to the idea of feedback. The advantage would be that participants across sites could share their experiences. Participants described similar forums run by NCAs in which individuals could seek clarification (e.g. how to identify the patient sample for a particular audit). It was noted that MINAP used to have an online forum and staff had found that useful. At site C, a member of IT staff commented that such forums are used a lot in the IT community, with benefits including that it is an asynchronous forum of communication and, therefore, does not depend on staff being in a particular meeting and could be searched through at any point in the future. However, some participants questioned whether or not staff would realistically have time to contribute to and make use of such a forum. As a source of support, the asynchronous nature of an online forum was perceived to be a disadvantage by some, as users would want an answer to their question straight away rather than waiting for someone to respond. The limited number of sites involved in the study was seen to affect this, in contrast to the MINAP forum, which could be accessed by every trust.

Recruiting champions and data uploaders

Informal agreement to become a champion from appropriate staff, for example consultants (usually our local collaborators), was often obtained at the adoption focus groups or, alternatively, focus group participants suggested appropriate colleagues. Prior to the installation of QualDash, potential champions were contacted by e-mail. Potential champions were provided with a description of the types of activities they would be asked to support as champion. The following list of activities was included in the e-mail:

• lead use of QualDash in appropriate settings (e.g. in meetings and report preparation)

• advocate the use of QualDash to other potential users

• support others to use QualDash if/when necessary

• act as point of contact for the QualDash team.

Appendix 9, Table 20, provides a list of the champions confirmed by site. In all sites, apart from the cardiology department at site C, clinical champions were identified. In most cases, champions were the local collaborator. Therefore, champions were already knowledgeable and supportive of the study aims and familiar with the research team.

Staff who would take responsibility for uploading data into QualDash were identified and confirmed in the same way as the champions and, in some cases, this role was performed by those acting as non-clinical champions (e.g. audit support staff) (details of who took on the role in each site are available in Appendix 9, Table 21). In the PICUs, staff who collected and uploaded data to the PICANet web agreed to upload data into QualDash. Staff who agreed to support MINAP data upload were based within and outside the cardiology service and performed different roles. Variation in background and experience of staff who agreed to upload data meant that they required different levels of support from the research team to learn how to complete the upload process.

E-bulletin and site demonstrations

To deliver information to potential users, in support of the champions’ efforts to raise awareness, two modes of dissemination were used. First, QualDash demonstrations were given in appropriate meetings that brought together clinicians (e.g. doctors and nurses) and other staff (e.g. service managers. Demonstrations were sometimes given by members of the research team, whereas in other cases the QualDash champions chose to provide a demonstration (see Appendix 9, Table 22, for overview). Second, an e-bulletin was drafted with the intention of providing it to the champions for them to disseminate via e-mail. However, this activity was suspended because of issues that emerged on installation of the dashboard (see Chapter 7).

Training materials

Crib sheets were produced that, with screenshots, provided step-by-step instructions to use the QualDash functions (e.g. how to add and remove variables from the subview and how to export visualisations). The crib sheets also included tips on interpreting data, including not making general claims based on small or unrepresentative samples, not mistaking correlation with causation and trying not to let preconceptions influence interpretation. The crib sheets (version 1) were sent out to one site when the installation of QualDash v1.0 was completed. However, based on user feedback, dissemination of the crib sheets was suspended to develop further iterations (e.g. to provide more information about each QualCard).

The research team also prepared a script for a training video and began recording the video to show the use of QualDash functions and to provide further guidance on data interpretation. However, as QualDash was introduced into each site, user feedback led to customisation of the QualCards (described in Chapter 7), meaning that a generic video would be potentially confusing if what it showed was different from what users saw when using QualDash. (QualDash v2.0, described in Chapter 7, included a documentation page with a demonstration video.)

A ‘QualDash set-up’ document was created with instructions for uploading data into QualDash, which was sent to relevant staff in advance of the installation visits.

Initial issues and their resolution

QualDash was developed and tested using simulated data. Therefore, the first time that QualDash was used with real MINAP and PICANet data was following installation and we anticipated that more validation and debugging would be required at that point. Two key sources of error were anticipated: (1) different data formats (e.g. time and date formats) supported on different server machines and (2) special cases that are encountered in only real-world data and are impossible to fabricate in simulated data. With new software, new requirements are also expected to arise as users start to use the software and, therefore, the research team anticipated that they would need to work to address new requirements to improve QualDash functionality for users.

The research team prepared to work with our champions to address major issues before wider dissemination of QualDash to avoid it being tarred with a ‘dodgy brush’ by users. To address issues efficiently, a process needed to be established for receiving and responding to user feedback. Hosting QualDash on local servers affected this process. Over 40 issues (a combination of bugs, where the software did not act as expected, and new requirements) were reported across sites between August and December 2019. Issues affecting correct data interpretation were considered to be top priorities. Figure 7 provides an overview of the process through which issues were identified and resolved by Mai Elshehaly, with the support of site staff and the research team.

FIGURE 7.

Process for identifying and resolving issues.

The research team could not access the QualDash software without being on site. Consequently, resolving reported issues had to be completed remotely, which could be a time-consuming and complex process, or Mai Elshehaly had to visit the site, which involved co-ordination with IT staff and/or clinical staff. Often, a site visit was required if the issue could not be satisfactorily resolved remotely. An example of this is provided in Box 5.

In addition, there was the challenge that when remote resolution required an update to the code that this was dependent on IT staff installing the updated code. For example, staff in the IT department at site B were sent an update for the PICANet dashboard in late October 2019, but at the end of November 2019 PICU staff reported that this update was not yet available to them.

Intuitive use compared with intuitive data interpretation

A feature of QualDash that was anticipated to support uptake and use, as formulated in the QualDash programme theory, was that it was intuitive to use. As described in Chapter 5, usability was rated as very good and observations in the think-aloud sessions suggested that participants found use to be intuitive. Therefore, there was confidence that QualDash enabled a certain level of intuitive use. Installation of QualDash v1.0 enabled site staff to view QualDash with real data for the first time. This prompted them to scrutinise QualDash in a way that they had not previously. For example, QualDash determined the labels for the pie chart segments based on what was stored within the data file, with the consequence that some pie chart segments were labelled with numbers as opposed to descriptors. QualDash’s interactive functions (e.g. using the pie chart segments to highlight the distribution of the variable across the months in the main bar chart) worked well. However, the labelling issue meant that users could not understand what the segment represented. This problem was a constraint on the use of the data interrogation functions, an attribute of QualDash linked in the programme theory to QI. As this had the potential to affect data interpretation across sites, the issue was prioritised for resolution. It was resolved by the creation of data definition files that translated these to actual labels wherever those labels were not made available by the audit file.

Alongside labelling, site staff familiar with service data identified what they perceived as ‘anomalies’ in the metrics displayed, which often transpired to be examples of where the configuration (i.e. metric calculation) did not meet their expectations. The programme theory hypothesised that, through displaying key metrics (and associated with the mechanisms) by site staff for monitoring and improving quality, QualDash would engage site staff in the use of NCA data and, therefore, the QualCard metrics were originally configured using variables and definitions provided by participants (from site A) at the phase 2 co-design workshops. These issues were not identified in the design workshops or adoption focus groups because simulated data were used to populate the dashboard. However, when champions scrutinised QualDash using real data, they identified discrepancies between the QualCard measure and their understanding of service performance. Examples of this are provided in Boxes 6 and 7.

Scrutinising QualDash with real data was an important part of the design and development process, surfacing the specific variables needed to calculate the key metrics that users wanted to use for quality monitoring. Adapting the QualCards became an important part of adoption, with learning from one site prompting discussion with, and revision of QualDash in, other sites.

Routines for data upload

Following initial upload of 3 years’ data into QualDash as part of the installation process, the intention was that those who had agreed to take on the role of data upload would learn to complete data upload autonomously and establish a routine for doing so, thereby providing users with access to data that was as timely as possible. The most routine and regular upload of data was in the PICU in site B, where the audit clerk reported uploading the data weekly. Despite not having an IT background, the audit clerk learned how to complete the data upload process with support from Mai Elshehaly. QualDash facilitated the reporting that the audit clerk had to do (described further in Time savings from using QualDash) and so they were motivated to learn how to complete data upload. In most sites, staff reported the intention to upload data monthly. However, staff changes complicated this. In the PICU at site C and in the cardiology departments at sites D and E, the person who originally agreed to complete the upload either left the organisation or moved to another role. In site E, data upload was further complicated by data extraction and upload not being completed by the same person (i.e. a clinical audit and effectiveness manager downloaded data from the supplier website and sent it to a member of staff in trust IT department to upload, neither of whom were users of QualDash).

Use of QualDash for monitoring care quality

Impact of QualDash on data quality

In line with the QualDash programme theory, a number of users anticipated that QualDash would have a positive impact on data quality. For example, the information analyst and champion in the cardiology department at site B wanted to enthuse consultants to use QualDash on the basis that they might complete MINAP data more fully if they use the data more, which would make the information analyst’s job easier. The information analyst also wanted to use QualDash to monitor and report on data quality, which could help them get more resources if they could provide evidence to managers of data being incomplete.

Similarly, the champion in site E, a cardiologist, highlighted positive impacts of QualDash on data quality:

The cardiologist says that when QualDash was first installed it enabled them to see where there was missing data and the audit team ‘cleaned up the data’ in response. Therefore, QualDash has already helped the service to get their data ‘a bit cleaner’.

The ITS analysis provided an opportunity to explore the impact of QualDash on data quality in terms of data accuracy and completeness. As described in Chapter 2, for MINAP, we looked at precision of arrival time (as a measure of accuracy) and whether or not ethnicity was recorded (as a measure of completeness). For PICANet, we looked at precision of admission time and discharge time and whether or not ethnicity was recorded. Times were considered precise if they were not on the hour or at half past the hour. The results are reported in Table 13. Analysis of MINAP ethnicity data for site E is not included, as there was only one record for which ethnicity was not known and, therefore, it was not appropriate for statical modelling of this kind. Similarly, analysis of PICANet ethnicity data for site A is not included, as ethnicity data were recorded for all patients, both before and after introduction of QualDash.

For MINAP, the proportion of records with a precise arrival time did not change following introduction of QualDash in site E, but rose in site B and declined in sites A and C (Figure 13). The proportion of records with ethnicity recorded did not change in site C, but rose in site B and declined in site A (Figure 14). These outcome patterns can to some extent – but not fully – be explained by drawing on the qualitative and log-file data. In site A, there was very limited QualDash use and, therefore, we would not expect to see an impact of QualDash, positive or negative, in this site, whereas in site B there was use of QualDash and an expressed intention to use QualDash to improve data quality. Although levels of QualDash use were similar in site C, our experience on the ground was that there was very little engagement with QualDash, in terms of both the dashboard and the study as a whole and, therefore, again, we would not expect to see an impact of QualDash, positive or negative, in this site. In site E, the champion perceived that QualDash did lead to an improvement in data quality, but this was not supported by analysis of the MINAP fields that we considered. However, we also know that MINAP data were not used routinely in this site and it may be that the absence of a routine that integrated QualDash into the work practices of the clinical team constrained the impact of QualDash.

FIGURE 13.

Proportion of precise MINAP arrival times for (a) site A; (b) site B; (c) site C; and (d) site E. Dotted orange line indicates installation of QualDash. Aqua line indicates the trend.

FIGURE 14.

Proportion of MINAP patients with known ethnicity for (a) site A; (b) site B; and (c) site C. Dotted orange line indicates installation of QualDash. Aqua line indicates the trend.

For PICANet, the proportion of records with a precise admission time did not change following introduction of QualDash in sites A and B, but declined in site C (Figure 15). The proportion of records with a precise discharge time did not change in site B, but rose in site A and declined in site C (Figure 16). The proportion of records with ethnicity recorded did not change in site B, but declined in site C (Figure 17). The PICU at site B is where we would expect to see an increase in data quality, as this is the site in which staff previously did not have access to data and in which there was significant QualDash use, yet the analysis does not support this hypothesis. The PICU at site A did not use QualDash and QualDash use in the PICU at site C was significantly limited and, therefore, we would not expect to see an impact of QualDash, positive or negative, in these sites. The decrease in data quality in site C is possibly explained by a change in who was responsible for entering the PICANet data that occurred during the study period.

FIGURE 15.

Proportion of precise PICANet admission times for (a) site A; (b) site B; and (c) site C. Dotted orange line indicates installation of QualDash. Aqua line indicates the trend.

FIGURE 16.

Proportion of precise PICANet discharge times for (a) site A; (b) site B; and (c) site C. Dotted orange line indicates installation of QualDash. Aqua line indicates the trend.

FIGURE 17.

Proportion of PICANet patients with known ethnicity for (a) site B; and (b) site C. Dotted orange line indicates installation of QualDash. Aqua line indicates the trend.

Time savings from using QualDash

As described above, the PICU at site B did not use PICANet data routinely for quality monitoring or improvement prior to introduction of QualDash. A major way in which the data were used, however, was to populate fields in the mandatory quarterly NHS England SSQDs. Before QualDash, collecting this information involved the audit clerk consulting a number of hard-copy and electronic data sources, including the ward diary, the ward admissions books and the patient administration system. The audit clerk also had to obtain bed occupancy rates from the matron and unplanned extubation rates from consultants. It used to take the audit clerk a half-day to collect these data (one-third of their part-time working week); however, following installation of QualDash, it took a few minutes. Not all SSQD information was included within PICANet (and, therefore, within QualDash) and some manual work was, therefore, still needed to finalise the submission, but QualDash’s introduction meant that, for example, the audit clerk no longer had to use a calculator to laboriously work out bed-days, 48-hour readmission rates and mortality rates. The audit clerk expressed their delight in an e-mail to the research team, following the 2019 SSQD data submission:

Just wanted to say QualDash has saved me hours of work with regards to data I submit to NHS England. It’s fab . . . Took me about 2 mins [minutes] to extract what I needed whereas before would have taken me an afternoon. So thank you.

Adoption activities for increasing uptake

The QualDash programme theory was expanded to include a number of CMO configurations that hypothesised how and why certain activities would support uptake and adoption of QualDash. We recruited champions and, in collaboration with them, QualDash was promoted at relevant meetings. The QualDash programme theory hypothesised that use of champions and promotional activities would support the adoption of QualDash, underpinned by leadership in its use, influencing others to use it by raising awareness about what QualDash offered and how it can benefit work processes. Owing to issues in dashboard design, we found that champions across most sites did not choose to lead use of QualDash because they wanted reassurance that metrics were configured as expected (in the case of consultants) before they used their influence to promote wider use or because they did not perceive benefits to their role (in the case of audit support staff). However, the idea that a clinical champion could lead QualDash use in appropriate settings was supported by qualitative data from site E where a clinical champion put QualDash on the agenda for their directorate meeting and presented QualDash within the meeting itself; however, based on the log files, this did not appear to prompt use by other staff. This may be because support for data collection was constrained, meaning that what was displayed was not an accurate measure of performance and, therefore, staff did not perceive it as optimally useful for QI.

Some support for CMO configuration A3 is provided by those cases where staff were motivated to use QualDash and did use QualDash, notably audit support staff in the cardiology department and the PICU at site B. These audit support staff appeared to find QualDash intuitive to use and this intuitive use is a feature of QualDash that is supported by the survey findings. However, these were staff who already had routines for using NCA data. By contrast, the clinical champion at site E was motivated to use QualDash but did not, reflecting that, not previously having access to MINAP data, they were not in the habit of doing so. This suggests that routines for using NCA data need to be established for QualDash to be used. In the revised programme theory, we have captured this additional contextual factor.

In CMO configuration A1, as a contextual factor, we did not initially include differences in resources in terms of (1) access to NCA data and (2) having audit support staff with the time and skills to produce visualisations of NCA data. However, our findings suggest that those sites that already have systems and processes in place for engaging with NCA data will be less likely to use a quality dashboard, such as QualDash. It is also questionable whether or not clinical staff would take over this work (vs. having audit support staff undertake this work). This suggests that the adoption strategy for such users should be directed at audit support staff, as QualDash should support the specific queries and tasks that these staff currently undertake, and the adoption strategy should not only involve promoting the features of QualDash but also show how it allows audit support staff to undertake their work more easily or quickly. Configuration A1 is also dependent on having the necessary computing resources, in terms of both hardware and software. Consequently, we break configuration A1 down into two CMO configurations to reflect such differences in context.

Configurations A4 and A5 are concerned with uploading data to QualDash. Again, our findings point to the need for necessary computing resources for audit support staff to be able to do this. As described in Chapter 7, the audit clerk in site A had agreed to upload data to QualDash, but was unable to.

Improving data quality and timeliness

In CMO configuration Q1, we hypothesised that, in certain contexts, QualDash may result in improved data quality and timeliness. Although some sites identified issues with data quality through using QualDash and sought to then improve data quality, our ITS analysis of audit data does not support the hypothesis that QualDash use results in an improvement in data quality. Although we identified the impact of reduced resources on the ability to engage with NCA data and saw QualDash as a way to address this, such reduced resources will also have an impact on the ability of teams to improve data quality.

Report generation

In CMO configuration Q2, we hypothesised that QualDash would be used because it facilitates report generation in sites that already produce reports using NCA and other data sources (e.g. in the cardiology departments and the PICUs in sites A and C). In practice, audit support staff in these sites chose to use their existing systems. This choice was influenced by the metrics that QualDash initially displayed, some of which were not configured as typically used by the service, and because QualDash was unable to display some metrics typically reported. In addition, difficulties in uploading data to QualDash meant that, for the PICU at site A, QualDash did not provide them with timely data. Again, computing resources also presented a constraint in terms of speed and having the necessary browser.

In CMO configurations Q5 and Q7, we hypothesised that teams would use QualDash to produce reports requested by groups outside the immediate clinical service team because of the ease of accessing NCA data and the ability to export visualisations, resulting in reduced time spent in report preparation and increased use of NCA data at divisional and organisational levels. Within our observations, we did not see the production of reports for divisional and organisational level staff, either using QualDash or by other means. The log files reveal that the functionality for exporting visualisations was used in all five cardiology departments, although we do not know if and how these visualisations were used. There was no evidence of this functionality being used in the PICUs and, based on our observations, it is likely that audit support staff in the PICUs in sites A and C would use their existing systems and processes for producing such reports, as they do for reports used within the PICU. For these audit support staff to use QualDash for this purpose, again our findings would suggest the need for visualisations that match the visualisations they currently prepare and the necessary computing resources. However, QualDash did lead to reduced time spent in preparation of reports to NHS England in the PICU at site B.

Reducing variation in extent of use of national clinical audit data

In CMO configurations Q3 and Q4, we hypothesised how and why QualDash may work to introduce NCA data into routine monitoring processes in sites where NCA data were used in a more limited capacity (i.e. sites B, D and E). In sites B and E, our findings present some evidence to support these hypotheses, suggesting that quality dashboards can reduce variation in the extent to which sites engage with NCA data. However, what we were not able to observe over the study period was whether or not this led to the introduction of any QI initiatives.

Optimising number and range of users

In CMO configuration Q6, we hypothesised that, via web-based access using a URL on any trust PC, QualDash would enable a greater range of staff to interact with NCA data. In sites where managers requested data reports from audit support staff, QualDash could be used by these staff to access the data directly. Although observations and informal conversations indicated that there may be a number of potential user groups beyond site champions and local collaborators, ongoing dashboard development was prioritised over delivery of dissemination activities planned to engage a wider range of users, which was subsequently prevented because of the COVID-19 pandemic. Consequently, our ability to test this hypothesis was limited.

Dashboard requirements

A programme theory of QualDash

We developed a programme theory of QualDash, as presented in Chapter 5. The QualDash programme theory built on the mechanisms, and contextual supports and constraints, elicited in phase 1 that explained the use and impact of NCA data. We described the resources that QualDash would offer to clinical teams when inserted into varying contexts, how we anticipated users would respond to these resources, how those responses would vary according to the user’s context and the subsequent impacts. Therefore, although the ordering of our objectives suggests that we would develop a programme theory focused on QualDash from the outset to inform dashboard design, the QualDash programme theory and CMO hypotheses could be formalised only after there was clarity regarding what functionality QualDash would provide.

Using programme theory to co-design QualDash

Developing a programme theory of NCAs enabled us to identify the metrics that engaged NHS staff in interactions with NCA data and what supported or constrained these interactions across sites. The NCA programme theory provided insight into what metrics should be prioritised for visualisation and how QualDash could incorporate functionality to address constraints on data use across sites, particularly variation in the ability to generate visualisations of NCA data and timeliness of such visualisations. Explicit direction regarding how these constraints should be overcome was not provided by the programme theory. Here, we drew on a combination of (1) our understanding from the phase 1 interviews of how the intended users already visualised NCA data, (2) the ideas and feedback interviewees provided and (3) the information visualisation expertise of team members. These details enabled us to move to development of the QualDash programme theory. This programme theory indicates the functionality that should be included in quality dashboards, separate from the underlying technology, providing guidance for future design of quality dashboards. Specifically, our findings suggest that those designing quality dashboards to support engagement with NCA data may find it beneficial to include the following:

• ‘at-a-glance’ visualisation of key metrics considered markers of safe and effective care on first logging into the dashboard

• simple visualisations, such as bar graphs and pie charts, configured in line with existing visualisations used by teams, with clear labelling of metrics

• functionality that supports current queries and tasks, including creation of reports and presentations

• the ability to interrogate the data, including exploring relationships between variables and drilling down to look at specific subgroups of patients.

Although the NCA programme theory highlighted areas where QualDash could provide benefit, it also revealed contextual factors that constrain the use of NCA data, which QualDash would not be able to address. Our focus was specifically on presentation of NCA data, with what data are collected, and how, being outside the scope of the study. However, as Chapter 4 reveals, the nature of the data collected and processes for data collection are intimately linked to how those data are subsequently used. Stewart et al. ,130 reporting on the state of NCAs in 2016, note that early NCAs were perceived to be relevant to doctors only and not to policy-makers, managers or other clinicians. Although some NCAs are notably interdisciplinary, PICANet was perceived by nursing staff to be of little relevance to them, in part, because it did not capture what they considered to be important markers of care quality. This has implications for the design of a quality dashboard, suggesting that the decision of which metrics should be displayed should be an interdisciplinary decision to ensure that the metrics have relevance to all members of the multidisciplinary team. Achieving this may also mean drawing on multiple data sources. This finding also has implications for NCAs more broadly, in terms of considering who they involve in making decisions about which data are to be collected. Although we found some use of NCA data at the organisational level, staff at this level perceived an imbalance between the benefits to their trusts from NCAs and the substantial resources consumed by participating in them, leading them to question the legitimacy of NCAs. This also has implications for NCAs more broadly. Possible measures to enhance the perceived legitimacy of NCAs could include NCAs moving from an emphasis on cumulative retrospective reports to real-time reporting, clearly presenting the ‘headline’ outcomes important to organisational level staff; further negotiation between trusts and NCAs about the nature and number of data to be collected; wider use by hospitals of routine clinical data to populate NCA data fields; and further use of technologies, such as dashboards, that help staff to explore and report audit data in meaningful ways.

Other learning that has come from this study is the critical role that audit support staff can play in supporting engagement with NCA data, through labour-intensive work to ensure data quality and the visualisation of data, and the importance of engaging audit support staff and drawing on their expertise in the process of design. As we move to a paperless NHS and a likely move to automatic gathering of NCA data from EHRs, our work is a timely reminder of the skilful work that goes into ensuring the quality of NCA data, enabling clinicians to trust the data and be willing to use it for QI. This is not to argue against automatic gathering. Rather, we suggest two factors that can facilitate this while maintaining data quality and clinicians’ trust in the data. First, the data that feed such systems must be accurate to avoid the problem of ‘garbage in–garbage out’,150 which may still require some of the time-consuming cross-checking that we identified. Second, software and data exchange interfaces between linked systems must be appropriately defined, both technically and semantically, and the complexity of the links between them navigated effectively. Crucially, both factors need input not only from IT specialists, but also from the audit support staff who understand the data and their contexts, meanings, dependencies, provenances, quality and limitations.

This study contributes to understanding the role that realist methods can play in informing intervention design. We would argue that developing a programme theory of the current programme, as we did with the programme theory of NCAs, is an important step because it provides an understanding of the constraints that need to be overcome. It should not be considered a weakness that the programme theory will not necessarily provide guidance on how those constraints should be addressed, as this provides the space to co-design solutions with intended users and draw on relevant subject expertise.

A benefit of developing the QualDash programme theory was that it made our reasoning explicit, providing a programme theory that could be tested and refined through testing of the CMO configurations in the evaluation. In this way, realist evaluation provides a way for design knowledge, in terms of what works and what does not, to cumulate. Most HITs are based on designers’ implicit theories about why certain features will provide benefit to users. If we become explicit about the theories that underlie our designs, we can then test those theories, using the refined theories to inform subsequent designs.

Using programme theory to co-design an adoption strategy

As with the second objective, it was not that the QualDash programme theory enabled us to develop the adoption strategy, but rather through the ideas elicited from stakeholders in phase 1 and working with the stakeholders to refine these ideas in phase 3, we were able to identify activities and their associated mechanisms that we hypothesised would support uptake and adoption. These activities and associated mechanisms were then incorporated into the QualDash programme theory. Again, this provided us with CMO configurations that could be tested and refined to allow knowledge of what works and what does not work to cumulate.

Our findings related to adoption fit with findings of other studies of dashboard use. A recent dashboard design study reported the need to add detailed definitions of the metrics used and the need to clarify section headers to improve comprehension and trust. 151 Our champions’ belief that QualDash should not be shared more widely before initial issues were addressed is supported by a study in which early issues created negative impressions that discouraged further use. 152 In the study by Bersani et al. ,152 it was felt that future efforts should focus on strong adoption planning and increased stakeholder engagement.

Phases 1 and 3 captured stakeholders’ perspectives on what is needed to successfully introduce a quality dashboard. Although our ability to fully test these theories was constrained by the COVID-19 pandemic, our findings suggest that, for health-care organisations seeking to introduce a quality dashboard, the following strategies may be beneficial.

Understanding how and in what contexts QualDash leads to improvements in care quality

The COVID-19 pandemic meant that we were unable to implement and evaluate a revised version of QualDash, limiting the opportunity to test and refine our QualDash programme theory. However, our study has provided the following insights into possible impacts of QualDash and the contexts in which they may occur:

• In contexts where clinical teams have been constrained in their ability to use NCA data, but where they have adequate resource to improve data quality, QualDash can support efforts to improve data quality via visualisation of missing data.

• In contexts where clinical teams have been constrained in their ability to use NCA data, QualDash can reduce the time spent in preparation of reports via automation of measure calculation and visualisation.

• QualDash can lead to reduced variation between sites in the extent to which sites engage with NCA data, due to clinical teams that have been constrained in their ability to use NCA data now engaging with those data via automatically generated visualisations.

However, what we were not able to observe over the study period was whether or not increased engagement with NCA data led to the introduction of any QI initiatives.

Feasibility of a trial of QualDash

As described in Chapter 8, it could be argued that all three of our progression criteria for a trial were met. Our findings have provided insight into a number of issues that need to be addressed for a successful trial of QualDash:

• QualDash would need to be hosted by a NCA supplier (or similar body), as hosting it within trusts presented challenges when working across five trusts and would not be feasible for wider roll-out. A particular challenge was inconsistency of data formats and labelling at individual NHS sites. Although this is not a challenge if QualDash is to be hosted by a NCA, it does, nevertheless, suggest a need for more work to standardise data formats and labelling if trusts want to procure software for exploring NCA data.

• QualDash uptake was limited, suggesting that encouraging QualDash use more widely could be challenging. However, this study has provided understanding of what constrains use of a quality dashboard, which can be categorised as those constraints that (1) could potentially be overcome by strategies to support adoption, (2) could potentially be overcome through functionality and features of the dashboard (including constraints that revisions to QualDash have already sought to address) and (3) cannot be overcome by a quality dashboard and need to be addressed by other means (e.g. lack of timely data as a result of human and technological resource constraints). Given the need for further research on effective adoption strategies that can be used by NCAs, we propose a trial that will not only assess the impact of QualDash, but will also provide rigorous evaluation of the adoption strategy. The design of such a trial is presented in Appendix 13.

• NCA data are complex, with the consequence that, even if a quality dashboard is intuitive to use, data interpretation is not necessarily intuitive. Although this may be addressed, in part, through clear labelling, we would seek to also evaluate the impact of training as part of a trial.

Our updated literature review suggests the need for more rigorous evaluation of the impact of computer-based quality dashboards on quality and safety of care. However, conscious of the resources that a trial of QualDash would require, the limited uptake and limitations of the data gathered to demonstrate that the progression criteria were met, we consider it is necessary to first refine QualDash and the adoption strategy to provide greatest potential for positive impact.

Intervention

Studies were included in the review if they described an evaluation of the use and impact of clinical or quality dashboards. We defined clinical dashboards as dashboards that use visualisation to inform the practice of health-care professionals and dashboards that may provide data at the level of the patient and health-care professional (showing all patients who they are caring for and comparing them with their peers and national benchmarks) or may allow the user to move between viewing information at both of these levels. 14 We defined quality dashboards as dashboards that provide a visual display of quality indicators at the ward or organisational level to inform operational decision-making and QI efforts. 18

The dashboard had to include some form of visualisation, such as graphical displays (e.g. bar graphs and pie charts) and/or colour coding of information. Dashboards that simply displayed information without the use of such techniques were excluded. A dashboard may be viewed on a computer screen or via another form of display, such as an interactive whiteboard. We did not include paper-based systems or systems that generated only a paper-based report.

Participants

All health-care professionals and managers at the organisational and/or ward/unit level within health-care organisations using the dashboard. Studies involving only health-care students were excluded.

Study design

All study designs were included in the review as long as they took place within a health-care organisation.

Performance and outcome measures

All reported performance and outcome measures were considered. This included qualitative and quantitative data on both measurable impacts and staff/patient perceptions. Studies were excluded where a dashboard was one part of an intervention and where it was not possible from the results to distinguish between the impact of the dashboard and the other intervention components.

Interventions

• Dashboard compared with no dashboard.

• Adoption and training compared with no adoption and training.

Primary outcome

The primary outcome would be emergency readmission within 48 hours of discharge (yes/no). This metric is considered an important indicator of care by PICU clinicians and one where there is substantial unexplained variation between PICUs.

Randomisation

Paediatric intensive care units would be randomised by the Clinical Trials Research Unit to one of eight groups (Table 27), using block randomisation with equal allocation to determine what intervention they receive in years 1–5 of the trial. Following randomisation, the Clinical Trials Research Unit would inform the chief investigator and PICANet of what each PICU will receive on a year-by-year basis, 1 year at a time, who will, in turn, inform the PICUs. A log will be maintained of who is unblinded to what at specific points in the trial.

Sample size

There are two principal comparisons of interest (dashboard vs. no dashboard and adoption and training vs. no adoption and training) that relate to the two main effects of the 2 × 2 factorial design. Assuming, based on the PICANet annual report (2019),156 that an average of 1.66% of admissions per year result in an emergency readmission within 48 hours of discharge in the control group with an intracluster correlation coefficient of 0.003846, then 32 PICUs each with an average of 627 admissions per year would provide 90% power to detect a reduction to 1.524% in the presence of a small synergistic interaction, with a two-sided 5% significance level.

Statistical analysis

The primary analyses will be conducted on an intention-to-treat basis using all available follow-up data from all consecutive admissions, imputing unavailable data where possible. The child-level binary primary outcome of emergency readmission within 48 hours of discharge will be analysed using the mixed-effects model suggested by Sundin and Crespi,157 with a random intercept for PICU, a fixed effect for time, treatment effects for interventions 1 and 2 and their interaction, and a residual. Consideration will be given as to whether or not to decompose the PICU-level variance into time-varying and time-invariant components. Supportive analyses will be conducted to assess the sensitivity of the conclusions of the primary analysis to assumptions about the missing data.