Interventions to optimise the outputs of national clinical audits to improve the quality of health care: a multi-method study including RCT

Research questions

Out of a set of recent, state-of-the-science, theory-informed suggestions for improving feedback, which are the most important and feasible to evaluate further within national audit programmes?

What is the effect of such modifications to feedback on intended enactment, comprehension, engagement among clinicians and managers targeted by national audits, user experience and preferences under ‘virtual laboratory’ conditions?

The 15 suggestions for improving feedback indicate a way forward but require further development and evaluation. 19 Rigorous evaluation methods, such as well-conducted cluster randomised trials, can establish the relative effectiveness of following such modifications to feedback. However, varying only five elements of feedback (e.g. timing, frequency, comparators, display and information credibility) produces 288 combinations – not allowing for replication of studies or the addition of other interventions, such as education meetings of outreach visits. 22 Given the multiplicity of factors that would need to be addressed, such an approach is not feasible; more efficient ways are needed to prioritise which of these to study. In objective 1, we undertook a fractional factorial screening experiment, building on current evidence and knowledge of behaviour change, and produced a statistical model to predict the effects of a large number of single and combined feedback modifications. This model can subsequently guide choices for further evaluation, as well as suggest practical ways of adapting feedback to enhance NCA impacts.

Research question

How do health-care organisations act in response to modifications of feedback from national audit programmes under ‘real-world’ conditions?

Our overall approach was consistent with the development, feasibility and (early) evaluation stages of the UK Medical Research Council guidance on complex interventions. 23 Having identified the most promising single and combined feedback modifications in a virtual experiment, we aimed to investigate how they work in ‘real-world’ conditions.

Our earlier programme, AFFINITIE (Audit and feedback interventions to increase evidence-based transfusion practice),24 evaluated the separate and combined effects of enhanced content of feedback and enhanced support following delivery of feedback with the National Clinical Audit of Blood Transfusion (NCABT). We identified marked variations in local NHS trust responses to blood transfusion audits, including a lack of clarity about who feedback should target and who is responsible for action;25,26 such problems are likely to apply to other national audits. Objective 2 aimed to evaluate how our different modifications of feedback from national audit programmes were delivered, perceived and acted on in health-care organisations, guided by Clinical Performance Feedback Intervention Theory (CP-FIT). 27 We worked with two national audit programmes that were introducing changes to how they delivered feedback.

The COVID-19 pandemic halted all non-essential research in the NHS, forcing us to abandon this objective in the early stages of fieldwork. With funder approval, we therefore modified our approach and drew on ‘expert’ interviews and CP-FIT to identify the strengths of the two national audit programmes, how their planned changes would strengthen their audit cycles, and further scope for strengthening their audit cycles.

Research question

What are the opportunities, costs and benefits of national audit programme participation in an international collaborative to improve audits through a programme of trials?

Large scale improvement initiatives, such as national audit programmes, continually aim to enhance their impacts, often by making incremental changes over time (e.g. in use of comparators, feedback displays). Given that such changes usually result in small to modest effects on patient care and outcomes, it is difficult to judge whether or not they are effective in the absence of rigorous experimental evaluations. There are potential significant returns on investment from NCA participation in a co-ordinated programme of research to improve effectiveness. We have proposed ‘implementation laboratories’ that embed research within existing large-scale initiatives such as national audit programmes. 28 Close partnerships between health-care systems delivering implementation strategies at scale and research teams hold the potential for a more systematic approach to identify and address priorities, sequential head-to-head trials comparing modifications to improvement strategies (e.g. of A&F), promoting good methodological practice in both improvement methods and evaluation, enhancing the generalisability of research and demonstrating the impact of improvement programmes. However, there is very limited experience of establishing such implementation laboratories. Objective 3 explored the opportunities, costs and benefits of national audit programme participation in a long-term international collaborative to improve audits through a programme of trials.

Myocardial Ischaemia National Audit Project

This project collects data from admissions in England, Wales and Northern Ireland for myocardial infarction. It aims to improve clinical care through the audit process and to provide high-resolution data for research. 29

Data span the course of patient care, from the moment the patient calls for professional help through to hospital discharge and rehabilitation. Clinicians and clerical staff in hospitals collect data on patient demographics, medical history, clinical assessment, investigations and treatments. Pseudonymised records are uploaded centrally to the National Institute for Cardiovascular Outcomes Research. In total, 206 hospitals submit more than 92,000 new cases to MINAP annually. The database currently holds approximately 1.5 M patient records. Vital status following hospital discharge is obtained via linkage to data from the Office for National Statistics. An annual report is compiled using these data, including individual hospital summary data.

National Diabetes Audit

The NDA programme is made up of four modules: the National Diabetes Core Audit, the National Pregnancy in Diabetes Audit, the National Diabetes Footcare Audit and the National Inpatient Diabetes Audit. 30 The NDA helps improve the quality of diabetes care by enabling participating NHS services and organisations to assess local practice against NICE guidelines, compare their care and outcomes with similar services and organisations, identify gaps or shortfalls that are priorities for improvement, identify and share best practice, and provide comprehensive national pictures of diabetes care and outcomes in England and Wales.

Our study focused on the National Diabetes Core Audit. For this, general practices and specialist services participate through the General Practice Extraction Service. Secondary care and structured education providers submit data manually via the Clinical Audit Platform. Audit reports provide national-level information for prevalence, care process completion, treatment target achievement, referral and attendance at structured education, comparisons for people with a learning disability, comparisons for people with a severe mental illness and complication rates. Reports also provide local-level information for registrations, demographics, complications, care process completion and treatment target achievement.

Paediatric Intensive Care Audit Network

PICANet was established to develop and maintain a secure and confidential high-quality clinical database of paediatric intensive care activity to identify best clinical practice, monitor supply and demand, monitor and review outcomes of treatment episodes, facilitate health-care planning and quantify resource requirements, and study the epidemiology of critical illness in children. 31

PICANet collects data from 30 hospitals providing specialist care. The core data set of demographic and clinical data on all admissions allows comparison of PICU activity at a local level with national benchmarks such as Paediatric Intensive Care Standards. This data set provides an important evidence base on outcomes, processes and structures that permits planning for future practice, audit and interventions. Each year, PICANet produces audit reports to show changes and comparisons over the 3-year reporting period.

Trauma Audit and Research Network

TARN is the NCA for traumatic injury and is the largest European Trauma Registry, holding data on > 800,000 injured patients, including > 50,000 injured children. 32 It aims to monitor processes and outcomes of care to demonstrate the impact of trauma networks, providing local, regional and national information on trauma patient outcomes and, thereby, help clinicians and managers to improve trauma services.

TARN collects data from 220 hospitals across England, Wales, Northern Ireland and the Republic of Ireland. Individual patient data are inputted manually at the trauma unit to an online data collection, validation and in-built reporting system, aiming to be available within 25 days of patient discharge or death. TARN produces annual national reports, triannual hospital network-level reports, triannual performance comparisons (e.g. hospital survival rates), quarterly patient-reported outcome measures (e.g. patient experience, return to work), and quarterly trauma dashboards for benchmarking against peers. TARN also provides continuous reporting and ad hoc analyses.

National Comparative Audit of Blood Transfusion

This programme of clinical audits examines the use and administration of blood and blood components in NHS and independent hospitals in the UK. 33 The programme aims to provide evidence that blood is being prescribed and used appropriately and administered safely, and highlight where practice is deviating from the guidelines to the possible detriment of patient care.

There is a rolling programme of audits and re-audits, with two or three taking place each year. Recent topics include the management of major haemorrhage, transfusion-associated circulatory overload, and patient blood management in adults undergoing elective, scheduled surgery. The NCABT contrasts with most other national clinical audit programmes, which consistently focus on a core, limited set of indicators. Consequently, the NCABT has to develop and implement different audit criteria and methods for the collection, validation and analysis of data within relatively short periods of time. The resultant feedback reports are subsequently uploaded and delivered to the hospital transfusion team via a hospital-specific NCA web page.

Design

We used a structured consensus process to guide the selection of feedback modifications for inclusion in the experiment. 37 Our method involved face-to-face meetings and discussion to elicit all views and promote transparent decision-making.

Design

Our approach followed human–computer interaction processes of UCD38 to design both the online versions of the modifications and other aspects of the experiment, including a questionnaire to measure participant responses. We started with two UCD workshops at City, University of London (May 2018 and June 2018), then undertook three iterative rounds of design and evaluation (summarised in Figure 1), followed by testing by the research team. Each of the three iterative rounds of UCD comprised designing a set of prototypes in consultation with team members familiar with each of the national audits, followed by formative evaluation of the prototypes with participants using the think-aloud technique. 39 This qualitative approach to design was intended to assess functionality, usability and user experience to optimise modification content and format.

FIGURE 1.

User-centred design process.

Throughout the development process, we held regular face-to-face, teleconference and e-mail discussions as a research team around the feasibility and detailed design of the candidate and evolving online modifications. We took notes and kept records of these exchanges.

Round 1: design and evaluation of sets of paper-based prototypes exploring the modifications and the design of the online audit report

Semistructured interviews gathered information about roles and typical audit report usage patterns, including familiar formats, navigational behaviour and attitudes to audit. Then, think-aloud interviews explored designs for an online audit report, including information architecture and navigational elements. Finally, think-aloud interviews evaluated prototypes of four of the seven selected modifications. We iterated content and designs between evaluation sessions.

Round 2: design and evaluation of sets of online prototypes refining modifications and the design of the study invitation e-mail, report iconography and terminology

Semistructured interviews gathered further information about roles, typical audit report usage patterns, including familiar formats, navigational behaviour and attitudes to audit. Think-aloud interviews gathered responses to online prototypes for all seven modifications. We also tested responses to a mock-up e-mail invitation, and a screen showing different icons paired with common audit terms.

Round 3: design and evaluation of a complete online prototype, refining the content, data, flow, and presentation of all screens in the experiment

We conducted ‘end-to-end’ usability testing of the prototype experiment to identify issues in the flow between screens, page interactions and content to ensure that the right information was passed between various components of the experiment.

Expert testing

Project team members, including those familiar with each national audit, undertook comprehensive ‘expert reviews’ of the live online experiment to identify programming bugs and usability issues and to ensure fidelity to modifications as intended ahead of the online launch. This involved several rounds of team testing of all aspects of the website, including rules and checks, randomisation and editorial content. Data were captured via self-reported issue logs and addressed by the web development team.

Recommend actions that are consistent with established goals and priorities

Recommend actions that can improve and are under the recipient’s control

Recommend specific actions

Shorthand reference

Specific actions (M3).

FIGURE 2.

Screenshot example of final ON version of modification specific actions (M3) (NCABT).

Provide multiple instances of feedback

Provide feedback as soon as possible and at a frequency informed by the number of new patient cases

Provide individual (e.g. practitioner-specific) rather than general data

Choose comparators that reinforce desired behaviour change

Selection for online experiment

Yes. Effective comparators (M7) would be ON if the feedback comparator showed recipient performance against that of the top 25% nationally. The modification would be OFF if the feedback comparator showed recipient performance against the mean average.

There was a wide range of options for varying comparators, including several variants of each of recipient performance over time, formal targets and peer comparisons. Although using several comparators might risk creating mixed messages for recipients, it might also maximise impact if the comparators are thoughtfully aligned. For example, recipients may see progress over time, note how this compares to others and be further motivated by explicit targets. We generated a (non-exhaustive) range of questions:

• Is a tailored approach (e.g. those below the mean see the mean, those above the mean see the top 10%) more effective than a standard approach (where all recipients see the same feedback)?

• Is adding an explicit target to peer comparisons more effective than not?

• Is adding individual peer performance scores (e.g. histogram in which the mean/top 10% is also marked) more effective than only the mean/top 10% summary statistics?

• Are identifiable peers more effective than anonymous peers (given that comparison is a social process?)

• Is feedback more effective if it is compared with summary statistics from one reference group (i.e. national) or multiple groups (e.g. national, regional)?

• Is a comparator more effective than no comparator?

The key issue for any such variant is which best focuses attention for driving behaviour change. Figure 3 illustrates the final design.

FIGURE 3.

Screenshot examples of final ON and OFF versions of effective comparators (M7) (NCABT).

Closely link the visual display and summary messages

Provide feedback in more than one way

Selection for online experiment

Yes. The modification would be ON if the feedback included a graphical display of performance data along with the textual information for effective comparators (M7). The modification would be OFF in the absence of a graphical display of performance data. Figure 4 illustrates the final designs.

FIGURE 4.

Screenshot examples of final versions of multimodal feedback (M9) when effective comparators (M7) is switched (a) ON and (b) OFF (NCABT).

Minimise extraneous cognitive load for feedback recipients

Selection for online experiment

Yes. In the second UCD workshop and subsequent team discussions, we debated the feasibility of operationalising extraneous cognitive load within an online experiment, recognising issues in standardisation and interpretation. We decided to develop a modification for the online experiment in which the control condition would comprise feedback with extraneous cognitive load. However, we had to be careful to ensure that the control condition was not degraded so much as to be unrepresentative of typical practice. We were also limited by programming capacity in how much we could achieve; increasing or reducing cognitive effort might require having two versions (ON and OFF) of all feedback pages and modifications. We therefore focused on ways of changing the content to increase or decrease cognitive load.

The modification would be ON in the absence of distracting detail. The modification would be OFF in the presence of distracting detail, such as additional general text not directly related to the audit criterion and feedback on other audit criteria. We recognised that this modification might interact with multimodal feedback (M9) and therefore anticipated a negative interaction in the analysis of the online experiment. Figure 5 illustrates the final design for the OFF version (high cognitive load).

FIGURE 5.

Screenshot example of a section of the final OFF version of cognitive load (M10) (NCABT).

Address barriers to feedback use

Provide short, actionable messages followed by optional detail

Selection for online experiment

Yes. The modification would be ON if the feedback included short messages with links to explanatory detail related to the audit criterion. The modification would be OFF in the absence of links to explanatory detail. This modification partly overlaps with specific actions (M3) and we anticipated a negative interaction in the online experiment. Similarly, this modification might also negatively interact with cognitive load (M10), given that adding information might distract participants. Figure 6 illustrates the final design.

FIGURE 6.

Screenshot example of final ON version of modification 12 (NCABT).

Address the credibility of the information

Prevent defensive reactions to feedback

Construct feedback through social interaction

Incorporate ‘the patient voice’

Selection for online experiment

Yes. The modification would be ON with the addition of a quotation from and a photograph of a fictional patient. The text would describe their experience of care, where possible, directly related to the audit criterion.

We aimed to embed one or more of the following behavioural change techniques69 in the quotation:

• feedback on outcome(s) of behaviour (i.e. stating how the patient benefited from clinical care consistent with recommended clinical practice).

• anticipated regret (i.e. suggesting that the feedback recipient might regret not following recommended practice).

• vicarious consequences [i.e. prompting observation of consequences for others (including rewards and punishments) when recommended practice is or is not followed].

• information about others’ approval (i.e. stating how the patient approves of the recipient following recommended practice).

The modification would be OFF in the absence of this information. Figure 7 illustrates the final design.

FIGURE 7.

Screenshot example of final ON version of M16: patient story (NCABT).

System scope

The experiment was to be delivered as a custom-built website that participants would access by clicking on a link in an e-mail invitation. The core functionality was to present participants with an audit page composed of the audit standard and combinations of ON or OFF versions of the 6 modifications (Appendix 2), followed by a questionnaire to measure their response. We decided to embed this functionality within a linear series of pages in designing the experiment (Table 4).

Requirements

The website was required to welcome the participant (step 1), obtain informed consent (step 2), allocate participants to one condition in the fractional factorial experiment (step 3), present feedback modifications (step 4), provide a questionnaire to collect outcome data for the experiment (step 5), allow participants to download ‘evidence-based tips’ for effective feedback and to claim a voucher and certificate upon completion (step 6). The website was also required to gather metrics, including time to completion and page visits. To maintain participant anonymity, the personal data required for making the voucher claim would not be linked to research data. We designed for anticipated behaviour, preventing user errors and a degree of misuse (e.g. we wanted to be able to detect repeat attempts from voucher request logs).

Round 1 sketches

We identified typical and salient content types from national reports published by our five collaborating national audits. We began to scope the information architecture for an interactive online version of the feedback excerpts (which effectively corresponded to a ‘mini-audit report’). We designed seven paper prototypes for ways of presenting an online audit report. We selected five sketches that best fulfilled our brief. The sketches illustrated ways that common online navigational design patterns could be applied to audit content. The sketches included a single scrollable web page with a hyperlinked table of contents, a website with a classic global navigation bar, one with a task-led dashboard, one with step navigation (‘next’ and ‘back’ buttons), and one with side tabs and ‘breadcrumb trail’ navigation. Breadcrumbs are a dynamic trail of links that allow users to traverse back through drilled content. We illustrated inline features to prioritise, sort and ‘bookmark’ recommendations, and buttons or links to related content (Table 5).

Round 1 findings

We received a broad mix of responses relating to the styles of navigation presented. The single scrolling page was deemed to be typical but not necessarily user-friendly. Participants responded positively to task-focused navigation and the classic global navigation with side-tabs. Participants gave mixed responses to the step navigation owing to the lack of signposting or menu in our sketch. They could not see how deep the system was or what information it contained. Participants also reported suitable or expected names for sections in order of importance, with the most salient information, ‘results’, presented first. Participants reacted positively to prioritisation features.

Round 1 requirements

The website was designed to:

• clearly convey the scope of the experiment and set respondent expectations before consent

• give a clear indication of progress through the experiment, such as pages viewed versus pages remaining

• use section titles that were familiar and easy to understand, such as ‘audit standard’, ‘results’, ‘recommendations’

• prioritise salient feedback over less critical information, such as recommendations

• present the audit report as a single page.

We were also able to draw on responses to the sketches to inform ways to operationalise certain modifications, such as optional detail (M12).

Round 2 prototypes

We built a Hypertext Markup Language (HTML) wireframe including the modification prototypes and two additional pages to evaluate the look and feel and wording of the e-mail invitation, appropriate pairing of icons with labels, and naming conventions for our information architecture. We published the HTML wireframe on Github Pages (GitHub Inc., San Francisco, CA, USA) to access during the evaluation session. We built and published a second standalone web-based mock-up of the audit report page with all modifications switched OFF. This mock-up included a range of designs for the overall look and feel and was shared with members of the project team only.

Round 2 findings

Positive comments from round 2 of the UCD concerned the clarity and simplicity of the e-mail wording. Mixed, neutral and negative comments oriented around issues of trust. Participants reported that the e-mail looked ‘too slick’ and ‘markety’. This raised questions about funding; one suggested that this was suspicious because it didn’t look like it was from the NHS. One participant felt that the question and answer format might be too conversational in tone and risk being dismissed by some clinicians. Participants suggested that increasing the prominence of collaborator and funder logos in the footer may increase trust. We also discussed the estimated completion time, considering that some would see a reduced amount of content. We were keen to set accurate expectations about the nature of the experiment and length of time to complete. We ultimately estimated the time to complete as 20–25 minutes. There was confusion around terminology, with some terms being used interchangeably, such as audit criteria and recommendations. The term ‘feedback’ was less clear than the term ‘results’. Participants reported a preference for naming conventions that followed standard scientific terminology, such as ‘results’.

Round 2 requirements

We provided consistent information between the e-mail invitation and the landing page and ensured that the funding body logo and collaborators were prominently presented. We simplified content (e.g. consistent use of ‘experiment’ versus ‘study’, avoiding repetition, removing question and answer format) and stipulated minimum system requirements. We chose to pair distinct blocks or sections of content, identified during round 1, to individual modifications to present them as part of a complete abridged version of an audit report. We defined baseline content in the report, including the report title, data period, audit criterion (named ‘audit standard’) and textual performance feedback in the ‘results’ section. This was the minimum viable amount of content that could be shown when all modifications were switched OFF.

Round 3 prototype

We built the final prototype (see Table 5), including the outcome questionnaire and peripheral pages such as the participant information sheet and ‘top tips’, using the popular bootstrap framework. Bootstrap is an open source toolkit for rapid development of responsive front-end prototypes using HTML, Cascading Style Sheets, and JavaScript. Responsive websites can detect and automatically fit a range of devices and screen resolutions. Although we did not design a platform that would be fully responsive to all devices, we were able to adapt to a broad range of screen sizes and resolutions. We designed the user interface to reduce differences in layouts between browsers to provide a comparable experience to all respondents. We optimised and tested the website in modern browsers [i.e. Firefox^® web browser (Mozilla Corporation, San Fransisco, CA, USA), Google Chrome^® browser, Microsoft Edge^® (Microsoft Corporation, Redmond, WA, USA) and Safari^® (Apple Inc., Cupertino, CA, USA)] and legacy browsers (i.e. Internet Explorer^® 7, 8, 9, 10 and 11). This restricted our application of all best-practice web development techniques, such as progressive enhancement and accessible design. Progressive enhancement is recommended for delivering websites that deploy enhanced features to browsers that support them so that designers can maximise impact without compromising the experience in other browsers.

Round 3 findings

We tested each iteration of the web prototype sequentially with the six participants in UCD round 3. We conducted end-to-end usability testing of the prototype to identify issues in the flow of the screens, layout, content, and look and feel of the user interface. We captured audio data and took observational notes that we analysed for emergent themes, usability issues and design suggestions. We identified 74 instances of usability issues (38 low to moderate, 36 severe to critical). Participants reported multiple issues that we sought to address by providing clearer information pertaining to the scope and purpose of the website on the landing page. One participant reported that they had expected the experiment to be more involved and wished to go back and re-read everything properly. There was extensive discussion between the project team about the issues and implications for randomisation if users accessed the back button. Although we could not prevent users from navigating back to the point at which they were randomised, we could prevent them from being reallocated. Only the first set of answers would be saved. Respondents were alerted prior to randomisation to avoid using the browser’s back button.

Round 3 requirements

We designed the landing page to include a welcome message describing the experiment’s purpose, minimum browser requirements and expected time to complete. We designed the system to take account of users who wished to move back and forth between pages of the experiment. We designed a pictorial, annotated overview of the steps involved in completing the study. We incorporated a responsive, rather than step, progress tracker that included a percentage to clearly orient the user, reduce abandonment and increase questionnaire completion rates. We implemented a discrete left-hand menu so users could navigate quickly to specific sections to improve the usability of the overall report page. We aimed to provide sufficient signposting for participants to move back and forth between the report page and the questionnaire page. We selected criteria to be a single outcome audit criteria statement. The section was clearly titled ‘audit standard’ to address the issue, observed in round 2, of participants mistaking the statement as feedback.

Round 3 expert evaluation

Stephanie Wilson conducted a design review on the finalised prototype to assess it for visual consistency, typography and editorial content. The prototype and content were finalised and shared with the developer, along with front-end code and assets. The website was built using ASP.NET Razor Pages in Microsoft Visual Studio^® (integrated development environment) with a Microsoft SQL Server^® database. Content was uploaded to the database.

Round 3 testing and deployment

We tested the front-end of the live website with eight project team members over 2 months to identify browser inconsistencies and issues with the flow between screens, page interactions, content, exceptions, form validation and error messages. We also wanted to ensure that the right information (i.e. the randomisation process and participant allocation) was passed between various components of the experiment. Project team members logged a total of 139 issues over several rounds of testing. These were collated into a single document and passed to the development team to fix. The issues identified fell into three main categories: functionality (n = 40), editorial content (n = 56) and cosmetic (n = 43).

Study design

We conducted a randomised, online, fractional factorial screening experiment. Six modifications to feedback were operationalised in two versions (ON/OFF) and applied within audit report excerpts for five different national clinical audits.

Participants were randomised to receive one of 32 combinations of the modifications, stratified by audit. After studying the audit excerpt, participants completed a short questionnaire to generate all study outcomes. Appendix 3 provides a full example of the ENACT (Enhancing NAtional Clinical audiT and feedback) online experiment interface. This study is reported as per the Consolidated Standards of Reporting Trials (CONSORT) guidelines for randomised trials.

Setting and participants

We collaborated with our NCA programmes, which covered a range of clinical topics in primary and secondary care. Three of the audits (NCABT, MINAP and TARN) each cover more than 150 acute NHS trusts in England alone, as well as other hospital sites in the devolved nations. PICANet presently covers 34 specialist-commissioner admission sites and can provide multiple respondents per site. The NDA covers all (approximately 7500) general practices in England.

Each audit shared an invitation, containing the link to the online experiment, via e-mail with their existing distribution list. These lists contain the recipients of their reports (i.e. clinicians, managers, nurses and commissioners), all of whom were potentially eligible to participate as regular recipients of each audits’ communications.

Prior to entering the experiment, participants could access the information sheet and were required to confirm their consent. This page stated that we were interested in the views of anyone who received and acted on the audit report as part of their professional role. There were no other eligibility criteria.

Participants were offered a £25 voucher and certificate of completion in recognition of their time. At the end of the experiment, participants were offered the opportunity to view evidence-based guidance (‘top tips’) on how to improve their own A&F practice.

To obtain the voucher and certificate of completion, participants provided their e-mail address on completing the experiment; to maintain anonymity, voucher and certificate requests were not linked to experiment data.

Protocol violations

After opening to recruitment, a serious risk to the integrity of the study was identified in which repeated (i.e. duplicate) participant completion of the experiment took place. The experiment was subsequently and temporarily closed while security was enhanced. Additional experiment entry criteria, applied prior to randomisation, required participants to provide their NHS or Health and Social Care Northern Ireland (HSCNI) e-mail address. This was validated to confirm that the participant had not previously completed the experiment; the anonymity of experiment data was retained.

As participants were not uniquely identifiable within the experiment data (to maintain anonymity), we defined a ‘contamination period’ over which repeated (i.e. duplicate) participant completion was known to have taken place. We further explored the time spent on questionnaire and other indirect, objective criteria with the aim of identifying and excluding repeated participation, thus providing a data set of independent participants.

For information about the protocol violations, contamination period and how we maintained study integrity, see Populations and Appendix 5.

Intervention

Following consent, participants indicated which audit was relevant to them, their role and organisation from a predefined list. Participants were then randomised to be presented with one of 32 versions of the excerpt of a modified audit report. Participants were informed that the excerpt contained hypothetical but realistic data.

Modified audit report excerpts followed a basic template. The page was titled with the relevant audit (e.g. ‘National Diabetes Audit Report’) and a statement that the data were collected in 2018. The excerpt showed an audit standard taken from the appropriate audit [e.g. ‘Patients with type 2 diabetes whose haemoglobin A_1c (HbA_1c) level is ≥ 58 mmol/mol (7.5%) after 6 months with single-drug treatment are offered dual therapy’] and the result [e.g. ‘Our practice achieved this standard of care for 86% (318/370) of patients’]. Audit standards were selected with input from each relevant NCA collaborator to ensure that they were perceived as valid and credible by experiment participants.

The remaining content was dependent on the version of feedback to which the participant had been randomised, according to a combination of six modifications (Table 7). The six modifications are described in detail in Chapter 2; the implementation of the modifications in the experiment are briefly outlined below and further illustrated in Appendix 2:

1. Effective comparators: feedback is typically given in the context of a comparator, and comparators can be important motivators for improvement or for maintaining high performance. The modification was ON when showing the top 25% nationally as the comparator and OFF when showing the mean average.

2. Multimodal feedback: presenting feedback in different ways may help recipients develop a more complete and memorable mental model of the information presented, allow them to interact with the feedback in a way that best suits them and reinforce memory by repetition. 19 The modification was ON if the performance result text was accompanied by a graphical display of performance data and OFF when the graphical display was absent.

3. Specific actions: specification of a desired behaviour can facilitate intentions to perform that behaviour and enhance the likelihood of subsequent action. 44 The modification was ON if the feedback provided specific recommendations for action (i.e. who needs to do what, differently, with/to whom, where and when) and OFF when such recommendations were absent.

4. Optional detail: it is recommended that feedback designers provide short, actionable messages with optional information available for interested recipients. The credibility of feedback can be enhanced if recipients are able to ‘drill down’ to better understand their data. 19,68 The modification was ON if short messages with clickable, expanding links to explanatory detail were included and OFF when these links were absent.

5. Patient voice: HQIP guidance for best practice in clinical audit recommends that PPI be included throughout the audit process as a marker of quality. 68 In principle, explicitly linking patient experience to audit standards may highlight the importance of providing high-quality care and, hence, increase motivation to improve practice. The modification was ON when a text box including a quotation from and a photograph of a fictional patient was added and OFF when the text was absent. The text described their experience of care, where possible, directly related to the associated audit standard.

6. Cognitive load: poorly presented and excessively complex feedback risks being misunderstood, discounted or ignored by recipients. 54 The modification was ON when distracting detail was absent, reducing cognitive load. The modification was OFF if distracting detail was added (i.e. additional general text not directly related to the audit standard and feedback on other audit standards).

Primary outcome

The primary outcome was participant strength of intention in relation to the target behaviour80 specified in each of the audit-specific materials presented.

A systematic review of the literature on intention–behaviour relations demonstrated ‘a predictable relationship between the intentions of a health professional and their subsequent behaviour’. 81 We recognised that (1) there is almost no intention–behaviour gap when intention is low, (2) there will be an intention–behaviour gap when perceived control over the behaviour is low and (3) there is an intention–behaviour gap when intention to do the ‘right’ thing (i.e. social desirability) is high. Although intention has a limited ability to predict subsequent behaviourr,82 it can be informative in intervention development and early evaluation. 23

We anticipated social desirability bias in responses. The randomised design ensured that this was balanced across the different combinations of modifications to the audit report excerpts. We aimed to minimise unintended ‘loading’ of potential responses of intention due to social desirability bias by presenting the target behaviour in the context of other behaviours that would be appropriate,84 including the introductory statement, ‘Considering the time and resources available to you and other clinical priorities . . . ’, and anchoring items over ‘the next three months’.

Intention was measured using three items (Q2–4 in Table 9, beginning with the stem statements ‘I intend’, ‘I want’ and ‘I expect’). 81,83 Each item used the same template and was followed by the appropriate audit standard, for example ‘Over the next 3 months, I [intend/want/expect] to ensure that our patients with type 2 diabetes whose HbA_1c level is 58 mmol/mol or above following 6 months with single-drug treatment are offered dual therapy’. Responses to each item followed a seven-point Likert scale and were scored –3 (completely disagree) through to +3 (completely agree); positive scores indicated greater intention.

Previous testing of these stems indicated that they measure the same concept, with Cronbach’s alpha values > 0.9. 84,85 We calculated the mean value across the three items to provide a measure of overall intention as the primary outcome.

Secondary outcomes

Proximal (‘upstream’) intention outcomes evaluated participants’ intention to undertake other actions in response to the feedback: bring the audit result to the attention of colleagues, set goals, formulate an action plan and review personal performance in relation to the audit standard. We defined these as ‘proximal’ outcomes on the assumption that they would generally precede actual enactment of the audit standard (i.e. the primary outcome). Responses to each item followed a seven-point Likert scale and were scored –3 (completely disagree) through to +3 (completely agree); positive scores indicated greater intention. Space was also provided for participants to report any further steps or actions that they would take to meet the audit standard.

Comprehension was evaluated using a single item (‘I found the information in this audit report excerpt easy to understand’) adapted from the website evaluation questionnaire (WEQ), a standardised questionnaire developed to evaluate website quality. 86 Responses followed a seven-point Likert scale and were scored –3 (completely disagree) through to +3 (completely agree); positive scores indicated higher comprehension.

User experience was defined as a ‘person’s perceptions and responses resulting from the use and/or anticipated use of a product’87 We use the positively worded two-item Lite version of the Usability Metric for User Experience questionnaire (UMUX-LITE)88 to evaluate whether or not the audit report and online interface were both useful and useable:

• This audit report excerpt met my information needs.

• This online audit report excerpt was easy to use.

These items were selected to minimise time, cost and user effort,89 while also providing a proxy for the longer and more commonly used System Usability Scale (SUS). 88,90

Responses followed a seven-point Likert scale and were scored –3 (completely disagree) through to +3 (completely agree); positive scores indicated better user experience. We calculated the mean value across the two items to provide a measure of overall user experience.

Additional outcomes

User engagement was measured by experiment diagnostics, including the length of time (in seconds) spent working through each combination of modifications and the number of ‘clicks’ within the audit report excerpt.

Sample size

Assuming similar effects of each modification across audits and roles, a total of 500 participants across the five NCAs provided 90% power to detect small to moderate main effects [i.e. 0.3 standard deviations (SDs)] for each modification (two-sided 5% significance test). This also provided approximately 80% power to detect main effects of 0.25 SDs and 70% to detect main effects of 0.22 SDs. No allowance for loss to follow-up was required, as data were collected at one time point only. Recruitment was permitted to exceed the 500 participant target, up to an overall maximum of 1200 participants and 480 participants per audit (15 replications of the 32 combinations of modifications). This increased the power to evaluate potential interaction effects within available resources. A 4-month recruitment period was originally planned.

Statistical analysis

All study data were extracted by City University, London (London, UK), and securely transferred to the Leeds Institute of Clinical Trials Research (Leeds, UK) for analysis. Analyses were conducted in SAS software, version 9.4 (SAS Institute Inc., Cary, NC, USA). The final version of a prespecified statistical analysis plan was approved prior to analysis commencing.

Populations

The intention-to-treat (ITT) population consisted of all randomised participants, grouped according to the combination of modifications they were randomised to receive, regardless of whether or not they completed the experiment.

Two modified ITT populations were defined owing to the detection of repeated participant experiment completion (see Appendix 4). As participants were not uniquely identifiable within the experiment data (to maintain anonymity), indirect, objective criteria were used to define the modified ITT populations, aiming to ensure the inclusion of unique participants.

The primary modified ITT population excluded all participants recruited during the contamination period. This is a conservative population, known to exclude a significant proportion of valid participants alongside repeated participation.

The secondary modified ITT population excluded participants who completed the experiment questionnaire in an unfeasibly fast time to allow for valid completion. A cut-off point of 20 seconds for questionnaire completion times was established according to a clear distinction in the distribution of questionnaire completion times for participants recruited within the contamination period (see Appendix 4, Table 30 and Figure 31).

Statistical considerations

Primary analysis was conducted on the primary modified ITT population, regardless of completion of the experiment or outcomes. An overall two-sided 5% significance level was used for all outcomes unless otherwise stated.

Distributional assumptions

Questionnaire responses followed a seven-point Likert scale, scored on an integer scale as –3 (completely disagree) through to +3 (completely agree); responses were quantitative with interval properties. Positive scores represented greater agreement with statements indicating intention, comprehension and user experience related to the audit report excerpt.

A continuous distribution was assumed for all outcomes; however, we anticipated that responses would likely be highly skewed because of social desirability bias. We explored the distribution of all outcomes using descriptive statistics and graphical display without reference to audit, participant role or randomised allocation.

We undertook appropriate model diagnostics to check the appropriateness of the distributional assumptions and validity of the statistical modelling. Based on the distribution of residuals within our analysis models, if approximate normality could not be assumed, we planned to dichotomise outcomes to form a meaningful binary outcome of agreement; however, diagnostic plots indicated approximate normality of residuals (see Appendix 6, Figure 46).

Covariates

Alongside the modifications under investigation, covariates for statistical analysis models included:

• Audit – MINAP, NCABT, NDA, PICANet, TARN. The audit with the largest number of randomised participants (i.e. NDA) forms the reference category.

• Randomised design block – block 1, block 2; using effect coding (–1, +1).

• Role – clinical (allied health professional, fully trained doctor, manager, nurse or nurse specialist, training doctor) or non-clinical (manager, audit and administration staff). Clinical roles formed the reference category.

Audit and role are included as covariates to allow direct estimation of effects. A randomised design block, generated by [B1] = ABF = CDE, is included as a covariate to account for its role in the design of the experiment; it is therefore considered a ‘nuisance’ parameter, rather than one of direct interest.

Primary outcome analysis

To identify and screen for potentially active modifications, the six experimental modifications and covariates (audit, design block, role) were included as independent variables in a multivariable linear regression model for the primary outcome.

Effect coding (–1, +1) was used for each modification to ensure that parameter estimates for all modifications and their interactions provide the main effect (rather than simple pairwise effect), that is the effect averaged across all combinations of levels of the other modifications. Parameter estimates are therefore on the same scale, unbiased and represent half the main, two-way and three-way modification interaction effects.

Analysis used a multistage approach. Available complete data were used to identify important modifications and interactions. The resulting model from available complete data was then applied using the primary modified ITT population, with multiply imputed missing data.

Stage 1: model selection using available complete data

We included main effects and two-way interactions of modifications in an ‘initial’ model, alongside key covariates and randomised design factors as follows (Table 10): randomised design block, audit, role, modification and two-way modification interactions.

As the experiment included replicated runs, we used the lack-of-fit test to obtain an unbiased, pure error estimate of the error variance and test the hypothesis that our ‘initial’ model was adequate.

Where lack of fit was observed, we used stepwise selection to identify additional important interactions (see Table 10) to include in a ‘full’ model. We used traditional entry and removal selection criteria based on a 15% significance level of the F statistic, reflecting the effect’s contribution to the model fit when it is added or removed, respecting the hierarchy of effects. The consistency of model selection using Akaike information criterion (AIC) and Bayesian information criterion (BIC) was explored.

Using the ‘initial’ or ‘full’ model as applicable, we identified the most important modifications78,91 by ranking the absolute standardised effect sizes of model parameters using Pareto and half-normal plots for each outcome. Based on the principle that a small number of parameters account for a large portion of the effect, Pareto plots provide a tool to separate the few vital factors from the many trivial ones.

A final ‘parsimonious’ model was obtained using backward selection (removal based on 15% significance level of the F statistic, respecting the hierarchy of effects) to simplify the model, while retaining important parameters identified via the Pareto plot, audit and randomised design block.

Type III sums of squares, mean squares and p-values are presented for overall effects and analysis of variance (model and lack of fit) in Results, informing the parameters included in the ‘initial’, ‘full’, and ‘parsimonious’ models.

Stage 2: primary intention-to-treat analysis

Primary analysis was based on the primary modified ITT population, using multiple imputation to account for missing outcome data.

We used the fully conditional specification (FCS), also known as multivariate imputation by chained equations,92,93 predictive mean-matching method of multiple imputation to impute missing data. A single missing data model was used to generate 50 imputations across all missing outcomes, with the following predictors (see Table 10): outcome, audit, role, audit*role interaction and modification (main effects, two- and three-way interactions). Additional interactions between modifications with audit and role were applied where model convergence allowed and included interactions between the modification main effects and two-way interactions, with audit and with role.

The models containing important effects identified via the analysis of available complete data were applied to the primary modified ITT population with multiply imputed missing data. Pareto plots from the ‘initial’ or ‘full’ model, as applicable, were compared to ensure that the appropriate parameters were included in the ‘parsimonious’ model. Where there were differences in the parameters meeting the threshold for inclusion from the analysis of available complete data and the primary modified ITT population, an inclusive approach was taken and parameters included from either analysis were included in the primary ITT ‘parsimonious’ model.

Presented parameter estimates, associated standard errors and p-values were calculated using Rubin’s rules. 94 Predicted plots are presented to illustrate the direction and strength of effect for identified main effects and interactions.

Secondary outcome analysis

The analysis strategy for secondary outcomes followed that of the primary outcome. Summary statistics were used to explore the length of time spent working through and engagement with the audit report by modifications and audit.

Sensitivity analysis

Sensitivity analysis using the available complete data in the secondary modified ITT population explored the consistency of findings across analysis populations for the primary outcome.

Recruitment

A total of 1241 randomisations were carried out across two recruitment phases (see Figure 9): 967 (77.9%) from 10 to 30 April 2019, including the randomisations during the contamination period; and a further 274 (22.1%) from 5 September to 18 October 2019.

The primary modified ITT population comprised 638 (51.4%) randomisations and excluded all 603 (48.6%) from the contamination period (Figure 8 and Table 11, for a detailed breakdown, see Table A.6). A larger proportion of randomisations occurred within the contamination period for the NDA, 380 (65.1%), than for the other NCAs. The secondary modified ITT population comprised 961 (77.4%) randomisations and excluded 280 (22.6%) where the questionnaire was completed in < 20 seconds.

FIGURE 8.

Experiment summary: participant flow.

Participation across audits in the primary and secondary modified ITT populations comprised, respectively, 204 (32%) and 457 (47.6%) randomisations in the NDA, 178 (27.9%) and 211 (22%) in MINAP, 118 (18.5%) and 130 (13.5%) in TARN, 102 (16%) and 129 (13.4%) in NCABT, and 36 (5.6%) and 34 (5.5%) in PICANet (Figure 9).

FIGURE 9.

Daily and cumulative online experiment recruitment.

A total of 617 (49.7%) randomisations were included in both the primary and secondary modified ITT populations, and 259 (20.9%) were excluded from both populations (Appendix 5, Table A.5).

Participant characteristics

Randomisation

Primary modified intention-to-treat population

A similar number and proportion of participants were randomised to each of the 32 combinations of modifications and, as a result, to each of the modifications (ON or OFF), both within and across audits (Figure 10, Appendix 5 ENACT Experiment Example Tables A.7–8). In the PICANet audit, owing to the small number of participants recruited and exclusions over the contamination period, only 6 of 32 combinations were replicated and there were six combinations with no participants.

FIGURE 10.

Randomly allocated modifications and modification combination by audit: (a) modification combination and (b) modification ON or OFF.

Experiment completion

Outcomes

Primary modified intention-to-treat population

The distribution of participants’ responses was variable across the outcomes; however, a negative skew, due to a ceiling effect for higher levels of agreement, is present across all outcomes (Figures 11 and 12). A greater skew with higher levels of agreement can be seen for the ‘want to ensure’ component of the primary intention outcome, for the proximal intention outcome to bring the audit report to the attention of colleagues, comprehension and user experience.

FIGURE 11.

Distribution of primary intention outcome components.

FIGURE 12.

Distribution of secondary outcomes on a scale of –3, completely disagree, to +3, completely agree: (a) attention; (b) goals; (c) action plan; (d) review performance; (e) comprehension; and (f) user experience.

Responses for two user experience components, ‘information met needs’ and ‘report easy to use’, were fairly consistent, with a Cronbach’s alpha of 0.85 and correlation of 0.75, suggesting that there was adequate internal consistency and reliability across the components (Appendix 5, Figure 34).

Primary outcome

Responses for three primary components, ‘I intend’, ‘I want’ and ‘I expect’, were fairly consistent across all participants, with higher intention observed for ‘I want’ and a more uniform distribution of responses across the neutral and agreement end of the scale for ‘I intend’ and ‘I expect’ to ensure that the audit standard is met (see Figure 11). The Cronbach’s alpha value, 0.85, and pairwise correlation coefficients suggest that there was adequate internal consistency and reliability across the components (Figure 13).

FIGURE 13.

Internal consistency of primary outcome components. α, Cronbach’s alpha; r, pairwise Pearson correlation coefficient.

The distribution of the primary outcome (combined on a scale of –3, completely disagree, to +3, completely agree) by audit shows variability in responses across the audits, with a more similar distribution across NDA, TARN and MINAP, albeit with the greatest intention observed in MINAP (Table 14 and Figure 14). There were fewer participants reporting the highest levels of agreement in the NCABT audit and, owing to the low number of respondents from PICANet, the distribution is relatively sparse.

TABLE 14 - Summary statistics for the primary outcome by participant characteristics and randomised modifications

	N	Missing	Mean (SD)	Median (IQR)
Audit
MINAP	158	20	1.4 (1.59)	2.0 (0.3–2.7)
NCABT	93	9	0.9 (1.31)	1.0 (0.0–2.0)
NDA	172	32	1.3 (1.30)	1.7 (0.3–2.0)
PICANet	33	3	1.6 (1.68)	2.0 (1.0–2.7)
TARN	110	8	1.5 (1.28)	1.8 (0.7–2.3)
Total	566	72	1.3 (1.41)	1.7 (0.3–2.3)
Role
Clinical	326	26	1.6 (1.29)	2.0 (1.0–2.3)
Manager	141	33	1.1 (1.28)	1.0 (0.0–2.0)
Audit and administration	99	13	0.9 (1.77)	1.0 (0.0–2.0)
Organisation
Commissioning	24	3	1.3 (1.62)	1.5 (1.0–2.3)
Community health-care trust	5	3	2.5 (0.77)	3.0 (2.0–3.0)
General practice	160	29	1.4 (1.24)	1.7 (0.3–2.0)
Hospital trust	377	37	1.3 (1.47)	1.7 (0.3–2.3)
Responsible for clinical care
Missing	0	72
Yes	394	0	1.6 (1.15)	2.0 (1.0–2.3)
No	172	0	0.6 (1.68)	0.7 (0.0–2.0)
Effective comparators (modification A)
ON	292	33	1.3 (1.41)	1.7 (0.3–2.3)
OFF	274	39	1.4 (1.42)	2.0 (0.3–2.3)
Multimodal feedback (modification B)
ON	289	31	1.2 (1.51)	1.7 (0.3–2.3)
OFF	277	41	1.4 (1.31)	1.7 (0.7–2.3)
Specific actions (modification C)
ON	284	34	1.4 (1.37)	1.7 (0.7–2.3)
OFF	282	38	1.2 (1.45)	1.7 (0.3–2.0)
Optional detail (modification D)
ON	273	39	1.4 (1.39)	1.7 (0.7–2.3)
OFF	293	33	1.2 (1.43)	1.3 (0.3–2.3)
Patient voice (modification E)
ON	280	40	1.4 (1.30)	1.7 (0.3–2.3)
OFF	286	32	1.3 (1.51)	1.7 (0.3–2.3)
Cognitive load (modification F)
ON	278	39	1.3 (1.41)	1.7 (0.3–2.3)
OFF	288	33	1.3 (1.42)	1.7 (0.3–2.3)
Number of modifications ON
0	18	3	1.1 (1.46)	1.3 (0.3–2.0)
2	264	32	1.3 (1.45)	1.7 (0.3–2.3)
4	268	35	1.4 (1.37)	1.7 (0.7–2.3)
6	16	2	1.3 (1.48)	1.8 (0.2–2.3)

FIGURE 14.

Distribution of the primary outcome by audit and role: (a) MINAP; (b) NCABT; (c) NDA; (d) PICANet; (e) TARN; (f) clinical; (g) manager; and (h) audit and administration.

Clinical participants reported higher levels of intention than non-clinical participants in manager or audit and administration roles (see Table 14 and Figure 14).

The distribution of the primary outcome varied across the versions of the modifications (Figure 15; see also Table 14). The greatest differences in the median level of intention were observed for the modifications ‘effective comparators’ (modification A, with lower median intention when a more specific comparator was used) and ‘optional detail’ (modification D, with higher intention when optional details were provided).

FIGURE 15.

Box plot of the primary intention outcome by modifications, indicating the mean (+), median, upper and lower quartiles (25th/75th percentiles), minimum and maximum values [most extreme value within 1.5 (IQR) of the 25th/75th percentile] and outliers (*).

Primary intention-to-treat analysis: model selection

Full details of model building based on available complete data and applied within the primary modified ITT population are available in Appendix 6.

Residual diagnostic plots (Appendix 6, Figure 46) indicated that, given the nature of the questionnaire scales, residuals across models based on available complete data were sufficiently normally distributed and homoscedastic with respect to fitted values to retain continuous outcomes.

Pareto plots presenting the relative magnitude and direction of standardised effects for parameters in the ‘full’ model are provided for the primary outcome and primary modified ITT population in Figure 16 and in Appendix 6, Figure 37, using the complete available data. No modifications were found to have an independent overall effect on the primary outcome intended enactment (see Figure 16); however, important interactions were identified across modifications and according to role. Audit and role had the greatest dominant influence on intention, followed by the interaction between multimodal feedback and optional detail (B*D) (see Figure 16). Less of a distinction was observed between subsequent effects with the greatest influence. These included some evidence for an effect according to multimodal feedback dependent on role (B*Role); dependent effects of effective comparators, multimodal feedback, and patient voice (A*B*E); optional detail and specific actions, both dependent on cognitive load (D*F, C*F); and only very limited evidence for an effect according to optional detail dependent on role (D*Role).

FIGURE 16.

Primary outcome: Pareto plot of standardised effects (full model).

Pareto plots and details of model building for secondary outcomes are available in Appendix 6 (Figure 40 and Tables 41–52).

Primary intention-to-treat analysis: model effects

Primary outcome

No modifications were found to have an independent effect on intended enactment (see Table 15). Audit and role had the greatest influence on intention, followed by a role-dependent interaction between multimodal feedback and optional detail (see Figure 16).

Intended enactment was lower in non-clinical participants than in clinical participants (–0.867, p < 0.001) in the NDA, with similar effects observed within MINAP, PICANet and TARN. This effect was not observed within NCABT, as intention was lower in clinical NCABT participants than in clinical participants in the NDA (–0.893, p < .001) and other audits (see Figure 18).

There was good evidence of an antagonistic interaction between multimodal feedback and optional detail (B*D, –0.112; p = 0.047); intention was reduced when both multimodal feedback and optional detail were applied (or not) in synergy, and improved when only one or the other was applied (see Figure 21). In non-clinical participants there was some evidence of a negative effect of multimodal feedback (B*Non-clinical, –0.196; p = 0.083), and a positive effect of optional detail (D*Non-clinical, 0.195; p = 0.087), with intention optimised when multimodal feedback was excluded and optional detail provided.

Predicted intended enactment of all 64 possible combinations of modifications ordered according to predicted intended enactment in clinical recipients is presented in Figure 17. As no interaction was detected between audit and modifications, the relative magnitude of effects of modification combinations is consistent across audits and predicted intended enactment is presented for the reference NDA only. Owing to interactions between participants’ role with multimodal feedback (B*Role) and optional detail (D*Role), the order of modification combinations differs for clinical (see Figure 17) and non-clinical recipients (Appendix 6, Figure 38).

FIGURE 17.

Predicted intended enactment for modification combinations ordered by recipients: (a) clinical; and (b) non-clinical.

The most effective modification combination in clinical recipients was multimodal feedback, specific actions, patient voice and reduced extraneous cognitive load (BDEF), with predicted intended enactment of 2.40 (95% CI 1.88 to 2.93) in the NDA (Table 16). However, including multimodal feedback while reducing extraneous cognitive load also resulted in the least effective combination of modifications when optional detail was provided (BDF), with predicted intended enactment of 1.22 (95% CI 0.72 to 1.72).

Audit and role

The predicted primary and proximal intention outcomes across audits and recipients role, averaged across modifications, are presented in Figure 18. Comprehension and user experience are not presented, as there was no evidence of an effect according to audit or role for these outcomes.

FIGURE 18.

Predicted agreement by audit and role for primary outcome intended enactment and secondary proximal intention outcomes: (a) intended enactment; (b) attention; (c) set goals; (d) action plan; and (e) review performance.

Intention for the primary outcome, intended enactment, was lower among non-clinical participants than among clinical participants (–0.867; p < 0.001), as was intention for the secondary outcomes of intention to formulate an action plan (–0.571; p = 0.015) and intention to set goals (–0.489; p = 0.03) in NDA, PICANet and TARN. There was reduced intention for the secondary outcome to review performance (–0.29; p = 0.043) for non-clinical recipients across all audits. There was no evidence of a difference in intention to bring the audit to the attention of colleagues according to role.

Among clinical participants, intention for the primary outcome, intended enactment, was lower in NCABT than in the NDA (–0.893; p < 0.001), as was intention for the secondary outcome intention of review performance (–0.565; p = 0.008). In the same group, intention was lower in both the MINAP and NCABT than in the NDA for the secondary outcomes intention to formulate an action plan (MINAP –0.482; p = 0.049; NCABT –0.603; p = 0.011) and intention to set goals (MINAP –0.445; p = 0.069; NCABT –0.638; p = 0.006). However, intention to bring the audit report to the attention of colleagues was higher in MINAP (0.521; p < 0.001), PICANet (0.755; p = 0.002) and TARN (0.304; p = 0.05) than in the NDA.

Effective comparators (modification A)

Using a comparator aiming to reinforce desired behaviour change, which showed recipient performance against the top quarter of performers, had no or minimal effect on the primary outcome, intended enactment and secondary proximal intention outcomes, including intention to bring the audit to the attention of colleagues, set goals, formulate an action plan and review performance.

On average, this modification did, however, reduce how easily participants understood the audit report (A, –0.091; p = 0.029) and their overall user experience (A, –0.087; p = 0.036). Synergistic interactions between the comparator and optional detail for these outcomes, comprehension (A*D, 0.078; p = 0.068) and overall user experience (A*D, 0.085; p = 0.043), meant that the negative comparator effect was not present when optional detail was provided (Figure 19).

FIGURE 19.

Predicted agreement by effective comparators and optional detail (A*D) for comprehension and user experience outcomes in NDA clinical recipients: (a) comprehension; and (b) user experience.

There was also weak evidence of a synergistic interaction with multimodal feedback (modification B) on intention to review performance (A*B, 0.115; p = 0.089), which suggested that multimodal feedback reduced intention unless accompanied by the effective comparators, but with optimal intention when neither modification was applied (Figure 20).

FIGURE 20.

Predicted agreement by effective comparators and multimodal feedback (A*B) for intention to review performance in NDA clinical recipients.

Higher-order three-way interactions detected between effective comparators, multimodal feedback and patient voice (A*B*E) and between effective comparators, specific actions and cognitive load (A*C*F) are described later, as are all other modification three-way interactions.

Multimodal feedback (modification B)

Using multimodal feedback to reinforce desired behaviour change, by including a graphical display of performance data alongside text, did not independently improve the primary outcome, intended enactment or the secondary outcomes of proximal intention, comprehension or user experience. Multimodal feedback did, however, interact with a number of other modifications, leading to variable effects on all outcomes.

A consistent antagonistic interaction was observed between multimodal feedback and optional detail, with good evidence of effect for the primary outcome, intended enactment (B*D, –0.112; p = 0.048), and secondary outcomes of user experience (B*D, –0.115; p = 0.006) and comprehension (B*D, –0.114; p = 0.008); and weak evidence for the secondary outcome of intention to set goals (B*D, –0.114; p = 0.067) (Figure 21). Outcomes were generally reduced when both multimodal feedback and optional detail were applied (or not) in synergy, and improved when only one of the modifications was applied. However, to improve the primary outcome, intended enactment, further interactions between multimodal feedback, optional detail and role (B*Non-clinical, –0.196; p = 0.083; D*Non-clinical, 0.195; p = 0.087) meant the effect was quite different in non-clinical participants than in clinical participants, and intention was optimised when multimodal feedback was excluded and optional detail provided (see Figure 21). There was also a positive interaction between multimodal feedback and the PICANet audit (B*PICANet, 0.607; p = 0.035), such that intention to set goals was improved when multimodal feedback was provided, irrespective of optional detail.

FIGURE 21.

Predicted agreement by multimodal feedback and optional detail (B*D) for outcomes: (a) intended enactment – clinical; (b) intended enactment – non-clinical; (c) comprehension – clinical and non-clinical; (d) user experience – clinical and non-clinical; (e) set goal – MINAP; (f) set goal – NCABT; (g) set goal – NDA; (h) set goal – PICANet; and (i) set goal – TARN.

There was a synergistic interaction between multimodal feedback and cognitive load for secondary outcomes intention to bring the audit report to the attention of colleagues (F, 0.126; p = 0.016; B*F, 0.087; p = 0.089) and to set goals in the PICANet audit (B*F*PICANet, 0.569; p = 0.046), such that intention was optimised when cognitive load was reduced and multimodal feedback was provided (Figure 22). An opposite antagonistic interaction was, however, observed for intention to review performance (B*F, –0.119; p = 0.085) and intention to set goals in the MINAP audit (B*F*MNAP, –0.4; p = 0.013) such that intention was optimised when cognitive load was reduced and multimodal feedback was not provided.

FIGURE 22.

Predicted agreement by multimodal feedback and cognitive load (B*F) for intention to bring audit report to attention of colleagues and to review performance in NDA clinical recipients and intention to set goals accounting for B*F*audit interaction in clinical recipients: (a) attention; (b) review performance; (c) set goal – MINAP; (d) set goal – NCABT; (e) set goal – NDA; (f) set goal – PICANet; and (g) set goal – TARN.

Specific actions (modification C)

Using specific actions to reinforce desired behaviour change, by including suggested specific recommendations, had no or minimal effect on secondary outcomes intention to bring the audit to the attention of colleagues and to set goals, or on comprehension.

Using specific actions to reinforce desired behaviour change improved the secondary outcomes intention to set an action plan and review performance, dependent on the inclusion of optional detail (modification D). The interaction between modifications was antagonistic for both intention to set an action plan (C*D, –0.107; p = 0.099) and intention to review performance (C, 0.118; p = 0.075; C*D, –0.119; p = 0.078), such that there was improved intention when specific actions were provided, but only when optional detail was not also provided (Figure 23). A weak synergistic interaction was also identified between specific actions and cognitive load (C*F, 0.093; p = 0.09) (Figure 24), for the primary outcome, such that intended enactment was optimised when both specific actions and reduced cognitive load were applied.

FIGURE 23.

Predicted agreement by specific actions and optional detail (C*D) for intention to make an action plan and review performance in NDA clinical recipients: (a) action plan; and (b) review performance.

FIGURE 24.

Predicted agreement by specific actions and cognitive load (C*F) for intended enactment in NDA clinical recipients.

Optional detail (modification D)

Adding optional detail to reinforce desired behaviour change, by including short messages with clickable, expanding links to explanatory detail, interacted with other modifications and participant role, resulting in effects on the primary outcome, intended enactment; secondary outcomes intention to set goals, make an action plan and review performance; and comprehension and user experience. Optional detail had no effect on the secondary outcome of intention to bring the audit to the attention of colleagues.

A weak but consistent antagonistic interaction was observed between optional detail and cognitive load for the primary outcome, intended enactment (D*F, –0.093; p = 0.089); intention to review performance (D*F, –0.123; p = 0.065); and comprehension (D*F, –0.073; p = 0.079) (Figure 25). Outcomes were generally reduced when both optional detail and reduced cognitive load were applied (or not) in synergy, whereas outcomes were improved when only one of the modifications was applied. To improve comprehension it was better to reduce cognitive load rather than include optional detail. To improve the primary outcome, intended enactment, a further interaction between optional detail and role (D*Non-clinical, 0.195; p = 0.087) meant that the effect was quite different in non-clinical participants for whom intention was optimised when optional detail was provided and cognitive load was not reduced.

FIGURE 25.

Predicted agreement by optional detail and cognitive load (D*F) in the NDA for intended enactment, accounting for D*Role interaction; intention to review performance in clinical recipients; and comprehension across recipients: (a) intended enactment – clinical; (b) intended enactment – non-clinical; (c) review performance – clinical; and (d) comprehension – clinical and non-clinical.

There was good evidence of a synergistic interaction between optional detail and patient voice such that including optional detail improved user experience when patient voice was included in the audit report, but including patient voice without optional detail reduced user experience (D*E, 0.085; p = 0.039) (Figure 26).

FIGURE 26.

Predicted agreement by optional detail and patient voice (D*E) in NDA clinical recipients for user experience.

Interactions between optional detail and effective comparators; multimodal feedback and specific actions for the primary outcome, intended enactment (B*D, –0.112; p = 0.047); the secondary outcomes intention to set goals (B*D, –0.114; p = 0.067), intention to set an action plan (C*D, –0.107; p = 0.099) and intention to review performance (C, 0.118; p = 0.075; C*D –0.119; p = 0.078); comprehension (A*D, 0.078; p = 0.068; B*D, –0.114; p = 0.008) and user experience (A*D, 0.085; p = 0.043; B*D, –0.115; p = 0.006) have been described previously.

Patient voice (modification E)

Using the patient voice to reinforce desired behaviour change by including a quotation and photograph from a fictional patient describing their experience of care related to the associated audit standard had no or minimal effect on the primary outcome, intended enactment, secondary proximal intention outcomes or comprehension.

Good evidence, however, of a synergistic interaction between patient voice and optional detail on user experience (D*E, 0.085; p = 0.039) has been described previously (see Optional detail).

Cognitive load (modification F)

The reduction of cognitive load, to reinforce desired behaviour change by removing distracting detail and additional general text not directly related to the audit standard and feedback, interacted with other modifications and audit, resulting in effects on the primary outcome, intended enactment; the secondary outcomes of intention to bring the audit to the attention of colleagues, set goals, make an action plan and review performance; and comprehension. Cognitive load had no effect on the secondary outcome user experience.

There was good evidence that intention to set goals (F*TARN, 0.357; p = 0.049) and intention to make an action plan (F*TARN, 0.407; p = 0.033) were improved when cognitive load was reduced for participants in the TARN audit (Figure 27).

FIGURE 27.

Predicted agreement by cognitive load and audit in clinical recipients for intention to set goals, accounting for B*F interaction (multimodal feedback*cognitive load) and intention to set an action plan: (a) set goals – multimodal feedback OFF; (b) set goals – multimodal feedback ON; and (c) action plan – multimodal feedback ON or OFF.

Interactions between cognitive load and other modifications are previously described. These include the interactions between cognitive load and:

• specific actions with an effect on the primary outcome (C*F 0.093; p = 0.09);

• multimodal feedback with an effect on intention to

  • bring the audit to the attention of colleagues (F 0.126, p = 0.016; B*F 0.087, p = 0.089)

  • review performance (B*F –0.119; p = 0.085),

  • set goals in the MINAP audit (B*F*MINAP –0.4; p = 0.013)

  • make an action plan in the PICANET audit (B*F*PICANet 0.569; p = 0.046

• optional detail with an effect on comprehension (F 0.103, p = 0.014; D*F –0.073, p = 0.079).

Three-way modification interactions

There was consistent weak to good evidence of a negative three-way interaction between effective comparators, multimodal feedback and patient voice across the primary outcome, intended enactment (A*B*E, –0.101; p = 0.072), intention to set goals (A*B*E, –0.121; p = 0.053) and to formulate an action plan (A*B*E, –0.137; p = 0.033). There was also good evidence of a positive three-way interaction between effective comparators, specific actions and cognitive load on the secondary outcome intention to formulate an action plan (A*C*F, 0.159; p = 0.015).

Owing to our fractional factorial design, three-way modification interactions were confounded such that the effect of A*B*E was confounded with C*D*F, and the effect of A*C*F was confounded with B*D*E (see Box 3). As a result, the identification of the interaction between effective comparators, multimodal feedback and patient voice (A*B*E) may reflect an interaction between the confounded modifications of specific actions, optional detail and cognitive load (C*D*F). Similarly, the identification of the interaction between effective comparators, specific actions and cognitive load (A*C*F) may reflect an interaction between the confounded modifications of multimodal feedback, optional detail and patient voice (B*D*E).

Sensitivity analysis

Sensitivity analysis used available complete data for the secondary modified ITT population to explore the consistency of findings across analysis populations for the primary outcome. Comparative Pareto plots for parameters in the full models for each population are provided in Appendix 6, Figure 39, and estimates for parameters in the ‘parsimonious’ models for each population are provided in Table 17.

Intended enactment is dominated by the effects of audit and role to a greater degree in the secondary modified ITT population (see Figure 50) than in the primary modified ITT population (Appendix 6, Figure 39), resulting in little relative input of modifications and no clear point at which to distinguish important modification effects.

Nevertheless, the modifications with the greatest influence and good evidence of an effect (p < 0.05) (see Table 17) include effective comparators (modification A) by audit (see Figure 51); the interaction between optional detail and cognitive load (D*F; see Figure 52); effective comparators, multimodal feedback and optional detail (A*B*D); multimodal feedback and optional detail (B*D; see Figure 53); and multimodal feedback and cognitive load by role (B*Role, F*Role).

Analysis of the primary modified ITT population did not detect an effect for effective comparators (modification A); the interaction between effective comparators, multimodal feedback and optional detail (A*B*D); or cognitive load by role (F*Role). There was also only some evidence of an interaction between optional detail and cognitive load (D*F; p < 0.1), rather than the stronger effect detected in the secondary modified ITT population; however, the magnitude and direction of effect were similar and this difference was probably due to the increased power that the larger sample size provided.

Analysis of the primary modified ITT population detected effects for optional detail by role (D*Role), and an interaction between effective comparators, multimodal feedback and patient voice (A*B*E); however, these effects were not detected within the secondary modified ITT population. There was also only very limited evidence of an effect for the interaction between specific actions and cognitive load (C*F; p < 0.15) (see Figure 54) within the secondary modified ITT population.

Secondary outcome: user engagement

User engagement measured through the length of time (in seconds) spent working through and the number of ‘clicks’ on the audit report excerpt showed a variable but skewed distribution across modifications and audits (Figure 28 and Appendix 7, Figure 55). Median time spent on the audit tended to be higher when each of the modifications were on, with the exception of reduced cognitive load. The greatest distinction in the number of clicks on the audit report was seen for the provision of optional detail with a greater number of clicks observed when this modification was ON than when it was OFF.

FIGURE 28.

Boxplot of user engagement – time on audit report – by (a) modification and (b) audit. Fourteen extreme outliers are not displayed.

Interpretation

There is accumulated evidence that different ways of providing feedback can change clinical practice. 11 There are many potential modifications to feedback and ways of combining them, and a limited understanding of how to optimise these. Our online screening experiment was designed to understand how modifications work, both overall and in combination, to robustly and efficiently identify optimal combinations of modifications that may have the most promising effects when subsequently applied to ‘real-world’ practice.

We found that the factors with the greatest influence on intended enactment of audit standards were audit and role (see Figures 16 and 18). The reasons for variations in intention by audit are not immediately apparent, but may be indicative of differences in audit and specialty engagement with the audit programmes. For example, there was a trend towards higher intended enactment in PICANet; this is a highly specialised audit with a relatively small number of participating sites and may, therefore, be a more cohesive, engaged and responsive network than other national audits. With the exceptions of MINAP and NCABT, intended enactment was generally higher for clinical than managerial, audit or administrative roles. This may be expected given that clinicians often have more direct influence over achievement of audit standards and this is consistent with one of the hypotheses from CP-FIT: that feedback recipients will act on goals perceived as relevant to their jobs. 27

We observed synergistic and antagonistic effects of different combinations of feedback modifications and participant roles. Some of these interactions may have intuitive explanations. For example, both providing optional detail and multimodal feedback entail giving additional information to audit recipients and, consequently, combining them means that their overlapping functions led to the result (intended enactment) being less than the sum of their parts (i.e. an antagonistic interaction). We found similar, if weaker, evidence for an antagonistic interaction between optional detail and cognitive load; in this case, providing optional detail might be expected to counter any benefits of reduced cognitive load given that additional information might have distracted participants. We also found weak evidence of a synergistic interaction between recommending specific actions and reducing cognitive load; both of these may have worked together and focused participants to act on audit standards. However, other interactions were difficult to explain and sometimes counterintuitive, such as both synergistic and antagonistic interactions between multimodal feedback and cognitive load for different outcomes, audits and roles.

Such a range of findings reflect the exploratory nature of this screening experiment, which aimed to detect the most promising signals of effects. It is notable that we did not find evidence supporting several hypotheses concerning the effects of single feedback modifications on recipient intentions, as predicted by the theoretical literature. 19,27 We were, uniquely, able to explore interactions between modifications and demonstrate the interdependence of feedback hypotheses. Considering potential real-world application, we would base our advice on interactions with the strongest evidence (reflected in Implications for practice).

We interpret these findings further, considering (1) design of the online feedback modifications, (2) participant engagement with the online feedback modifications and (3) sensitivity and ceiling effects of intended enactment and proximal intention.

First, we prioritised and deliberately selected feedback modifications that would be relatively amenable to online experimentation and could be operationalised with sufficient fidelity to the original 15 suggestions for effective feedback. 19 Modifications were developed through UCD to understand user preferences and comprehension. We presented the feedback modifications within credible, if hypothetical, excerpts of audit reports. Nevertheless, where anticipated modification effects were not detected, we must consider whether or not the online feedback modifications were optimised to deliver a sufficient ‘dose’ to bring about changes in recipient responses and use these findings to pinpoint where additional work is needed to ‘enhance’ modifications. One case in point is multimodal feedback; although the Cochrane review11 indicated that feedback may be more effective when it combines both written and verbal communication, we were able to operationalise this modification only by adding graphical information to textual information. Another is that specific actions did not feature among the most dominant effects on intentions to set goals, as may have been expected.

Second, considering participant engagement, we noted that 11.3% of participants (most commonly managers) dropped out of the experiment prior to questionnaire completion. This suggests a modest degree of self-selection, so that those who completed the experiment may have perceived it as more relevant to their roles than those who did not. Most of our feedback modifications were designed with an assumption that limited exposures could influence responses (e.g. use of different comparators, recommending specific actions). We had originally estimated a completion time of 20–25 minutes for the audit excerpt and survey; participants in the primary analysis spent a much lower median time of < 5 minutes on the experiment. Specifically, they spent a median of 66.5 seconds (IQR 31–136 seconds) on the audit report excerpt, with NDA participants (mostly general practice staff) spending just over half as long as all others on the audit report (see Table 32). Although these short durations reflect only limited engagement, it is not known how long feedback recipients would typically spend examining feedback in actual practice settings; it may still be relatively brief given competing pressures on attention. Therefore, this aspect of our experiment may still have reasonable external validity, especially given that much NCA feedback is already presented online or sent electronically.

Third, we set out to design a screening experiment that would be relatively sensitive in detecting changes in proximal outcomes of behaviour change, primarily intended enactment of audit standards. We would expect some attenuation of any effect on intended enactment when the feedback modifications are applied in ‘real-world’ practice, largely because of many post-intentional influences on practice (e.g. time and resource constraints). Further considering possible ceiling effects, we had anticipated that the primary and secondary outcomes measuring intentions would exhibit skew towards higher intention, partly because of social desirability bias (i.e. participants would be reluctant to state that they did not want to perform recommended practice). We therefore attempted to neutralise some of this bias by offering statements that recognised that participants would have competing priorities in normal practice. However, our intention measures were still generally skewed towards higher intentions, which imposed a ceiling effect on our ability to detect change.

Comparison with existing literature

We briefly compare our methods and findings with those of similar studies modelling interventions to change professional practice. Three have assessed feedback interventions. In the first, a factorial trial, Bonetti et al. 35 randomised GPs twice to receive or not receive A&F and educational reminder messages regarding ordering of lumbar spine X-rays. Both paper-based interventions changed simulated behaviour (responses to vignettes), but not behavioural intention. The effect of A&F on simulated behaviour was mediated through perceived behavioural control. This finding is noteworthy given that we found the role of the recipient was associated with intended enactment; clinicians may have considered that they had greater control over their ability to implement audit standards than those in other roles. In the second study, Gude et al. 95 conducted an online before–after study with intensive care staff to assess the effect of feedback on intentions to change pain management across a set of indicators. Intentions to improve practice, consistent with actual gaps in performance, improved modestly following feedback. In the third study, Gude et al. 96 conducted an online before–after study to assess whether or not feedback changed intentions of cardiac rehabilitation staff to adhere to performance indicators. Intention outcomes were assessed according to whether or not an indicator was selected for improvement. Lower performance at baseline was associated with greater intentions to change practice after feedback.

Two further studies modelled different interventions targeting professional behaviour. Eccles et al. 36 randomised older people’s mental health teams to one of three paper-based interventions incorporating persuasive messaging to promote the disclosure of a diagnosis of dementia or to no intervention (control). 36 There were no significant differences in intention or simulated behaviour between the trial groups. Explanations considered for this null finding included that the paper-based self-administered nature of the intervention was not ‘potent’ enough and ceiling effects on intention. Bonetti et al. 70 randomised dentists to a paper-based intervention, rehearsing alternative actions, to change intention to implement evidence-based practice for third molar management or to no intervention (control). 70 This intervention significantly influenced intention to extract third molars in line with recommended practice.

Together, these studies found variable effects of simulated interventions on professionals’ behavioural intentions. These variable effects may, in part, reflect differences in how intentions were assessed.

Strengths and limitations

We have conducted a unique experiment in collaboration with five NCA programmes. Our fractional factorial randomised design increased confidence in the internal validity of our findings and allowed evaluation of a set of feedback modifications, in terms of both their individual average effects and in combination by exploring interactions.

The five NCAs provided diversity in audit methods, topics and targeted audiences, thereby increasing confidence that our findings are relevant to a wider range of clinical audits.

We were able to efficiently and simultaneously investigate the effects of six different modifications using a well-powered randomised design. Our primary and secondary analysis populations exceeded our sample size requirements of 500 participants, providing > 90% power to detect small to moderate main and interaction effects for each modification.

Our fractional factorial design enabled us to provide direct information on which modifications provide an active effect and whether or not there was any interaction between modifications, providing the gold-standard approach for developing multicomponent interventions. 78 Indeed, we identified multiple synergistic and antagonistic effects between modifications, which would not have been feasible within the standard parallel-designs framework for each modification or even particular combinations of modifications.

We highlight three limitations. First, as we have emphasised earlier, this was a screening experiment using proxy outcomes theorised to influence actual clinical behaviour. We have, therefore, identified feedback modifications and combinations of modifications worthy of further real-world evaluation in pragmatic trials before findings can directly inform widespread adoption in NCA programmes. However, our findings suggest that there are some practical considerations for the design and delivery of A&F programmes.

Second, we evaluated the effects of the feedback modifications on different audit standards across the five NCAs. These audit standards may have been of varying quality and familiarity with audit teams. We could not separate the effects of feedback modifications related to the audit standards chosen for the experiment from wider differences in the national audits and their participants. The levels of agreement with outcome statements related to audit standards and the distribution of agreement may have varied across outcomes according to the level of commitment or effort required to achieve each target audit standard. Indeed, outcomes related to audit standards and intention varied considerably by audit and role, whereas comprehension and user experience did not.

Third, the integrity of the experiment was threatened by a significant number of duplicative responses by a single participant. As a consequence of designing our experiment to maintain participant anonymity of responses, we were unable to identify the duplicative participant within the experiment data. We therefore minimised the impact that duplicative responses had by removing all 603 (49%) participants who took part during the contamination period to ensure that only genuine, independent responses were included in the primary analysis. This approach simultaneously discarded unidentifiable genuine responses, representing a waste of research resources and participant time.

We conducted sensitivity analysis on a secondary modified sample, excluding only participants who spent < 20 seconds completing the experiment questionnaire. This resulted in far fewer exclusions (280, 23%) and a greater proportion of participants included from general practice. Sensitivity analysis of the primary outcome largely supported the modification effects identified; however, additional effects not detected in the primary analysis were identified, in part due to the increased sample size but also due to differences in participants and responses between the populations.

Implications for practice

Our findings have some general and specific implications for the design and delivery of NCAs. In general, although the six feedback modifications we evaluated did not improve intended enactment of audit standards in isolation, they did all have effects on a range of intended actions, comprehension and user experience. Indeed, the relative magnitude of predicted intention across possible combinations of modifications demonstrates interdependent effects of the modifications. For example, predicted intention in clinical participants was both optimised and reduced when multimodal feedback was provided and cognitive load reduced owing to the inclusion and exclusion of other modifications working with and against these two modifications.

Our results need to be contextualised in the wider theoretical and empirical literature on A&F,11,19 and indicate the need for careful consideration before adopting widespread changes. For example, we observed both synergistic and antagonistic effects when a number of modifications to feedback were combined. This suggests that NCA programmes need to explicitly consider how different features of their feedback are likely to act together.

Partly because of the nature of the online experiment, the modifications we examined are unlikely to require major resources for them to be implemented in NCA programmes. Most audit leads may wish to user-test any feedback changes prior to national roll out and this would probably be the main implementation cost.

We observed differences between national audits in intended enactment of the audit standards used in the experiment. Although we cannot specify which aspects of the audit standards, audit organisation or targeted recipients account for these variations, our findings suggest that NCA programmes should explicitly review the strengths and weaknesses in their whole audit cycles to identify priorities for change. 27 We also found that clinical recipients were more likely to report higher intended enactment of audit standards than managers, administrative and audit staff; we acknowledge that national audits may already face considerable challenges in ensuring that feedback is disseminated to those who are most likely to be able to act on it. 26

We make some specific suggestions based on modification effects supported by good or consistent evidence for the combined analysis of five audits:

• Using a comparator aiming to reinforce desired behaviour change (effective comparators), which shows recipient performance against the top quarter of performers rather than a comparison against overall mean performance, may reduce how easily recipients understand the audit report and their overall user experience unless accompanied by short, actionable messages with progressive disclosure of additional information (optional detail).

• Combining optional detail and a quotation and photograph from a fictional patient describing their experience of care related to the associated audit standard (patient voice) may improve recipient experience.

• Combining multimodal feedback with optional detail may reduce intentions to implement audit standards and set goals, and reduce comprehension and recipient experience.

• Many recipients may invest a relatively brief time in digesting feedback. Minimising cognitive load, by removing distracting detail and additional general text not directly related to the audit standard and feedback, may improve comprehension and, especially when combined with multimodal feedback, intention to bring the audit report to the attention of colleagues.

• The variations in effects by national audit programme suggest that such programmes can learn from one another in comparing how they design and deliver both audit and feedback methods.

Implications for research

Online screening experiments have an appeal in their potential to identify promising intervention components with relative efficiency prior to scaled-up definitive evaluations. We highlight one theoretical and one practical lesson respectively from our experience.

First, it is worth considering whether or not intention is the most appropriate primary outcome to use in screening experiments of A&F. CP-FIT hypothesises that a number of other factors both upstream and downstream to intention affect the ability of A&F to change clinical behaviour. 27 Upstream influences include interactions with and perceptions and verifications of feedback data. We found that reducing cognitive load improved comprehension of data and increased the self-reported intention to bring audit findings to the attention of colleagues when accompanied by multimodal feedback. Therefore, any future experiments could use a wider range of outcomes that reflect different aspects of the whole audit cycle. 27

Second, we have documented the protocol violations, contamination period and measures to protect study integrity (Appendix 4). We offer some practical suggestions to reduce the risk of deliberate or inadvertent duplicated participation in online experiments offering incentives:

• Consider what is essential to meet ethical safeguards and data protection and whether or not there is a strong reason to remove linkage of personal and (non-sensitive) study data.

• Assess the balance between study security and ease of participation. Requesting limited personal information, such as e-mail addresses, at study entry may not have an obvious effect on response rates.

• Attempt to visualise problematic scenarios. Only one individual is necessary to exploit an existing vulnerability. Consider study vulnerabilities and how they might be exploited. Consider whether or not and how it would be possible to identify and exclude suspect data with confidence if a problem arose.

• Regularly monitor aspects of collected data. Simple checking of recruitment totals may mask problems until it is too late.

• Unless there is high confidence in study security, use manual rather than automated delivery of incentives.

• Ensure that at least one person can access study systems and extract detailed monitoring data. Rapid responses can be critical in damage limitation.

Summary

Overall, although we recognise the limitations of our online experiment, we have demonstrated that it is feasible to conduct a large-scale study across different audit programmes and health-care settings. Doing so at scale allows the efficient evaluation of multiple features of feedback interventions, including the critical detection of synergistic and antagonistic effects. Our demonstration and lessons learned should therefore be of interest to the wider, international research community.

Original study design

We had planned a qualitative case study of four localities, with two purposively selected sites for each audit. We had intended to maximise diversity by recruiting sites on the basis of previously documented high and low performance in relation to audit criteria. Performance was selected as a sampling criterion because feedback may be more effective when baseline performance is low. 11

Data collection was to follow the trail from the receipt of feedback data through to any actions taken to improve clinical care over a 6-month period. We would have collected data through interviews, meeting observation and document review in the four localities, interviewing clinicians, managers, board members and lead commissioners (nine interviews per locality; 36 interviews in total).

Through discussions with our partner NCA programmes from objective 1, we became aware that two were in the process of introducing changes to their feedback delivery. Both changes aligned with suggestions for effective feedback:19

• The NDA was increasing the frequency of data sharing with recipients. Its annual report was to be supplemented by quarterly data releases, a change that met the suggestion to provide feedback as soon as possible, particularly for outcomes with many patient cases (i.e. suggestion 5 in Box 1). We had been unable to operationalise this suggestion for effective feedback in our online experiment.

• TARN had developed a new tool, ‘TARN Analytics’. It allowed feedback recipients to examine data in greater depth and ‘drill down’ beyond the headline messages. This aligned with the suggestion that key feedback messages are supported by optional detail to enhance credibility (suggestion 12 in Box 1).

Clinical Performance Feedback Intervention Theory

Clinical Performance Feedback Intervention Theory (Figure 29) resulted from a metasynthesis of qualitative studies of A&F interventions. 27 The model builds upon 30 existing theories and frameworks of behaviour change, including feedback intervention theory,98 the Capability, Opportunity, Motivation and Behaviour (COM–B) model,99 and the Consolidated Framework for Implementation Research. 100 It proposes that effective A&F is a cyclical process consisting of goal setting and audit, feedback message production, perception and acceptance of feedback message, recipient desire and intention to respond, action (at both individual and organisational levels) and, ultimately, quality improvement. The cycle will be weakened, and progress halted, if any individual process fails.

FIGURE 29.

Clinical Performance Feedback Intervention Theory. The diagram presents the theory’s variables and explanatory mechanisms, and their influence on the feedback cycle. Reproduced with permission from Brown et al. 27 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure below includes minor additions and formatting changes to the original figure.

Reproduced with permission from Brown et al. 27 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure below includes minor additions and formatting changes to the original figure.

The authors offer 42 ‘high-confidence hypotheses’ based on the theory. These illustrate how CP-FIT variables and explanatory mechanisms influence the feedback cycle. These are all in the format of ‘Feedback interventions are more effective when . . .’. For example, for the variable, ‘importance’, the hypothesis is ‘. . . they focus on goals recipients believe to be meaningful and often do not happen in practice’.

Revised study

We halted the planned study in March 2020 following the emergence of the COVID-19 pandemic and the cessation of non-essential research in NHS settings. Consequently, we successfully adjusted our design to ensure that it could proceed with minimum disruption to the wider programme. Our revised study avoided any burden on the NHS and fully exploited the collective knowledge and experience of our project team and extended network of advisors and stakeholders. The audits, their modifications and the extent of their alignment with CP-FIT would all remain core elements of our study.

The revised study allowed us to partly address our original objective and make a contribution to the field. First, our revised sample incorporated a wider range of perspectives, reducing clinical and commissioning contributions by necessity and bringing different knowledge, experience and perspectives of A&F researchers, patient and public representatives, and clinicians with experience of managing and receiving audit in other clinical fields. Second, we delivered an innovative, rapid method of evaluating feedback against a comprehensive, integrated theory (CP-FIT).

We applied CP-FIT to identify strong and weak points in the A&F cycles for both the NDA and TARN and how those may be addressed by their intended innovations. Thus, we aimed to highlight good practice and identify evidence-based suggestions on how each audit could enhance its impact.

Design

This was a rapid, theory-informed interview study.

Participants

We invited our co-investigators, Project Steering Group members, PPI panel members, reference panel members (from objective 1) and selected nominees from TARN and the NDA to participate. We aimed for a sample of around 20 individuals, seeking representation across the various roles and backgrounds available.

Interview procedure

We sent participants a table of the CP-FIT feedback cycle processes prior to the discussion (Appendix 9). This was to improve familiarity with the theory, help to concentrate attention on specific feedback processes and frame discussion. We provided a question for each process to aid participant interpretation and understanding (e.g. for verification, the question was ‘Can the recipient interrogate the data?’).

Interviews lasted approximately 30–45 minutes and were conducted remotely. We structured our topic guide around CP-FIT and the individual processes of the feedback cycle. Some interviews covered the entire cycle, whereas others focused on specific processes, depending on interviewee experience and preferences. Where appropriate, follow-up questions were used to clarify understanding.

The first interviews were with audit managers from the NDA and TARN. These established understanding of how the audits worked, recent changes made and their reflections on the feedback cycles for their audit programmes. Their responses helped us to develop short summaries of both audits (Appendix 10) for subsequent participants who did not already possess a detailed knowledge of these audits.

We invited participants to discuss one or both of the audits, and to raise any issues related to the audit cycle that particularly interested them or that they wanted to discuss after reading the summaries. We also prompted participants on aspects of the audit cycle that had not been covered in other interviews. We were open to other findings that did not fit within the theory and disconfirming cases. Participants were given opportunities to indicate issues that did not clearly fall within CP-FIT. For instance, we asked ‘If you could change one thing about this audit, what would it be?’ or ‘Is there anything that you wish to discuss that has not been covered by the cycle?’. Interviews were audio-recorded and transcribed verbatim.

National Diabetes Audit

Trauma Audit Research Network

Comparison to existing literature

Brown et al. 27 specifically refer to how researchers can use CP-FIT to evaluate feedback interventions and explain their observed or predicted effects. In completing a rapid, framework-driven analysis, we have demonstrated an approach that others may use to evaluate other feedback interventions and identify both their strengths and weaknesses.

In our evaluation, we paid particular attention to two of the suggestions for effective feedback identified by Brehaut et al. 19 This work builds upon our online experiment (objective 1) to explore how two of the suggestions may work in practice and has highlighted some of the complexities involved. For example, releasing data more frequently is recommended, but our findings illustrate that this is likely to be insufficient to prompt clinical improvement on its own. Timely feedback is unlikely to offer any advantages if it struggles to reach its target audience and, if it does reach them, is considered incomprehensible.

Participants, particularly those with primary care roles, identified a potential conflict between the NDA and the performance management system for general practices, QOF. There are overlaps between the clinical standards included, and our results indicated that QOF was prioritised. Interventions that target processes of diabetes care must, therefore, account for QOF. We drew a similar conclusion in an earlier trial of an intervention including A&F that failed to improve diabetes management in primary care. 106 The associated process evaluation suggested that, although the intervention was aligned with existing priorities and initiatives, it was not sufficiently distinctive to stand out and engage clinical teams. Work around the monitoring and achievement of the QOF is often embedded within general practices and is therefore likely to be resistant to change. However, it is possible that making trend or benchmarking data more accessible within NDA outputs would offer practices valuable information that the QOF cannot provide.

Strengths and limitations

A particular strength of the study is our use of CP-FIT, enabling a state-of-the-science analysis of two NCAs. Furthermore, we drew upon wide-ranging participant experience of audit and research.

We acknowledge three main study limitations. First, this was a rapid study; we opted to modify our work in response to the COVID-19 pandemic. It was not possible to complete an in-depth, qualitative analysis of the interviews within our programme. However, rapid approaches to qualitative research are gaining recognition as a valid and viable methodology within a limited time frame. 101–103 They have been used successfully in other contexts, most recently in completing qualitative research into the COVID-19 pandemic. 103 This rapid approach therefore allowed us to identify meaningful, timely findings.

Second, pandemic restrictions limited our potential participant sample. In particular, we were prevented from approaching clinical teams, as per the original design. This was perhaps of greater significance regarding TARN: our wider network of collaborators included only one health professional with direct knowledge and experience of TARN outputs; by contrast, for the NDA, the sample included three GPs. Consequently, our analysis is weighted towards the perspectives of a single individual, although we have introduced external perspectives wherever relevant. A related issue is the risk of bias in our sample. We interviewed people with an interest in the use of A&F to improve health care. Further research with participants who are less interested in A&F and more typical of targeted recipients would be necessary before drawing firm conclusions from our results.

Third, several participants had no prior knowledge of both audits under consideration. To inform discussion we provided summaries of both audits and shared links to recent outputs. Most interviewees were experienced in A&F and familiar with the evidence around how to make it more effective, perhaps with more critical distance than those with direct stakes in both audits.

Implications for practice

Our findings have direct implications for both of the evaluated NCAs. We identified three strong features of their feedback likely to facilitate progress through the audit cycle, as well as three others requiring more attention to enhance feedback effectiveness. For the NDA, three strengths were:

1. The move to quarterly data release. The NDA’s recent move to increase the frequency of feedback was welcomed, appears to meet the needs of audit users and is in line with best evidence about improving feedback’s effectiveness. It is likely to have facilitated the acceptance and intention processes of the audit cycle.

2. Automated data collection. The routine extraction of data minimises the impact on primary care teams and helps to ensure a large and accurate data set. It also reduces complexity and strengthens the initial processes of the audit cycle.

3. Respected source and widely accepted indicators. The feedback is considered to come from a respected, credible source and the clinical indicators reported are recognised by users as relevant and concerning an important public health issue. There was no indication that users would question their worth, nor the need to monitor achievement.

We suggested three areas for further attention:

1. Delivery to target recipients. There were indications that the feedback is failing to reach much of its intended audience. The data may be publicly available, but our interviews suggested that GPs are often unaware of the feedback. Consequently, they will not discuss the data with their practice teams, and it is not surprising that the audit is not a prompt for clinical improvement. There were suggestions that the audit is overshadowed by QOF and has not done enough to define its own position and importance. Potential solutions might be to increase efforts to ‘push’ the information to CCGs and practices, and accompany this with messages highlighting the importance of the data (‘observability’).

2. Presentation. Following the cycle, when the feedback does reach its target audience, it is far from certain that it will be read and understood. Several interviewees commented on the impenetrability of the content and found the Microsoft Excel datasheet to be off-putting. Thus, consideration should be given to how the data might be made clearer for users with limited time and resources.

3. Interpretation. Although comparators are available within the feedback, participants felt that more could be done to make this process more straightforward and accessible. Inclusion of a meaningful comparator is recommended for effective feedback. In the feedback provided by the NDA, it was felt that too much was required of analysts and others skilled in manipulating data to produce the required comparators. Wherever possible, potential barriers to understanding should be removed, for example, providing more accessible comparator information [e.g. practice previous performance (trend), or performance of similar practices and CCGs (benchmarking)].

Similarly, for TARN, we identified three features consistent with best practice:

1. Enhanced interactivity. The new TARN Analytics tool was considered an important and useful addition, and aligned with recommended practice. The enhanced ability to interrogate and ‘drill down’ into the data (in some cases to the patient level) is likely to strengthen the verification and acceptance processes of the audit cycle.

2. Use of comparator. The presentation of comparative performance is consistent with evidence of how to enhance feedback effectiveness. Incorporating social influence helps to facilitate perception and can help to guide intention and behaviour.

3. Respected source. The feedback is considered to originate from a respected, credible source, and not just by those working within trauma. Audit commissioners and clinicians from other fields recognise TARN as an exemplar of good practice from which other audits could learn. A commitment to ensuring data accuracy and validity has contributed to this reputation.

We suggested three areas for further attention:

1. Action planning. Providing suggestions of specific actions to undertake to improve performance can enhance feedback effectiveness. Interviewees perceived the absence of such options in the TARN feedback, observing that merely showing achievement is not sufficient to bring about improvement. Recipients have the opportunity to explore a wealth of data, but action plans and guidance on what could be done to improve would probably prompt further achievement by making the feedback more actionable and facilitating intention and behaviour.

2. Greater flexibility around comparators. Although it remains uncertain whether or not particular types of comparator are more effective than others, providing a higher benchmark (e.g. the top quartile) could potentially serve as a greater spur for improvement than showing average national performance. This hypothesis is under investigation elsewhere,107 but our own objective 1 screening experiment failed to support it. Nevertheless, providing users with the ability to select their own comparators and benchmarks may be beneficial.

3. Evidence of impact. Participants generally believed that engagement with TARN had resulted in improvements to patient care. However, no specific examples of this were provided and participants who searched for such examples on the TARN website were unable to locate them. Clearer demonstration of the impact of TARN – including presenting this in a manner that is accessible to patients and the public – would likely strengthen the audit cycle, as well as support continued audit engagement and acceptance.

Clinical Performance Feedback Intervention Theory was developed with specific attention on feedback interventions in health care. We suggest that our findings are of interest to three stakeholder groups. First, we have identified aspects of audit programmes that attenuate the audit cycle, or even cause it to break down entirely. We gathered information from experts and audit recipients about how feedback innovations are perceived and acted on (or not). Some of our findings and lessons are likely to be applicable to other NCAs. Second, we have indicated features to consider in the external review and commissioning of national audit programmes, together with recommendations on how they might be made more effective. Third, patients’ groups and charities will be interested to learn how they might support audits and encourage improvement. We found demand for audits to make greater efforts to publicise the impact they had on health-care quality, including making this impact more accessible to the public.

Implications for research

We have demonstrated an approach to applying CP-FIT to two national audits. Further work could examine relationships between key features of A&F programmes, as delineated by CP-FIT, and their effects on clinical practice and outcomes.

Summary

To our knowledge, this is the first application of CP-FIT to evaluation of feedback delivered by individual NCAs. We considered two examples of feedback in relation to the audit cycle and associated high-confidence hypotheses. Such an application may be of value to other national audit programmes seeking ways to improve the impact that they have on patient care.

Participants

We used purposive and snowball sampling to achieve a heterogeneous sample of participants encompassing three groups: A&F researchers, members of clinical audit programme staff and recipients of feedback (typically health professionals). We deliberately included participants with little or no experience of embedded experiments because their expectations and concerns are important to understand when planning new programmes. We aimed for 10 participants from each group and identified them through a combination of our contacts in this field and existing networks. 116 Participants possessed varying experience of involvement in research evaluating A&F. Participant recruitment was not limited to the UK as we wished to generate lessons relevant to an international audience. Several participants had contributed to an international meeting to establish a network for A&F research (The 4th Annual International Symposium of Advancing the Science and Impact of Audit & Feedback, 23–4 May 2019, Amsterdam, the Netherlands). 117 The final sample total was guided by evidence of data saturation. 118

Development of interview schedule

We used a semistructured interview schedule (Appendix 11), informed by our previous work in this field and existing literature on evaluations of major initiatives involving research–practice partnerships. Our AFFINITIE programme24 identified several influences on the participation of audit programmes in research. These included alignment of timelines and human resources, ensuring secure sharing of data and negotiation of an understanding of equipoise in experimenting with enhanced feedback interventions. We also considered the lessons generated from other relevant initiatives, such as UK National Institute for Health and Care Research Collaborations for Leadership in Applied Health Research and Care and the former Veteran’s Administration Quality Enhancement Research Initiative (VA-QUERI). 119–121

Our interview schedule drew upon the theoretical domains framework (TDF). 122 The TDF synthesises 33 behaviour change theories into 14 domains representing a range of individual (e.g. knowledge, beliefs about consequences), sociocultural (e.g. social influences, role and identity) and environmental (e.g. context and resources) barriers to and enablers of behaviour change. Use of the TDF helped to ensure that we would comprehensively identify and explore the potentially amenable behavioural influences on the planning and conduct of A&F research. The interview schedule was refined following input from our programme reference group. This group included patient representation and members with experience in behavioural science, clinical work and management of NCAs.

We (SLA, TAW, SWo) pilot tested the interview schedule to ensure comprehensibility and answerability of the questions. We independently coded the data from three pilot interviews, assigning initial codes and then allocating each code to a single TDF domain. We reviewed coding for agreement to TDF domains and resolved any differences through discussion with Robbie Foy and Fabiana Lorencatto. No further changes were made to the interview schedule.

Data collection and analysis

We (SLA, TAW, SWo) conducted interviews face to face, by video or by telephone, according to participant preference, from May 2019 to October 2019. All interviews were audio-recorded and transcribed verbatim. Transcriptions were imported into NVivo 12 and de-identified. We analysed data by coding transcripts into theoretical domains in a recursive process. 123 Specifically, Sarah L Alderson, Thomas A Willis and Su Wood independently coded data from interviews they had conducted themselves and assigned initial codes before assigning each code to a theoretical domain. All codes within each domain were reviewed and differences were resolved through discussion (with RF and FL). Finally, we conducted further inductive analysis of coding to populate each theoretical domain with content that described participants’ beliefs and to check for any further beliefs not accounted for by the framework. This generated overarching themes that encapsulated participants’ experiences of barriers to and enablers of embedding A&F research in NCAs. We made no changes to the interview guide after piloting three interviews and analyses.

Resources

Interviewees who had roles within NCAs typically described their teams as small and under-resourced. This limited their ability to participate in A&F research (Environmental context and resources). Moreover, their practical and intellectual capacity was consumed by existing priorities (Memory, attention and decision-making processes). There was variation between different audit programmes’ funding models, with many staffed by volunteers (Beliefs about capabilities). Changes in practice (e.g. adjusting the format of reports) were often resource-intensive and there was a risk of overstretching the team in committing to new projects (Environmental context and resources and Emotion). Some audit programmes worked with clinicians and external parties for limited periods only, such as a single audit cycle, hindering the continuity necessary for research purposes (Environmental context and resources). In some cases, clinicians had a role in identifying audit criteria; this was often seen as an opportunity to further their own research and leadership profiles, rather than a chance to improve health care or audit programme effectiveness (Social and professional role and identity).

Audit staff often work to strict timelines for delivering feedback, and schedules are also informed by funding cycles. Therefore, collaborative research must understand and align with these constraints (Environmental context and resources). Research partners must be responsive to the needs of the collaborating audit programme (Skills) and additional costs associated with the research would likely need to be met by the research partner for collaboration to be possible (Environmental context and resources).

You are having to align fairly complex research governance processes with those external deadlines and that . . . is definitely a challenge!

P11, feedback researcher and audit staff

There’s a lot of audits that are running on a shoestring as well! So a lot of people that want to improve what they do, you know I’m talking about in terms of delivering their audit . . . but they’re running on a shoestring financially.

P15, audit staff

Logistics

Participants identified several logistical barriers to embedding research within audit programmes. Feedback researchers described how longer-term, programmatic funding was necessary to develop and test different ways of designing and delivering A&F over multiple audit cycles (Knowledge). However, there was a perception that this was not considered a ‘sexy’ topic, restricting the likelihood of obtaining research funding (Knowledge). Furthermore, the necessary research costs might be high, particularly with the involvement of a clinical trials unit (Environmental context and resources). Convincing funders of the need for research in this area was seen as one of the biggest barriers, despite the potential benefits for health-care quality (Goals and Environmental context and resources).

Issues around data quality were described by participants. One interviewee from a national audit programme explained how participating in embedded research had made the programme aware of their own data quality deficiencies (Social and professional role and identity). Some reported being aware of weaknesses and a lack of rigour in the processes of other audit programmes (Skills and Beliefs about capabilities). It was noted that there is variability in the credibility of the indicators used by audit programmes, and in the extent to which they are evidence based. In some cases, required standards had been set based on current achievement in care rather than an (non-existent) evidence base of best care. Some audit staff felt that further research in this area should be prioritised before embedding research within audits as any improvement shown would not necessarily equate to improved care (Goals).

Other logistical barriers included the constraints of the electronic health system used by health care providers (Environmental context and resources) and the risk of contamination between trial arms when participants share or compare feedback with one another (Social influences). Participants with a background in research considered organisational restructuring of health care, particularly UK primary care, to be a particular risk to trial feasibility (Environmental context and resources). To be successful, embedded research required stable organisations and relationships.

Ethics barriers were not considered an issue. Indeed, not conducting research in this context might be considered unethical, as changes were typically made to feedback methods without formal evaluation (Environmental context and resources). Several audit programmes had data-sharing agreements that permitted the use of data for research, although there was less certainty regarding data ownership in countries outside the UK (Environmental context and resources):

I don’t think any funders would consider it . . . maybe sexy, for instance? And might, well if not be aware on panels, I don’t think they often will be aware of . . . of the area . . .

P18, feedback researcher

No one knows what the, what a good induction rate is or a good elective caesarean section rate is so, . . . that’s quite tricky to then work with. Some would say well . . . the audit could pick one. But then I would anticipate we would have a lot of backlash from people. Some people would say you’re too high; some people, you’re too low and everything in-between.

P21, audit staff

Leadership

Feedback researchers and audit staff felt that the leadership skills of key individuals within audit programmes were very important to the potential success of a collaboration (Skills and Social and professional role and identity). An enthusiasm for research was necessary, together with an understanding of the equipoise surrounding the most effective design and methods in feeding back clinical performance to health-care providers (Knowledge). An ability to convince others of the importance of research was essential, particularly funders and key stakeholders (Optimism and Social and professional role and identity). Audit staff and health-care providers wanted reassurance that (assumed) beneficial content would not be removed, suggesting that there was a lack of equipoise in effective feedback design (Beliefs about consequences).

A personal interest in conducting embedded research was considered an important quality for audit leaders that would boost their team’s motivation towards the work. Team learning culture and inquisitiveness were thought to depend on their leader (Skills, Optimism and Social and professional role and identity). Health-care professionals felt encouraged to participate in feedback research that was led by an enthusiastic and respected audit programme team (Skills, Beliefs about capabilities and Social influences). The potential for conflict arose when audit staff believed recipients would not want to be in a control arm, given perceived reductions in the quality of feedback delivered (Beliefs about consequences). All participants described how a steer from lead commissioners and key local individuals may help to legitimise the research and increase involvement, bolstering the chances of successful collaboration (Skills and Social and professional role and identity):

So, you might have a clinical lead for [audit programme] locally in a trust who is loving a current data visualisation and, you know that person changes at exactly the time, you know you switch the visualisation; and the fact, the drop off isn’t that the visualisation has gone, the drop-off is that you’ve lost the key person locally.

P12, audit staff

They genuinely seem to have been interested and, and, and keen to learn from the findings.

P11, feedback researcher and audit staff

Relationships

Participants described how relationships and trust between audit programmes and researchers took time to develop (Reinforcement and Social influences). In particular, audit staff expressed a preference to begin with simple studies to establish benefits and procedures, rather than moving straight to a series of large-scale trials and jeopardising their ability to fulfil existing demands (Environmental context and resources, and Reinforcement).

Both feedback researchers and audit staff identified the need for diplomacy to maintain relationships when difficulties arose (Skills and Intention). There was also nervousness regarding loss of control; researchers wanted more control over data gathering and audit staff wanted to retain control over audit content (Beliefs about consequences and Memory, attention and decision-making processes). Patience was needed by all parties, particularly in the set-up phase (Skills and Social influences). Co-design and involving audit recipients in the research were identified as means of establishing trust between researchers, audit programmes and the health-care system (Goals and Social influences).

Participants were generally consistent in their view that audit programme involvement in embedded research should not be onerous, and balance was required between the ambitions of researchers and pragmatism (Beliefs about capabilities). If this was achieved, the ongoing partnership would be valuable and worthwhile (Reinforcement). All interviewees felt that the benefits of embedding experimentation with audit programmes outweighed the challenges (Intentions and Goals):

So the kind of skills I need are a bit of diplomacy, a bit of prompting, a bit of time management, a bit of sort of people management in that respect, but also the ability to step back and not say ‘this is what I think we should do’. Um, and then just tenacity er, in diplomacy again to smooth over some trouble waters, bits and pieces, keep going.

P6, audit staff

The clinical teams you know, may be interested in improvement, but often they’re interested in doing their clinical work and not being bothered too much. And the clinical audit leadership wants to demonstrate that the audit is, is worth it and that it’s producing value. So I can see that that would be one of the first tasks is umm, is reaching a shared understanding.

P17, feedback researcher

Perceived risks

The potential for negative unintended consequences was raised by audit staff and health-care professionals (Beliefs about consequences and Emotion). For example, failure of an experiment to show an improvement in audit programme effectiveness might jeopardise continued funding and recommissioning, as well as threaten job security of employed staff (Beliefs about consequences and Environmental context and resources). Some worried that a ‘ceiling effect’ on improvement would mean that trials would fail to show benefit.

There was also concern about the potential for damaging the relationship with an audit programme’s end-users in health-care delivery (Beliefs about consequences). It was feared that changing the format or design of a now-familiar audit would alienate recipients and lead to disengagement that would affect both the audit programme and experiment findings (Reinforcement and Beliefs about consequences). Audit staff in particular emphasised the importance of maintaining the audit programme ‘brand’ (Social influences). Most participants considered this type of research to be of low risk and low cost to health-care providers (Beliefs about consequences and Reinforcement); however, balancing the needs of all stakeholders was seen as a significant challenge (Social influences):

They were very open about being worried about what we would find. They have pressures of their own around . . . the commissioning of the audit and the reputation of their organisation.

P11, feedback researcher and audit staff

People get used to our reporting format. They get, they finally got, got that now! You know I understand what that’s showing me now! We go ‘Wee! We’ve changed it!, You know like, no, so what we might think is terribly good in their space, they might go ‘God, I don’t understand it now!’. You know back to square one!

P15, audit staff

Opportunities and benefits

All roles provided examples of how NCA programmes might benefit from embedded experimentation (Optimism and Beliefs about consequences). These included opportunities to gain new skills, new ideas and visions being shared with the audit team, as well as increased financial investment and new collaborations leading to further opportunities (Skills, Beliefs about consequences and Reinforcement).

It was considered that part of an audit programme’s role was to continually strive to increase its effectiveness. Thus, embedding research could be considered a strategic decision that would raise awareness of the programme, demonstrate to funders that improvement work was taking place, provide evidence of impact and help satisfy their objective of improving patient care (Social and professional role and identity and Goals).

All groups of participants saw a high potential for auditing to have an impact on health care and patient benefit, and considered audit programmes to be an underused research resource (Goals, Intentions and Beliefs about capabilities). Embedding research in existing structures was seen to be an efficient model of quality improvement and would enhance the evidence-base for optimising A&F (Reinforcement and Memory, attention and decision-making processes). Audit staff and health-care professionals were optimistic about the future of embedded experimentation and all groups expressed enthusiasm about being involved in such work (Goals, Emotion and Optimism):

When we retender for running the national clinical audits, it’s useful to have an evidence base on where we’re going for focus. We want to do lots of things, we’re limited in terms of capacity in what we can realistically implement. So knowing that we’re implementing something that’s going to make more of a difference and then have a knock on impact hopefully on patients.

P4, audit staff

It’s likely that the new discoveries are likely to plateau and really now the bigger challenges putting into effect the medicines and treatments that we know work . . . I think has gotta be the kind of highest priority really because there’s not really any point in developing new treatments if we’re not using the ones we’ve have currently as effectively as we could.

P23, audit staff and health-care professional

Comparison to existing literature

There is scarce literature on embedding research in implementation laboratory settings. The concept shares similarities with that of ‘learning health systems’, in which evidence is gathered from routine care and deployed quickly into practice via cycles of continuous learning and improvement. 124 Initial efforts to implement learning health systems have identified requirements for success that align with our findings: adequate funding, robust data systems, and an organisational culture that values quality improvement. 125 We suggest ensuring that audit programme partners are fully represented and involved from the initial stages of collaboration. This may help to increase the likelihood of success by helping to overcome institutional pressures in the design phase where audit programmes have considerable financial and organisational challenges.

Our research augments work on stakeholder perspectives regarding research–practice partnerships. Such collaborations may provide both structure and opportunity for developing a shared cognitive space around which collective action can be organised. A process of consensus building, although time-consuming, may support the alignment of shared priorities, formation of a trusting relationship through the relinquishing and sharing of power, and identification of the possible long-term benefits of embedded programmes of trials in quality improvement activities. 113 The barriers to achieving, and sustaining, a partnership are similar to those found in other contexts,114 with the need for appropriate structures (including leadership and establishing specific roles) and processes to facilitate optimal conditions for genuine and collaborative action. 113

Strengths and limitations

This is the first in-depth exploration of the feasibility issues, barriers and opportunities to embedding research within NCA programmes. Existing research has focused on the use of audit data in clinical research (e.g. epidemiological studies) or by clinicians to improve health care. 12 The qualitative nature of our study provides subjective evaluations of the potential impact of embedding research within audit programmes, but further work is necessary to objectively evaluate or observe research–practice partnerships to understand the associated challenges, risks and benefits more thoroughly. Our theory-guided approach allowed us to identify the cognitive, affective, social and environmental influences on behaviour and potential reasons for implementation problems in embedding research within NCA programmes.

We recognise several study limitations. The majority of health-care staff we recruited had some current or previous involvement with NCA programmes, potentially subjecting our findings to social desirability bias. Our links with the A&F ‘MetaLab’,110 an international collaboration, enabled us to leverage a reasonably diverse range of perspectives to draw upon international examples of embedded research. The health-care professionals we interviewed were mainly secondary care based. However, inclusion of staff from primary care, where audit programmes are limited, ensured that we captured insights from those not involved with audit programmes. All participant groups identified the same optimal conditions, potential risks and benefits, suggesting that when developing research–practice partnerships the majority of challenges are predictable and could be overcome through thoughtful planning and communication.

Implications for research, practice and policy

Implementation laboratories, in the form of embedded programmes of trials in NCA programmes, have been suggested as a means of enhancing the impact that A&F has and also producing generalisable knowledge about how to optimise effectiveness. 12,111 Sequential, head-to-head trials testing different feedback methods in a national audit programme provide a robust empirical driver for change. Modifications identified as more effective than the current standard become the new standard; those that are not are discarded. The testable recommendations for feedback modifications proposed by Brehaut et al. ,19 such as using a comparator based on the average or high performers, have minimal cost implications. However, our findings suggest that there are resource implications for audit programmes that are not currently met. Marginal gains in the effectiveness of A&F are likely to be worthwhile at a population level. Supported by adequate resources and funding, they would also be feasible to test within an implementation laboratory. 111

Our interviewees consistently stated that the benefits of embedding experimentation within audit programmes would outweigh the risks and challenges. We have demonstrated the willingness of audit staff and health-care professionals (i.e. the audit programme’s end-users) to participate in an implementation laboratory to enhance the effectiveness of A&F. Drawing on our findings, we have produced 10 ‘top tips’ to develop successful collaborations between researchers and audit programmes in a programme of trials (Table 21).

Summary

We applied the TDF to understand different perspectives of embedding experiments within national audit programmes. Overarching themes concerned resources, logistics, leadership, relationships, perceived risks, and opportunities and benefits. Considering these factors may help create the optimal conditions for sustainable collaboration between feedback researchers and clinical audit programmes.

Round 1

Controllable actions (M2) was not included.

Round 2 prototypes

We designed and tested two prototypes with content from a hypothetical NDA report to explore responses to different audit criteria wording using process and outcome standards. Processes were those standards that related to a clinical process of care likely to be within a feedback recipient’s control. This represented the ON version of the modification. Outcomes were those standards that were not directly within the control of the recipient (e.g. patient outcomes following a given treatment). This represented the OFF version of the modification (Table 22).

Round 2 findings

Participants struggled to make sense of the audit criteria statements in round 2. We observed confusion relating to the context and labelling of the criteria, and participants reported that they found both ON and OFF statements to be ambiguous.

We further encountered two practical problems with operationalising controllable actions (M2) in the online experiment:

• Paired process of care and outcome indicators were not available for all five NCA programmes participating in the experiment.

• Operationalising both process of care and outcome indicators would have prohibitively increased the complexity of content and programming for the online experiment (e.g. in requiring differently worded feedback excerpts and outcome measures).

We therefore dropped this modification later during the UCD work and included only one audit criterion per national audit.

Round 1

Specific actions (M3) was not included in round 1.

Round 2 prototypes

We designed and tested two prototypes with content from a hypothetical NDA report to explore responses to different wording of recommendations (Table 23). We drew on the AACTT framework45 to develop specific recommendations. Below is an example recommendation developed for the NDA.

The general practitioner or nurse in the practice with a lead role in diabetes should review patients with type 2 diabetes at least quarterly if their HbA_1c level is ≥ 58 mmol/mol and they have been treated with a single drug for ≥ 6 months. The lead general practitioner or nurse should ask reception to invite these patients for a consultation to discuss treatment options. During the consultation, clinicians should discuss treatment options with the patient and offer dual-therapy, if appropriate.

For further information on target levels and treatment options, see NICE guidance at URL: https://cks.nice.org.uk/diabetes-type-2#!scenario (accessed 2 January 2022).

Round 2 findings

There was high agreement between participants relating to the OFF version, with all comments being mixed/neutral or negative. Participants reported that the recommendation wording was too long, too vague and predicted that it would be inaccessible to the average recipient. Participants reported that the visual appeal and legibility of the recommendations list would be improved by bullet points.

There was low agreement overall relating to the ON version. Participants reported that although they found the wording clearer and more specific, this may not always make it more effective. Participants alluded to tensions between the convenience and replicability of receiving instructional recommendations compared with the autonomy and advocacy associated with more general guidelines that ‘[get] you to think about it’ (P09), to design well-fitting actions. It was suggested that health professionals in more senior roles may prefer more control in this regard.

Round 3 prototypes

We decided to operationalise the ON version of specific actions (M3) as a list of recommendations formatted in a grid layout and the OFF version as the absence of recommended actions. We designed and tested six iterations of this design using content for the NDA, PICANet and TARN.

Round 3 findings

Participants liked that recommendations were succinct and appeared in a single screen. We identified one notable usability issue: one participant wished to access additional detail behind the recommendations, such as the source. However, it was not possible to implement within this section of the report given an overlap with optional detail (M12). We responded to this suggestion by including full and adequate references to support the recommended actions.

Final design

The final prototype of this modification provided a specific step-by-step actionable guide to achieve the standard provided (see Figure 2). This appeared within the body of the one-page report. We enhanced the bulleted list with positive green tick icons following participant feedback. We included a subscript to show the source of the recommendations. Participants found the icons visually appealing and eye-catching.

Round 1

Effective comparators (M7) was not included in round 1.

Round 2 prototypes

We designed four prototypes populated with content from a hypothetical NDA report. The prototypes were bar graphs displaying different types of comparator (Table 24).

Round 2 findings

Participants reported that legends on graphs were too detailed, ran to too many lines and were too small to read. There were low levels of agreement relating to the types of comparator displayed. Participants were largely indifferent to the top 10% shading. We had a broader mix of responses relating to the prototype containing a CCG comparator for general practices. Participants were generally interested to know how they compared with their peers but for a national guideline felt that national comparison was most appropriate. Concerns about peer comparisons ranged from demotivation associated with poor performance when compared with peers (‘you don’t have to rub my nose in it’, P10) to concern that this level of reporting might risk ‘naming and shaming’ if recipients can identify sites in smaller CCGs.

Round 3 prototypes

We operationalised this modification as a textual comparator of results. This would serve two purposes: it would allow us to explore two modifications within one report section, ‘Results’, and would ensure that salient detail was included in the body of the experiment report rather than appended as a legend. We decided to broaden the target percentile performance to the top 25%, bringing together guidance from specialist team members, responses from UCD and from existing reporting. We aimed to strengthen the effect and validity of the content by using achievable comparators typical of collaborating audits. Participants in all rounds of UCD and all modifications expected to be presented with at least one comparator. We therefore selected to include national ‘mean average’ as a baseline comparator when OFF.

We designed and tested six iterations of this design using content for the NDA, PICANet and TARN. We presented effective comparators (M7) as a textual summary message above a chart for multimodal feedback (M9) in the ‘Results’ section of the report page. We included two versions of the bar chart, one for when M7 was OFF and one for when it was ON. This allowed us to retain the link between the visual display and the summary message, to be consistent with recommended best practice. 19

Round 3 findings

We identified one major usability issue early on in round 3 when testing the ON version of effective comparators (M7). The participant reported that the summary legend was lost below-the-fold and it would be more helpful to read this before viewing the graph to support comprehension when assessing the visual display. We received positive responses to the use of national upper quartile performance and describe this in more detail in relation to the visual display presented for multimodal feedback (M9).

Final design

The final prototype for this modification was a statement including either the ‘mean average’ or top 25% performance comparator:

• ON: ‘The top quarter of hospitals achieved this for X of patients’.

• OFF: ‘This compares to the national mean achievement of X’.

As well as changing the order of the effective comparators (M7) feedback statement and the multimodal feedback (M9) bar chart, we increased the font weight to emphasise the quality indicator as had been suggested in earlier UCD rounds. We subsequently presented effective comparators (M7) above the bar chart and this issue was no longer reported. We also decided to report audit performance as a percentage rather than a number, although for baseline feedback we included numerator and denominator data. This was supported by UCD participants, clinical members of the team and audit collaborators (see Figure 3).

Round 1 prototypes

We designed 10 prototypes with content from hypothetical MINAP, NCABT and TARN audit reports. The prototypes involved different combinations of textual and graphical feedback and explored combinations of bar, line and funnel charts, textual quality indicators and graphical icons to illustrate trend, based on designs from earlier feedback research programmes24,129 (Table 25).

Round 1 findings

Positive responses related to the use of the bar graph as a universally liked and easy-to-understand data visualisation. Participants responded well to typographical emphasis on the textual quality indicator. Participants preferred a combination of denominators alongside percentage quality indicators in textual feedback, allowing users to validate the data and judge the direction and scale of change from one year’s figures to the next. We observed that modality and variety in feedback may support different user needs, providing that they do not negatively affect cognitive load. Participants suggested that we cater for the majority of users when choosing a mode of data presentation, such as delivering feedback as a bar chart rather than a funnel plot, or incorporating features to support multimodal data presentation to allow switching between different types of graphs. It was suggested that interactivity might also be useful for report users such as front-line staff, to alter variables and support them ‘playing with the data’, to make predictions or judgements about clinical interventions. Reduced scrolling was also relatively important.

Negative responses related to the over-simplistic OFF versions and the use of funnel plots. Funnel plots presented during round 1 were the most contentious of the data visualisations. They were reported to be visually appealing but open to misinterpretation:

[W]e know how to read them, you know the funnel plots, but whenever we put them up in boards or anything people really struggle to, to read them.

P07

The use of the word ‘We’ in the feedback caused some confusion when users had not been offered a means to select their unit, department or hospital.

First impressions of the use of colour were positive. Visual coding systems such as traffic light systems (red denoting bad, amber denoting a warning and green denoting good) are a familiar and favoured mechanism for reducing cognitive burden. Participants also reported that they were useful, to produce summary presentations of feedback for colleagues. We observed participant confusion and mixed sentiment relating to a visual indicator showing temporal change. Our prototype provided a prominent green trend arrow alongside the quality indicator but lacked labelling or a clear key, creating confusion:

[T]he green arrow, I guess is telling me that I’m good and I’m on my way up. So does that mean I don’t need to do anything; I’m not sure what that means. Umm, or does that mean I need to go up further, more?

P06

Round 2 prototypes

We shortlisted bar graphs and line graphs, plus an audio feature, to augment textual feedback in round 2. We designed and tested three prototypes using content from a hypothetical NDA report. These included textual, tabular, graphical and audio feedback (see Table 25).

Round 2 findings

We evaluated with four participants. There were mixed responses to all three prototypes. Participants were unanimously critical of the audio feature. There was minor discussion about what the audio content might contain and who it would be for but none of the participants thought it would be a feature they would use. Usability issues included the use of a small font for the legend and a lack of comparator in the graph.

Round 3 prototypes

We chose to present multimodal feedback (M9) using a bar graph including effective comparators (M7) quartile shading. Bar graphs were deemed most appropriate for implementing snapshots of aggregate overviews that recipients would be able to understand without expecting to be provided with additional detail. 73,130 Both modifications were ON during round 3 testing.

We designed and tested six versions using content for the NDA, PICANet and TARN. We incorporated effective comparators (M7) and multimodal feedback (M9) into the ‘Results’ section of the report.

Round 3 findings

We identified two usability issues relating to multimodal feedback (M9) display. Early testing revealed that the graph was too small and needed to be enlarged. Participants also reported that they needed clear legends to explain the marks on the graph. Participants responded positively to the familiarity of the bar graph, visual appeal of quartile shading on the graph and general ease of use of the ‘Results’ section.

Final design

In the final prototype, we used a bar chart and comparator markings for the ON version of multimodal feedback (M9) (see; Figure 4a). The quality indicator in the textual feedback was emboldened to emphasise headline figures. Owing to the interaction with effective comparators (M7), the quartile shading on the graph was only visible during certain combinations of randomised modifications. The bar chart was sized to ensure that font sizes were readable, and to prevent excess scrolling. We implemented a prominent and user-friendly key to describe the coloured marks. The bar chart was absent when the modification was OFF (see Figure 4b).

Round 1 prototypes

We designed and iterated thirteen prototypes for cognitive load (M10) in round 1 populated with content from hypothetical audit reports for MINAP, NCABT and TARN (Table 26). Each participant saw up to a maximum of 10 prototypes. The prototypes incorporated features of reports that are known to load and ease mental processes. We tested a large information-heavy set of feedback, with poor formatting and no chunking compared with a range of chunking methods such as lists, tables, filterable lists, user-targeted content and filterable tables. We also explored different graph types, signposting elements such as a progress tracker and preferred hierarchy of the audit criterion compared to recommendations for action.

Round 1 findings

Participants agreed that too much text was overwhelming. They were reluctant to read content in small, dense fonts. We recorded two positive comments relating to the control/OFF versions, but these were in response to the graphical percentage shading in early iterations of feedback features. The shading was reported to add visual interest and helped to quickly convey context against which participants could judge their practice’s performance. This indicated that additional visual detail might help to reduce cognitive load. We also observed that ‘chunked’ information (in this case audit recommendations) improved visual appeal and readability.

Round 2 prototypes

We reduced the number of prototypes to exclude content types that were selected for other modifications. We explored different ways of presenting information from a single and comparable data set to assess features that might variably affect cognitive load.

We designed and tested four prototypes with content from a hypothetical NDA (see Table 26). We were at this point exploring a suitable ON and OFF version. Our prototypes presented a single information loaded data set containing process of care data, adjusted by degrees to reduce cognitive load. We compared the use of tables, filter features and typographical design elements from earlier feedback research programmes. 24,129

Round 2 findings

We evaluated with four participants. As in round 1, participants agreed that OFF versions containing small fonts and lots of information were ‘not user-friendly’ and reduced motivation: ‘Horrible . . . what is my motivation to try and go through all this?’ (P08). They suggested that headline information should provide key results as a ‘teaser’ at the top. There was limited agreement relating to additional controls to reduce cognitive load on tabular process of care data. These were felt to be more visually appealing and ‘easier to look at’ (P10), but problematic when applied to the data set. Participants reported that, for clarity, both denominator and percentage should be provided. This finding informed the implementation of baseline content in the wider experiment.

Round 3 prototype

We used extraneous text (e.g. audit background) and two small tables (e.g. patients not meeting the standard) as ‘background noise’, providing information that was not directly related to the audit criterion. We followed suggestions derived from team meetings, findings from the UCD and audit collaborators to use information that would promote intended behaviours. For the final prototype, we presented extraneous distracting detail for the OFF version.

We designed and tested six iterations of cognitive load (M10) with content for the NDA, TARN and PICANet. Distracting detail was presented as a section of supplementary information and data focused on the number and percentage of patients who receive or fail to receive care processes to achieve the standard. The modification was ON when this was not shown. We tested the OFF version, with extraneous detail being shown, in the context of all other modifications being ON.

Round 3 findings

We evaluated with six participants. There was a high level of agreement regarding the inclusion of extraneous content. Participants either neglected to read this content or were confused by its purpose in relation to the criteria and the feedback. Participants also reported navigational fatigue when all modifications were visible. The extended page length created excessive scrolling and there was a requirement to provide a means to navigate between sections within the page.

Final design

We fine-tuned content for this modification for all audits. The section title for cognitive load (M10) was iterated to ‘About the National Diabetes Audit’. This improved the clarity of the section, indicating to the user that this was additional general information not directly related to the audit criterion. We later finalised this section title as a generic ‘About this audit’ to simplify the programming (see Figure 5).

Round 1 prototypes

We designed and iterated five prototypes for optional detail (M12) populated with content from hypothetical audit reports for MINAP, NCABT and TARN (Table 27). The prototypes used features that support progressive disclosure such as hyperlinks, tabs and mouse activated dynamic annotations (tool tips). We iterated prototypes by modifying designs to include ideas that came out during testing, such as including higher fidelity content to get more value from the sessions.