Educational interventions for preventing vascular catheter bloodstream infections in critical care: evidence map, systematic review and economic evaluation

Risk of bias

The concept of risk of bias indicates whether study outcomes are likely to be valid and, hence, whether they may be trusted in the data synthesis. 58 Project scoping searches indicated that before-and-after studies were likely to be the most frequent research approach used for evaluating the effectiveness of educational interventions for preventing catheter-BSI. Our quality assessment criteria therefore focused on before-and-after studies, with additional criteria provided for assessing the quality of randomised controlled trials (RCTs), where appropriate. Risk of bias criteria have been established by the Cochrane Collaboration for RCTs58 and could, in principle, be adapted or developed for assessing some aspects of bias in non-randomised studies. 59 Before-and-after studies without a concurrent control group may be particularly at risk of performance bias, as investigators may be unable to isolate effects of their intended changes in health-care practice (e.g. implementation of an intervention) from other changes that could influence patient outcomes (e.g. intrahospital, regional or national changes in health-care practices or policies). Before-and-after studies may also be at risk of selection bias if the population characteristics of the intervention and comparator (baseline) groups differ systematically, and attrition bias if availability of data differs between the intervention and comparator groups.

Specific criteria for assessing risk of bias have not been published for before-and-after studies. We developed assessment criteria for risk of selection bias, performance bias and attrition bias (Table 4), based on the general format of the Cochrane risk of bias tool, in which risk of bias is judged either as low, high or unclear. 58 Judgements of ‘unclear’ risk of bias were made if insufficient information was reported to assess a given risk of bias criterion. The risk of bias criteria were agreed by the review team and then applied independently by two reviewers to the studies included in the systematic review. The criteria were revised in light of disagreements between the reviewers' judgements. The final risk of bias criteria (see Table 4) are intended to assist interpretation of the current data synthesis [see Chapter 4, Synthesis of effectiveness (primary outcomes)] and may not be applicable outside this current systematic review. It should be recognised that assessment of risk of bias is an imperfect science, as subjectivity of interpretation and inter-reviewer disagreements occur even with the well-established Cochrane risk of bias criteria for RCTs. 60

For RCTs, we applied the risk of bias criteria listed in Table 4 to before-and-after comparisons within each study arm so as to enable comparisons with the non-randomised studies; and we also assessed risks of selection bias according to the adequacy of randomisation and allocation concealment, according to the standard Cochrane risk of bias criteria for RCTs. 58

In addition to the risks of selection, performance and attrition bias reported above which we assessed using the criteria specified a priori (see Table 4), any other possible sources of bias noted by the reviewers in the primary studies were recorded at the data collection stage, in a free text field of the data extraction form.

Reporting of data collection in the primary studies

We recorded whether the methods of data collection were reported for clinical outcomes, infection surveillance and staff performance. Judgements were agreed by two reviewers and were recorded as yes, no, partly or unclear, together with an explanatory statement, in the data extraction form for each study. If data collection processes were reported, we also recorded whether they were shown to be validated and reliable.

Geographical locations

Of the 74 primary research studies reporting educational interventions for preventing catheter-BSI which we included in the evidence map, most (65%) have been conducted in North America. Seven studies have been conducted in Spain (10%),64,65,68,69,121,122,127 five in Brazil (7%),108,109,112,118,132 two each in Argentina,129,130 France,70,117 Switzerland,94,144 and the UK,110,128 and one each in Australia,83 Belgium,111 Canada,119 Italy,120 South Korea142 and Mexico. 103 The UK studies were conducted in Northern Ireland128 and Scotland110 in paediatric and adult critical care units, respectively. No multinational studies were identified.

Spatial and temporal scales

Most (58) studies (78%) were conducted in single hospitals, with 40 of the studies (54%) conducted in single critical care units. Of the included studies, 11 were conducted in 10 or more critical care units. 34,68,82,83,85,91,107,115,133,140,141 The largest study conducted was the Michigan Keystone ICU project,34 which included 103 critical care units in 67 hospitals in Michigan, USA.

The duration of interventions ranged from < 1 day to 7 years. In approximately one-quarter of the studies the intervention duration was unclear owing to inadequate or ambiguous reporting (Figure 2).

FIGURE 2.

Duration of interventions

Study designs

The designs of the interventional studies are summarised in Table 5, based on a classification system proposed by the Cochrane Collaboration. 58,59 The order of study designs in Table 5 reflects a hierarchy of the reliability of evidence, with risk of bias likely to be lower in well-conducted RCTs than in cohort studies.

The most frequently used study design, in 48 studies (65%), was a prospective before-and-after study (see Table 5). Eight studies81,96,111,116,134,137,141,142 were historically controlled before-and-after studies. Three studies68,136,145 were controlled trials but only two of these were randomised. 136,145 Speroff and colleagues136 conducted a cluster RCT in the USA, in which 60 hospitals were randomised to either a virtual collaborative quality improvement (QI) programme or a toolkit-based approach for preventing catheter-BSI. Khouli and colleagues145 conducted a RCT in the USA in which medical residents were randomised in a simulation laboratory to simulation-based plus video training for CVC insertion or video training alone. Palomar-Martinez and colleagues68 conducted a study in Spain in which 17 critical care units received either an intervention based on the Michigan Keystone ICU project or served as controls. Two studies used an interrupted time series approach. 88,115 In the remaining 13 of the 74 studies (18%) the design was judged unclear, either because it was unclear whether pre-intervention groups were selected prospectively or retrospectively (four studies100,106,120,121), or because too little information about the study methods was reported to classify the design (nine studies78,80,82,90,101,102,105,113,125).

Critical care specialties

The majority of studies were in adult medical, surgical and cardiac critical care units (Table 6). The specialties were not described consistently in the studies and in some cases were unclear. Numbers in Table 6 do not sum to 74, as some studies covered multiple specialties.

Intravascular devices

The studies in general provided very little information about the vascular devices used. Sixty-nine studies (93%) referred to central lines or CVCs. Five studies65,94,105,117,131 stated that they included arterial catheters as well as CVCs, and one study51 specifically excluded arterial catheters. Many of the studies did not report the insertion site (64% of the studies), lumen material (95%), whether antimicrobial-impregnated catheters were used (74%) or the number of lumens used (91%) (Table 7). Comparisons between studies are problematic, as unreported differences in vascular devices could contribute to interstudy differences in catheter-BSI incidence rates. Only three studies83,90,131 reported catheter-BSI data separately for different types of vascular catheter.

Description and definition of catheter-bloodstream infection

In 36 of the 74 studies (49%) the authors described BSIs as catheter-associated (CABSI) and in 31 of the studies (42%) the authors described BSIs as catheter-related (CRBSI). Three studies70,104,117,125 described BSIs in other ways and the remaining three studies80,86,118 provided no description for the infection data they presented. Twenty-eight of the 74 studies (38%) provided a definition of catheter-BSI within their report and 56 studies (76%) cited a reference to a definition. Most of the studies defined catheter-BSI according to criteria of the US CDC NNIS, which operated up to 2004, or the CDC NHSN, which replaced the NNIS in 2005. Although the CDC definitions have changed through time, studies continued to cite old versions. For example, at least seven studies published since 2005 referred to infection definitions published in 1988.

Aims of the studies

Most (57) of the 74 studies (77%) stated that their aim was specifically to prevent catheter-BSI in critical care. The remaining studies mostly aimed to prevent catheter-BSI together with ventilator-associated pneumonia (VAP),82,102,128,136 together with VAP and urinary tract infections (UTIs),86,104,114,117,130,137 or together with VAP and sepsis prevention. 91,101 Five studies89,93,94,118,120 each included prevention of catheter-BSI as part of a more general aim (Table 8).

Types of educational intervention

Of the 74 studies, 25 (34%) provided information on development or testing of their interventions but only three77,102,119 (4%) reported that their interventions were based on educational theory.

The most frequently addressed clinical practices were hand hygiene (49 studies), catheter insertion site preparation (typically providing guidance on the use of chlorhexidine or povidone–iodine) (45 studies), use of full barrier precautions (43 studies), and catheter insertion site selection (typically discouraging use of the femoral site) (36 studies). The studies were variable in the extent to which they reported the clinical practices addressed by their interventions, and it was not always clear which clinical practices were included in education. For example, of the 49 studies with interventions that targeted hand hygiene behaviour, only 37 studies mentioned explicitly that intervention included education about hand hygiene (Table 9).

Twenty-two studies (30%) reported interventions that were purely educational and 52 studies (70%) reported that their interventions contained components beyond education (e.g. provision of antiseptic, or a catheter supplies cart).

The educational forum types (approaches to education delivery) are summarised in Table 10. Studies often gave rather general descriptions of educational approaches in which specific details were not reported. For this reason, the numbers of studies that used particular educational forum types may have been underestimated.

Thirty-one studies (42%) reported infection surveillance feedback and 36 studies (49%) reported performance feedback, but in nine studies79,80,85,115,118,119,125,128,142 it was unclear whether feedback approaches were used. In several studies, feedback approaches that were described as being part of an intervention also appear to have been used during the pre-intervention period. 50,87,136,138,139

Characteristics of the educational approaches are summarised in Table 11 and also may have been underestimated owing to incomplete reporting by the study authors. Active learning approaches that reinforced education delivery by repeated information provision, testing and assessment and/or feedback approaches were reported in 49 of the 74 studies (66%), but in 18 studies (24%) it was unclear whether active or passive (non-reinforced) educational approaches were used. It is difficult to draw firm conclusions about the use of didactic and interactive educational approaches because often teaching sessions were described too superficially for us to be sure of their approach. At least 31 of the studies (42%) used an interactive approach to learning which encourages self-discovery of information. Of the 74 studies, 21 studies (28%) made it clear that their educational sessions were mandatory, 46 studies (62%) conducted at least some of the training in-service, and group-based education appears to have been more frequently used than individual training (see Table 11).

Knowledge of the providers and recipients of the education (Table 12) and the intensity (concentration) of education (Table 13) is important for determining the resources that would be required if the intervention were to be replicated in another setting. Only 27 of the studies (36%) provided full information on who delivered the education, whereas 47 of the studies (64%) fully reported the intended recipients. In 42 of the studies (57%) it was unclear how many educational sessions were used and in 54 (73%) it was unclear how much time the educational sessions occupied.

Concurrent changes in clinical practice, policy or infrastructure in addition to the intended intervention for preventing catheter-BSI were evident in 17 studies (23%). These included changes in the use of antimicrobial catheters, mechanical ventilation and/or UTI prevention bundles, hand hygiene programmes, staffing or critical care bed number. Where such changes occurred it would be difficult to isolate the effects of the intended intervention for prevention of catheter-BSI. Only 7 of the 74 studies (9%) explicitly stated that other concurrent interventions or changes in patient care did not occur during a study. 70,77,85,86,121,122,145

Study designs

The 24 studies included in the systematic review had the following designs. Most (20) were prospective before-and-after studies and these involved either single critical care units,50,51,87,93,94,97,108,110,138 multiple critical care units for which results were pooled across the units,34,83,98,126,129,130,135,139 or multiple critical care units for which results were presented separately for each unit. 103,109,122,144 One RCT136 was based on clusters, with hospitals randomised to interventions, but implementation was at the level of the critical care unit, comparing ‘Virtual Collaborative’ and ‘Toolkit’ CQI programmes in 60 critical care units. One non-randomised study68 compared a CQI intervention and control (no intervention) in 17 critical care units. The remaining two studies83,117 were single-cohort CQI programmes without a true baseline period in which data from early in the intervention period served as the baseline (Table 15).

Populations

Characteristics of the critical care patient populations were generally reported superficially. Only 10 out of the 24 studies (42%)50,87,93,103,109,110,129,136,139,144 reported the age and/or sex of their patient population (Table 16). The youngest patients reported were on average in their early 40s, in the studies by DuBose and colleagues93 and Higuera and colleagues. 103 The oldest critical care population group reported was in the study by Rosenthal and colleagues,129 in which the mean age was close to 72 years. Where reported, the studies included mixed-sex populations, with the proportion of men ranging from 46% to 79%. Only one study mentioned the patients' ethnicity, stating that 94% of the population was Caucasian. 139 Comorbidities were reported inconsistently, with some studies providing considerable detail and others providing no information (see data extractions in Appendix 5). In most studies the population characteristics either did not differ between the intervention and comparator groups or were not reported. Where differences between groups were evident we considered the risk of bias (reported below).

The critical care specialties most frequently reported were medical, surgical and cardiac. Three of the regional-scale studies33,83,136 were not conducted entirely in adult critical care units but we judged the studies to have met the population inclusion criterion for the systematic review because the proportion of non-adult critical units was small. Burrell and colleagues83 included two paediatric critical care units (95% were adult units), Pronovost and colleagues35 included one paediatric unit (99% were adult units) and Speroff and colleagues136 included two paediatric units in each study arm (93% were adult units). Palomar Martinez and colleagues68 appeared to focus on adult critical care but did not state this explicitly (the study authors were contacted but had not clarified this at the time of writing).

Vascular devices

All 24 studies included in the systematic review reported that they used CVCs, but it was not always clear whether other catheter types were also used. Misset and colleagues117 included arterial catheters, whereas Warren and colleagues51 specifically excluded these. Antimicrobial impregnated catheters were used routinely by DuBose and colleagues93 and Warren and colleagues139 but were not used by Sherertz and colleagues135 or Warren and colleagues. 51 Longmate and colleagues110 specified that their default catheters were non-impregnated and had four or five lumens, whereas Guerin and colleagues98 used antimicrobial-impregnated catheters unless patients were hypersensitive to them. Coopersmith and colleagues50,87 used four-lumen antimicrobial-impregnated catheters before their educational intervention but stated that their ‘accessibility was limited’ after the implementation of education (we discuss the implications of this for risk of bias below). The only other studies that reported lumen characteristics were by Wall and colleagues,138 who used triple-lumen catheters; Render and colleagues,126 who specified that 60% of the catheters were multilumen; and Misset and colleagues,117 who specified that single-lumen or multilumen catheters were used as clinically required.

Definitions of bloodstream infections

Of the 24 studies, 12 defined their infections as, or equivalent to, catheter-associated (CABSI), 11 defined their infections as, or equivalent to, catheter-related (CRBSI), and the remaining study117 gave an unclear definition (Table 17). Only eight studies provided definitions that agree with those of the Matching Michigan programme. In one of these studies, by Palomar Martinez and colleagues,68 definitions equivalent to CABSI and CRBSI were both accepted but were not separated when reporting the results (see Table 17). In support of their reported infection definitions, 28 of the studies also cited published references to definitions of CABSI or CRBSI. However, the most frequently-cited of these references, by Garner and colleagues,147 does not actually define CABSI or CRBSI.

Types of educational intervention

The types of educational approach used in the 24 included studies are summarised in Table 15. As noted in the protocol (see Appendix 1), educational interventions were defined in a broad sense to capture any type of information provision relating to the prevention of catheter-BSI by critical care staff, including the use of checklists, performance feedback and information surveillance feedback. Fourteen of the studies (58%)50,51,87,93,94,108,109,122,129,130,135,138,139,144 used interventions that we classified as purely educational. These ranged from the provision of single lectures109,122 to multimodal combinations of bedside teaching,51,94,139 in-services 50,87,94,139 self-study,50,51,139 practical demonstrations,87,94,144 slide shows or videos,94,144 lectures,51,87,109,139 discussion groups or classes,51,130 supervision,138 simulations,135 and/or posters and fact sheets50,51,87,108,139 (see Table 15). In one study129 the educational approach was not reported and it is possible that other studies may not have fully reported all of the educational approaches that they used. These purely educational interventions were all implemented at a local scale, mostly in single critical care units.

A total of 1034,68,83,97,98,103,110,117,126,136 of the 24 studies (42%) were classified as having intervention components beyond education (a classification previously used by Safdar and Abad37 in a systematic review of interventions for preventing health care-associated infections). These studies included some of the educational approaches referred to above but, in addition, they used changes in equipment (e.g. the provision of a catheter supplies cart or alcohol for skin antisepsis) or infrastructure (e.g. the provision of a team). Five34,68,83,126,136 of these studies implemented their interventions at a regional scale, in 8,126 17,68 37,83 60136 or 10334 critical care units (see Table 15). These regional-scale studies included the Michigan ‘Keystone ICU’ project conducted in the USA;34,123,124 the ‘CLAB ICU’ project,83 which sought to replicate aspects of the Keystone ICU project in Australia; the ‘Bacteraemia Zero’ project,68 which sought to replicate aspects of the Keystone ICU project in Spain, and the RCT referred to above, which compared Virtual Collaborative and Toolkit approaches in the USA. 136 Four of these interventions could be described as CQI programmes34,83,126,136 in which iterative improvements in clinical practices became embedded over time, whereas the fifth68 was a short (3-month) pilot study of a CQI programme. A further two CQI programmes were implemented at a local scale in individual critical care units in France117 and Scotland. 110 The remaining three studies97,98,103 that implemented interventions with components beyond education at a local scale (in one to three critical care units) included a catheter insertion care bundle,97 a catheter ongoing care bundle,98 and an educational intervention with provision of alcohol hand rub. 103

The duration of interventions included in the systematic review ranged from a brief 15-minute lecture reported by Perez Parra and colleagues122 to up to 6 years in the case of a multimodal educational intervention reported by Eggimann and colleagues94,95 (see Table 15). With the exception of five studies,51,108,122,135,139 the studies monitored effects of the interventions on catheter-BSI incidence density only during the period of intervention implementation, without post-intervention follow-up. This reflects an intention in many of the studies that the interventions would become embedded into routine clinical practice.

Formal and informal education

Formal education implies that participants set aside some time for structured learning, for example to participate in classes, lectures, seminars, view slide shows, or take self-study modules. Informal education may involve passive information dissemination, for example in-service discussions during daily rounds, posters, newsletters, or supervision. The distinction is important from the perspective of resource provision since staff engaging in formal learning will be taken away from their critical care duties. Nearly all of the educational interventions contained formal education components (Table 18), implying that staff cover would need to be provided for those staff participating in an intervention. However, the concentration of the education (duration and frequency of sessions and whether they were periodically reinforced) was rarely reported (see data extraction forms in Appendix 5).

Educational theory

None of the 24 included studies specified an educational or behavioural theory. Pronovost and colleagues124 referred to the ‘4E’ framework proposed by Heifetz, in which the technical aspects (i.e. scientific evidence; definitions of measures standardised across hospitals) are classed as education and evaluation, whereas the adaptive elements (i.e. implementation of measures; interventions modified to fit the local context of a clinical area) are classed as engagement and execution. The authors stated only that the technical functions in the study were centralised.

Comparison of educational interventions with UK practice

The Matching Michigan programme is reflective of current NHS practice for preventing catheter-BSI (see Chapter 1) and educational interventions included in the systematic review shared some of the approaches used in Matching Michigan to varying degrees (Table 19). Of the 24 included studies, 11 studies34,68,83,93,97,108–110,126,136,138 used checklists to improve compliance with best practices for prevention of catheter-BSI. Three studies34,83,98 empowered their staff to halt CVC insertion procedures if protocols for infection prevention were not followed. Fourteen studies34,50,83,87,97,108–110,117,126,130,136,138,139 included infection surveillance feedback in their interventions, in which catheter-BSI incidence rates were reported to critical care staff at regular intervals, either at meetings or using posters or fact sheets. However, for six of these studies, infection surveillance feedback was already in place during the baseline period and would not explain any differences in catheter-BSI incidence densities observed between baseline and intervention periods. Fourteen studies involved interventions that provided some form of performance feedback to the critical care unit staff. The feedback primarily provided information on compliance with interventions or specific intervention components (e.g. hand hygiene110), but in some studies feedback was given on problems encountered109 or on staff competence assessed through tests of knowledge or skills, with staff required to repeat training if satisfactory scores were not achieved. 50,51,98 In three studies68,122,139 the critical care staff were given tests or questionnaires, but it is unclear whether the results were fed back to the staff.

Clinical practices recommended in the Department of Health ‘High Impact Intervention No. 1’ for CVC insertion and maintenance28 (see Tables 2 and 3) and used in Matching Michigan were included to varying degrees in the primary studies (Table 20). Where studies had two interventions,109,136 the interventions within each study did not differ in the clinical practices addressed and have been summarised together in Table 20. The most frequently addressed of the recommended clinical practices were hand hygiene prior to patient contact, and the use of maximal sterile barrier precautions and antiseptic insertion site preparation. Only three studies93,129,139 did not address hand hygiene practices, although in five studies68,97,122,130,135 there was ambiguity as to whether hand hygiene was prior to or after contact with the patient (see Table 20). Generally, fewer studies addressed practices related to CVC ongoing care than practices related to CVC insertion. When interpreting Table 20, it is important to bear in mind that authors of the primary studies might not have reported all of the clinical practices that their interventions addressed, as studies were variable in the amount of detail they provided about their methods.

The most comprehensive interventions in terms of the number of evidence-based clinical practices they included were education-only local-scale interventions reported by Eggimann and colleagues,94 Lobo and colleagues108 and Warren and colleagues,51 and regional-scale interventions with components beyond education reported by Burrell and colleagues83 and Pronovost and colleagues. 34 Among these studies, the Burrell study (the CLAB ICU project)83 appears most relevant to current UK clinical practice, as it sought to replicate the Keystone ICU project at a regional scale.

Summary of study characteristics

Twenty-four studies were included in the systematic review. Most studies were conducted in medical, surgical and cardiac critical care units but the studies provided little information about their study populations and the vascular devices used. Where reported, patients' ages ranged from early 40s to early 70s. Half of the studies were conducted in the USA,34,50,51,87,93,97,98,126,135,136,138,139 with only one study conducted in the UK110 (a CQI programme in a single critical care unit in Scotland). The majority of studies used a single-cohort uncontrolled before-and-after design, with only one RCT included. The interventions evaluated in the studies were diverse in their educational approaches, ranging from single lectures in single critical care units to regional-scale CQI programmes in up to 103 critical care units. Fifty-eight per cent of the interventions reported were purely educational and 42% included components beyond education. Nearly all interventions included formal education approaches that would take staff away from bedside patient care. The interventions varied in the extent to which they addressed evidence-based clinical practices relevant to the NHS for preventing catheter-BSI. Most interventions focused on catheter insertion rather than catheter ongoing care, with hand hygiene, maximal barrier precautions and antiseptic preparation of the insertion site being the most frequently addressed practices. Different educational approaches, which were unique to individual studies, were used to address the same clinical practices. Although studies had to provide a definition of catheter-BSI to be included in the systematic review, only eight studies provided definitions consistent with those used in the NHS for the Matching Michigan programme. Two of the regional-scale interventions68,83 aimed to replicate the Keystone ICU project intervention in different countries, and, of these, the CLAB ICU project83 appears most relevant to NHS practice, although it was conducted in an Australian health-care setting.

Quality assessment: risk of bias

The included primary studies were difficult to assess for risk of bias because in most cases insufficient information was reported, resulting in a judgement of ‘unclear’ (Table 21). Methodological information supporting the judgements reached by the reviewers is given in the data extraction forms (see Appendix 5). When interpreting risk of bias it is important to bear in mind that, as judgements of bias are constrained by the availability of information, studies classified as having high or low risk of bias may not necessarily be at greater or lower risk of bias than those studies classified as unclear.

Studies judged to be at high risk of bias

Three studies68,109,144 were judged to be at high risk of selection bias, as the numbers of patient-days and CVC-days were consistently lower in the intervention period than in the pre-intervention period;109 baseline incidence of catheter-BSI was higher in control than in intervention critical care units;68 or the McCabe rapid fatality score, number of trauma patients and number of patients who had a CVC inserted were statistically significantly higher in the intervention than in the baseline period. 144

Four studies68,109,129,130 were judged to be at high risk of performance bias owing to a potential influence of other concurrent or overlapping interventions as the planned intervention overlapped with different hand hygiene interventions;129,130 staff in a critical care unit receiving one intervention moved to a different critical care unit receiving another intervention;109 or staff in control critical care units were able to attend training for the intervention. 68

Two studies50,87 were judged to be at high risk of performance bias owing to staff or policy changes because they reported changes in antibiotic prescribing patterns and accessibility of quadruple-lumen catheters,50 or reported that the number of ICU beds was higher in the intervention period (six beds; 33% increase). 87

Four studies50,51,93,110 were judged to be at high risk of performance bias because critical care staff assessing outcomes (the staff collecting and culturing blood samples and analysing the results) were not blinded. The remaining 19 studies did not report whether the staff were blinded but it seems unlikely that authors would have blinded study participants without reporting so.

One study50 was judged to be at high risk of performance bias, as the authors implied that there was a procedural difference in diagnosis and/or reporting of infections between the baseline and intervention periods (it was stated that treatment of catheter colonisation could have accounted for a large number of documented infections in the pre-intervention time period).

Studies judged to be at low risk of bias

Studies were judged to be at low risk of bias if their population characteristics did not appear to differ between baseline and intervention periods94,103,139 (low selection bias risk); they stated that no other concurrent interventions were conducted,94,97,122 or no changes in critical care structure or staffing occurred94,117 that could potentially affect catheter-BSI incidence; they stated that catheter-BSI definitions and diagnosis procedures did not change93,122 (low performance bias risk); or missing data were deducible and did not appear to differ between baseline and intervention periods94,144 (low attrition bias risk).

Risk of bias in randomised controlled trials

Only one RCT, by Speroff and colleagues,136 was included in the systematic review. In addition to the risk of bias criteria reported above for before-and-after studies, which we used to assess before-and-after comparisons within each of the RCT arms, we assessed the risk of selection bias, according to the adequacy of randomisation and allocation concealment, using the Cochrane Collaboration criteria for risk of bias in RCTs. 58 The RCT136 was judged to be at unclear risk of selection bias because insufficient information was provided about the methods of random sequence generation and allocation concealment.

Other bias risk

Information collected in the data extraction forms (see Appendix 5) suggests that studies may have been at risk of self-reporting bias (study outcomes were reported directly by the investigators without independent checking). For example, self-reporting of outcomes was routine110,136 or dependent upon staff availability;83 an audit tool and intervention were implemented by the same team that designed them;87 the study co-ordinator who was involved in delivering an intervention checked data sheets for missing items and errors;129 and it was assumed that nurses accurately captured information and completed a checklist for every insertion, without formal validity and reliability analyses. 138

In three studies110,117,136 the intervention as reported by the authors was not intended solely for the prevention of catheter-BSI: Longmate and colleagues110 reported an intervention for preventing CRBSI, VAP and MRSA; Misset and colleagues117 reported an intervention for preventing CRBSI, VAP and UTIs; and Speroff and colleagues136 reported two interventions, each for preventing both central line-associated bloodstream infection (CLABSI) and VAP. In these studies, clinical effectiveness must be evaluated for the intervention as a whole, as effects of the individual intervention components that target the different types of infections are not separable.

Reporting of data collection in the primary studies

Of the 24 primary studies,34,50,51,68,83,87,93–95,97,103,108–110,117,122–124,126,129,130,135,136,138,139,144,146 none provided full details of their data collection processes. The most detailed descriptions of data collection were reported by Burrell and colleagues83 and Wall and colleagues,138 but even their descriptions were incomplete. Burrell and colleagues83,146 stated that data entry was manual, with an infection nurse checking data on each form. Collected data were received and collated by the Clinical Excellence Commission. Missing and invalid data were followed up and validity of reported central line-associated bacteraemia (CLAB) were confirmed with individual critical care units. However, many critical care units did not have microbiological support and reported CLAB through discussion with senior critical care staff, while improved understanding of surveillance definitions versus clinical definitions led to some CLAB cases being reclassified. Wall and colleagues138 reported that upon completing a checklist, a nurse detached the top page (with all items readable) and dropped it in a secure lockbox. The second page remained on the patient's chart with the sensitive items blacked out. The infection control practitioners collected the checklists daily and scanned the de-identified forms into a pre-established computerised database using scanning software which read pre-established fields on the checklist into a spreadsheet database. These data were stored on a secure computer at the Center for Clinical Improvement for future statistical analyses.

Only one94 of the 24 studies reported that their data collection approach was reliable (pre-tested and standardised in several pilot phases), for infection surveillance. One study98 mentioned that device-day data collected by critical care unit nursing staff were compared with data collected daily by the intravascular catheter management team to confirm the accuracy of data collection but did not provide any supporting data. Three studies110,136,137 stated explicitly that validity and/or reliability of the data collection method was not assessed. It seems improbable that validity and reliability of data collection would have been assessed in the remaining studies, as no mention was made in the publications.

Summary of quality assessment

Overall, the methodological quality of the 24 included primary studies was difficult to assess owing to limited or unclear reporting of study populations and research methods. Nine studies50,51,68,87,93,109,110,129,130 were judged to be at high risk of bias, but, as risk of bias could be assessed only for well-reported studies, the extent of risks of bias among the studies may have been underestimated. Several different types of data were collected in the primary studies, including infection surveillance information, results of tests and assessments, and information on patient outcomes. However, none of the included studies fully reported its data collection methods and the majority of studies did not report whether data collection approaches had been shown to be valid and/or reliable. In most cases the staff who were involved in data collection were not specified, and studies may have been at risk of self-reporting bias (we did not formally assess whether staff involved in data collection were independent, but the authors of five studies83,87,110,129,136 implied that they were not).

Incidence density of catheter-bloodstream infection

In 10 studies, insufficient data were reported for the calculation of incidence density RRs with 95% CIs or there were ambiguities in the published data,83,93,94,97,117,126,130,135,136,138 and we attempted to contact the authors for clarification. Six authors responded with information, three did not respond and one responded stating that relevant data were not available. Data that were provided by the study authors in response to us contacting them are indicated in Appendix 7.

To enable comparisons of clinical effectiveness among the included interventions, incidence density RRs with their 95% CIs are displayed in forest plots, with the interventions grouped according to their spatial and temporal scales: regional-scale interventions (see Figure 4), local-scale interventions of duration ≤ 12 months (see Figure 5), and local-scale interventions of duration of > 12 months (see Figure 6). Within each forest plot, studies are ordered by effect size (larger effects of interventions relative to comparators are indicated by smaller incidence density RRs). As this is a narrative synthesis, pooled-effect estimates are not displayed in the forest plots. The amount of information available varied considerably among the studies, with some studies providing RRs for more than one intervention scenario or time period (multiple within-study effect estimates are not statistically independent but are included in the forest plots, as pooled-effect estimates are not calculated). The full data used in calculating RRs, including the baseline and intervention catheter-BSI incidence densities for each study, are given in Appendix 7.

Educational interventions were classified either as effective at reducing catheter-BSI incidence density, lacking convincing evidence for effectiveness, or ineffective according to the following criteria:

• Effective The incidence density RR for intervention compared with comparator (baseline) was < 1.0 and the 95% CI of the RR did not include 1.0.

• Ineffective The 95% CI of the incidence density RR for intervention compared with comparator (baseline) included 1.0.

• Lack of convincing evidence for effectiveness Effectiveness was marginal, or serious limitations of the study methodology cast doubt on the reliability of the results (explanations are provided below on a case-by-case basis).

Regional-scale interventions

Five studies34,68,83,126,136 investigated regional-scale CQI interventions (Figure 4; for the full data see Appendix 7, Data for forest plot: regional-scale interventions). These studies included the Keystone ICU project,34,123,124 the ‘CLAB ICU’ project,83,146, which sought to replicate aspects of the Keystone ICU project in Australia, and the ‘Bacteraemia Zero’ project,68 which sought to replicate aspects of the Keystone ICU project in Spain. Three of the studies used uncontrolled before-and-after designs, whereas two (a RCT by Speroff and colleagues136 and the non-randomised controlled study by Palomar Martinez and colleagues68) included two parallel comparison groups of critical care units (virtual collaborative and toolkit groups136 or intervention and control groups68). The study by Palomar Martinez and colleagues68 reported historical baseline data on infection incidence for 3 years prior to the prospective intervention (2004–6), of which we report only the 2006 data, as these are most directly relevant for comparison with the study period (2007) (for full data see the data extraction form in Appendix 5). All of the regional-scale studies calculated the number of device-days based on presence/absence of vascular catheters, which does not take into account the number of concurrent catheters that a patient may have.

FIGURE 4.

Incidence density RRs (± 95% CI) for regional-scale interventions (multiple catheters per patient not counted separately in device-days). a, CI not calculable

Clinically effective interventions (with unclear risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, the CLAB ICU project83,146 and Keystone ICU project34,123,124 interventions appear to have been effective in reducing catheter-BSI incidence densities relative to baseline, although both studies had unclear risk of bias (Box 1). The Keystone ICU project achieved a reduction in catheter-BSI incidence density after 3 months, which subsequently persisted through the 36-month intervention monitoring period. The CLAB ICU project achieved a reduction in catheter-BSI incidence density after 6 months, which was mostly sustained until the end of the 18-month intervention monitoring period (statistically significant in quarters 3, 5 and 6) (see Figure 4). These studies both have some relevance to clinical practices for prevention of catheter-BSI in critical care in the NHS and are considered in more detail below.

Clinically ineffective interventions

The two interventions implemented in the RCT by Speroff and colleagues136 were clearly not effective at reducing catheter-BSI incidence density, as acknowledged by the study authors (Box 2). During the 18-month study period, catheter-BSI incidence density for the virtual collaborative intervention actually increased relative to baseline (see Figure 4).

Process evaluation conducted by Speroff and colleagues136 (discussed below: see Process evaluations, facilitators and barriers) indicated that adoption of intervention components was consistently higher in the virtual collaborative intervention arm than the toolkit approach arm, suggesting that failure of the interventions to reduce catheter-BSI incidence was not simply related to the extent of intervention implementation.

Interventions lacking convincing evidence for effectiveness

The two remaining regional-scale interventions, by Render and colleagues126 and Palomar Martinez and colleagues,68 appear at first sight to have been effective at reducing the incidence densities of catheter-BSI. However, upon closer inspection the findings of these studies are difficult to interpret.

Published data for the study by Render and colleagues126 indicate an incidence density RR below 1.0 but the publication did not provide a CI or sufficient data for us to calculate one. Further data obtained on request from the author (see Appendix 7, Data for forest plot: regional-scale interventions) enabled us to calculate an incidence density RR with a 95% CI, but the CI indicates lack of effectiveness (see Figure 4). On balance, the available information for the study by Render and colleagues126 does not provide convincing evidence that the intervention was effective at reducing the incidence density of catheter-BSI (see Box 2).

The results of the Bacteraemia Zero project reported by Palomar Martinez and colleagues68 are difficult to interpret because incidence density RRs were significantly lower than 1.0 both for the intervention and control groups of critical care units, with a larger effect (smaller RR) evident in the control group (see Figure 4). The Palomar Martinez study68 is notable in that we judged it to be at high risk of selection bias because the control group had higher baseline incidence density of catheter-BSI than the intervention group. We also judged this study to be at high risk of performance bias because staff in control critical care units were able to attend intervention group meetings at which intervention information and materials were disseminated. On balance, this 3-month pilot study by Palomar Martinez and colleagues68 does not provide convincing evidence of effectiveness of their CQI programme at reducing the incidence density of catheter-BSI.

Local-scale interventions of up to 12 months in duration

Twelve studies investigated local-scale interventions of up to 12 months' duration. Of these, 10 studies50,51,98,103,108,109,122,129,139,144 provided sufficient data for the calculation of incidence density RRs (Figure 5; for the full data, see Appendix 7, Data for forest plot: local-scale interventions of duration up to 12 months). These studies all calculated the number of device-days based on the presence/absence of vascular catheters, which does not distinguish multiple concurrent catheters in a patient. The educational interventions were diverse and included single lectures on practices related to CVC care;109,122 structured and interactive education modules focusing on hand hygiene and CVC care144 multimodal education of various types with performance feedback;50,51,103,129,139 multimodal education with infection surveillance feedback;108 and a post-insertion central line care bundle. 98 Three of these studies each involved a single critical care unit;50,51,108 three studies involved multiple critical care units and reported results pooled across the units;98,129,139 and four studies presented results separately for different critical care units. 103,109,122,144 Lobo and colleagues109 compared a single lecture in one critical care unit (‘ICU A’) with a tailored continuous educational intervention in another critical care unit (‘ICU B’) located at the same hospital, whereas Higuera and colleagues,103 Perez Parra and colleagues122 and Zingg and colleagues144 compared results from different critical care units which had received similar interventions (see Figure 5).

FIGURE 5.

Incidence density RRs (± 95% CI) for local-scale interventions of up to 12 months in duration (multiple catheters per patient not counted separately in device-days). a, Confidence interval not calculable

Clinically effective interventions (with unclear or high risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, seven of the interventions50,51,98,103,129,139,144 would appear to have been effective in reducing the incidence density of catheter-BSI (Box 3), as incidence density RRs were significantly below 1.0 (see Figure 5). The studies by Coopersmith and colleagues50 and Warren and colleagues51 were judged to be at high risk of performance bias because the authors stated that the staff assessing outcomes were not blinded to the intervention. However, it seems unlikely that staff would have been blinded in any of the other studies, although this was not reported. Two studies50,129 were judged to be at high risk of performance bias for other reasons. Effects of the educational intervention implemented by Coopersmith and colleagues50 appear to have been confounded with changes in antibiotic prescribing practice and availability of four-lumen catheters, whereas effects of the educational intervention implemented by Rosenthal and colleagues129 overlapped with a separate hand washing intervention. One study by Zingg and colleagues144 was judged to be at high risk of selection bias, as the study characteristics of the population differed between the baseline and intervention periods.

In two103,129 of the seven studies that appeared clinically effective, further caution on interpretation is required. In the study by Rosenthal and colleagues129 incidence density RRs were significantly < 1.0 for education or education and performance feedback periods together but not for the performance feedback period alone. These results are difficult to interpret because the education and performance feedback periods were sequential, and performance feedback may not have been independent of the effects of the preceding education. Interpretation should therefore be restricted to the education phase, which itself appeared to be clinically effective, although this was only monitored for 1–2 months (see Figure 5). In the study by Higuera and colleagues103 the intervention was effective when data for two critical care units were pooled but found to be effective in only one of the units when they were analysed separately (see Figure 5). It is notable that the medical–surgical critical care unit had a higher baseline incidence density of catheter-BSI per 1000 catheter-days (57.4) than the neurosurgical critical care unit (32.8). These baseline incidence densities and also the baseline catheter-BSI incidence density in the study by Rosenthal and colleagues128 (45.9 per 1000 catheter-days) are higher than typically found in European studies.

For these seven studies50,51,98,103,129,139,144 that appeared to be effective at reducing incidence of catheter-BSI (albeit with the caveats noted above), statistical significance indicated by the CIs of RRs in Figure 5 is consistent with the primary study publications, which, in all cases, claimed that effects of the interventions at reducing catheter-BSI incidence were statistically significant.

Two51,139 of the studies that appeared effective at reducing catheter-BSI incidence (see Box 3) reported post-intervention follow-up monitoring, which may provide an indication of the longer-term persistence or attenuation of intervention effectiveness. Warren and colleagues139 conducted 10 months of follow-up after a 3-month intervention; and Warren and colleagues51 conducted 23 months of follow-up after an intervention of about 1 month. Although the RRs based on the whole follow-up period appear encouraging, monthly data provided for the latter study51 (data extraction form – see Appendix 5) show that incidence densities varied considerably from month to month, and returned to baseline levels within the first 3 months of the 23-month follow-up period.

Clinically ineffective interventions

For two109,122 of the local-scale short-term studies, the 95% CIs for the RRs indicate that the reduction in catheter-BSI incidence density was not statistically significant (see Figure 5). In contrast, the primary publications for these studies both claimed statistically significant effects of the interventions. 109,122 Lobo and colleagues109 based their conclusion on incidence data rather than incidence density. Perez Parra and colleagues122 stated that they initially conducted a Wilcoxon rank sum test and then to control for (unspecified) confounding effects of external events and (unspecified) secular trends that occurred during the study period they used a Poisson regression approach. The statistical significance reported122 varied among the critical care units (overall: p = 0.3; general post surgical: p = 0.05; cardiac post surgical: p = 0.12; medical: p = 0.31) and it is not clear to which of the analytical approaches the p-values refer. On balance, we conclude that these two studies were not effective at reducing catheter-BSI incidence density (see Figure 5 and Box 4) and the claims of statistical significance in the primary publications do not provide sufficient grounds for us to alter our conclusion.

Interventions lacking convincing evidence for effectiveness

The intervention reported by Lobo and colleagues108 has a RR bordering on statistical non-significance (upper limit of the 95% CI 0.97) and a non-significant RR for the follow-up period suggesting that the intervention was only briefly effective (see Box 4) at reducing the incidence of catheter-BSI (see Figure 5). The publication for this study did not report statistical significance. 108

Local-scale interventions of more than 12 months in duration

Seven studies87,94,95,97,110,138 investigated local scale interventions of > 12 months in duration, of which five provided sufficient data for the calculation of incidence density RRs (Figure 6: for the full data see Appendix 7.3, Data for forest plot: local-scale interventions of duration of > 12 months). The studies were all conducted in single critical care units, apart from one which involved three units. 97 The interventions included multimodal education with slide shows and bedside in-services, repeated for up to 6 years;94,95 multimodal education with self-study, bedside in-services and performance feedback over 15 months;87 a central line bundle deployed over 19 months, including checklist, infection surveillance feedback, performance feedback and catheter supplies cart;97 a CQI programme implemented over 3 years, including checklist, infection surveillance feedback and performance feedback;110 and a CQI programme focused on real-time performance feedback over 2 years, including a checklist, supervision of insertions and web-based tutorial with competence assessment. 138

FIGURE 6.

Incidence density RRs (± 95% CI) for local-scale interventions of more than 12 months in duration (multiple catheters per patient not counted separately in device-days unless stated)

Three87,97,138 of these five studies calculated the number of device-days based on presence/absence of vascular catheters, which does not distinguish multiple concurrent catheters in a patient. Eggimann and colleagues94 and Longmate and colleagues110 calculated the number of device-days in two ways: based on the presence/absence of vascular catheters; and by counting each concurrent catheter as a separate device-day. In both studies the incidence density RRs were similar for both methods of calculating the device-days (see Figure 6).

Clinically effective interventions (with unclear risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, three of the interventions conducted by Eggimann and colleagues,94,95 Longmate and colleagues110 and Galpern and colleagues97 appear effective in reducing the incidence density of catheter-BSI (Box 5), with incidence density RRs significantly lower than 1.0 (see Figure 6). Exceptions were that the RRs were not significantly different from 1.0 on all of the monitoring dates reported by Eggimann and colleagues94,95 and Longmate and colleagues. 110 Data from the Eggimann study94,95 suggest that the intervention was effective for the initial 3 years of implementation but not consistently so thereafter. Data from the Longmate study110 suggest that the intervention did not become effective at reducing catheter-BSI until the third year of implementation (see Figure 6). The publications reporting these three studies94,95,97,110 either claimed statistically significant intervention effects97,110 or did not mention the statistical significance of effects. 94,95

The study by Longmate and colleagues110 was judged to be at high risk of performance bias, as the authors stated that the staff assessing outcomes were not blinded. However, as mentioned above, it is unlikely that staff would have been blinded in any of the other studies, although this was not reported. The other studies were judged to be mostly at unclear risk of bias, although (as mentioned in the section on quality assessment above) the Eggimann study94 was judged to be at low risk of bias for four of seven bias domains assessed (the other three were judged unclear).

Clinically ineffective interventions

Two87,138 of the five studies that implemented local-scale interventions of > 12 months' duration, by Coopersmith and colleagues87 and Wall and colleagues,138 were not effective at reducing catheter-BSI incidence density (Box 6), as the incidence density RRs were not significantly different from 1.0 (see Figure 6). Coopersmith and colleagues87 acknowledged that effects of their intervention were not statistically significant. Among possible reasons for the lack of success of the intervention, Coopersmith and colleagues87 suggested that that diffuseness of the educational message may have been a contributory factor and that ideally didactic education should be specifically targeted to support best clinical practices. Wall and colleagues138 did not report the statistical significance of the effectiveness of their real-time process feedback approach for reducing catheter-BSI. The number of infections appeared to decrease dramatically (from 25 per 24 months to 6 per 24 months),138 but there was also a marked decrease in the number of CVC-days during the study, which explains the lack of statistical significance when intervention effects are analysed in terms of the incidence density of catheter-BSI.

Overview of intervention effects on catheter-BSI incidence density

Assuming that the incidence density RRs reflect those of the planned interventions, then a range of different types of educational intervention would appear to have been effective at reducing catheter-BSI incidence in critical care units (Boxes 1, 3 and 5). A key proviso is that some studies were judged to be at high risk of specific types of bias, but risk of bias was generally unclear and it is not possible to objectively classify the studies on bias risk. The interventions that appeared effective, subject to the caveats above, include regional-scale CQI programmes34,83 or a local-scale CQI programme;110 local-scale multimodal education with or without performance feedback and infection surveillance feedback;50,51,94,103,129,139,144 a catheter insertion bundle;97 and a catheter ongoing care bundle. 98 However, single lectures on CVC care and infection prevention were not effective as a means of reducing catheter-BSI incidence densities. 109,122 There is no clear evidence to suggest that regional-scale studies were more effective than local-scale studies, or that short-term studies (up to 12 months' duration) had different effectiveness than long-term studies (exceeding 12 months' duration). All three groups of studies (regional scale, local scale short-term and local-scale long-term) contain examples of effective interventions (Boxes 1, 3 and 5) and ineffective interventions (Boxes 2, 4 and 6). The studies included in the systematic review provide no clear evidence that performance feedback and/or infection surveillance feedback were influential in achieving effectiveness. Among the 12 interventions classed as clinically effective, eight (67%) included performance feedback,50,51,83,97,98,103,110,129 four (33%) included infection surveillance feedback,34,83,97,110 three (25%) included both types of feedback83,97,110 and two (17%) did not include either type of feedback. 94,144 Among the eight interventions that were not classed as clinically effective, the respective proportions were similar: four (50%) included performance feedback,109,126,136,138 three (38%) included infection surveillance feedback,108,109,126 two (25%) included both types of feedback109,126 and one (13%) did not include either type of feedback. 122 Apart from single lectures being ineffective, there appears to be no clear evidence that interventions which included components beyond education (e.g. provision of antiseptic or a catheter supplies cart) were more or less effective than interventions that consisted of education alone.

Most studies did not separately count multiple vascular catheters per patient when calculating the number of device-days. In two studies that compared different approaches for calculating the number of device-days, the calculation approach used did not appear to influence the incidence density RR for catheter-BSI.

Mortality

Five studies94,103,110,139,144 reported mortality as an outcome but they did not specifically report mortality due to catheter-BSI or its sequelae.

Only one of these studies110 reported mortality specifically for critical care patients with CVCs (those patients in the critical care unit for > 2 days with a CVC for at least part of their critical care stay). Mortality rates significantly decreased during the study period, being 21.2% during the baseline period, and 20.9% and 16% during years 3 and 4 of the intervention, respectively (year 3 vs. 1, p = 0.328; year 4 vs. 1, p = 0.013).

Three94,103,144 of the five studies reported unadjusted mortality rates for critical care patients. Of these, two reported that there were no statistically significant differences between the baseline and intervention periods. 94,144 An exception was that mortality due to cardiac arrest was significantly more frequent during the intervention period of one study94 (p < 0.05). The third study103 reported statistically significantly lower rates of unadjusted mortality per 1000 critical care unit discharges during the intervention period than the baseline period (64/132 = 48.5% and 111/338 = 32.8%), respectively [RR = 0.68 (95% CI 0.50 to 0.91), p = 0.01].

The remaining study139 reported that mortality for an ambiguous population (‘in-hospital mortality rate for catheterised patients’) did not differ significantly between the baseline and intervention periods.

In summary, insufficient data are available for us to draw any firm conclusions about the effects of the educational interventions, or the effects of catheter-BSI, on rates of mortality among patients in critical care units.

Length of stay

Seven studies50,94,109,110,117,139,144 reported LOS. One of these studies110 explicitly stated that their LOS data were for critical care unit patients who had vascular catheters. One study144 compared LOS in patients with and without CRBSI. For the remaining five studies, LOS data appear to refer to all critical care unit patients, not limited to those with vascular catheters.

Longmate and colleagues110 reported both mean and median lengths of stay for patients who were in the critical care unit for > 2 days and had a CVC for at least part of their critical care stay. Mean (± SD) lengths of stay significantly increased [5.4 ± 3.9 days during year 1 (baseline period), 5.3 ± 3.7 days during intervention year 3, and 5.9 ± 3.7 days during intervention year 4] (differences from baseline, p > 0.05). The corresponding median [interquartile range (IQR)] lengths of stay decreased, and were, respectively, 9.7 (4–20) days, 8.9 (4–13) days and 6.0 (3–11) days (differences from baseline, p < 0.05).

Zingg and colleagues144 reported median overall LOS for patients in five critical care units, during a 4-month baseline and 5-month intervention period. LOS was statistically significantly longer during the intervention period. The median (IQR) LOS for baseline and intervention periods respectively were 3 (2–7) days and 4 (2–9) days – difference, p < 0.001. Corresponding mean LOS during these periods were 5.9 days and 7.5 days, respectively. Zingg and colleagues also reported that median (IQR) LOS was 15.5 (10–25) days in patients with CRBSI and 5 (3–12) days in patients without CRBSI (difference reported as 10.5 days).

In summary, two studies110,144 reported length of critical care unit stay in relevant populations of patients with CVCs. One study,144 a 9-month education intervention by Zingg and Colleagues, reported significantly longer duration of stay during the intervention period, whereas the other, a 4-year continuous QI programme by Longmate and colleagues,110 found significantly shorter duration of stay in the intervention period. In addition, Zingg and colleagues144 reported that the median LOS for critical care patients with CRBSI was 10.5 days longer than for critical care unit patients without CRBSI (reported as 15.5 and 5 days, respectively).

Attitudes

One of the included studies by Sherertz and colleagues135 reported staff attitudes as a quantitative outcome. The proportions of postgraduate year 1 physicians-in-training who perceived a need for povidone–iodine, gloves, gowns and full sterile drapes increased significantly following a formal one-day education course (full data are in the data extraction form – see Appendix 5). However, incidence density RRs could not be calculated for this study135 so its clinical effectiveness at reducing catheter-BSI incidence density is unclear.

Four studies68,83,110,122 mentioned qualitative observations relating to the attitudes of critical care staff towards evidence-based infection prevention practices. Burrell and colleagues83 observed that some clinicians considered the incidence of CLAB in New South Wales to be low and doubted the value of the project, as existing Australian practice was felt to be equal to or better than the methods informing the project. Some clinicians doubted the evidence even with supportive CDC guidelines. Hat-wearing was a contentious element of the physician bundle: clinicians cited lack of evidence for hat use and four critical care units elected to omit their use as standard practice for CVC insertion. 83 The remaining studies provided only brief mention of staff attitudes. Longmate and colleagues110 reported that two consultant clinicians doubted the need for full aseptic technique for CVC insertion, although this was resolved after evidence sharing. Perez Parra and colleagues122 reported that staff incorrectly assumed that small drapes were sufficient for catheter-BSI prevention, although it is not clear whether they were referring to the baseline or intervention period. Palomar Martinez and colleagues68 reported that some practitioners expressed dissatisfaction with chlorhexidine for skin antisepsis and doubted its effectiveness.

Although limited in detail and lacking quantitative analysis, these observations of the attitudes of critical care staff highlight that staff may be liable to question the need for evidence-based infection prevention practices, even when presented with supporting evidence.

Knowledge

Two50,122 of the included studies indirectly reported improvements in knowledge, expressed as changes in the proportion of staff with correct test scores50 or the percentage of correctly answered questions. 122 In the latter study, the test questions most often answered incorrectly (fewer than 50% correct) were those about the need for full sterile barriers during CVC insertion and on the choice of antiseptic for skin disinfection. 122 These studies suggest that educational interventions can improve critical care staff knowledge of infection prevention practices but they did not report the types of knowledge gained through participation in the interventions.

Compliance

Compliance with evidence-based practices for preventing catheter-BSI was reported in 19 out of the 24 included studies.

Compliance with hand hygiene

Seven studies reported compliance with hand hygiene behaviour (Table 22). They all reported improvements in compliance relative to baseline, although these were not statistically significant in all cases. In most of the studies hand hygiene compliance rates did not reach 100% during educational interventions. In one study, the change was modest (final compliance 30%) and non-significant, possibly because hand hygiene was not a specific target of the education. 87

Five103,108,109,130,144 of the studies that achieved statistically significant improvements in compliance had specified hand hygiene as a main108,109,130 or joint103,144 component of their education. High baseline variability in compliance rates is notable, both within and between the studies. For example, Lobo and colleagues108 found baseline compliance with hand hygiene at CVC insertion was already 100% but compliance with hand hygiene before line manipulation was only 5%. Overall, these findings suggest that hand hygiene behaviour in relation to the insertion and management of CVCs is complex and variable, and can differ considerably between the stages of intravascular catheter site preparation, insertion and ongoing management. Owing to the relatively small number of studies that reported hand hygiene compliance it is unclear whether the degree of compliance with hand hygiene practices had any bearing on the effectiveness of the interventions for preventing catheter-BSI.

Compliance with barrier precautions

None of the seven studies87,108,109,126,135,138 that measured compliance with barrier precautions (Table 23) had specified this as a target behaviour change, although most of the studies included an element of education about sterile barrier precautions in their interventions. Compliance with barrier precautions was highly variable at baseline, ranging from 0% to 100%. Final compliance with barrier precautions after study interventions ranged from 65% to 100%, indicating improvement, but this was reported to be statistically significant for only two out of eight comparisons.

The studies appeared to differ in their ability to detect statistically significant changes in compliance. A 30% increase in compliance with maximal sterile barrier use was not significant (p = 0.29) in one study,86 whereas a 21% increase in compliance with sterile drape use was significant (p < 0.001) in another study. 135 Wall and colleagues138 reported that a decline in compliance with maximal barrier precautions during the intervention period was caused by lack of use of patient drapes. Compliance with maximal barrier precautions improved after the team purchased new sterile kits pre-packaged with drapes, and confirmed providers had completed a tutorial. Wall and colleagues138 also mentioned that use of the femoral site, compared with other insertion sites, was associated with statistically significant lower compliance with hand washing, chlorhexidine skin antisepsis and maximal barrier precautions.

Compliance with dressing management

None of the six studies that measured compliance with dressing management (Table 24) explicitly specified this was a target behaviour, although dressing care was clearly stated as a topic in the educational interventions of the studies by Coopersmith and colleagues,87 Higuera and colleagues103 and Lobo and colleagues. 109 Compliance with various aspects of CVC dressing care ranged from 11% to 84% at baseline and improved to 21% to 97% after interventions, with the improvements all being reported to be statistically significant, although compliance with correct dating of CVC dressings reached only 21% after the intervention by Coopersmith and colleagues. 87 In the study by Rosenthal and colleagues,129 compliance with placing gauze dressings and with checking the condition of dressings both increased by a greater degree following performance feedback than following education alone. However, as these intervention components were implemented sequentially, the effects of performance feedback may not have been independent of the education.

Compliance with skin antisepsis prior to catheter insertion

Skin antisepsis prior to CVC insertion was a common element of the education in many of the studies included in the systematic review, but no studies specified explicitly that it was a target behaviour. Baseline rates of compliance with skin antisepsis ranged from 9% to 57% in the three studies that reported this outcome, and in all cases interventions resulted in improvements, with final compliance ranging from 82% to 100%. The change was statistically significant in one study108 but significance was not reported in the other two studies126,138 (see Table 25).

Compliance with hub disinfection and catheter set dating

Compliance rates for line protection and hub disinfection were reported in two studies108,109 and ranged from 33% to 69% at baseline and were improved by interventions, with final compliance in the range 82% to 98% (all improvements were reported statistically significant). However, there was a notable difference between two studies in compliance with the dating of intravenous administration sets, although in both cases increases in compliance occurred, which were statistically significant. Baseline compliance was only 0.57% and initially fell to 0% after education in the study by Rosenthal and colleagues129 but then reached 74% after a phase of performance feedback. In the study by Higuera and colleagues,103 compliance with set dating rose from 40.69% to 93.85% (see Table 25).

Compliance with overall bundles

Three studies83,98,110 reported compliance with overall care bundles (Table 25). These were ‘patient’ and ‘clinician’ bundles82 a CVC insertion bundle110 and a post-insertion care bundle. 98 Compliance improved in the studies by Burrell and colleagues83 and Longmate and colleagues110 but was reported to be statistically significant only in the former study. The intervention by Guerin and colleagues98 did not appear to affect compliance with the CVC post-insertion care bundle. It is notable that compliance with all four bundles was already high at baseline, ranging from 74% to 94%, and improvements appeared modest (increases of only 7% and 11% in the Burrell study83 despite being statistically significant, and approximately 20% in the Longmate study110).

Burrell and colleagues83 demonstrated relationships between compliance with the two care bundles and the effectiveness of their CQI intervention at preventing catheter-BSI (central line-associated bacteraemia). Risk of catheter-BSI was reduced if insertion was conducted by physicians compliant in both bundles [RR = 0.5 (95% CI 0.4 to 0.8); p = 0.004], but risk was increased if insertion was conducted by physicians not compliant with the clinician bundle [RR = 1.62 (95% CI 1.1 to 2.4); p = 0.018. Most (94.0%) cases of non-compliance with the clinician bundle were due to failure to wear a hat, mask and eyewear.

Longmate and colleagues110 reported that all-or-nothing compliance with the CVC insertion bundle component of their CQI programme fluctuated between 80% and 100% during an 18-month period. The authors suggested that low initial compliance was attributed to a lack of checklist stickers early in the period; improved compliance later may have been due to introduction of a CVC insertion pack and the creation of subteam nurses to ‘own’ CVC processes, and also to greater scrutiny and follow up of episodes of incomplete compliance. 110

In addition to the studies listed in Table 25, Speroff and colleagues136 provided extensive data on the adoption of various components of flexible QI interventions in critical care units of hospitals that had been randomised to toolkit-based and virtual collaborative QI approaches (full data are given in the data extraction form: see Appendix 5). Adoption of intervention components was consistently higher in virtual collaborative hospitals than toolkit hospitals. However, as mentioned above, neither of these intervention approaches was successful in reducing catheter-BSI incidence density relative to baseline. Risk of catheter-BSI appeared higher under the virtual collaborative (see Figure 4), which had the more frequent adoption of intervention tools and strategies, suggesting that failure to reduce catheter-BSI incidence density was not simply related to the extent of intervention implementation.

Further data on compliance with evidence-based practices were reported by Coopersmith and colleagues,50 DuBose and colleagues,93 Lobo and colleagues,109 Palomar Martinez and colleagues,68 and Render and colleagues. 126 These data (see Appendix 5) are not discussed here as they are based on very small or unclear sample sizes, or it is unclear to which study periods they refer.

Summary of compliance

Overall, nearly all of the studies that reported compliance with evidence-based infection prevention practices reported improvements relative to the baseline period, although the uncontrolled studies cannot definitively exclude an influence of secular trends. Most of the information on compliance relates to hand hygiene, barrier precautions and CVC dressing care. The studies are difficult to compare, however, as in some studies small changes in compliance were reported to be statistically significant, whereas in other studies large changes were reported not statistically significant. Interventions targeting specific behaviours appear more likely to improve compliance but this is difficult to assess critically as target behaviours were not always clearly specified in the primary studies. Baseline compliance rates varied considerably between studies and for hand hygiene they varied markedly between the different stages of CVC site preparation, insertion and ongoing care. It is not clear from these data whether compliance with evidence-based practices was an important mediator of intervention effectiveness at preventing catheter-BSI, as baseline compliance rates were often already high. An exception is the study by Burrell and colleagues,83 which found that compliance with two care bundles significantly reduced the risk of catheter-BSI.

Other secondary outcomes

Our systematic review protocol (see Appendix 1) specified two secondary outcomes that we would assess if reported in the primary studies: (1) the reaction of critical care staff to education and (2) critical care staff practical skills in relation to infection prevention. However, none of the 24 studies included in the systematic review reported these outcomes.

Process evaluations, facilitators and barriers

In addition to the assessments of attitudes, knowledge and compliance reported above, six studies34,68,83,110,126,136 provided qualitative observations relevant to understanding intervention processes, facilitators and barriers, although none of these studies carried out a full process evaluation.

Three of the CQI studies83,110,136 identified a lack of adequate infrastructure to support data collection and dissemination as being a barrier to successful implementation of the interventions. Burrell and colleagues83 stated that reliable baseline data did not exist prior to the project owing to variable reporting mechanisms. Some critical care units were hesitant to accept previously reported rates, and inadequate staffing rates in some critical care units impacted on data capture rates. Difficulties were also encountered regarding data collection and associated information technology, such that the project team resorted to hard copy receipt of checklists. The lack of a continuous and sustainable data collection system involving cross-specialty collaboration was seen as a serious risk to sustainability of the CLAB ICU project principles. Speroff and colleagues136 commented that the lack of appropriate infrastructure to support data-driven QI was a significant barrier and that systematic standardised data collection was initially lacking in many of the study hospitals. Early effort was therefore needed to deploy a system-wide standardised infection control database registry. Longmate and colleagues110 reported that investigators were initially unable to reach agreement on a system – which had full support of both clinicians and data analysts – for collecting process measurements.

Three studies34,68,110 reported difficulties around the use of chlorhexidine for skin antisepsis. These related to difficulty in obtaining supplies (implying uneven compliance across critical care units)34,68 and the fact that chlorhexidine is colourless, making the skin area prepared with antiseptic difficult to see. 68,110 In one study110 it was agreed that povidone–iodine could be used to colour the skin, followed by chlorhexidine.

Two studies68,136 commented that their8 interventions provided insufficient time for some components to be implemented. Speroff and colleagues136 reported that implementation of checklists was slow, suggesting that beneficial translation of desired changes may take more than 18 months to achieve. Palomar Martinez and colleagues68 reported that none of the critical care units was able to implement catheter equipment carts within the 3-month duration of their pilot study.

Other aspects of intervention process evaluation were identified in specific studies:

Burrell and colleagues83 reported difficulty in ensuring application of, and adherence to, infection surveillance definitions. They also commented that the CLAB ICU project methodology was based on a single successful collaborative (the Keystone ICU project) without a detailed analysis of all available evidence, which allowed criticism of methodology to be an excuse for non-compliance.

Palomar Martinez and colleagues68 reported that critical care staff were reluctant to take a test as part of the training for the CQI intervention, as the test was not mandatory, results were not anonymous and there was no credit given for participation in the training.

Pronovost and colleagues34 reported that the Keystone ICU project intervention was modestly more effective in small hospitals, with an incidence rate ratio of 0.97 (95% CI 0.96 to 0.99, p < 0.001) for each 100-bed decrease in the size of the hospital. Although Pronovost and colleagues34 did not conduct a detailed process evaluation during the Keystone ICU project, Dixon-Woods and colleagues149 developed an ex-post theory of the processes that contributed to the project's success at preventing catheter-BSI. Although not noted by Dixon-Woods and colleagues,149 an important difference between the Keystone ICU project34 and other regional-scale CQI approaches that we reviewed68,83,110,136 could be that the Keystone ICU project was based on an established data collection system.

Render and colleagues126 provided a table of facilitators and barriers in their publication but it is unclear whether this was based on quantitative evidence. The study authors reported that after 6 months the project leader and project co-ordinator reviewed their detailed notes from the monthly reporting meetings to independently identify themes contributing to or delaying project success. The list of barriers and facilitators was then validated by the project leaders. However, the validation process was not explained.

Halton and colleagues

Halton and colleagues151 conducted an economic evaluation of the effectiveness of a CVC care bundle for preventing CRBSI relative to current practice, which was defined as a non-bundled approach to central line management with the use of uncoated catheters. The study setting was an adult ICU in Australia. The intervention ran over an 18-month period. The CVC care bundle encompassed five elements: optimal hand hygiene, chlorhexidine skin antisepsis, maximal barrier precautions for catheter insertion, choice of optimal insertion site and prompt catheter removal. The intervention also included a programme to educate staff about clinical leadership and the risk of infection.

The total costs of implementing the intervention were estimated in two parts: the costs directly associated with the components of the intervention and the costs related to the monitoring, education and leadership activities. A Markov model was constructed and was used to estimate the cost-effectiveness of the CVC care bundle for different combinations of antimicrobial and uncoated catheters. The analysis was conducted from the Australian health-care payer perspective with the costs reported in Australian dollars at 2006 prices. The study used various sources including the published literature, primary data from studies, national statistics, population surveys and health-care databases to derive costs and the associated health benefits. The economic outcomes from the model were summarised in terms of QALYs and total costs.

Modelling approach

The structure of the model was based on the patient clinical pathway. A Markov state–transition decision model, which had previously been developed to evaluate the cost-effectiveness of antimicrobial CVCs in an Australian setting, was adapted. 155 The Markov model consists of short- and long-term components: short term for the hospital stay and long term for the remainder of the patients' life after hospital discharge. The patients enter the model with the CVC in situ. The short-term Markov model consists of daily cycles whereby patients may develop CRBSI, remain as an ICU patient with the CVC or have their catheter removed. Patients who develop CRBSI have an increased risk of death in hospital (Figure 8).

FIGURE 8.

Markov model of the health states for ICU patients with a CVC. 151

The daily probabilities of catheter removal and CRBSI were estimated by fitting a Weibull distribution to data from an epidemiological study of CVCs. 156 In the long-term Markov model, the surviving cohort was followed for the remainder of their lifetimes. Utilities associated with different health states were assigned to cycles spent in the ICU and 6 months after discharge. Costs and QALYs were discounted using a rate of 3%.

Cohen and colleagues

Cohen and colleagues76 conducted a retrospective analysis of the costs from a simulation-based educational intervention in CVC insertion as a means of reducing CRBSI among patients in a medical intensive care unit (MICU) in Chicago, USA. The study estimated the financial implication of the educational intervention by comparing the rates of CRBSI during the intervention period (December 2006 to November 2007) and the year before the intervention. The intervention included a 2-hour lecture, ultrasound training and organised practice with a CVC simulator, as well as feedback from an instructor. The education intervention involved emphasis on the core evidence-based protocol for reducing CRBSI, which includes hand washing, full sterile barrier technique, chlorhexidine skin preparation, avoidance of the femoral site and prompt CVC removal.

The cost-effectiveness model used regression analyses to estimate the costs from the non-randomised before-and-after study. The resources used and cost estimates for the intervention were derived from hospital cost accounting data. The costs included were the cost of the intervention and the associated costs for the hospital stay.

Bond and King

Bond and King152 conducted an economic evaluation of the theoretical impact of an educational intervention to improve the safety of CVC insertion. The study setting was a health-care system that included a tertiary care centre, a community hospital and an emergency department in the USA. The educational intervention consisted of a CVC education course. It was a day-long programme with brief introductory lectures followed by hands-on procedural simulation in CVC insertion, using training mannequins, appropriate sterile procedures, ultrasound imaging and feedback from instructors. In addition, doctors and nurses were taught the Institute for Healthcare Improvement (IHI) bundle of barrier precautions and new processes to encourage, ensure and track compliance.

A decision-analytic model was constructed and was used to estimate the cost-effectiveness of the education intervention. The authors did not state the perspective of the analysis, nor the base year for the costs. The study used various sources including data from published primary studies. The results from the model were presented in US dollars in terms of net monetary benefits of the education intervention compared with no education.

Description of the model structure

A decision-analytic model was designed to estimate the cost-effectiveness of a CVC care bundle for preventing catheter-BSI in critical care units compared with remaining with current clinical practice. A diagram of the model is shown below (Figure 9). The model follows cohorts of patients from their admission to the critical care unit. The cohorts of patients compared are those who receive the CVC care bundle and those who receive current clinical practice. The numbers of critical care patients infected with catheter-BSI depend upon the catheter-BSI incidence rate, the proportion of patients with a CVC, and the effectiveness of the intervention (CVC care bundle or current clinical practice) for preventing infections. Patients may die during their hospital stay and the risk of mortality is greater for those with catheter-BSI. Furthermore, LOS will be greater for patients with catheter-BSI. The model estimates the number of people who contract catheter-BSI, those who die in hospital and the total LOS for the two cohorts. The long-term survival of patients after discharge from the critical care unit is estimated using a simple Markov model with states for alive and dead. Long-term QALYs are estimated for these patients using general population age-related HRQoL utility values. The model calculates the costs associated with LOS and the treatment and diagnosis of the catheter-BSI infections. There are also costs of implementing the CVC care bundle.

FIGURE 9.

Cost-effectiveness model for the CVC care bundle to prevent catheter-BSI

In the model, the total costs and discounted QALYs are calculated for both cohorts and thus the cost-effectiveness of the CVC care bundle is calculated:

The model is based on the following assumptions:

• We assumed for the purposes of the model that CABSI and CRBSI are synonymous, and are collectively referred to as catheter-BSI. It is not possible to distinguish between the surveillance definition of catheter-associated BSI and the clinical definition of catheter-related BSI. Although in theory CABSI overestimates the true incidence of CRBSI (for definitions see Chapter 1), our review of clinical effectiveness (see Chapter 4) and another systematic review21 found that the definitions are used interchangeably and inconsistently in primary research studies.

• We assumed that there is no difference in mortality during the hospital stay following critical care discharge for patients who had catheter-BSI in the critical care unit compared with those who did not. We were unable to find any data for the mortality during the hospital stay following critical care discharge for these two groups.

• We assumed that there was no difference in mortality after hospital discharge for those who had catheter-BSI in the critical care unit compared with those who did not.

The model does not consider the HRQoL of patients in critical care units, because the time spent in critical care is very small compared with the lifetime horizon, and therefore any QALY gains during this period are insignificant. In addition, non-fatal adverse events associated with catheter-BSI were included within the model as a cost, but not as a utility decrement.

The economic evaluation does not include non-tangible benefits or disbenefits associated with the intervention, such as changes to staff morale and public confidence in the health-care system.

Evaluation of uncertainty

There are uncertainties in the evaluation of the cost-effectiveness of the CVC care bundle. These reflect a lack of information about the content of interventions, their costs and resource use, and the extent to which they were implemented in primary research studies, as well as a general lack of information about the effects of catheter-BSI on patient survival and HRQoL. Uncertainty was evaluated using deterministic and probabilistic sensitivity analyses (PSAs). One-way deterministic sensitivity analyses (described below) were conducted to evaluate the influence of individual parameters on the model results and to test the robustness of the cost-effectiveness results to variations in the structural assumptions and parameter inputs.

Multiparameter uncertainty in the model was addressed using PSA. 162 In the PSA, probability distributions were assigned to point estimates of all parameters used in the base-case analysis. The model was run for 1000 iterations, with a different set of parameter values for each iteration, by sampling parameter values at random from their probability distributions. The uncertainty surrounding the cost-effectiveness of the CVC care bundle is represented according to the range of cost-effectiveness results. The parameters included in the PSA, the distribution used for sampling each parameter, and the upper and lower limits assumed for each variable are reported in Appendix 9.

Model validation

The economic model was validated by checking the model structure, calculations and data inputs for technical correctness by an independent health economist. The structure was reviewed by two clinical experts for its appropriateness for the disease process and the treatments considered. The robustness of the model to changes in input values was tested using sensitivity analyses to ensure that any changes to the input values produced changes to the results of the expected direction and magnitude. Finally, the model results were compared with those from previous published cost-effectiveness analyses.

Data sources

Sensitivity analysis

Probabilistic sensitivity analyses

In the PSA, all parameters were sampled probabilistically from an appropriate distribution162 using similar ranges as used in the deterministic sensitivity analyses (see Appendix 9). The parameters sampled were: catheter-BSI incidence rate, critical care mortality, catheter-BSI mortality risk, catheter utilisation, critical care unit and ward bed-day costs, treatment costs for catheter-BSI, critical care and ward LOS and CVC care bundle effectiveness and cost.

One thousand simulations were run. The PSA results are presented in Table 35 and show similar results to the deterministic analyses (see Tables 32–34) with an ICER of –£488 per QALY gained. The variability of the results is explored in more detail in Table 36, by showing the IQRs for model outputs. The CVC care bundle cost varies between £1377 and £1702, and this is offset by the savings from the inpatient bed-day cost (–£3647 to –£1553). The scatterplots and histogram for cost and health outcomes for the PSA are shown in Figures 10 and 11. The majority (83%) of the results show that the bundle is cost saving compared with current clinical practice. The cost-effectiveness of the bundle was < £5000 per QALY gained for all simulation results.

TABLE 35 - Baseline PSA cost-effectiveness results

Strategy	Cost, £		Life-years		QALYs		ICER (£/QALY)
	Median	IQR	Median	IQR	Median	IQR	Median	IQR
Current clinical practice	4903	3351–6831	878.9	865–894	673.7	663–685	–	–
CVC care bundle	3489	2817–4462	882.7	868–897	676.6	666–688	–	–
Difference	–1414	–2582 to –293	3.78	2.43–4.58	2.90	1.83–3.51	–488	–827 to –131

TABLE 36 - Summary of PSA cost-effectiveness results for the difference between the CVC care bundle cohort and the current clinical practice cohort

Outcome	Median	IQR
Patient with catheter-BSI in critical care (cases per 100 adult critical care patients)	0.76	0.54–1.02
Total mortality, critical care (cases per 100 adult critical care patients)	0.28	0.20–0.39
Additional critical care LOS for catheter-BSI (days per 100 adult critical care patients)	–0.93	–1.60 to –0.49
Additional ward LOS for catheter-BSI (days per 100 adult critical care patients)	–3.98	–5.81 to –2.63
Inpatient bed-day cost, £	–2358	–3647 to –1553
Intervention cost, £	1542	1377–1702

FIGURE 10.

Histogram of ICERs for CVC care bundle compared with current clinical practice from the PSA simulation runs

FIGURE 11.

Scatterplot for PSA

Scenario analyses

In addition to the sensitivity analyses, additional scenario analyses were undertaken to investigate the uncertainty in the model results for simultaneous changes to more than one parameter.

Effect of changing the age of patients in critical care

A scenario analysis was conducted for alternative mean starting ages for patients in the critical care unit, using the analysis described above (see the section ‘Estimation of HRQoL and long-term survival’). The calculated mean discounted life expectancy from hospital discharge for a typical patient of age 50 and 70 years was 15.8 and 7.6 years respectively and the corresponding mean discounted QALY was 12.6 and 5.6.

For a cohort of patients with mean age of 50 years, the cost-effectiveness is slightly improved at –£413 per QALY gained. At mean age of 70 years, the cost-effectiveness is slightly worse at –£926 per QALY gained.

Threshold analysis

Table 37 shows the results of a threshold analysis in which all parameters were varied until the CVC care bundle became more costly than the current clinical practice. For this scenario, the CVC care bundle would only be more expensive for any feasible changes to four of the parameters, and would remain cost saving for the other parameters. The results show that the bundle would no longer be cost saving if the bundle cost per patient was > £30, the bundle effectiveness incidence density RR was > 0.7, the catheter-BSI incidence density of < 1.8 per 1000 catheter-days, or the proportion of critical care patients with a CVC of < 0.35.

TABLE 37 - Model scenario analysis results

Parameter	Value at which CVC care bundle becomes more expensive than current practice
Catheter-BSI incidence, per 1000 catheter-days	< 1.8
Proportion of patients with CVC	< 0.35
CVC care bundle cost, per patient, £	> 30
CVC care bundle effectiveness	> 0.7

Two-way sensitivity analysis comparing central venous catheter care bundle cost versus effectiveness

Figure 12 shows a two-way sensitivity analysis of the effect on the model results of simultaneously varying both bundle cost and bundle effectiveness. These results show that, even when using extreme values for the parameters, for example with a bundle effectiveness of 0.7 and bundle cost of £85 per critical care patient, the bundle (although no longer cost saving) remains cost-effective with an ICER of < £5000 per QALY gained.

FIGURE 12.

Two-way sensitivity analysis comparing CVC care bundle cost vs. effectiveness

Two-way sensitivity analysis comparing catheter-bloodstream incidence versus additional critical care length of stay

Figure 13 shows a two-way sensitivity analysis of the effect of simultaneously changing both catheter-BSI incidence and additional critical care LOS for catheter-BSI on the model results. These results show that even when using extreme values for the parameters, for example with an incidence density of 1.0 catheter-BSI per 1000 catheter-days and no additional critical care LOS for catheter-BSI, the CVC care bundle is no longer cost-saving, but remains cost-effective with an ICER of < £2000 per QALY gained.

FIGURE 13.

Two-way sensitivity analysis comparing CRBSI incidence vs. critical care additional LOS

Two-way sensitivity analysis comparing catheter-bloodstream infection incidence versus bundle effectiveness

Figure 14 shows a two-way sensitivity analysis of the effect of simultaneously changing both catheter-BSI incidence and CVC bundle effectiveness on the model results. These results show that, even when using extreme values for the parameters, for example with a bundle effectiveness of 0.7 and incidence density of 1.0 catheter-BSI per 1000 catheter-days, the bundle, although no longer cost saving, remains cost-effective with an ICER of about £3000 per QALY gained.

FIGURE 14.

Two-way sensitivity analysis comparing catheter-BSI incidence vs. bundle effectiveness

Estimating the cost of national implementation

The national costs and benefits of implementing the CVC care bundle were estimated using the model based on the total annual number of admissions to critical care units during 1 year (2009/10) (ICNARC). In that year, there were 96,810 admissions to 188 NHS adult general critical care units in England, Wales and Northern Ireland. Note that this may slightly underestimate the total number of admissions, as not all critical care units participated in the data collection. The results are shown in Table 38.

For England, with 90,000 critical care patients per year, the model estimates that implementing the CVC care bundle would reduce the number of catheter-BSI infections by > 700 (IQR 482–914) and save 270 (IQR 184–348) lives per year. The yearly additional cost to implement the intervention used in England would be £1.4M (IQR £1.2M–£1.5M). However, if the intervention was implemented, not only would the cost of implementation be recouped, but also there would be a net saving from implementing the intervention of £1.5M, largely as a result of the savings in costs related to reduced LOS (£2.4M, IQR £1.4M–£3.3M). The CVC care bundle remains cost saving up to an annual implementation cost of £2.7M (equivalent to £30 per critical care patient).

Study designs

The only studies with concurrent control groups68,109,136 included in the systematic review did not demonstrate clinical effectiveness of their interventions. The entire evidence base for clinical effectiveness thus comes from single-cohort studies that would be classed as low quality according to the criteria for grading quality of evidence and strength of recommendations (GRADE). 181

A problem with these poor-quality studies is that they cannot usefully inform infection prevention guidelines such as ‘epic2’, which need to be based on reliable evidence. More rigorous study designs are needed to enable the determination of cause–effect relationships. In the Bacteraemia Zero project, for example, the intervention was no more effective than the control (i.e. no-intervention group), but within the intervention group of critical care units catheter-BSI incidence density was significantly lower during the intervention period than at baseline. This example illustrates how potentially misleading single-cohort studies can be for deducing effectiveness, as they do not control for secular trends. It is important that the study design employed is appropriate for the research question and capable of determining causality. Although not always feasible, RCTs may be appropriate for, and have been used in, evaluations of the effectiveness of educational interventions. 180 As there are often two or three levels of infrastructural organisation, i.e. critical care units grouped within hospitals, and hospitals grouped within regions, a cluster RCT design may be appropriate. Scales and colleagues181 provided an example of the utility of a cluster RCT approach for evaluating the effectiveness of a QI improvement programme on adoption of evidence-based practices in critical care units. An alternative approach to the use of concurrent control groups is to use an interrupted time series (ITS) design. An ITS design requires sufficiently fine resolution of temporal monitoring of outcomes (e.g. weekly or monthly) to enable detection of any changes in outcomes when an intervention is implemented, against any background secular trends. However, there is no clear consensus on how many temporal outcome assessments would be needed for a study to be classified as ITS. Only two88,115 of the 74 studies included in the evidence map (none of which was included in the systematic review) employed an ITS approach (a further study127 claimed to use an ITS approach but was reported as a single-cohort before-and-after study design).

Study reporting

Studies that have been conducted need to be more clearly and fully reported. Guidance on the reporting of observational studies of educational interventions is available from several sources, including the STROBE checklist (strengthening the reporting of observational studies in epidemiology),182 the TREND checklist (improving the reporting quality of nonrandomised evaluations in behavioural and public health interventions)55 and the ORION statement (reporting intervention studies of nosocomial infections). 57 Item 7 in the STROBE checklist encourages study authors to report all potential confounders and effect modifiers. This item should be particularly emphasised by peer reviewed journals as, in this review, a failure of authors to disclose this information was a major reason why many primary studies were judged to be at unclear risk of performance bias. Another key deficiency in the primary studies was the failure to report whether valid and reliable data collection methods had been used. A potential problem with these checklists and statements is that when reporting non-randomised educational interventions for preventing catheter-BSI it may not be obvious which checklist(s) the authors should use. Our assessment of study quality suggested non-compliance of authors with the checklists and statements is frequent.

Populations

The eligibility criteria for the systematic review developed in consultation with the project AG focused on adult critical care patients, who make up the majority of the critical care population and were identified in the evidence map as having the most primary evidence. Neonatal critical care units were not specified in the systematic review eligibility criteria as they are very different to those of adult specialties and experience higher rates of BSI-related mortality. 24 In practice it was not feasible for us to completely exclude children from the systematic review since some adult critical care specialties may include a small proportion of children, but the studies rarely reported this level of detail about their populations. Three of the regional-scale interventions34,83,136 were not conducted entirely in adult critical care units but we judged them to have met the population inclusion criterion for the systematic review because the proportion of paediatric critical units they included was small (in all cases < 9%), whereas a fourth regional-scale intervention did not explicitly state the population but we assumed it to be primarily adult. 68 If we had strictly limited the systematic review to entirely adult critical care patients then we would have excluded several studies of relevance to NHS practice, notably the Keystone ICU project,34 CLAB ICU project83 and Bacteraemia Zero project. 68

Primary outcomes

All 24 of the studies included in the systematic review reported the incidence of catheter-BSI but 10 studies did not report sufficient data for us to calculate the incidence density RR and its 95% CI, and we therefore had to contact the study authors for this information. The incidence density expressed per 1000 catheter-days is a common metric used to compare infection prevalence, but studies reporting catheter-BSI incidence density typically follow the NNIS and CDC approach,22 which does not count separately any individual catheters present in the same patient during a 24-hour period. A more precise definition employed by HELICS174 defines a catheter day as a patient having a single catheter for a whole or part 24-hour period, whereas two catheters for a part or whole 24-hour period would be defined as two device-days. Only two of the studies included in the systematic review94,110 used both definitions of catheter-days, enabling comparisons. The limited data available from these two studies suggest that the definition of catheter-days did not have an appreciable impact on incidence density RRs.

The temporal and spatial resolution of outcome assessments varied considerably among the studies, illustrating why standardisation to catheter-days is necessary to enable comparisons. Most studies reported incidence density RRs for the cumulative number of catheter-BSI and catheter-days registered during a single intervention period compared with a single baseline period. Where additional finer-resolution (e.g. monthly) data were also provided,50,51,103,109,122,139 (data in Appendix 5) these illustrated considerable short-term temporal variability in catheter-BSI incidence density that was not reflected in overall estimates for intervention and baseline periods. Studies which included multiple critical care units usually provided pooled estimates of catheter-BSI incidence density for all the units combined. Where catheter-BSI incidence density was also reported separately for different critical care units103,122,144 considerable spatial variability in catheter-BSI incidence density was evident, which would be obscured when pooling incidence densities across units. We recommend that primary studies should always provide estimates of the temporal and spatial variability of their data when reporting outcomes.

Relatively limited data were provided about mortality and LOS in the primary studies. It would be helpful in future studies for records to be kept of the mortality and LOS for critical care patients with and without catheter-BSI, to clarify the burden of catheter-BSI in settings relevant to the NHS.

Secondary outcomes and process evaluations

The Medical Research Council (MRC) guidance on developing and evaluating complex interventions183 stresses the importance of conducting process evaluations to help in understanding the reasons why complex interventions may or may not be effective. Assessments of attitudes, knowledge, compliance and other aspects of intervention processes identified several barriers or potential barriers to the successful implementation of the educational interventions included in the systematic review. Staff attitudes towards the need for evidence-based infection-prevention practices appear to be a barrier, as several studies reported staff resistance to the use of sterile drapes, gloves, masks and hats, with some staff questioning evidence-based guidance. In the CQI programmes a notable issue was a lack of existing systems and infrastructure for data collection. This seems to be an important difference between the Keystone ICU project, which was based on an existing data collection system, and the other CQI programmes that appeared to have had to develop data collection systems for their interventions. These observations of process improve understanding of potential facilitators and barriers to the successful implementation of educational interventions for preventing catheter-BSI but they were primarily based on ad hoc observations reported by the study authors, and it may be that other important facilitators or barriers were identified but not reported.

Educational approaches

Although many of the interventions included in the systematic review addressed similar sets of clinical practices, these were often addressed using very different, and often unclear, educational approaches. In many cases educational interventions were described superficially and it was difficult to tell whether all of the educational approaches used had been fully reported. Ideally, educational interventions should be reported in sufficient detail, and the resource requirements needed to implement the education elucidated, so they could be replicated. The information needed would include, for example, the total staff time involved and the cost of all materials, preferably itemised to allow interpretation of how costs might vary if intervention concentration (i.e. frequency and duration of sessions) were varied. The types of education should be clearly reported, as formal educational approaches that take staff away from bedside patient care will have associated opportunity costs. An example of good reporting of the education frequency/duration and staffing requirements was provided by Zingg and colleagues. 144 Educational interventions tend to evolve once they are implemented,149,184 so it is important that any differences between the intended and actual implementation are reported. Where possible, primary outcomes research studies should include an integral cost-effectiveness evaluation.

Instead of each intervention using its own unvalidated educational approach, as was the case in the studies we reviewed, there is a good case to be made for educational approaches to be developed in a more co-ordinated way so that they integrally support the national guidance on care bundles and can be deployed in a more consistent and similar manner. The educational interventions which we reviewed do not draw upon pedagogical, theoretical or conceptual frameworks that would enable understanding of why a particular approach works, or does not work. Consequently there are no generalisable lessons to inform the ‘epic2’ guidelines on which types of educational approach should be used in support of infection prevention practices. Unless some action is taken to effect a change, it seems likely that the current primary research activity in this area will continue to develop further ad hoc and unvalidated educational approaches. A necessary first step to address this issue would be to determine a responsible organisation for considering the harmonisation of educational approaches (e.g. an evidence-based-practice guidelines development group). Involvement of educationalists in the design of interventions would help to ensure that the interventions are well supported by relevant theory and are reliable and generalisable.

Information feedback approaches

Infection surveillance feedback, in which results of infection surveillance are provided to health-care workers to inform their clinical practices, is thought to be an effective strategy for reducing nosocomial infection incidence. 185 Our clinical effectiveness findings suggested that infection surveillance feedback and/or performance feedback were not essential for effecting reductions in catheter-BSI incidence density. However, the feedback approaches varied considerably among the primary studies, with some studies providing active feedback (e.g. at meetings), whereas others provided passive feedback (e.g. in wall charts) and none of the studies clearly reported the data collection methods used or whether they were valid and reliable. It may be that the interventions included in our systematic review were too heterogeneous in their intervention characteristics for any underlying influences of feedback approaches on catheter-BSI to have been detectable. For some studies, it became apparent only when the publications were scrutinised in detail that infection surveillance feedback was already practised in the pre-intervention period, and would therefore not explain any observed changes in catheter-BSI incidence when interventions were implemented.

Definitions of catheter-bloodstream infection

During the preparation of this report we identified citations to more than 15 different published sources of CDC definitions in support of CABSI or CRBSI. The most frequently cited publications containing CDC definitions were by Garner and colleagues,147 NNIS186 and Horan and colleagues. 187 However, although two of these references define LCBSI,147,187 none of them actually defines CABSI or CRBSI. A systematic review on 191 studies of patients with cancer published by Tomlinson and colleagues in 201121 found that CABSI and CRBSI are defined inconsistently in primary research studies. Our findings (see Chapter 4) provide further evidence that CABSI and CRBSI are inconsistently defined in primary research. For the NHS, the definitions of CABSI and CRBSI that were used during Matching Michigan (see Table 1) appear appropriate and could be formally recommended for wider use.

Influence of updated literature searches on the clinical effectiveness results

Bibliographic searches, which were rerun in March 2012 using the original search strategy, identified 933 potentially relevant new titles and abstracts published between February 2011 and March 2012, of which 19 full-text publications met the criteria for inclusion in the evidence map (see Appendix 6, Clinical effectiveness full-text records identified in search updates) and 12 potentially relevant new conference abstracts were identified (see Appendix 6, Clinical effectiveness conference abstracts identified in search updates).

Of the 19 new full-text publications, 18 reported 18 new primary research studies and one publication188 reported on an existing study already in the evidence map. 83 The new studies published during February 2011 to March 2012 would increase the size of the evidence map from 74 studies to 92 studies (i.e. a 24% increase), indicating that the evaluation of educational interventions for preventing catheter-BSI is an active area of research. This appears to be particularly the case for paediatric and neonatal critical care (60% increase and 36% increase, respectively).

Among the 18 new studies identified that would be eligible for inclusion in the evidence map, three studies would appear to also meet the inclusion criteria for the systematic review, although this was not assessed formally. Render and colleagues189 evaluated a regional-scale CQI programme conducted in 174 critical care units in 123 hospitals in the USA, which was based on a centralised infrastructure to support the co-ordinated implementation of care bundles containing infection prevention learning modules and five evidence-based practices (hand hygiene, maximal barrier precautions, skin antisepsis, avoidance of femoral insertions, removal of unnecessary CVCs). Kim and colleagues190 evaluated a catheter care bundle in six critical care units in a hospital in the USA, including checklist, supplies cart, catheter need review, avoidance of the femoral site, staff empowerment to halt incorrect procedures, staff education, and infection surveillance and performance feedback. Seddon and colleagues191 evaluated a CQI programme in a single critical care unit in New Zealand based on a care bundle, insertion and maintenance checklists and performance feedback but we noted that this intervention coincided with a major expansion of the critical care unit, which could be a confounding factor.

These three newly published studies all used before-and-after designs and claimed statistically significant reductions in the incidence density of catheter-BSI, although we did not check their calculations or methodological quality. The Render study189 is notable in being the largest study we have come across in terms of the number of critical care units involved (174 units were included). The Seddon study191 is notable in being the only study conducted in New Zealand. If these studies were, upon more rigorous scrutiny, shown to be clinically effective at preventing catheter-BSI then their findings would be broadly consistent with those of our systematic review which demonstrated both local-scale and regional-scale CQI programmes including CVC care bundles appear effective at preventing catheter-BSI. However, as with all the before-and-after studies, a key assumption is that the observed effects were a result of the planned interventions.

Searches of databases of ongoing research did not identify any other relevant primary research studies taking place up to May 2012.

In summary, the updated searches indicate that educational interventions for preventing catheter-BSI is an active area of research, but did not identify any new evidence that would appear to change the conclusions of our systematic review.

Findings from other systematic reviews

Systematic reviews of interventions for preventing health care-associated infections identified before this project commenced are summarised in Chapter 1. The most directly relevant and recent of these were published in 2008 by Safdar and Abad37 and in 2010 by Cherry and colleagues. 49

Safdar and Abad37 conducted a systematic review of educational interventions for preventing health-care associated infections, not limited to catheter-BSI or critical care. Of 26 primary studies they included, 10 were also included in our evidence map. The main conclusion from their systematic review, which was limited to English-language publications, was that implementation of educational interventions may reduce health care-associated infections considerably but the before-and-after design of studies precluded drawing firm conclusions. Safdar and Abad37 recommended cluster randomised trials using validated educational interventions and costing methods should be developed to determine the independent effect of education on reducing healthcare-associated infections and the cost-savings that may be realised with this approach.

Cherry and colleagues49 aimed to determine individual features of educational interventions that impact on competence in aseptic insertion technique and maintenance of CVCs by health-care workers. Their review was not limited to catheter-BSI or critical care and it is unclear whether they applied publication language restrictions. Of 47 primary studies included, 28 were also included in our evidence map. Their main conclusions were that educational interventions appear to have the most profound and prolonged effect when used in conjunction with audit, feedback, and availability of new clinical supplies consistent with the content of the education provided, and if baseline compliance to best practice is low. Cherry and colleagues49 did not include informal learning and they did not explicitly report data to support their conclusions concerning audit and feedback.

Our searches did not find any more recent relevant systematic reviews than those by Safdar and Abad37 and Cherry and colleagues. 49 These reviews are now relatively old, with searches conducted up to November 2006 and August 2008, respectively. Our current evidence synthesis is more rigorous than these preceding systematic reviews in that we were not limited to English-language publications and we formally assessed study quality using risk of bias criteria. In contrast, Cherry and colleagues49 used a composite quality score expressed as a percentage which is difficult to interpret. The quality score focused on the quality of education rather than other aspects of methodological quality and some studies that were also included in our systematic review50,108 received a maximum quality score (100%) from Cherry and colleagues49 despite having methodological limitations (see Chapter 4).

Systematic review of clinical effectiveness

One of the strengths of this review is its adherence to rigorous systematic review methods. We conducted exhaustive searches that were not limited to English-language publications, applied explicit inclusion criteria to the search results, critically appraised the included studies, and used transparent methods to synthesise study findings. A further strength is our inclusion of an initial descriptive mapping stage followed by a systematic review of a subset of studies. This process facilitated the involvement of clinical experts in the design of the review. The review team included clinical experts in infection prevention, critical care and medical education and was supported by an AG that included clinical experts directly involved in implementing NHS policies for the prevention of catheter-BSI.

Despite its strengths the review had limitations. It was preferable for the subset of the studies in the systematic review to be homogeneous in terms of the intervention characteristics to ensure that their aggregation statistically would be meaningful and appropriate (i.e. comparing like with like) but, in accordance with the systematic review eligibility criteria developed in consultation with clinical experts, a wide variety of intervention types was included. Although this diversity of interventional approaches does appear relevant to NHS practice, it was not appropriate to pool outcomes for different interventions in a quantitative meta-analysis and so we instead conducted a narrative synthesis. The systematic review was restricted to studies that defined catheter-BSI so that we could investigate any impact of differing infection definitions on study outcomes. However, upon scrutiny of the studies we found that definitions of CABSI and CRBSI were applied inconsistently and generally did not agree with the definitions used in NHS practice. Therefore, we could not explore the potential influence of different infection definitions on effectiveness outcomes.

There are some uncertainties in our systematic review of clinical effectiveness. All studies that appeared to demonstrate clinical effectiveness of their educational interventions were single-cohort studies without concurrent control groups. The extent to which secular trends might have played a part in determining the observed changes in catheter-BSI incidence densities is unclear. The systematic review focused on adult critical care specialties that represent the majority of critical care specialties in the NHS, but it is not known whether similar findings would have been obtained for studies in paediatric and neonatal critical care units (it was not feasible to conduct a systematic review on all specialties in this project owing to the review team resources that would be required). Other uncertainties concerned poor reporting in the primary study publications: intervention details, especially the concentration of education and resources needed to implement education were often unclear or not reported, and nearly all of the studies were judged to be at unclear risk of bias owing to inadequate reporting of key methodological information.

Although it would have been preferable to have included the Matching Michigan programme in our evidence map and systematic review of clinical effectiveness (subject to meeting the inclusion criteria), this was not feasible as results from Matching Michigan were not available at the time of our analyses. However, where possible we have used interim information from Matching Michigan to inform our economic evaluation.

Economic evaluation

Our economic evaluation is the only published example of an assessment of the cost-effectiveness of educational interventions for prevention of catheter-BSI in the UK. The model specifically addresses clinical practices in critical care units in NHS trusts in England but the results are probably generalisable to the wider UK. The economic evaluation was informed by systematic reviews of effectiveness and cost-effectiveness, and systematic searches for input parameter data. The model was developed following a structured and objective process in accordance with standard NICE practice, and the model structure and data inputs are clearly presented in this report to facilitate replication and testing of our model assumptions.

Despite these strengths, the economic evaluation has some limitations. Owing to lack of data, we had to assume that mortality rates during the hospital stay following critical care discharge, and after hospital discharge, do not differ between patients who have catheter-BSI in the critical care unit and those who do not. We also had to assume that CABSI and CRBSI are synonymous, as the systematic review of clinical effectiveness had indicated that these definitions were not meaningfully separable in the primary research literature. The model does not consider the HRQoL of patients in critical care units because the time spent in critical care is very small compared with the lifetime horizon, and therefore any QALY gains during this period are insignificant. In addition, non-fatal adverse events associated with catheter-BSI were included within the model as a cost but not as a utility decrement. The economic evaluation does not include non-tangible benefits or disbenefits associated with the intervention, such as changes to staff morale and public confidence in the health-care system.

There are uncertainties around the cost of the CVC care bundle intervention. The implementation of the intervention is likely to vary widely in practice between critical care units. Different critical care units are also likely to vary in the extent to which they would have already implemented components of the CVC care bundle at baseline. Some model input parameters could not be sourced from the UK and clinical effectiveness estimates were obtained from a relevant Australian study. Where possible we conducted sensitivity analyses to illustrate the influence of these uncertainties on the cost-effectiveness estimates produced by the model.

5.1 Catheter-related bloodstream infections (CRBSI) in critical care

Intravascular catheter placement is an important cause of bloodstream infections (BSI)1,2, and is the commonest source of hospital-acquired bacteraemia in hospitals in England3. CRBSIs (CRBSI) are a particular problem in critical care due to the high frequency of intravascular catheter placement and increased susceptibility to infections among critical care patients. CRBSI are associated with morbidity and, especially in paediatric critical care, also mortality4. Estimates of the additional length of stay per CRBSI episode in UK critical care units have ranged from 1.9 days5 to 11 days6. Owing to a lag in the publication of infection rates, there is uncertainty as to whether these published data are representative of current rates of CRBSI in UK critical care units. 1

(1 Recent unpublished data from UK critical care units suggest that CRBSI rates may in some cases be much lower than those reported in the published literature (Dr D. Wyncoll, personal communication). In the current project, as indicated in section 8.2, the most relevant data to UK critical care units will be used to evaluate cost-effectiveness of educational interventions.)

CRBSI result from inadequate hygiene and suboptimal catheter management procedures. These include among others inadequate hand hygiene of hospital staff, inadequate skin hygiene at the site of patients' catheter insertion, suboptimal location of catheters, and unnecessary placement of catheters. CRBSI are believed to be largely preventable following work in the UK that has successfully reduced the number of cases of MRSA BSI. It has been proposed that the majority of CRBSI could be prevented using evidence-based educational interventions to ensure that doctors and nurses are committed to a culture of safety and follow best practice to achieve this7,8.

5.2 Educational interventions for preventing CRBSI

6.1 Existing research

6.2 Research objectives

The aim of this project is to conduct an evidence synthesis of the clinical and cost-effectiveness of educational interventions aimed at hospital staff in critical care (doctors and nurses) for preventing CRBSI. An economic model will be devised by adapting an existing cost-effectiveness model or constructing a new one using the best available evidence to determine cost-effectiveness in a UK critical care setting. The project aims also to provide recommendations that will be sufficiently specific to be of use to those implementing infection-prevention strategies in the NHS and for further research.

The main objectives will be as follows:

1. To systematically review: (a) the clinical effectiveness; and (b) the cost-effectiveness of educational interventions for the prevention of CRBSI.

2. To use an evidence mapping approach to describe the scope of the clinical effectiveness evidence base in terms of the different types of educational interventions, critical care settings, study designs and their theoretical basis, types and duration of education reinforcement, and outcomes reported including evidence of sustainability of effect. The evidence map would: (a) provide an overview, classification and characterisation of relevant educational interventions to enable complex interventions to be visualised and, where appropriate, compared; (b) provide a classification and report of other key study attributes, for example illustrating how CRBSI and CABSI are defined and applied in the studies; and (c) use recognised criteria to screen studies in terms of their relevance to the NHS.

3. To apply the evidence mapping exercise results to prioritise a subset of studies of highest relevance for detailed appraisal in the systematic review of clinical effectiveness.

4. To develop a decision-analytic model to determine and compare cost-effectiveness of relevant groups of interventions and settings identified through evidence mapping, either by adapting an existing economic model or constructing a model for the UK de novo.

5. To identify future research needs and make specific recommendations about the implementation of educational interventions for preventing CRBSI that are relevant to service users in the NHS.

7.1 Systematic review of clinical effectiveness

7.1.4 Evidence mapping

All studies that meet the inclusion criteria will be entered into a mapping exercise in order to clarify the structure of educational interventions and identify those that are potentially of most relevance to the NHS.

The mapping exercise is summarised in Figure 2 and will follow principles developed by the Global Evidence Mapping Initiative45 to classify and summarise the evidence base as well as guidance from the Medical Research Council43 and the National Institute for Health and Clinical Excellence (NICE)46 on the reporting and evaluation of complex interventions. The key objectives of this step will be to: (a) provide an overview, classification and characterisation of relevant educational interventions to enable complex interventions to be visualised and, where appropriate, compared; (b) provide a classification and report of other key study attributes, for example illustrating how CRBSI and CABSI are defined and applied in the studies, and whether studies included process evaluations, information on potential facilitators or barriers to implementation, or other secondary outcomes; and (c) identify studies that are of most relevance to the NHS which should be prioritised for full evidence synthesis. The mapping exercise will be conducted as follows (Figure 2):

1. With assistance from the Project Advisory Group (section 10), the characteristics of studies to be included in the descriptive map will be determined and made into a list;

2. In a pilot exercise involving two researchers, keywords will be developed that reliably and reproducibly describe each of the study characteristics in the list;

3. Each included study will be mapped by one researcher and the agreed keywords relevant to describe the characteristics of each study will be entered into a Microsoft Excel or Access database such that study interventions and keywords can be cross-tabulated;

4. Keyword assignments and database entries for each study will be checked by a second researcher;

5. Entries in the database will be used to concisely summarise the structure and composition of the study interventions using numerical, graphical and/or narrative methods where appropriate.

FIGURE 2.

Overview of the evidence mapping procedure for studies of clinical effectiveness.

Depending upon the overall quality and quantity of evidence available, the evidence mapping exercise could include a preliminary appraisal of methodological quality to help decide which studies are prioritised for detailed full evidence synthesis (e.g. based on study design and sample size). A thorough appraisal of methodological quality, using risk of bias criteria, will be applied later to those studies that are identified as being of most relevance to the NHS and which proceed for full data extraction (section 8.1.5).

Depending upon the overall quality and quantity of evidence available, the evidence mapping exercise could include a preliminary appraisal of methodological quality to help decide which studies are prioritised for detailed full evidence synthesis (e.g. based on study design and sample size). A thorough appraisal of methodological quality, using risk of bias criteria, will be applied later to those studies that are identified as being of most relevance to the NHS and which proceed for full data extraction (section 8.1.5).

7.2 Economic evaluation

The cost-effectiveness of educational interventions in preventing CRBSI in critical care will be assessed in two stages: a systematic review of cost-effectiveness and development of a decision-analytic economic model (Figure 1).