Development and validation of a risk model for identification of non-neutropenic, critically ill adult patients at high risk of invasive Candida infection: the Fungal Infection Risk Evaluation (FIRE) Study

Analysis of risk factors

Eight articles examined risk factors for IFD, each of which is described briefly below.

Surgery

Seven studies25,26,35,36,38–40,43 examined the association between surgery and IFD. The type and timing of surgery varied across the studies, with two36,38 looking at abdominal surgery and the others looking at any surgical procedure. Five of the seven studies25,26,35,36,38,40 reported a significant association between surgery and IFD on both univariable and multivariable analysis.

Total parenteral nutrition

Six studies25,26,28,35,37,38,40 examined the association between TPN and IFD. All six found a significant association with IFD on univariable analysis. Four of the six studies25,26,35,37,38 also found a significant association on multivariable analysis.

Fungal colonisation

Five studies25,26,35,36,39,43 examined the association between fungal colonisation and IFD. Four of the five studies26,35,36,39,43 reported an association on both univariable and multivariable analysis. Sites of fungal colonisation examined and modelling approaches varied across the studies.

Renal replacement therapy

Seven studies26,28,35,37,38,40,41,43 examined renal replacement therapy as a risk factor for IFD. Five of the seven studies26,28,35,37,38,40 found a significant association on univariable analysis. Three of the seven studies28,35,38 demonstrated a significant association on multivariable analysis. Only one of the two EPCAN articles demonstrated a significant result on multivariable analysis. The type and exposure time to dialysis varied across the studies; some looked at pre-admission dialysis, whereas others examined haemofiltration in the unit.

Infection

Five studies25,26,38,41,43 examined the relationship between infections/sepsis and IFD. Three of the five studies26,38,41 demonstrated an association on multivariable analysis. The source and site of infection varied across the studies. One examined bacterial infection/bacteraemia without specifying type and source of infection. 41 Another examined enteric bacteraemia, which included enterococcal, Bacteroides and other Gram-negative bacilli bloodstream infections. 38 One demonstrated an association with severe sepsis, although the infection source was not specified. 26

Mechanical ventilation

Five studies25,26,35,37,40,41 examined the association between receipt of mechanical ventilation and IFD. Two of the five studies40,41 reported a significant association on multivariable analysis. Both timing and duration of mechanical ventilation varied across the studies, with one study finding presence of mechanical ventilation was significant after day 3 of critical care unit admission,40 and the other finding that a duration of mechanical ventilation > 10 days was significant. 41

Diabetes

Four studies26,28,36,41 examined whether or not a past medical history of diabetes mellitus was a risk factor for IFD. Two of the four studies28,41 demonstrated a significant association on both univariable and multivariable analysis.

Acute severity score

Eight studies25,26,28,35–37,39,43 examined whether either the APACHE II or APACHE III (Acute Physiology And Chronic Health Evaluation) score was a risk factor for IFD. Two of the eight studies39,43 demonstrated a significant association on both univariable and multivariable analysis.

Other risk factors

A number of other risk factors were identified to be significantly associated with IFD on multivariable analysis in single studies. These included CBP time, acute renal failure, broad-spectrum antibiotic use, red blood cell transfusions, antifungal medication use, CVC use, diarrhoea and peripheral catheter use (see Table 2). Of note, two studies27,35 examined the association between neutropenia and IFD, neither of which demonstrated a significant association. Similarly, none of the five studies26,28,36,37,43 looking at immunosuppressant use demonstrated an association with IFD.

Risk models and clinical decision rules

Four of the studies26–28,43 developed a risk model or clinical decision rule for IFD (Table 4) and one evaluated a clinical decision rule for IFD, all in the critical care setting.

Leon et al. 26 developed and validated a risk model from which they derived a bedside scoring system to inform early antifungal therapy in non-neutropenic, critically ill patients. The study was a prospective cohort of 1699 patients, of whom 980 with colonisation or infection were included in the model development with 97 cases of IFD. Multifocal Candida colonisation, surgery directly prior to critical care unit admission, severe sepsis and TPN were included in the final risk model. The optimal score from the model gave a sensitivity of 81% and a specificity of 74%.

Ostrosky-Zeichner et al. 27 developed a number of clinical decision rules for IFD in the critical care setting. The study was a retrospective chart review of 2890 patients from 12 participating centres with 88 cases of IFD. Several clinical decision rules, with varying combinations of risk factors, were developed and tested. The best performing rule consisted of the following risk factors: any systemic antibiotic, presence of a CVC, and at least two of the following – TPN, any dialysis, any major surgery, pancreatitis, and use of steroids or other immunosuppressants. The model gave a sensitivity of 34% and a specificity of 90%.

Paphitou et al. 28 developed and validated a number of clinical decision rules from a single centre, retrospective cohort study of 327 critically ill patients. There were nine cases of proven IFD with 27 probable/possible cases.

Several combinations of risk factors were evaluated, of which any combination of diabetes mellitus, new-onset haemodialysis, use of TPN or receipt of broad-spectrum antibiotics was considered the most useful. The model gave a sensitivity of 78–83% and specificity of approximately 50%.

Pittet et al. 43 developed a number of clinical decision rules based on intensity of Candida colonisation from a single-centre, prospective cohort study of 29 critically ill patients with significant Candida colonisation of whom 11 had severe Candida infection. The best-performing rule, developed post hoc to give perfect discrimination in the small data set, was a Candida-corrected colonisation index (ratio of highly positive fungal screening samples to the total number of samples) of 0.4 or more.

Finally, Piarroux et al. 42 evaluated the clinical decision rule developed by Pittet et al. 43 whereby patients admitted to a single SICU were screened for fungal colonisation and pre-emptively treated with fluconazole if the Candida-corrected colonisation index was ≥ 0.4. Using same centre, historical control subjects, from a time period prior to offering prophylaxis, a reduction of unit-acquired IFD from 2.2% to 0% (p < 0.001) was reported.

Design and development of data set and protocol

A list of the key risk factors for IFD, identified from the systematic review of the literature, was compiled. Through consultation with the clinical experts on the FIRE Study Steering Group, the list was added to and refined after comprehensive discussion, to produce a final data set and definitions.

Data were collected at three different decision time points: on admission to the critical care unit; at the end of the first 24 hours; and at the end of the third calendar day. Outcomes data were collected until discharge from critical care or death. The rationale for the time points is as follows:

• On admission Allowed a record of the risk factors to which the patient was exposed in the period up to 7 days prior to admission and provided the first decision point for antifungal prophylaxis.

• At the end of the first 24 hours Given the interventions performed in the first 24 hours of care in the critical care unit, this allowed an updated record of the risk factors to which the patient was exposed and provided a second decision point for antifungal prophylaxis. Data are routinely collected in the first 24 hours for the Case Mix Programme (CMP), the national clinical audit for adult critical care, which reduced duplication of data collection effort.

• At the end of calendar day 3 With the median stay in adult general critical care of 53 hours, patients still on the critical care unit at this time point are expected to be long-stay patients and at higher risk of IFD. This allowed an updated records of the risk factors to which the patient was exposed and provided the third and final decision point for antifungal prophylaxis. End of calendar day 3 was selected to coincide with data collection in adult general critical care units for the Critical Care Minimum Data Set (CCMDS).

Primary outcome

The primary outcome for the FIRE Study was IFD, defined as a blood culture or sample from a normally sterile site (including, but not restricted to, cerebrospinal fluid, peritoneal fluid, pleural fluid and pericardial fluid, and excluding bronchoalveolar lavage, urine and sputum) that was positive for yeast/mould cells in a microbiological or histopathological report. This definition was chosen to best capture Candida IFD and was recognised to be likely to under-represent IFD due to other species.

For statistical and economic modelling, the primary outcome was invasive Candida infection, defined as IFD (as above)-positive for Candida species in a microbiological or histopathological report.

Data

Data collected for each patient:

• CMP admission number (for data linkage with the CMP; see below)

• hospital number (for local retrieval)

• NHS/Community Health Index (CHI) number

• date of birth

• sex

• date of admission to hospital

• date and time of admission to the critical care unit.

Data collected at each of the three time points:

• lines in arteries (number)

• major intra-arterial devices (any)

• catheters in central veins (number, position)

• peripheral lines (any)

• intracranial devices/perineural lines (number)

• drains (number)

• enteral feeding tube

• urinary catheter

• organ support (advanced respiratory support, renal support)

• TPN

• steroids (high or low dose)

• immunosuppressives

• existing diagnosis of diabetes mellitus (admission only)

• antimicrobial drugs (last antimicrobials prior to admission and first antimicrobials following admission)

• neutropenic status (end of 24 hours only).

Data collected on admission and at any time up to discharge from the critical care unit or death:

• surgery (condition requiring surgery, urgency of surgery, unexpected complications and open abdomen following surgery)

• fungal colonisation (numbers of samples reported and numbers positive)

• IFD (date/time, organism and site)

• antifungal drug use (topical and systemic, initial regimen and date/time of first administration).

Fungal colonisation was defined as the presence of yeast colonisation in any sample reported on a microbiology system and was recorded as the date that a positive report was received, i.e. the point at which a treatment decision could be made based on this knowledge.

Research governance

The FIRE Study was sponsored by ICNARC. An application was made to the Bolton NHS Research Ethics Committee (REC) following confirmation of funding, and a favourable opinion was received on 15 December 2008. The Scotland A REC reviewed the protocol on 26 October 2009 and concluded that the project could be conducted as an extension to existing audit and was not classified as research.

The FIRE Study was piggybacked on to the CMP in England, Wales and Northern Ireland, and linked with data provided by the Scottish Intensive Care Society Audit Group (SICSAG) in Scotland. The CMP is the national clinical audit of adult general critical care units in England, Wales and Northern Ireland, established in 1995. Trained data collectors collect the raw data to precise rules and definitions. The data then undergo extensive local and central validation for completeness, illogicalities and inconsistencies prior to pooling. SICSAG is the national clinical audit of adult general critical care units in Scotland, established in 1995. Data are collected on local software and undergo logical checks on data entry. Monthly case note validation is undertaken on a random sample of 10% of records. Both the CMP and SICSAG databases have been independently assessed against 10 criteria for coverage and accuracy by the Directory of Clinical Databases (DoCDat; www.icapp.nhs.uk/docdat/) and achieved mean quality scores of 3.7 and 3.8 (on a scale of 1 = worst to 4 = best).

The CMP has approval under Section 251 of the NHS Act 2006 (originally enacted as Section 60 of the Health and Social Care Act 2001) to hold limited patient identifiable data (date of birth, sex, postcode, NHS number) without consent (approval number: PIAG 2–10(f)/2005). No additional patient identifiable data were required for the FIRE Study. In June 2008, the Patient Information Advisory Group (PIAG), since superseded by the National Information Governance Board for Health and Social Care (NIGB) Ethics and Confidentiality Committee, approved the extension of the Section 251 approval for the CMP to cover the FIRE Study.

Each participating critical care unit in England, Wales and Northern Ireland completed local research and development (R&D) approvals prior to commencing recruitment. In accordance with the guidance given by the Scotland A REC, each participating critical care unit in Scotland obtained approval from their local Caldicott Guardian prior to commencing recruitment.

Patient information sheets (see Appendix 3) and posters were displayed in participating critical care units so that patients and families/close friends would be aware that the unit was taking part in the FIRE Study, which would not affect their care. Patients or families/close friends were able to opt out from participation and their data were removed from the FIRE Study database. Patient information sheets and posters to be displayed in critical care units in Scotland were adapted to reflect the classification as an extension to existing audit rather than as research.

Critical care unit recruitment

All adult general critical care units in England, Wales and Northern Ireland participating in the CMP were initially invited to take part in the FIRE Study. Subsequently all adult general critical care units in Scotland were also invited to participate. Separate, standalone, high dependency units (HDUs) and specialist units (neurosciences, cardiothoracic, etc.) were not eligible for participation in the FIRE Study. Staff in participating critical care units collected data on consecutive admissions to their unit.

Dataset familiarisation courses

Regional Dataset Familiarisation Courses were held across England and Scotland. Staff from critical care units who were unable to attend on any of these days were provided training via teleconference.

The Dataset Familiarisation Courses were one-day events where the background, aims and rationale for the FIRE Study were discussed with the collaborating clinicians, research nurses and data clerks. This was followed by a detailed explanation of the definition for each field in the data set with opportunities for questions and examples. Each delegate was given a FIRE Study Data Collection Manual to take back with them to their unit for reference.

The Data Collection Manual contained precise, standardised definitions for each field, along with data collection forms and flows (see Appendices 4 and 5) to guide them through the data collection process. From the data collection forms, data were entered on to a dedicated, secure web-based data entry system developed and hosted by ICNARC. Data collection manuals, flows and forms, frequently asked questions (FAQs), definitions and error checking were also available, either for download or built into the design for the web portal. Web portal pages were regularly reviewed and new versions released to ensure up-to-date clarity and to answer common queries.

Maintenance and motivation of units

During the course of the study, quarterly newsletters were sent to all participating critical care units. Newsletters were used as an opportunity to clarify any data issues, as well as to maintain motivation and encourage involvement through regular updates.

The Study Coordinator maintained close contact with all units by telephone and e-mail throughout the study.

Support costs

The FIRE Study was set up at the same time as the new system for supporting research in the NHS through Comprehensive Local Research Networks (CLRNs) was evolving. Initially, funding for NHS support costs was sought through local systems in each CLRN. In June 2010, support for data collection was centrally approved by the Central and East London CLRN to an equivalent of a 0.5 WTE (whole-time equivalent) Band 6 Research Nurse for a critical care unit with 800 admissions per year.

Sample size calculation

Assuming a 1% incidence of invasive Candida infection among non-neutropenic, adult patients admitted to UK critical care units,4,5 a sample size of 40,000 patients in the development sample was selected to give 20 EPV for consideration of 20 candidate variables in the risk model. This sample size was also sufficient to give 80% power to detect, as statistically significant (p < 0.05), a risk factor present in 10% of the population associated with a 50% increase in the risk of invasive Candida infection. Simulation modelling indicated that this sample size calculation was robust to clustering of both risk factors and outcomes at the critical care unit level. An additional 20,000 admissions were recruited for the validation sample. The 60,000 admissions target was based on the assumption that 80 critical care units would participate in, and complete, data collection and validation.

Data management

Data management was an ongoing process. Data were monitored and validated throughout the data collection period in order to ensure that the database was as complete and accurate as possible during the study, and to minimise the time between the end of data collection and start of data analysis.

Data linkage between the FIRE and CMP databases was performed regularly, to ensure complete capture of admissions. Data collectors were notified of the missing records and asked to update the portal with this information.

Data validation reports

Data validation checks were run periodically on each record on the web portal. These checks identified any incomplete data (missing values) and inconsistent data (unusual, although not impossible, data) both within and across data fields. Following receipt of a Data Validation Report (DVR), data collectors either updated/corrected the data on the web portal or responded to the FIRE Study Team to confirm data were correct.

Data linkage with the Case Mix Programme and Scottish Intensive Care Society Advisory Group

FIRE data were linked with the corresponding CMP/SICSAG data and any discrepancies in patient identifiable data (date of birth, sex, NHS number, date of admission to hospital and critical care) raised and resolved.

Reliability study

At the end of data collection, a reliability study was conducted to confirm that all cases of IFD were correctly recorded. Each critical care unit received a mixed, blinded list of all of the reported IFD-positive cases from the unit, along with a 2% random sample of non-IFD cases. For each patient, the local principal investigator was required to recheck the original hospital notes and microbiology records and make an independent decision on the IFD status originally recorded. Reliability study results were completed and signed off by the local principal investigator at each unit and returned centrally to the FIRE Study Team for verification against the original IFD data. The reliability study was conducted following all other validation, once access to the web portal had been disabled, to ensure that the data on it were not used to complete the Reliability Study. Where there were discrepancies between original and reliability study data, the IFD status provided by the local principal investigator was accepted as final. Any diagnosis of IFD in a sample from a non-sterile site was followed up with the local principal investigator to determine whether or not IFD had also been found in a sterile site as per the definition.

Critical care unit recruitment

Recruitment of critical care units took place between April 2008 and December 2010 (England, Wales and Northern Ireland) and February 2010 and October 2010 (Scotland). One hundred and three critical care units expressed an interest in taking part in the FIRE Study and were sent a Principal Investigator Details Form and Site-Specific Information (SSI) Form to complete. Of these, 100 (83 in England, Wales and Northern Ireland, and 17 in Scotland) critical care units sought and gained approval from either their local R&D Departments (England, Wales and Northern Ireland) or Caldicott Guardians (Scotland) and commenced data collection. R&D approval took a median of 45 days [interquartile range (IQR) 28.5 to 79.5] from submission of SSI Form. R&D approval to start of FIRE Study data collection took a median of 24 days (IQR 8.0 to 52.5). Four critical care units withdrew from the study owing to local staffing and data collection issues, giving a final total of 96 participating units. Eleven critical care units stopped data collection early and the remaining 85 continued data collection to 31 March 2011 (Figure 2). Five critical care units had periods of 1–2 months' data excluded from the final data set owing to failure to capture all consecutive admissions as a result of temporary local staffing issues.

FIGURE 2.

Participation timeline.

Each critical care unit was represented at a Dataset Familiarisation Course. Representatives from 90 units attended one of the nine regional Dataset Familiarisation Courses and a further 10 units, who were unable to attend one of these courses, were provided individual or group Dataset Familiarisation by teleconference.

The final 96 participating critical care units were representative of all UK adult general critical care units in terms of geographical distribution, teaching status of hospital located within and number of critical care unit beds (Table 6).

Recruitment of admissions

The final data set contained a total of 60,778 admissions. Individual critical care units recruited between 58 and 2061 admissions (median 503, IQR 368 to 890; Figure 3).

FIGURE 3.

Recruitment by critical care unit.

Reliability study

Ninety-six reliability study reports were sent out for a total of 1293 cases originally reported with a status of IFD and 1289 reported as non-IFD. A total of 917 discrepancies were identified; 913 of these were cases originally reported as IFD not confirmed as IFD by the principal investigator in the reliability study. In the majority of these cases, the positive sample was not from a normally sterile site and this had been recorded incorrectly in the original data. These cases were amended in the final data set. Only four cases which were originally recorded as non-IFD were amended in the final data set, following the reliability study, to IFD.

Descriptive analysis

Admissions with IFD were classified according to species, site(s) of IFD and timing of IFD (pre-admission or day of admission). Case mix was summarised in terms of age, sex, past medical history (severe comorbidities, diabetes mellitus and neutropenia), classification of surgery within up to 7 days prior to admission, acute severity of illness and primary reason for admission to the critical care unit. The following were reported as therapies received during the first 24 hours following admission to the critical care unit: TPN, systemic antimicrobial use, immunosuppressive use and CVC use. Corticosteroids were included as immunosuppressives. Organ support was reported if received at any time during the critical care unit stay. Fungal colonisation was reported according to the time of the report (pre-admission or identified in unit). Outcomes reported were mortality in the original critical care unit and at ultimate discharge from acute hospital, length of stay in the original critical care unit and total length of stay in acute hospital, stratified by survival status. Use of topical and systemic antifungal therapy was reported by time of initiation (pre-admission or in unit). For admissions with IFD identified in the unit, the timing of the first systemic antifungal was also reported relative to the timing of IFD.

For the purpose of summarising case mix, outcomes and antifungal use, the cohort was divided into groups of admissions by fungal species. The groups consisted of admissions with IFD-positive for Candida albicans (Candida albicans IFD); admissions with IFD-positive for other Candida species (Candida non-albicans IFD); and admissions with no IFD either prior to or during the critical care unit stay (no IFD). Admissions with IFD-positive for Candida of unknown species or non-Candida species were excluded from these grouped comparisons owing to small numbers. Admissions with IFD-positive for both Candida albicans and Candida non-albicans were included in the Candida albicans subgroup.

Additional data definitions for linked data from the Case Mix Programme and the Scottish Intensive Care Society Audit Group

Severe comorbidities were defined using the APACHE II method48 and must have been evident in the 6 months prior to critical care unit admission. The following severe comorbidities were reported: very severe cardiovascular disease (fatigue, claudication, dyspnoea or angina at rest – New York Heart Association Functional Class IV); severe respiratory disease (permanent shortness of breath with light activity due to pulmonary disease or receipt of home ventilation); chronic renal failure (current requirement for chronic renal replacement therapy for irreversible renal disease); chronic liver disease (biopsy proven cirrhosis, portal hypertension or hepatic encephalopathy); haematological malignancy (acute or chronic myelogenous/lymphocytic leukaemia, multiple myeloma or lymphoma); metastatic disease (distant metastases, documented by surgery, imaging or biopsy); and immunocompromise due to disease or treatment (human immunodeficiency virus/acquired immunodeficiency syndrome, daily high-dose steroid treatment, radiotherapy, chemotherapy, or congenital immunohumoral or cellular immune deficiency state).

Acute severity of illness was measured using the APACHE II Score48 and the ICNARC Physiology Score49 assessed during the first 24 hours following admission to the critical care unit. The APACHE II score comprises the APACHE II Acute Physiology Score (weightings for 12 physiological variables: temperature, mean arterial pressure, heart rate, respiratory rate, oxygenation, pH, sodium, potassium, creatinine, haematocrit, white blood cell count and Glasgow Coma Score) plus weights for age and severe conditions in the past medical history. The ICNARC Physiology Score comprises objective weightings for 12 physiological variables (temperature, systolic blood pressure, heart rate, respiratory rate, oxygenation, pH, sodium, urea, creatinine, urine output, white blood cell count and Glasgow Coma Score).

The primary reason for admission to the critical care unit was recorded using the ICNARC Coding Method, a hierarchical system with five tiers: surgical status, body system, anatomical site, physiological or pathological process, and medical condition. 50 Admissions were grouped by medical and surgical primary reasons for admission according to body system.

Organ support consisted of advanced cardiovascular, advanced respiratory, renal, gastrointestinal and neurological support, defined according to the CCMDS. Advanced cardiovascular support was defined as receipt of multiple intravenous and/or rhythm-controlling drugs, of which at least one must be vasoactive, used simultaneously to support or control arterial pressure, cardiac output or organ/tissue perfusion; continuous cardiac output monitoring; intra-aortic balloon pump or ventricular assist device; or temporary cardiac pacemaker. Advanced respiratory support was defined as receipt of invasive mechanical ventilation; bilevel positive airway pressure (BiPAP) via a translaryngeal tracheal tube or tracheostomy; continuous positive airway pressure (CPAP) via a translaryngeal tracheal tube; or extracorporeal respiratory support. Renal support was defined as receipt of acute renal replacement therapy or chronic renal replacement therapy where other acute organ support was administered. Gastrointestinal support was defined as receipt of parenteral or enteral nutrition. Neurological support was defined as central nervous system depression sufficient to prejudice the airway and protective reflexes (not caused by intentional sedation or deliberate overdose) or receipt of invasive neurological monitoring or treatment (e.g. intracranial pressure monitoring, jugular bulb sampling, external ventricular drain); continuous intravenous medication to control seizures and/or for continuous cerebral monitoring; or therapeutic hypothermia.

Critical care unit mortality was defined as death in the original critical care unit. Ultimate acute hospital mortality was defined as death before final discharge from acute hospital and included deaths after direct transfer to another acute hospital from the hospital housing the critical care unit. Length of stay was reported as both critical care unit and total acute hospital stay. Total acute hospital stay included continuous stay in acute hospital, even if transferred from/to another acute hospital.

Statistical methods

Analyses were performed using Stata version 10.1 (StataCorp LP, College Station, TX, USA). Variables were summarised as either mean and standard deviation (SD) or median and IQR for continuous variables and as frequency and percentage for categorical variables. All analyses were univariable and no statistical testing was undertaken.

Development and validation samples

The FIRE Study data set was divided into the following groups to form the development and validation samples for the risk models:

1. Development sample All admissions to a random sample of participating critical care units in England, Wales and Northern Ireland, from July 2009 to December 2010 (approximately 40,000 admissions).

2. Random validation sample All admissions to the remaining participating critical care units in England, Wales and Northern Ireland (approximately 5000 admissions).

3. Temporal validation sample All admissions to critical care units in the development sample, from January to March 2011 (approximately 10,000 admissions).

4. Geographic validation sample All admissions to participating critical care units in Scotland (approximately 5000 admissions).

Patient inclusion/exclusion criteria

Data for the FIRE Study were collected on all admissions to participating critical care units. The following exclusions were applied to the data set for the development and validation of risk models.

For model 1, at admission to the critical care unit:

• age < 18 years on admission;

• second and subsequent admissions of the same patient during the same acute hospital stay

• neutropenia (absolute neutrophil count < 1 × 10⁹/l)

• active haematological malignancy evident during the 6 months prior to admission to the critical care unit

• admission to the critical care unit following solid organ transplant

• IFD identified within up to 7 days prior to admission to the critical care unit; and

• receipt of systemic antifungals within up to 7 days prior to admission to the critical care unit.

For model 2, at 24 hours following admission to the critical care unit, as model 1 plus:

• death or discharge from the critical care unit within 24 hours

• IFD identified during the first 24 hours following admission to the critical care unit; and

• receipt of systemic antifungals during the first 24 hours following admission to the critical care unit.

For model 3, at the end of the third calendar day following admission to the critical care unit, as model 2, plus:

• death or discharge from the critical care unit before the end of the third calendar day

• IFD identified before the end of the third calendar day following admission to the critical care unit; and

• receipt of systemic antifungals before the end of the third calendar day following admission to the critical care unit.

Selection of candidate variables

Variables considered for the three models were taken from the FIRE Study database having been selected based on their association with IFD in previous literature and on expert clinical opinion. Owing to the low number of events, and to maximise the number of EPV, each potential variable included in the data set was considered for inclusion in the modelling process with selection of candidate variables based on the strength of evidence in the existing literature (see Chapter 2), prevalence, data completeness, and consideration of whether or not the same concept was better captured by alternative variables. Risk factor data for interventions, organ support, therapies and fungal colonisation were updated using additional data available at the second and third time points (24 hours and the end of the third calendar day following admission to the critical care unit). At the second and third time points, candidate variables were included in the full model if they were either included or close to being included in the final model from the previous time point or if the prevalence of the risk factor had increased substantially from the previous time point. Physiological variables were only available from the first 24 hours following admission to the critical care unit; they were therefore introduced at the 24-hour time point and retained in the model at the end of the third calendar day (still using data from the first 24 hours) if included in the final model at 24 hours.

Handling of missing data

Extensive data validation was used to ensure that the data were as complete as possible (see Chapter 3). A complete case analysis was undertaken when the proportion of missing data was low (< 1%). Candidate predictors with substantial numbers of missing data were excluded if the problem was thought likely to be due to inherent difficulties in data collection, rendering any model using such variables difficult to apply in practice. When the proportion of missing data was larger than 1% but not considered to be a problem likely to recur then patients without evidence indicating the presence of the risk factor were assumed not to have it.

Model development

Logistic regression models were derived to model the risk of subsequently developing invasive Candida infection based on information available at the three time points. Robust standard errors were used to allow for clustering within critical care units. Candidate variables were identified and alternative approaches to modelling each individual risk factor were compared and evaluated in univariable analyses. All candidate variables were then included in a ‘full’ multivariable model and the model was progressively simplified using backwards stepwise selection with the least statistically significant being removed at each step.

Variables that remained in the final admission model were considered for inclusion in the full 24-hour model along with additional physiology variables measured only after admission and variables where lack of effect in the admission model might have been due to low prevalence. Variables updated at each time point were updated in each model. The surgery variable was recoded at this stage to combine the ‘elective with no complications’ and ‘elective with complications’ categories. A similar approach was taken to decide on candidate variables for the 3-day model.

At each stage the model was fitted in the development sample and the performance of the model was assessed. Model discrimination was assessed with the c-index,30 equivalent to the area under the ROC curve,31 calibration by graphical plots of observed against expected risk, and overall fit by Brier's score. 63

Internal validation

Bootstrapping with 200 bootstrap samples was used on the development sample to internally validate the final selected model at each time point and to estimate optimism adjusted measures of the model discrimination and overall fit. 64 Overfitting is a phenomenon whereby the process of selecting variables for a risk model and estimating the coefficients results in a model that has very good fit in the data set used to develop the model but will fit less well when applied to other data. The bootstrap procedure refits the model in each bootstrap sample and compares the performance of the refitted model in the bootstrap sample with that of the refitted model in the original development data set. This gives, for each bootstrap sample, an estimate of the optimism in each of the measures of model performance that can be averaged and subtracted from the original values to give optimism-adjusted measures.

External validation

The final selected model at each time point was evaluated in the three external validation samples, chosen to test different aspects of future performance. Evaluation in the random validation sample, drawn from a random selection of critical care units withheld from the development sample, tested the performance of the model when applied to different settings within the same health system and time period. The temporal validation sample, drawn from the final 3 months' data collection from the critical care units in the development sample, tested the performance of the model in the same critical care units over time. Finally, the geographical validation sample, drawn from critical care units in Scotland, tested the performance of the model when applied in a different geographical region with a different (but similar) health-care system. The models were evaluated in each validation sample separately and then in all three samples combined. Model performance in the validation samples was assessed using the same measures as in the development sample.

Comparison with existing clinical decision rules

The performance of the risk models at the end of calendar day 3 was compared with that of existing clinical decision rules identified from the systematic literature review (see Chapter 2) using the full validation data set. Models at earlier time points were not considered, as all existing rules had been developed using only patients with stays in the critical care unit of at least 3 days. For comparison with the clinical decision rules, three alternative thresholds were applied to the risk predictions FIRE Study models: 0.5%, 1% and 2%. The corresponding clinical decision rules have been identified as F1, F2 and F3, respectively. The following existing clinical decision rules were included in the comparison:

• Three rules presented in Ostrosky-Zeichner et al. :27

  • (any antibiotic use days 1 to 3) and (central venous catheter days 1 to 3) OZ1

  • (any antibiotic use days 1 to 3) and (central venous catheter days 1 to 3) and (at least one of surgery days −7 to 0, immunosuppressive use days −7 to 0, pancreatitis, TPN days 1 to 3, renal support days 1 to 3, steroid use days −7 to 3) OZ2

  • [(any antibiotic use days 1 to 3) or (central venous catheter days 1 to 3)] and (at least two of surgery days −7 to 0, immunosuppressive use days −7 to 0, pancreatitis, TPN days 1 to 3, renal support days 1 to 3, steroid use days −7 to 3) OZ3

• Three rules presented in Paphitou et al. :28

  • (at least one of diabetes, TPN days −7 to 0, renal support days 1 to 3) P1

  • (at least one of diabetes, TPN days –7 to 0, renal support days 1 to 3, broad-spectrum antibiotic use days –7 to 3) P2

  • (at least one of diabetes, TPN days –7 to 0, renal support days 1 to 3) and (broad-spectrum antibiotic use days –7 to 3) P3

Here days –7 to 0 denotes the time period within up to 7 days prior to admission to the critical care unit, days 1 to 3 denotes the time period from admission to the end of the third calendar day, and days –7 to 3 denotes the entire time period from up to 7 days prior to admission until the end of the third calendar day following admissions. For applying these rules in the FIRE Study data set, antibiotic use within up to 7 days prior to admission to the critical care unit was approximated by the last antibiotic use prior to admission, and antibiotic use by the end of calendar day 3 following admission to the unit was approximated by the first antibiotic use following admission.

Clinical decision rules were assessed for sensitivity (the proportion of admissions with subsequent invasive Candida infection identified as being high risk), specificity (the proportion of admissions without subsequent invasive Candida infection identified as being low risk), PPV (the proportion of those identified as high risk who subsequently developed invasive Candida infection) and NPV (the proportion of those identified as low risk who did not subsequently develop invasive Candida infection).

Development and validation samples

The data set was divided into development and validation samples as follows:

1. development sample 39,685 admissions to 70 units

2. random validation sample 4669 admissions to 10 units

3. temporal validation sample 11,051 admissions to 66 units

4. geographic validation sample 5373 admissions to 16 units.

Patient inclusion/exclusion criteria

Figure 7 illustrates the selection of admissions for inclusion in the risk models at the three time points. Following exclusions, validated data on 35,455 admissions were available for the development of the admission model with data available on a further 18,834 for validation. Similarly data were available on 26,540 admissions for the development of the 24-hour model with data on a further 13,832 for validation. Finally, data were available on 16,405 for the development of the 3-day model with 8488 for validation. Table 13 describes the populations included in the development and validation samples for each of the three models.

FIGURE 7.

Flow diagram of patient inclusion/exclusion for model development and validation. D, development sample; V, full validation sample; VG, geographical validation sample; VR, random validation sample; VT, temporal validation sample.

Selection of candidate variables

Table 14 lists all of the variables considered and the rationale for inclusion or not as candidate variables in the modelling process.

Model development

The following modelling approaches were selected. Variables relating to surgery within up to 7 days prior to admission were combined and modelled as a single categorical variable with the following categories: no surgery, elective/scheduled surgery with no unexpected complications, elective/scheduled surgery with unexpected complications, and emergency/urgent surgery. An additional categorical variable was created representing gastrointestinal surgery with and without an open abdomen following surgery. Categorical variables were created for the number of lines in arteries (none, one, two or more), number of catheters in central veins (none, one, two or more), number of intercranial devices (none, one, two or more), number of drains (none, one to three, four or more) and number of microbiological samples positive for fungal colonisation (none, one, two or more).

For the model at 24 hours, highest central temperature (or non-central temperature if no central temperature available), highest respiratory rate, lowest systolic blood pressure, and highest heart rate were all included in the full model as continuous variables. However, those remaining in the final model (highest heart rate and lowest systolic blood pressure) were categorised to aid implementation of the model in practice. Highest heart rate was categorised above or below 100 beats per minute and lowest systolic blood pressure > and < 90 mmHg. This simplification resulted in a slight improvement in calibration. Lowest platelet count and lowest white blood cell count both had a higher proportion of missing data which was considered acceptable for a complete case analysis and were therefore categorised, before inclusion in the full model, to categories of lowest platelet count (above and below 190 × 10⁹/l) and lowest white blood cell count (above and below 12 × 10⁹/l). Patients with missing data for these variables were then assumed to be in the low-risk category (i.e. unmeasured laboratory values were expected to be normal). In the final models, the number of microbiological samples positive for fungal colonisation were recategorised as < 2 and ≥ 2 for the model at admission and as none and ≥ 1 for the models at 24 hours and the end of the third calendar day.

The full admission model (19 candidate variables with 27 parameters) had a c-index of 0.720 when fitted in the development sample. Following backwards elimination, the final admission model contained seven variables with 11 parameters (c-index 0.705). The full 24-hour model (16 candidate variables with 19 parameters) had a c-index of 0.828 when fitted in the development sample. Following backwards elimination, the final 24-hour model contained seven variables with 10 parameters (c-index 0.824). The full 3-day model (nine candidate variables with 13 parameters) had a c-index of 0.837 when fitted in the development sample. Following backwards elimination the final 3-day model contained five variables with seven parameters (c-index 0.835). Full details of the model selection process are reported in Table 15 and the final risk models are presented in Table 16. The distribution of predicted risk from the final risk models is shown in Figure 8, and the calibration of the final risk models based on three equal-sized groups on predicted risk is shown in Figure 9.

FIGURE 8.

Distribution of predicted risk from the final risk models in the development and validation samples. Patients with predicted risk of > 5% not shown: 65 (0.1%) at admission (44 development, 21 validation); 198 (0.5%) at 24 hours (119 development, 79 validation); 296 (1.2%) at end of calendar day 3 (170 development, 126 validation).

FIGURE 9.

Calibration of the final risk models in the development and validation samples.

Internal validation

Optimism adjusted estimates of the c-index and Brier's score are reported in Table 17. As anticipated, some overfitting was present, resulting in small reductions in the estimated model performance.

External validation

There was a further loss of model performance in the validation samples (see Table 17), reflecting the fact that the bootstrap procedure applied in the development sample did not account for all aspects of the modelling procedure that may have introduced overfitting (in particular, the stepwise selection of variables for the final models).

Comparison with existing clinical decision rules

Table 18 reports the sensitivity, specificity, PPV and NPV in the full validation sample for clinical decision rules based on risk predictions from the FIRE Study model at the end of the third calendar day following admission to the critical care unit compared with existing clinical decision rules. The comparison of sensitivity and specificity is illustrated in Figure 10. The most promising of the existing rules was that identified as the ‘best performing’ by Ostrosky-Zeichner et al. 27 (OZ3). This rule had better sensitivity, specificity, PPV and NPV than rules P1 and P3 from Paphitou et al. ,28 whereas the remaining rules (OZ1, OZ2 and P2) gave higher sensitivity but at the cost of treating 43–83% of admissions. Applying cut-points of 0.5% (F1) or 1% (F2) to risk predictions from the FIRE Study model gave similar performance to rule OZ1, with rule F1 giving slightly higher sensitivity but lower specificity, and rule F2 giving slightly lower sensitivity but higher specificity (see Figure 10).

FIGURE 10.

Comparison of sensitivity and specificity of clinical decision rules at the end of the third calendar day following admission to the critical care unit in the full validation sample (n = 8488).

Missing data

For the model on admission to the critical care unit, 74 candidate variables were identified. Of 54,289 admissions in the data set, only 15 had complete data for all of these variables (although it should be noted that many of these missing values were missing for structural reasons, e.g. data items related to surgery that were only collected on those who underwent surgery).

An attempt was made to estimate the missing values in the admission data set using multiple imputation66 via the Amelia algorithm,67 as provided by the amelia{Amelia} function written in R (www.r-project.org) (this report uses the R convention of writing the name of a function and its parent package as functionName{packageName}); however, the algorithm did not converge after 2500 iterations within the first imputation.

As a result of the difficulty in implementing multiple imputation, default-orientated cold-deck imputation was used, in which missing values were filled either with values corresponding to default values or, when a default value was not obvious for a variable, with a value indicating ‘not known’. For example, missing values of severe conditions in the past medical history and of fungal colonisation were set to ‘No’, and those of ethnicity were set to ‘Not reported’. The only continuous numerical variable was age, which was complete, and so hot-deck imputation was not required.

A similar approach was applied to the data sets for the models at 24 hours (36,943 admissions, 140 candidate variables) and the end of calendar day 3 (22,726 admissions, 148 candidate variables). Application of the Amelia multiple imputation algorithm was again unsuccessful owing to non-convergence. Cold-deck imputation was applied to categorical variables, as for the admission model. Imputation was then concluded for the numerical variables, such as the physiological measurements from the first 24 hours, by using k-NN hot-deck imputation (with k = 1) through the imputation{rminer} function in R.

Unbalanced data

The FIRE data set was extremely unbalanced (i.e. one outcome was much more common than the other). In the admission data set, following imputation, only 216 of 54,289 admissions (0.4%) had invasive Candida infection. A consequence of the extreme imbalance is that the application of a machine learning method such as tree induction or a neural network will result in a model that will be heavily biased toward correctly classifying the majority class (i.e. no invasive Candida infection) at the expense of the minority class (invasive Candida infection).

A standard approach to an unbalanced data set is to use undersampling, in which majority-class records are randomly selected from the data set until the number selected is equal to the number of minority-class records present in the data. If this is done repeatedly, a set of pseudo-samples is created from which a corresponding set of models can be induced;68 however, a problem with this approach is that, if the number of pseudo-samples does not adequately cover the majority-class cases, the resulting model may be far from optimal.

Another approach to an imbalanced data set is to create a set of n records from the majority class, equal in size to the set of minority-class records, which is somehow representative of the majority class. One way of attempting this is to apply learning vector quantisation69 to the majority-class cases. 70 If n vectors are used, they will converge towards becoming n centroids that collectively represent the distribution of the majority-class records. This was attempted with n = 216 using the lvq3{class} function, but the running time was too slow.

Deepa and Punithavalli71 proposed using a genetic algorithm72 to evolve a near-optimal pseudo-sample. This is done by defining a chromosome to be a collection of n genes, each gene corresponding to a majority-class record row number. In other words, each chromosome corresponds to a random selection of majority-class records. This approach was attempted using the galgo package, but more work was required to determine the minimal size of the genetic algorithm population to evolve a representative chromosome of cases.

Chawla et al. 73 describe the synthetic minority over-sampling technique (SMOTE) data rebalancing algorithm, which uses a combination of undersampling and oversampling to produce a near-balanced pseudo-sample. The SMOTE method is available from the SMOTE{DMwR} function, which was applied to the post-imputation data set. This quickly created a balanced pseudo-sample with 2808 records from the minority class (invasive Candida infection) and 2592 records from the majority class (no invasive Candida infection).

The data sets for the models at 24 hours and the end of calendar day 3 were similarly unbalanced, with 141 (0.4%) and 112 (0.5%) admissions with invasive Candida infection. The SMOTE method was applied to the imputed data sets, which resulted in a pseudo-sample for the 24-hour model consisting of 2115 records with and 1974 records without invasive Candida infection, and a pseudo-sample for the end of calendar day 3 model consisting of 1120 records with and 1008 records without invasive Candida infection.

Feedforward neural networks

The term ‘artificial neural network’ covers a wide variety of classification and regression models,74,75 and these networks have been applied to a variety of medical problems. 76,77

Logistic regression analysis is an established classical statistical technique for developing predictive models for critical care. 78 Feedforward neural networks (FFNNs) are regression models that consist of nested regression functions, and this creates highly flexible non-linear models. 79 Jaimes et al. 80 compared logistic regression models with neural networks for predicting mortality in ICUs, and they found that the FFNN gave better discrimination, as measured by the area under the ROC curve (0.7517 and 0.8782 for the logistic regression model and the neural network, respectively, p = 0.037). Clermont81 states that FFNNs do have benefits, such as adaptability to unforeseen interactions, so long as issues such as overfitting82 are addressed. Lukaszewski et al. 83 were able to develop an FFNN for the prediction of sepsis in intensive care patients based on biomarkers, such as cytokine and chemokine with high sensitivity and specificity (91.4% and 80.2%, respectively).

In the FIRE Study protocol, the use of FFNNs was proposed as an alternative to logistic regression modelling; however, there are some potential issues with the use of FFNNs. First, the issue of overfitting; however, this can be controlled by ‘early stopping’ and regularisation. 74 Second, and more importantly, the existence of local error minima. The training of a FFNN involves a search across an error surface that starts at some point in the space of all possible weights for a FFNN and hopefully finishes at the global minimum. Standard optimization algorithms such as gradient descent always aim to decrease error at every step during training, but this creates a potential problem. Error surfaces often consist of many minima, and the attempt by an algorithm to always decrease the error function at each step can result in a vector of weights reaching a minimum that is not the global minimum, with the weight vector remaining stuck at the local minimum. Consequently, the optimum weight vector is not reached. This problem can be reduced by restarting the training at many different starting points on the error surface, but this will substantially increase the training time of a FFNN.

The local minima problem with FFNNs led to the consideration of two alternative approaches to developing accurate classification models for the FIRE Study: support vector machines and random forests.

Support vector machines

A support vector machine maps the set of feature vectors in a training set to a corresponding set of points in a higher-dimensional space such that the associated classes are linearly separable by a hyperplane. Moreover, the classes are separated by a gap (margin) that is as wide as possible and which is defined by support vectors. 84

Verplancke et al. 85 compared the use of support vector machines with logistic regression for the prediction of hospital mortalities among patients with haematological malignancies, but they did not find the discrimination of support vector machines to be statistically better than that of logistic regression. In contrast, Van Looy et al. 86 compared the ability of support vector machines to predict tacrolimus blood concentrations with that of linear regression and found the support vector machines to be significantly better.

An approach to feature selection with support vector machines is as follows. A support vector machine (M₁) with feature vector x₁ is trained and tested on data using cross-validation. The total misclassification rate ψ₁ is recorded. The sensitivity of each x in x₁ is determined, where the sensitivity of x is the change in the output of the support vector machine for a unit change in x, whereas all of the other variables in x₁ are held at their median values. This gives the relative importance of the variables in x₁ with respect to M₁. If x_• is the least important variable, a support vector machine (M₂) with feature vector x₁ − {x_•} is trained and tested on the same data and its misclassification rate ψ₂ is recorded. If ψ₂ ≤ ψ₁ then (M₂) becomes the current model and the above comparison is repeated. This continues until either ψ₂ > ψ₁ or |x₁| = 1, whereupon x₁ is the selected feature subset.

The above approach was tested on a number of test data sets, such as the Hosmer and Lemeshow87 low-birthweight data, with the fit{rminer} function in R, which utilises Gaussian radial basis functions, but the results varied. It is believed that this is due to the support vector machine parameters being non-optimal, an area in which more work is required.

The svm.fs{penalizedsupport vector machine} function was also investigated. This function provides both the LASSO (least absolute shrinkage and selection operator)88 and SCAD (smoothly clipped absolute deviation)89 support vector machine feature selection techniques; however, the results varied with the test data sets, presumably because only linear kernel functions are provided by the function.

As a number of issues with the implementation of support vector machines for the FIRE Study were unable to be resolved, attention turned to the use of random forests to model the FIRE Study data.

Random forests

Tree induction is an established method for modelling data. It is based on the greedy, recursive partitioning of a feature space into disjoint rectangular regions, the optimal split at each successive partition being based on minimising a measure of class heterogeneity, such as the Gini index:90

where p^υ,k is the relative frequency of class k at node υ. Alternatively, splits can be based on maximising the reduction in a measure of tree deviance. 75

The larger the tree, the more likely that it will overfit the data; therefore, some form of pruning is used to obtain a subtree that generalises beyond the training data. Essentially, there is a trade-off between tree complexity (size) and goodness of fit to the data (tree node purity) according to a cost complexity function.

Breiman91 showed that significant improvements in accuracy could be achieved by using a collection of trees called a random forest. Formally, a random forest can be defined as a learning ensemble consisting of a bagging of unpruned decision tree learners with a randomised selection of features at each split.

Random forests go beyond the standard tree-induction process by introducing two additional types of randomisation. First, random-forest construction uses the concept of bootstrap aggregation (or bagging) in which ‘learners’ are trained independently on distinct bootstrap samples. Classification is determined by majority vote from the learners. Breiman92 proved that bagging decreases misclassification rates by reducing prediction variance. Second, random-forest construction selects a random subset of candidate predictor variables at each node split. This further reduces variance,91 as well as reducing search times for variable selections at the nodes.

Misclassification errors can be determined while a random forest is grown. When the training set for a tree is drawn by sampling with replacement during bootstrapping, about one-third of the cases are left out of the sample. The remaining out-of-bag cases are used as a test set for the tree. This approach provides an estimated misclassification error for a random forest that is very accurate in practice.

The random forest algorithm is summarised below:

Let D be the data set with N records and set of candidate predictors V. Let N_trees be the number of trees required for random forest F.

1. F ← ∅

2. for i ∈ {1, … ,N_trees} do

   1. Select bootstrap sample x* of size N from D. Let d be the set of cases in D not in x*.

   2. Grow an unpruned tree T_i on x* during which, at each node, randomly select m_try (< |V|) candidate predictors from V and determine the best split with respect to the Gini index using only the selected predictors. Do not prune T_i.

   3. Classify the records in d using Ti.

   4. F ← F ∪ {Ti}

3. Let k be the class that got most of the votes every time record x was out-of-bag. The proportion of times that k is not equal to the true class of x, averaged over all records, is the out-of-bag error estimate.

4. return F and the out-of-bag error estimate.

Random forests can be induced by a number of packages written in R, one of which is randomForest. The function randomForest{randomForest} was applied to the balanced pseudo-sample for the admission model (following imputation and balancing with the SMOTE algorithm) to create a random forest consisting of 100 trees. The number of variables randomly sampled as candidates at each split of a tree node (m_try) was set at √(v), where v = 74 was the number of candidate variables.

The out-of-bag estimated misclassification rate was 4.13%. This error rate as a function of the number of trees used in the random forest is shown in Figure 11. From the out-of-bag data, the estimated sensitivity, specificity and PPV were 99.5%, 92.4% and 92.5%, respectively.

FIGURE 11.

Out-of-bag estimated misclassification error, positive predictive error (1 − PPV) and negative predictive error (1 − NPV) as a function of the number of trees added to a random forest for the FIRE Study data set at admission.

The relative importance of the variables present amongst the trees of a random forest was determined by measuring, for each variable, the mean decrease in the Gini index of that variable across the nodes of all the trees (Table 19).

The technique described above for the creation of a random forest for the admission model was repeated for models at 24 hours and the end of calendar day 3 following admission to the critical care unit. The out-of-bag estimated overall misclassification rate for the 24-hour model was 2.86% (Figure 12). From the out-of-bag data, the estimated sensitivity, specificity and PPV for this model were 99.8%, 94.6% and 94.7%, respectively. The out-of-bag estimated overall misclassification rate for the end-of-calendar-day 3 model was 4.98% (Figure 13). The estimated sensitivity, specificity and PPV for this model were 97.7%, 92.4% and 92.8%, respectively. The relative importance of the variables for the 24-hour and end-of-calendar-day 3 models were determined from the mean decrease in the Gini index of those variables, as for the admission model. Tables 20 and 21 list the 20 most important variables for each model.

FIGURE 12.

Out-of-bag estimated misclassification error, positive predictive error (1 − PPV) and negative predictive error (1 − NPV) as a function of the number of trees added to a random forest for the FIRE Study data set at 24 hours.

FIGURE 13.

Out-of-bag estimated misclassification error, positive predictive error (1 − PPV) and negative predictive error (1 − NPV) as a function of the number of trees added to a random forest for the FIRE Study data set at the end of calendar day 3.

Population of interest

The population of interest, represented by the development and validation samples described in Chapter 5, was defined as non-neutropenic adult patients admitted to NHS critical care units, and excluded those with IFD prior to a decision time point and those receiving systemic antifungal therapy as part of routine clinical practice prior to a decision time point. This last criterion excluded a number of patients that received systemic antifungal therapy in the absence of IFD, and may therefore have represented use of antifungal prophylaxis. Figure 14 reports the proportion of patients that were excluded at each time point due to receiving systemic antifungal therapy that would otherwise have met the inclusion criteria for the models. Overall, 1635 patients (3.0%) otherwise eligible for the models were excluded for this reason at any of the three time points. There were 26 cases of invasive Candida infection among these patients, and assuming a RR of invasive Candida infection associated with antifungal prophylaxis of approximately 0.519 we may anticipate that a further 26 cases may have been prevented.

FIGURE 14.

Current use of antifungal prophylaxis and potential impact of antifungal prophylaxis on observed outcome. a, Assuming a RR of invasive Candida infection of 0.5 associated with antifungal prophylaxis;19 b, prior to 24 hours; c, prior to end of calendar day 3.

Strategies under comparison

The economic evaluation defined three decision time points at which to consider assessment of the risk of invasive Candida infection. These were:

• on admission

• at the end of the first 24 hours, and

• at the end of calendar day 3.

The alternative strategies recognised these time points (Table 22). Each time point defined a decision node, with two possibilities: either assessment of the risk of invasive Candida infection was not undertaken (no risk assessment) or risk assessment was undertaken (risk assessment). For example, under risk assessment on admission, the patients' risk was assessed according to the predicted risk from the on admission FIRE Study risk model. Those patients whose predicted risk of invasive Candida infection during the critical care unit stay exceeded the specified threshold (P_T) were designated ‘high risk’. These high-risk patients were then assumed to receive a single course of prophylaxis with fluconazole.

Strategy 1 assumed that there was no assessment of the risk of invasive Candida infection, as is usual practice in UK critical care units for the majority of patients (see Chapter 4). Strategies 2–4 assumed that risk assessment was performed using a single FIRE Study risk model at a single point in time – either on admission, at the end of 24 hours or at the end of calendar day 3. So, for example, strategy 4 assumed that patients' risk was only assessed at the end of calendar day 3 with prophylaxis initiated for those patients defined by the end of calendar day 3 FIRE Study risk model as having high risk of invasive Candida infection. Strategies 5–8 allowed for risk assessment at multiple time points. At any time point, risk assessment was only considered for the subgroup who were still in critical care, not already receiving systemic antifungal therapy, and without IFD. It was assumed that prophylaxis was initiated for those newly defined as high risk at the particular time point.

The risk thresholds (P_T) defined a priori according to the literature and expert opinion were 0.5%, 1%, 2%, 5% and 10%. However, the low incidence of Candida infection in the FIRE Study meant that under some of the strategies there were no infections in those designated high risk, and so 2% was the highest risk threshold considered in the analysis. In strategies involving risk assessment at multiple time points, the same risk threshold was applied at all time points.

Model structure

The model included a hypothetical cohort of 1000 cases with characteristics defined by the patients who met the FIRE Study inclusion criteria. The model structure (Figure 15) recognised the alternative strategies and time points described in Table 22.

FIGURE 15.

Structure of the model comparing alternative strategies for assessing risk and initiating prophylaxis for non-neutropenic, critically ill adult patients.

If, on admission, no risk assessment was undertaken (strategies 1, 3, 4 and 6), patients faced the risk of either having (R₁) or not having (1 − R₁) invasive Candida infection within the first 24 hours in the critical care unit. From the ‘no invasive Candida infection’ health state, patients faced a baseline risk of all cause death. For patients predicted to develop invasive Candida infection, an excess risk of death was applied. Under the strategies where risk assessment was undertaken on admission (strategies 2, 5, 7 and 8), the predicted probability of invasive Candida infection at any time during the critical care stay was estimated from the on-admission FIRE Study risk model. The proportion of patients (P₁) whose predicted risk of infection was higher than the risk threshold (e.g. 2%) were judged ‘high risk’ and assumed to receive prophylaxis. For these patients, the probability of developing invasive Candida infection at any time during critical care was estimated by multiplying the PPV from the ‘on-admission’ FIRE model (PPV₁) by the RR of invasive Candida infection associated with receiving antifungal prophylaxis versus no prophylaxis. 19 The proportion of patients (1 − P₁) whose predicted risk of infection was lower than the risk threshold (e.g. 2%) were judged ‘low risk’ and assumed not to receive prophylaxis. [For these patients, the probability of developing invasive Candida infection prior to the next decision time point in the strategy under consideration (or at any time during critical care, if admission was the only decision time point, i.e. strategy 2) was estimated as one minus the corresponding NPV from the ‘on admission’ FIRE model (1 − NPV₁). Note that the NPVs were calculated for each time point because the patients that did not receive antifungal prophylaxis may be reconsidered for risk assessment and prophylaxis at subsequent time points. By contrast, as antifungal prophylaxis was a ‘one-off’ treatment, those that received antifungal prophylaxis had a subsequent risk of invasive Candida infection at any time during the critical care stay.]

The risk of death, conditional on presence or absence of invasive Candida infection, was assumed the same whether or not patients received antifungal prophylaxis.

At the second assessment time point, the model considered those patients still on the critical care unit, not receiving systemic antifungal therapy and without IFD at the end of 24 hours. The strategies with no assessment at this time point (strategies 1, 2, 4 and 7) allowed for patients to face a baseline risk of invasive Candida infection (R₂) before the end of calendar day 3. For strategies with risk assessment at this time point (strategies 3, 5, 6 and 8), prophylaxis was initiated for the proportion of patients (P₂) whose predicted risk from the 24-hour FIRE Study risk model exceeded the risk threshold. The subsequent probability of invasive Candida infection was then calculated by multiplying the PPV of invasive Candida infection from the 24-hour FIRE Study risk model (PPV₂) by the RR of invasive Candida infection. For the proportion of patients (1 − P₂) whose predicted risk from the 24-hour FIRE Study risk model was below the risk threshold, the subsequent probability of invasive Candida infection prior to the next decision time point was estimated as one minus the corresponding NPV from the 24-hour FIRE Study risk model (1 − NPV₂).

At the third assessment time point, the model considered those patients still on the critical care unit, not receiving systemic antifungal therapy and without IFD at the end of the third calendar day. The strategies with no assessment at this time point (strategies 1, 2, 3 and 5), allowed for patients to face a baseline risk of invasive Candida infection (R₃) over the remaining critical care stay. For strategies with risk assessment at this time point (strategies 4, 6, 7 and 8), prophylaxis was initiated for the proportion of patients (P₃) whose predicted risk from the end of calendar day 3 FIRE Study risk model exceeded the risk threshold. The subsequent probability of invasive Candida infection was then calculated by multiplying the PPV of invasive Candida infection from the end of calendar day 3 FIRE Study risk model (PPV₃) by the RR of invasive Candida infection. For the proportion of patients (1 − P₃) whose predicted risk from the end of calendar day 3 FIRE Study risk model was below the risk threshold, the subsequent probability of invasive Candida infection at any subsequent time during critical care was estimated as one minus the corresponding NPV from the end of calendar day 3 FIRE Study risk model (1 − NPV₃).

At each decision node, a proportion of patients left the model because they died or were discharged from critical care. After the last decision node (end of calendar day 3), remaining patients were assigned the mean LOS and cost of those remaining in critical care in the FIRE Study for > 3 days, according to whether or not they had invasive Candida infection. These patients were then assigned lifetime quality-adjusted life-years (QALYs) according to the assumptions detailed below.

Model input parameters

The decision problem required information on the following input parameters: transition probabilities (risk of invasive Candida infection without prophylaxis, PPV and NPV following risk assessment, probabilities of death with and without invasive Candida infection, RR of invasive Candida infection after prophylaxis); costs; and lifetime QALYs. The estimation and sources for each set of input parameters are detailed below.

Transition probabilities

Costs

Risk assessment was assumed (based on expert opinion) to require 10 minutes of nurse time, giving a cost of £8.67. 96 Prophylaxis costs were calculated assuming a standard regimen of 400 mg for 10 days, with unit costs taken from the British National Formulary (BNF)97 (Table 26). This unit cost recognises that, according to the BNF, non-proprietary fluconazole intravenous infusion was available from September 2011. The recommended dose of 400 mg per day would be given as two 100 ml (200 mg) infusions. This would be an appropriate regimen to use for non-neutropenic, critically ill patients when the source of infection is unknown and there is a very high suspicion of invasive Candida infection. Once the patient can absorb, they may, even within critical care, be switched to the lower-cost oral formulation and this, together with the possibility of local discounts, is considered in the sensitivity analysis. The resultant unit cost for intravenous fluconazole of £7.78 per day replaces the unit cost of £45.74 per day used in a previous analysis of the FIRE Study, which was taken from a 2006 HPA report,4 inflated to 2010–2011 prices. The previous unit cost did not reflect the price reductions for the generic indication. Note also that, unlike the previous version, the base case assumes a more realistic treatment duration of 10 days rather than 14 days. The net effect is that in the base case the unit cost of antifungal prophylaxis is £77.80 per day, not the £640 assumed previously. Morbidity costs were included from critical care admission until ultimate discharge from acute hospital. These costs were calculated by estimating LOS both within and after critical care. Each day of critical care was classified according to Healthcare Resource Group 4 (HRG4) category derived from organ support data in the CCMDS, which forms part of the routine CMP data collection. Each bed-day was costed with the corresponding cost per bed-day from the UK ‘Payment by Results’ database. 98 Costs per bed-day in hospital after discharge from critical care were taken from the literature. 99 No costs after the initial hospital episode were considered. All costs were adjusted to 2010–11 price levels. 96

Lifetime quality-adjusted life-years

The main outcome measure was the lifetime QALY. This measure required using data on mortality from the original hospital episode (for patients with and without invasive Candida infection), and all-cause mortality after hospital discharge to project life-years following each strategy. These estimated life-years were combined with estimates of health-related quality of life (HRQOL) to project lifetime QALYs for each patient. 100 It was recognised that critical care survivors have a higher risk of death than the age-/ sex-matched general population. 101,102 There is a lack of work defining the size and duration of excess mortality following critical care survival, both generally and specifically for non-neutropenic adult patients. For example, for adult patients with severe sepsis (including sepsis shock) the strongest evidence is in support of an excess mortality of approximately 20% for up to 4 years after discharge from critical care,103 although some previous work has applied excess mortality for up to 25 years. 104 In this evaluation, the base-case analysis followed previous studies in taking a conservative approach and applied excess mortality of 20% for up to 4 years (see subsequent sensitivity analysis). 100,102,105 Future costs and outcomes were discounted at the recommended rate of 3.5%. 106

Base-case analysis

The probabilistic sensitivity analysis recognised parameter uncertainty by resampling the input parameters 5000 times from the designated distributions. Each iteration processed the patient cohort through each of the eight strategies described. For each strategy the model reported process measures and short-term end points, including the proportion of patients predicted to receive antifungal prophylaxis, the proportion having invasive Candida infection, and the mortality within critical care. The model also reported final end points including lifetime costs (£) and QALYs per patient for each strategy. Incremental costs, QALYs and incremental net benefits (INBs), at a threshold of £20,000 per QALY, were calculated as the differences in mean end points following each prophylaxis strategy compared with current practice. Across the 5000 runs, means were reported together with the 2.5 and 97.5 percentiles to give the limits of the 95% credible intervals. Cost-effectiveness acceptability curves (CEACs) were calculated according to the proportion of replications for which each strategy was the most cost-effective, i.e. had the maximum net monetary benefits across all eight strategies, at different levels of willingness to pay for a QALY gain (£0 to £50,000 per QALY gained). The analyses were repeated for the risk thresholds of 0.5%, 1% and 2%.

Recommendation 1: Further research is required to consider the full costs of antifungal prophylaxis

The economic analysis of the FIRE Study identified potentially cost-effective treatment strategies that would involve a substantial proportion of patients receiving antifungal prophylaxis. Few data currently exist on the potential downstream impact of such strategies on the management of future patients and the consequent impact on costs and outcomes as a result of the potential to promote antifungal resistance. Future research should consider the additional burden to patients whose treatment with antifungal agents becomes inappropriate owing to increased resistance. This research can inform future decision analytic models required to incorporate additional parameters such as the resistance rate and the ensuing effect on patient morbidity and mortality.

Recommendation 2: Further research should be conducted to inform the long-term survival, including quality and costs of survival, for the population of patients admitted to UK adult general critical care units

Very limited information was available to inform the estimates of long-term survival and quality of life in the decision model and many assumptions needed to be made. Increasing the available information on these important parameters would inform all future economic evaluations in this area. Further information on the long-term survival of critically ill patients may be obtained at low cost through record linkage between the CMP and the NHS Central Register. To date, linkage has been prevented by the lack of algorithms at the NHS Information Centre to link on NHS Number alone, without the need for more detailed patient identifiable data (the collection of which cannot be justified in the context of an ongoing national clinical audit using patient data without individual patient consent). Such algorithms are in the process of being developed, and we are working with the NHS Information Centre to conduct such record linkage in the future. More detailed information on the quality and costs of survival would require specific data collection, with patient consent, in the form of long-term follow-up studies.

Recommendation 3: Future research into treatment strategies for selecting patients for antifungal prophylaxis should consider combining clinical risk estimates, such as those from the FIRE Study risk models, with novel diagnostic tests based on biomarkers

The treatment strategies considered within the FIRE Study consisted of either giving or not giving antifungal prophylaxis to patients identified as high risk by the risk models at one or more of the time points. More complex, but potentially more beneficial, treatment strategies may incorporate a two-stage process, whereby the risk models are used to identify patients as being at high risk of IFD based on their clinical characteristics, and that these patients (rather than receiving antifungal prophylaxis directly as a result of the risk model) are provided a diagnostic test such as quantitative-PCR117 or (1→3)-β-d-glucan assay. 118 Alternatively, patient groups for application of rapid diagnostic tests may be selected based on individual clinical criteria, such as fever. Once suitable biomarkers are identified and appropriately evaluated, such treatment strategies should be evaluated using a similar decision model structure to that used in the FIRE Study.

Recommendation 4: Further research should be considered to inform estimates of the positive predictive value and negative predictive value of the FIRE Study risk models among non-neutropenic, critically ill adult patients admitted to UK adult general critical care units

Given the low event rate observed in the FIRE Study, the estimates of PPV and NPV were uncertain. In addition, the threshold levels of risk that could be considered in the decision model were limited by the sparsity of the data. Obtaining more information on these parameters would improve the decision model and may allow consideration of strategies involving use of prophylaxis at higher estimates of risk; however, such a study would be more costly to undertake as it would involve further primary data collection on risk factors as well as outcomes (although not the number of potential risk factors considered in the FIRE Study) and may therefore be best focused only on admissions remaining in the critical care unit at the end of calendar day 3, the decision time point at which treatment strategies appeared most likely to be cost-effective.

Recommendation 5: Further research should be considered to inform estimates of baseline risk of invasive fungal disease and associated outcomes among non-neutropenic, critically ill adult patients admitted to UK adult general critical care units

Owing to the low event rate, estimates of the baseline risk of IFD and associated outcomes were uncertain. Further data collection on these parameters would both provide a larger sample size to give more accurate estimates and permit monitoring over time in order to observe and evaluate trends. Research to inform estimates of the baseline risk of IFD and associated outcomes could be conducted at low cost by collecting a very small amount of additional information (at the minimum, fields for the presence of IFD and the date/time) either within or alongside the CMP. However, in considering such a study, it would be necessary to take note of the difficulty in applying the definitions of IFD that was highlighted through the reliability study. It may therefore be best to focus such work on monitoring rates of IFD in blood, for which the definitions may be most reliably applied.

Recommendation 6: Results of recommendations 1, 2, 4 and 5 (above) should be re-evaluated for their impact on the decision model and value of information analyses

As more information becomes available on any of the parameters in the decision model, the results of the model should be updated to evaluate whether or not the conclusions and/or the estimates of the value of further research have changed.

Recommendation 7: Further research into machine learning techniques should be considered to establish whether or not current barriers to their implementation at the bedside can be overcome

A number of technical and practical barriers were identified in the application of machine learning techniques to this decision problem. However, the random forests method showed considerable promise in producing potentially highly accurate models, and further consideration should be given to support vector machines, as the literature suggests that this approach may produce equally accurate but simpler models. Research looking to overcome these barriers and provide reliable, accurate models that can be delivered simply at the bedside should therefore be considered.