The future for diagnostic tests of acute kidney injury in critical care: evidence synthesis, care pathway analysis and research prioritisation

Phase 1: systematic review

Phase 1 consisted of three distinct systematic literature searches and reviews. Review 1 was broad and inclusive, analogous to a horizon scan, and sought to identify candidate or in-development relevant diagnostic tests that could be used in critical care to identify AKI. Review 2 employed a focused search that identified current evidence on the analytical validity, clinical validity, clinical utility and cost-effectiveness of the prioritised tests identified in search 1. The purpose of review 3, which is reported in the health economics chapter (see Chapter 5), was primarily to inform the design and parameterisation of the economic model for assessing cost-effectiveness. Using a series of highly focused literature searches it aimed to identify information describing the clinical care pathway and standard care for AKI in critical care, including investigation, clinical management, interventions, health-care resource use, morbidity, mortality, quality of life and other relevant outcomes descriptors.

Phase 2: evidence synthesis and meta-analysis

Phase 2 aimed to combine the findings from phase 1 and enable summary estimation of the relevant metrics for the diagnostic properties of tests, predominantly sensitivity and specificity for relevant outcomes.

Phase 3: care pathway analysis

Phase 3 was conducted in parallel to literature search 3, using formal consultation with relevant experts and patients with the primary aim of defining standard care for patients at risk of and experiencing AKI in critical care. This was necessary as part of the economic model development process, including the identification of key decision points at which tests change the process of care, mechanisms for changing the process of care, relationships and downstream knock-on effects that may be influenced by tests and key surrogate end points. In addition to expert consultation and information obtained from literature search 3, recent UK clinical trial data sets and UK registration study data sets were identified and analysed to inform the model structure and parameters.

Phase 4: decision-analytic model

Phase 4 consisted of the construction, parameterisation and analysis of a health economic decision model based on the care pathway developed in phase 3. The model was initially designed to calculate expected costs and quality-adjusted life-years (QALYs) for the current AKI care pathway. AKI diagnostic tests were then incorporated into the model at key decision points, as indicated by the current evidence and recommendations from the specialist advisory group. Divergence in the clinical pathway consequent on test results was modelled based on the diagnostic properties and decision impact of the tests.

Phase 5: sensitivity analysis and value of information analysis

Phase 5 consisted of a series of sensitivity analyses undertaken to explore the impact of parameter and structural uncertainties on the expected cost-effectiveness of each test. In addition, VOI analysis was undertaken in an attempt to quantify the value of further publicly funded research into diagnostic tests. VOI analysis is a method based on Bayesian decision theory that can be used to characterise the burden of uncertainty on a NHS reimbursement decision-maker or commissioner. 16,17 The results were presented to guide a future research programme in this area, as needed prior to adoption of any new tests by the NHS.

Objective

The objective of this search was to identify candidate or in-development relevant diagnostic tests that could be used in critical care to identify AKI.

Identification of studies

A broad strategy was employed to support a horizon-scanning search for tests and biomarkers that are, or can be, used for AKI diagnosis but that are not currently used as standard practice in emergency and critical care. Our search included ongoing studies in trials registers, conference proceedings and recently published studies in PubMed to ensure that all emerging test and biomarker evidence was identified. Inclusion of the major databases such as MEDLINE, EMBASE and Science Citation Index (via Web of Science) enabled comprehensive identification of relevant literature and allowed assessment of the likely weight of evidence (volume of research) for different tests should they be included in the next stage (search 2, evidence on candidate tests). Searches were carried out in September 2014 in ClinicalTrials.gov (US National Institutes of Health) (accessed 29 September 2014), Cochrane Central Register of Controlled Trials (via Wiley Online Library) (Issue 9 of 12, September 2014), Cochrane Database of Systematic Reviews (via Wiley Online Library) (Issue 9 of 12, September 2014), Conference Proceedings Citation Index – Science (Thomson Reuters’ Web of Science) (1990 to September 2014), Database of Abstracts of Reviews of Effect (via Wiley Online Library) (Issue 3 of 4, July 2014), EMBASE Classic and EMBASE (via Ovid) (1947 to 25 September 2014), International Clinical Trials Registry Platform (World Health Organization) [www.who.int/ictrp/en/ (accessed 29 September 2014)], MEDLINE (via Ovid) (1946 to September Week 3 2014), MEDLINE In-Process & Other Non-Indexed Citations (via Ovid) (25 September 2014), metaRegister of Current Controlled Trials (mRCT) (accessed 29 September 2014), PubMed (US National Library of Medicine) (1946 to September 2014) and Science Citation Index (Thomson Reuters Web of Science) (1900 to September 2014).

The search terms included index terms, free-text words, abbreviations and synonyms. Terms for AKI and specific biomarkers were identified from known relevant papers, database thesauri and suggestions from clinical members of the team. Searches for diagnostic studies are known to often retrieve very large numbers of citations. Although the use of diagnostic search filters is cautioned against – because of poor reporting and indexing – for pragmatic reasons we included some diagnostic terms in these scoping searches. We combined our search concept for AKI with a ‘fluid’ search concept as only tests using fluid samples (rather than tissue samples) could be considered for use in emergency and critical care. A variety of terms, headings and subheadings were combined to identify ‘tests’. The concepts used were:

1. AKI or conditions that imply AKI (e.g. tubular necrosis)

2. diagnostic tests or biomarkers [generic ‘test’ terms; MeSH terms, e.g. Acute-Phase Proteins/du (Diagnostic use); specific test terms, e.g. NGAL]

3. plasma, blood, urine and serum samples or specimens

4. evaluation of a test to predict or diagnose (e.g. sensitivity, accuracy, monitor, etc.).

The research team agreed that biomarkers developed and evidence gathered during the previous 10 years would be most relevant for the scoping review. All sources were searched from 2004 onwards, except for Web of Science, which was searched from 2008 onwards. The complete search strategies are available in Appendix 1.

Selection of studies

Studies were included if the participants were adults or children with new or existing AKI and (1) they were based in critical care (or another clinical setting if using a candidate test for future use in clinical care), (2) they evaluated a fluid biomarker not currently used as part of routine care for AKI, (3) they used the biomarker for AKI diagnosis, risk stratification or prediction of treatment benefit and (4) they involved at least 52 subjects. The participant criterion was based on the hypothesis that useful biomarkers in the literature have been evaluated in underpowered studies. Thus, we made the following assumptions: in order to be published, markers have a sensitivity and specificity of 0.6; to be inclusive in terms of power (1 – β) and significance level (α), we assumed power to be 0.7 and the significance level to be 0.2; and ideally a marker would be most useful if it had a sensitivity and specificity of 0.7 or 0.8 (although 0.8 is unlikely). To identify the most relevant literature, we anticipated that the majority of early-stage clinical validation studies would have equal numbers of cases and controls. In addition, to include studies with some chance of identifying markers with a sensitivity/specificity of > 0.8, we selected papers that studied at least 52 patients (26 per arm in trials) (Table 1).

Studies were excluded if the participants had CKD only, if they used a tissue biomarker or imaging technology for AKI diagnosis, or if the biomarker was used only for monitoring events subsequent to an AKI diagnosis (e.g. post discharge). Biomarker discovery and preclinical studies were also excluded, along with those evaluating the stability and/or storage of biomarker samples. No restrictions on language or study design were applied (Table 2).

Literature yield

Scoping searches conducted at the project proposal stage suggested a pool of around 2500 references; however, the final search identified 6329 articles, which was reduced to 4804 following exclusion of duplicates. Given this almost twofold increase in the volume of literature, and in a change to the protocol, the decision was taken to appraise articles in a single stage (title and abstract screen) rather than in two stages (title and abstract screen followed by full-text review). At the outset of the process, a training session was held to facilitate standardisation in screening and selection and ensure that each reviewer was aware of and understood the explicit inclusion and exclusion criteria. Abstracts were then screened for relevance by individual reviewers (with independent double assessment of 25% of articles), with 487 relevant articles identified.

Identification of candidate biomarkers

Data from each of the 487 studies were extracted by one reviewer (EDM) and used to produce a longlist of potential biomarkers. In total, 153 individual biomarkers (excluding serum creatinine) were identified. Non-novel biomarkers, that is, those that were already used as part of standard care in the diagnosis of kidney function, were excluded (n = 11), as were those for which the citation did not report complete details of the population studied (n = 19). The remainder (see Appendix 2) were then tabulated on four dimensions: volume of evidence, currency of evidence, total population included and biological or mechanistic plausibility (inflammatory marker, function marker, damage marker and cell cycle marker). Pragmatic limits were then set on each of the dimensions to enable the longlist to be reduced while trying to ensure that the focus of search 2 would be on those tests that would be likely to produce the most evidence. The limits and their rationales were as follows:

• the biomarker must have been considered in six or more studies – to try to ensure that there were sufficient data to allow appropriate synthesis for each test

• studies must have been published in the previous 5 years – to try to ensure that only the most promising biomarkers with recent evidence were included

• the biomarker must have been used with ≥ 1500 subjects in total (i.e. across studies) – to try to ensure that the biomarker had wide clinical use and would provide sufficient data for synthesis

• biomarkers from all four plausibility dimensions should be represented – to ensure that the focus did not exclude biomarkers with a specific biological or mechanistic function.

Application of these limits provided a shortlist of 10 candidate tests (Table 3), which, following review by the Project Delivery Board and ratification by the specialist advisory group, were used in search 2.

Objective

The objective of this search was to identify current evidence on the analytical validity, clinical validity, clinical utility and cost-effectiveness of the diagnostic tests identified in search 1.

Identification of studies

The world literature from 2004 to November 2015 was reviewed to identify existing research describing the diagnostic accuracy, analytical validity or cost-effectiveness of the 10 candidate biomarkers. It became clear when developing our search strategy that we would identify considerably more literature than suggested by our prestudy scoping searches. We therefore took the decision to carry out a more sensitive search over a shorter time period rather than a precise search covering the full duration of the databases. As our aim was to identify novel tests, we believed that this approach would be more inclusive, identifying the diversity of newer biomarkers and reducing the identification of older, established biomarkers. We searched the following databases for published and unpublished literature: ClinicalTrials.gov (US National Institutes of Health) (accessed 30 November 2015), Cochrane Central Register of Controlled Trials (via Wiley Online Library) (Issue 10 of 12, October 2015), Cochrane Database of Systematic Reviews (via Wiley Online Library) (Issue 11 of 12, November 2015), Conference Proceedings Citation Index – Science (Thomson Reuters Web of Science) (1990 to November 2015), Database of Abstracts of Reviews of Effect (via Wiley Online Library) (Issue 2 of 4, April 2015), EMBASE Classic and EMBASE (via Ovid) (1947 to 24 November 2015), Health Technology Assessment database (via Wiley Online Library) (Issue 4 of 4, October 2015), Health Management Information Consortium database (1983 to November 2015), International Clinical Trials Registry Platform (World Health Organization) (accessed 30 November 2015), MEDLINE (via Ovid) (1946 to November Week 2 2015), MEDLINE In-Process & Other Non-Indexed Citations (via Ovid) (24 November 2015), NHS Economic Evaluation Database (via Wiley Online Library) (Issue 2 of 4, April 2015), PubMed (US National Library of Medicine) (1946 to November 2015) and Science Citation Index (Thomson Reuters Web of Science) (1900 to November 2015).

The search consisted of index terms and text words for AKI, AKI synonyms and name variants for the 10 biomarkers: brain natriuretic peptide (BNP), cystatin C, IL-18, IL-6, KIM-1, L-FABP, N-acetyl-beta-D-glucosaminidase (NAG), Nephrocheck, NGAL and tumour necrosis factor alpha (TNF-α). The searches were limited by date of publication (from 2004) but no restrictions on language were applied (non-English-language papers were included at this stage but would be included for full-text assessment only if translation of the study data would be possible within the project timescale). All study designs except for single case studies were included. A diagnostic search filter was tested but not used in the final searches as it excluded potentially relevant abstracts. Search strategies for each database are provided in Appendix 3.

Selection of studies

Studies were included if they evaluated at least one of the outcome areas of clinical validity (including utility), analytical validity or cost-effectiveness and:

• the participants were adults or children with new or existing AKI

• they were based in critical care or the emergency department or included patients undergoing cardiac surgery

• they evaluated one or more of the 10 candidate biomarkers (fluid only)

• they included serum creatinine or another candidate biomarker as a comparator

• they assessed use of the biomarker for AKI-related decision-making

• they involved at least 50 subjects (unless reporting an aspect of analytical validity or cost-effectiveness).

The emergency department and cardiac surgery (including presurgery) were included as the most important alternative settings to consider when trying to capture relevant tests used outside ICUs/high-dependency units that might be transferable to the critical care setting. A cut-off point of ≥ 50 participants was used to include studies with some chance of identifying markers with a sensitivity/specificity of > 0.75 (see Table 1). This threshold was lowered slightly from search 1 to be as inclusive as possible when obtaining evidence relevant to the candidate biomarkers.

Studies were excluded if they involved kidney transplant patients only, studied a tissue biomarker or imaging technology for AKI diagnosis or used the biomarker only for predicting transplant rejection. Studies considering risk factors for AKI itself or combining the results for individual tests (with the exception of algorithmic biomarkers) were also excluded, as were case studies and descriptive or commentary pieces (Table 4).

Identified citations were stratified according to the test under study and in relation to whether they considered single or multiple biomarkers and were reviewed on a group-by-group basis. Titles and abstracts were screened for eligibility by one reviewer, with a random sample (15%) of articles independently screened by a second reviewer. Full-text articles of potentially relevant studies were then obtained and independently assessed by two reviewers to determine their inclusion status. Differences of opinion were discussed until a consensus was reached. To facilitate consistency in appraisal, the three main reviewers (EDM, NC and NW) initially assessed a batch of 14 papers independently and then met to discuss outcomes and to ensure clarity around the inclusion and exclusion criteria.

Data extraction

Data extraction was carried out by a single reviewer using a bespoke proforma (see Appendix 4). Data extracted included study methodology, country of study, study duration, setting and patient population (including baseline characteristics), type of candidate test and parameters, details of the diagnostic ‘gold standard’ and AKI classification system used, outcome measure(s) studied and findings. Analytical and validation factors associated with the physical measurement of a biomarker [including sensitivity, specificity, precision, parallelism, recovery, selectivity, limit of quantitation (LOQ) and vulnerability to interferences] were also sought for review in line with current US Food and Drug Administration (FDA) best practice guidelines18 and Clinical and Laboratory Standards Institute (CLSI) principles (see Table 38). 29 Pre-analytical variables that might influence the quality, integrity or composition of samples, including biological factors (such as within-patient variability, sample timing, medical history and diet and lifestyle) and technical factors (such as sample collection, processing, shipping and storage conditions), were also obtained. Prior to use, the data extraction proforma was piloted by the reviewers and team statisticians on a small number of studies and refined where necessary. Once the process of data extraction began, the statisticians then reviewed a sample of completed proformas to ensure that the relevant data were being extracted.

Quality assessment

Quality assessment for the evidence synthesis was carried out by one main reviewer (EDM) using the QUADAS-2 (quality assessment of diagnostic accuracy studies) tool,30 a diagnostic test-specific approach to determine potential study bias. In this system, papers are evaluated on the basis of patient selection, interpretation of the index test, appropriateness and interpretation of the reference standard and flow of patients and timing of tests. The applicability of each study to the question under review is also assessed (see Appendix 5). In keeping with good practice, studies at risk of bias were not excluded from the meta-analysis. Instead, an appraisal of the strength of the existing evidence has been reported and the findings are interpreted in light of this.

Results of the systematic review

As was the case with search 1, broadening the search strategy to include tests used following cardiac surgery impacted significantly on the volume of literature identified compared with that anticipated by the original scoping strategy, with a total of 5045 articles identified. Given the time implications associated with assessment and data extraction in a complex review such as this, we took the decision to focus initially on three of the candidates: a recently developed test that is receiving considerable marketing and is the only FDA-licensed test for AKI (Nephrocheck, which detects two biomarkers, TIMP-2 and IGFBP-7) and the two biomarkers for which there was the greatest amount of evidence (cystatin C and NGAL). The choice of these three tests as the focus for study was ultimately a pragmatic one, determined with advice from the specialist advisory group. Although fewer studies have used Nephrocheck than some of the other shortlisted tests, it is one of the most recent biomarkers that has shown enough promise to be approved by the FDA and so there is much interest in its clinical utility. We were aware that there would be more publications on the older biomarkers; however, our approach seemed an intelligent approach that avoided ignoring a new and potentially novel biomarker. Between them, these markers were considered in 3260 of the identified citations (65%), with 605 articles meeting the inclusion criteria for detailed review (Figure 2). We were unable to locate three papers and 23 non-English-language papers were set aside. Most of the excluded studies were either conference abstracts with no subsequent publication or studies that did not focus on the population or setting under review (Table 5). In total, 207 eligible papers31–237 were included in the review (see Appendix 6).

FIGURE 2.

Flow of studies into the review.

Location and setting

The majority of the eligible studies were carried out in Europe (n = 114, 55%) and North America (n = 63, 30%), with the USA being the single most prolific country (Table 6). Fifteen studies involved centres in multiple countries. The most commonly reported clinical setting was cardiac care (n = 86, 42%) – including 11 studies on contrast-induced nephropathy – followed by critical care (n = 85; 41%), the emergency department (n = 20; 10%) and the laboratory (n = 11, 5%). The setting of five studies was unclear. Less than one-quarter of studies (n = 46, 22%) reported the involvement of multiple centres (median 3, range 2–35) (see Appendix 6). Study duration ranged from 1 month to 7 years and most papers were published between 2012 and 2015 (70%).

Population

Most studies were small in scale (mean 227, median 112 participants), with the smallest being an analytical validity study involving 17 patients undergoing surgery for congenital heart disease and the largest being a study of 1635 adults admitted to the emergency department (see Appendix 6). Two studies did not specify population size: one on the analytical validity of NGAL and the other using decision analysis to model the cost-effectiveness of NGAL following cardiac surgery. In more than three-quarters of studies the population was adult patients (n = 166, 80%), with three studies including both adults and children. The source of the samples was not specified in two studies on the analytical validity of NGAL.

Biomarkers and outcome areas

The most commonly studied biomarker was NGAL (n = 145, 70%) followed by cystatin C (n = 91, 44%) and Nephrocheck (n = 10, 5%). One study evaluated all three tests, 35 evaluated cystatin C and NGAL and three evaluated Nephrocheck and NGAL. The Nephrocheck test is carried out using urine only, but cystatin C and NGAL were evaluated on a range of sample matrices, most commonly serum for cystatin C and urine for NGAL (Table 7). Forty-one studies evaluated two different matrices for the same or multiple biomarkers (n = 14 for cystatin C, n = 35 for NGAL). Although there is a standard definition for the diagnosis of AKI, various classifications can be used in clinical practice to grade the level of injury (Table 8). In this review, the most commonly used criteria in studies determining clinical validity were the RIFLE (Risk, Injury, Failure, Loss of kidney function and End-stage kidney disease) criteria,241 either alone or in conjunction with the Acute Kidney Injury Network (AKIN) criteria242 (n = 80, 42%). Almost one-fifth of studies did not report using one of the standard classifications.

Of the outcomes areas evaluated in this review, the most commonly considered was clinical validity (i.e. diagnostic accuracy; n = 193, 93%). A smaller number of studies focused on analytical validity (n = 12, 6%), with only two studies focusing on the cost-effectiveness of biomarker use (both in cardiac surgery, one looking at NGAL use in adults and the other looking at the use of cystatin C and NGAL in children). Almost one-quarter of studies (n = 46, 22%) reported on clinical validity alongside some aspect of analytical validity, usually related to brief details on limits of detection or inter-/intra-assay variation. In most of the studies dealing with clinical validity, the purpose of the use of the biomarker was AKI diagnosis, either solely (n = 126; 65%) or alongside risk prediction (n = 34, 18%) or prognosis (n = 21, 11%).

Quality assessment

With few exceptions, a cohort study design was the most frequently used study design in biomarker evaluation (n = 181, 87%), perhaps unsurprisingly given the topic under review. There were four RCTs, all of them relatively small scale (n = 71–204 adult patients), one in critical care and three involving cardiac surgery. One paper reported subgroup analysis from a larger trial but did not provide details of the parent study; the focus of two others was on the use of therapeutic drug treatment to prevent renal damage [Probucol (Sanofi Aventis, Paris, France) and erythropoietin].

Only 14% of studies (n = 29) reported a power calculation to justify the included sample size and less than half provided details of patient throughout [n = 88, 43%; Consolidated Standards of Reporting Trials (CONSORT) diagram, n = 38; written statement, n = 50]. For the most part, the reporting of methodology was poor and few studies stated that they had adhered to a quality standard when describing their study [Standards for Reporting Diagnostic Accuracy (STARD) guidance,243 n = 4; Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidance,244 n = 3; both STARD and STROBE, n = 1]. Fewer than two-thirds of clinical validity studies provided sensitivity and specificity data for a given biomarker cut-off threshold (n = 113, 59%). Only 42 eligible studies (20%) provided sufficient data to allow population of a confusion matrix and, therefore, their inclusion in meta-analysis: four studies evaluating Nephrocheck, 17 studies evaluating cystatin C and 35 studies evaluating NGAL (see Chapter 3).

Risk of bias among studies included in the evidence synthesis

When considered across the four domains – patient selection, index test, reference standard and flow and timing – only six45,48,50,54,58,66 of the studies included in the evidence synthesis had a low risk of bias for all; two studies49,72 did not report enough information to be able to allocate a level of bias for any of the domains (Table 9). Three studies had a high risk of bias for one domain: one36 that used a prespecified threshold for the biomarker cut-off point (potential index test bias), one in which the analysis did not include all patients39 and one that had a prolonged interval between the tests (both flow and timing bias). 57 In all of the remaining studies (n = 31, 74%), the level of bias was unclear in at least one of the domains. This was especially true for bias related to the reference standard, for which just over half of studies (n = 22, 52%) did not provide information on blinding, that is, whether or not the index test results were interpreted without knowledge of the results of the reference standard. There was little concern that the included studies were not applicable to this review.

Primary objective

The primary objective was to estimate pooled means and variances of sensitivity and specificity for each of the diagnostic tests considered. When appropriate data were available, these estimates were obtained separately for each of the health-care settings considered and for each of the sample media considered. When only one study was available, no meta-analysis was undertaken.

Identification of studies

Details on the search strategies used to identify studies and the process for study screening and evaluation, data extraction and quality assessment are presented in Chapter 2.

Full papers were retrieved for studies of all patients (< 18 years, ≥ 18 years) in which AKI diagnosis had been evaluated using any one or multiples of the three diagnostic tests considered (Nephrocheck, NGAL, cystatin C), in any of the sample media considered (blood serum, blood plasma or urine), in either of the health-care settings considered (critical care unit or cardiac surgery). Studies were excluded if they were not primarily located in either of these health-care settings.

Study methods

The gold standard for determining AKI diagnosis was defined as diagnosis according to the RIFLE,241 AKIN242 or KDIGO (Kidney Disease: Improving Global Outcomes)238 diagnostic and classification system, based on an assessment of serum creatinine levels and urine output (see Chapter 2).

Outcome measurements

The primary outcomes were sensitivity (the probability of the test being positive given that the true diagnosis is positive) and specificity (the probability of the test being negative given that the true diagnosis is negative), which are determined by comparison of the results of the experimental diagnostic test with the results of the gold standard method used in the study. Studies were excluded if the gold standard method used to determine the outcome was not described in sufficient detail. Studies were not excluded if the cut-off point used to assess the positive and negative status of the outcome in the experimental diagnostic test was not reported.

Diagnostic and staging systems for acute kidney injury

A number of diagnostic and staging systems for AKI have been used in diagnostic accuracy studies. The most commonly used make use of repeated serum creatinine measurements and measurement of urine output to diagnose and stage AKI. Three commonly used systems are the RIFLE, AKIN and KDIGO systems.

Key data extracted

The primary data extracted for inclusion in the meta-analysis are shown in Table 11. It is recommended in the STARD statement that a cross-tabulation of the index test results by the results of the reference standard is included in any study report,251,252 but it was anticipated that this information would not be present in all study reports. In this situation the elements of the confusion matrix were calculated using information describing the diagnostic outcomes and estimates of sensitivity and specificity. For example, if the sensitivity (s) and number of true diagnoses [given by the sum of the number of true positives (TPs) and the number of false negatives (FNs), i.e. (TP + FN)] were reported in the study then the number of TPs could be calculated as s.(TP + FN). A similar calculation for specificity (p) allowed the estimation of the number of true negatives (TNs): p.(FP + TN), where FP represents the number of false positives. Finally, given these estimates for TP and TN and the numbers of true outcomes [(TP + FN) and (FP + TN)], simple subtraction provided estimates for FN and FP.

Study exclusion

Studies were excluded from the meta-analysis if it was not possible to estimate values for the elements of the confusion matrix or if other key data could not be extracted. Further reasons for the exclusion of studies were if diagnosis was carried out in the emergency department rather than in the critical care unit and if the biomarker was measured on a relative scale rather than an absolute scale, for example unit of biomarker per unit of serum creatinine.

Data analysis

Simple diagnostic accuracy summaries [sensitivity, specificity and the diagnostic odds ratio (DOR) and its components – positive likelihood ratio (LR+) and negative likelihood ratio (LR–)] were produced for each study included in the meta-analysis. The sensitivity of a diagnostic test (T) is defined formally as the probability that the test will give a positive result if the patient has the disease (D+), in this case AKI. This is often referred to as the TP rate for a diagnostic test and can be expressed as a conditional probability:

The specificity of a diagnostic test is the probability that the test will give a negative result if the patient does not have the disease (D–), which is equivalent to 1 minus the FP rate for the test and can be expressed as the conditional probability:

Confidence intervals (CIs) were estimated for sensitivity and specificity based on the Wilson score interval method. 253

The LR+ of a diagnostic test is the probability of a patient with disease (D+) having a positive test result divided by the probability of a patient without disease (D–) having a positive test result:

Similarly, the LR– of a diagnostic test is the probability of a patient with disease having a negative test result divided by the probability of a patient without disease having a negative test result:

Confidence intervals for the LR+ and LR– were estimated using the method of Koopman. 254

The DOR for a test is the ratio of the odds of a positive test result for a patient with disease relative to the odds of a positive test result for a patient without disease:

Confidence intervals for log(DOR) were estimated based on the assumption that, as an odds ratio, the DOR is normally distributed. Estimates for DOR were then obtained by back-transformation.

The method of meta-analysis for diagnostic accuracy studies used here was the bivariate meta-analysis proposed by Reitsma et al. ,255 based on the methodology of van Houwelingen et al. 256 Briefly, if logit sensitivity (µ_si) and logit specificity (µ_pi) are

and

for each study i (with k studies included in the meta-analysis), the true logit sensitivity and logit specificity are then assumed to have a bivariate normal distribution across studies:

where σSPn is the covariance between logit sensitivity and logit specificity. This model is extended by incorporating the variability due to sampling through the variance of sensitivity (sS,i2) and specificity (sP,i2), as measured in each study:

and

assuming that 0 < p and s < 1 and that the number of subjects used to estimate sensitivity and specificity is large. 256 The final model is then a bivariate random-effects model of the form:

This model was estimated using likelihood-based methods using the mada package257 in the R Environment for Statistical Computing (The R Foundation for Statistical Computing, Vienna, Austria). It has been shown that this method is equivalent to the hierarchical regression meta-analysis proposed by and further developed by Rutter and Gatsonis when there are no study-level covariates. 258–260

Separate meta-analyses were conducted for each diagnostic test, sample media and health service setting. Pooled estimates of sensitivity, specificity, LR+, LR– and DOR can be estimated from back-transformed parameter estimates. Estimates from each study and pooled estimates from the meta-analysis are presented in forest plots. A summary receiver operating characteristic (SROC) curve was estimated, with estimates of the confidence and prediction region. 255,259 Approximate estimates of the variance of sensitivity and specificity for use in the economic model were determined using the delta method. 261

Tests of heterogeneity were not used, as such statistical methods (Cochran’s Q, I²) do not account for heterogeneity explained by phenomena such as positivity threshold effects and are not recommended by the Cochrane Diagnostic Test Accuracy Group. 262 Estimating the prediction region in the SROC curve is one way of examining the extent of heterogeneity by depicting a region within which, assuming that the model is correct, we have 95% confidence that the true sensitivity and specificity of a future study would lie. 260

Nephrocheck

Critical care unit: urine

Summaries of the baseline characteristics and test parameters for the included urinary Nephrocheck studies are shown in Table 12. Three studies were included,33,44,45 with a total of 1289 patients [199 patients (15.4%) with a diagnosis of AKI and 1090 patients (84.6%) without a diagnosis of AKI]. The sample for use in the test was taken on enrolment in all of the included studies. The outcome used to define the presence of AKI was consistent across each of the three studies (KDIGO stage 2 or 3). Similarly, the threshold used to define a positive test was consistent in all included studies [(TIMP-2) × (IGFBP-7) = 0.3]. Diagnostic accuracy summaries for the included studies are shown in Table 13.

TABLE 12 - Characteristics of studies included in meta-analysis for Nephrocheck in the critical care unit health-care setting using urine

+, with a diagnosis of AKI; –, without a diagnosis of AKI; IQR, interquartile range (quartile 1, quartile 3); SD, standard deviation.

First author and year	Patient group	Age (years)	Timing of test sample	Threshold	Outcome	Outcome
						+	–
Kashani 201345	Patients admitted to the critical care unit	Median (IQR): ± 64 (53, 73)	Within 18 hours of enrolment	0.3	KDIGO (2 or 3) within 12 hours	101	627
Bihorac 201433	Patients admitted to the critical care unit	Mean (SD): ± 63 (17)	On enrolment (median 15 hours from admission)	0.3	KDIGO (2 or 3) within 12 hours	71	337
Hoste 201444	Patients admitted to the critical care unit	Median (IQR): + 64 (54, 75); – 65 (54, 78)	On enrolment (within 24 hours of admission)	0.3	KDIGO (2 or 3) within 12 hours	27	126

TABLE 13 - Diagnostic accuracy summaries for studies included in meta-analysis for Nephrocheck in the critical care unit health-care setting using urine

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Kashani 201345	90	313	11	314	0.891 (0.815 to 0.938)	0.501 (0.462 to 0.540)	1.785 (1.609 to 1.980)	0.217 (0.124 to 0.382)	8.21 (4.305 to 15.7)
Bihorac 201433	65	182	6	155	0.915 (0.828 to 0.961)	0.460 (0.407 to 0.513)	1.695 (1.502 to 1.914)	0.184 (0.085 to 0.399)	9.21 (3.891 to 21.9)
Hoste 201444	24	59	3	67	0.889 (0.719 to 0.961)	0.532 (0.445 to 0.617)	1.898 (1.510 to 2.387)	0.209 (0.071 to 0.615)	9.09 (2.602 to 31.7)

Figure 3 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for Nephrocheck in the critical care unit using patient urine samples. The pooled sensitivity estimate was 0.90 (95% CI 0.85 to 0.93) and the pooled specificity estimate was 0.49 (95% CI 0.46 to 0.53). Figure 4 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction and confidence regions are small, suggesting limited heterogeneity. This is to be expected given the highly controlled similarity in these studies.

FIGURE 3.

Forest plot for studies included in the meta-analysis and pooled estimates for sensitivity and specificity Nephrocheck in the critical care unit health-care setting using urine.

FIGURE 4.

Summary receiver operating characteristic curve for studies included in the meta-analysis for Nephrocheck in the critical care unit health-care setting using urine.

Neutrophil gelatinase-associated lipocalin

Critical care unit: plasma

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 16. Eight studies were included,35,43,48,49,51,59,66,100 with a total of 1670 patients [381 patients (22.8%) with a diagnosis of AKI and 1289 patients (77.2%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies, with six studies35,43,49,59,66,100 using a diagnostic outcome and two studies48,51 using a failure-type outcome. There was also heterogeneity in the time at which the outcome assessment occurred (unclear, 48 hours and 7 days). Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 242 pg/ml49 to 558 ng/ml. 43 Diagnostic accuracy summaries for the included studies are provided in Table 17.

TABLE 16 - Characteristics of studies included in meta-analysis for NGAL in the critical care unit health-care setting using plasma

+, with a diagnosis of AKI; –, without a diagnosis of AKI; IQR, interquartile range (quartile 1, quartile 3); SD, standard deviation; ULN, upper-limit of normal.

Threshold measured in ng/ml unless stated otherwise.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)a	Outcome	Outcome
						+	–
Constantin 201036	Patients admitted to the critical care unit	Mean (SD): ± 57 (16)	Within 2 hours of admission	303	RIFLE (≥ R) within 7 days of admission	7	81
de Geus 2011100	Patients admitted to the critical care unit	Median (IQR): + 62 (50, 68); – 58 (43, 68)	On admission	417	RIFLE (≥ F) within 7 days of admission	56	461
Kokkoris 201248	Patients admitted to the critical care unit	Median (IQR): + 63 (50.3, 80.8); – 49 (35.0, 66.3)	Within 13 hours of admission	62	RIFLE (≥ R) within 7 days of admission	36	64
Linko 201351	Critically ill patients receiving ventilator support	Median (SD): ± 61 (51, 73)	Enrolment (at least 6 hours’ ventilator support)	304	RRT (AKIN 3, KDIGO 3)	87	282
Hjortrup 201543	Patients admitted to the critical care unit with severe sepsis and in need of fluid resuscitation	Median (IQR): ± 66 (63, 85)	Enrolment	558	KDIGO (≥ 1) within 48 hours of enrolment	31	100
Legrand 201549	Patients admitted to the critical care unit with oliguria (diuresis < 0.5 ml/hour/kg for > 6 consecutive hours)	Median (IQR): + 55 (41, 70); – 55 (41, 70)	At time of oliguria diagnosis	242 pg/ml	KDIGO (≥ 1) within 7 days of admission	41	70
Palazzuoli 201559	Acute heart failure (evidence of volume overload, pulmonary congestion or BNP greater than the ULN for age)	Mean (SD): + 78 (9); – 80 (8)	Within 24 hours of admission	134	AKIN (≥ 1) during the hospitalisation period	78	125
Shum 201566	Patients admitted to the critical care unit and expected to stay for > 24 hours	Median (IQR): + 74 (60, 83); – 64 (54, 78)	6 hours after admission	230	AKIN (≥ 1) within 48 hours of admission	45	106

TABLE 17 - Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the critical care unit health-care setting using plasma

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Constantin 201036	43	1	9	35	0.827 (0.703 to 0.906)	0.972 (0.858 to 0.995)	29.7 (4.29 to 206.4)	0.178 (0.098 to 0.323)	167.2 (20.20 to 1384)
de Geus 2011100	39	46	17	415	0.696 (0.567 to 0.801)	0.9 (0.869 to 0.924)	6.98 (5.04 to 9.65)	0.337 (0.227 to 0.502)	20.70 (10.85 to 39.49)
Kokkoris 201248	26	14	10	50	0.722 (0.560 to 0.842)	0.781 (0.666 to 0.865)	3.302 (1.992 to 5.473)	0.356 (0.207 to 0.612)	9.286 (3.628 to 23.77)
Linko 201351	59	73	28	209	0.678 (0.574 to 0.767)	0.741 (0.687 to 0.789)	2.62 (2.05 to 1.60)	0.434 (0.318 to 0.594)	6.033 (3.577 to 10.18)
Hjortrup 201543	18	24	13	76	0.581 (0.408 to 0.736)	0.76 (0.668 to 0.833)	2.42 (1.53 to 3.83)	0.552 (0.359 to 0.847)	4.385 (1.877 to 10.24)
Legrand 201549	33	14	8	56	0.805 (0.660 to 0.898)	0.8 (0.692 to 0.877)	4.024 (2.460 to 6.583)	0.244 (0.130 to 0.459)	16.50 (6.259 to 43.50)
Palazzuoli 201559	66	25	12	100	0.846 (0.750 to 0.910)	0.8 (0.721 to 0.861)	4.231 (2.942 to 6.083)	0.192 (0.113 to 0.326)	22.00 (10.34 to 46.82)
Shum 201566	26	25	19	81	0.578 (0.433 to 0.710)	0.764 (0.675 to 0.835)	2.45 (1.60 to 3.74)	0.553 (0.386 to 0.790)	4.434 (2.111 to 9.134)

Figure 5 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the critical care unit using patient plasma samples. The pooled sensitivity estimate was 0.72 (95% CI 0.65 to 0.79) and the pooled specificity estimate was 0.81 (95% CI 0.75 to 0.86). Figure 6 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction interval shows that there is a degree of heterogeneity in sensitivity and specificity, probably reflecting the variability in outcome measures and cut-off points, as well as other unidentified sources of heterogeneity.

FIGURE 5.

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the critical care unit health-care setting using plasma.

FIGURE 6.

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the critical care unit health-care setting using plasma.

Critical care unit: urine

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 20. Six studies were included,32,34,35,43,48,100 with a total of 1194 patients [283 patients (23.7%) with a diagnosis of AKI and 911 patients (76.3%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies. Of the studies included, five had a similar end point,32,34,35,43,48 being at least the least severe stage of the RIFLE, AKIN or KDIGO classification system. There was heterogeneity in the time up to which the outcome assessment occurred (from 28 hours up to the end of the hospital stay). Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 29.5 ng/ml32 to 1310 ng/ml. 100 Diagnostic accuracy summaries for the included studies are shown in Table 21.

TABLE 20 - Characteristics of studies included in meta-analysis for NGAL in the critical care unit health-care setting using urine

+, with a diagnosis of AKI; –, without a diagnosis of AKI; IQR, interquartile range (quartile 1, quartile 3); SD, standard deviation; SE, standard error of the mean.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
de Geus 2011100	Patients admitted to the critical care unit	Median (IQR): + 62 (50, 68); – 58 (43, 68)	On admission	1310	RIFLE (≥ F) within 7 days of admission	56	461
Chen 201234	Patients admitted to the critical care unit	Mean (SE): 66 (1)	On admission	110	AKIN (≥ 1) within 48 hours of admission	43	107
Kokkoris 201248	Patients admitted to the ICU	Median (IQR): + 63 (50.3, 80.8); – 49 (35.0, 66.3)	Within 13 hours of admission	58.5	RIFLE (≥ R) within 7 days of admission	36	64
Aydogdu 201332	Patients admitted to the ICU (without previous history of renal disease)	Mean (SD): + 70 (13) (sepsis); – 66 (10) (no sepsis); – 67 (15) (sepsis)	Daily throughout hospital stay	29.5	RIFLE (≥ R) during hospital stay	63	88
Cho 201335	Patients admitted to the medical or surgical ICU	Mean (SD): + 65.4 (14.8); – 60.4 (17.4)	On admission to the ICU	251	AKIN (≥ 1) within 5 days of admission	54	91
Hjortrup 201543	Patients admitted to the critical care unit with severe sepsis and in need of fluid resuscitation	Median (IQR): ± 66 (63, 85)	Enrolment	558	KDIGO (≥ 1) within 48 hours of enrolment	31	100

TABLE 21 - Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the critical care unit health-care setting using urine

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
de Geus 2011100	31	46	25	415	0.554 (0.424 to 0.676)	0.900 (0.869 to 0.924)	5.55 (3.87 to 7.96)	0.496 (0.370 to 0.665)	11.2 (6.09 to 20.6)
Chen 201234	28	17	15	90	0.651 (0.502 to 0.776)	0.841 (0.760 to 0.898)	4.098 (2.516 to 6.675)	0.415 (0.273 to 0.629)	9.882 (4.380 to 22.295)
Kokkoris 201234	28	18	8	46	0.778 (0.619 to 0.883)	0.719 (0.599 to 0.814)	2.765 (1.801 to 4.246)	0.309 (0.165 to 0.581)	8.944 (3.438 to 23.271)
Aydogdu 201332	55	24	8	64	0.873 (0.769 to 0.934)	0.727 (0.626 to 0.809)	3.201 (2.247 to 4.560)	0.175 (0.090 to 0.338)	18.333 (7.623 to 44.092)
Cho 201335	40	27	14	64	0.741 (0.611 to 0.839)	0.703 (0.603 to 0.787)	2.497 (1.753 to 3.555)	0.369 (0.230 to 0.590)	6.772 (3.177 to 14.435)
Hjortrup 201543	17	23	14	77	0.548 (0.378 to 0.708)	0.770 (0.678 to 0.842)	2.38 (1.48 to 3.85)	0.587 (0.392 to 0.877)	4.07 (1.74 to 9.48)

Figure 7 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the critical care unit using patient urine samples. The pooled sensitivity estimate was 0.70 (95% CI 0.59 to 0.80) and the pooled specificity estimate was 0.79 (95% CI 0.71 to 0.86). Figure 8 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction region shows that there is a degree of heterogeneity in sensitivity and specificity, probably reflecting the variability in outcome measures and cut-off points, as well as other unidentified sources of heterogeneity.

FIGURE 7.

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the critical care unit health-care setting using urine.

FIGURE 8.

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the critical care unit health-care setting using urine.

Cardiac surgery: plasma

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 22. Eight studies were included,31,40,47,60–63,67 with a total of 2644 patients [286 patients (10.8%) with a diagnosis of AKI and 2358 patients (89.2%) without a diagnosis of AKI]. The outcome used to define a diagnosis of AKI was largely consistent across the studies, although there was some heterogeneity in the time period in which the outcome was assessed. However, one study, used an end point that could not easily be mapped to the considered criteria. 67 Each of these could be considered to be somewhere between the least two severe categories of the RIFLE, AKN or KDIGO classification system. Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 150 ng/ml31 to 426 ng/ml. 67 Diagnostic accuracy summaries for the included studies are shown in Table 23.

TABLE 22 - Characteristics of studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using plasma

+, with a diagnosis of AKI; –, without a diagnosis of AKI; CABG, coronary artery bypass graft; IQR, interquartile range (quartile 1, quartile 3); SCr, serum creatinine; SD, standard deviation.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Haase 200931	Patients undergoing cardiac surgery	Mean (SD): + 74.2 (6.9); – 68.3 (10.3)	6 hours post CPB surgery	150	AKIN (≥ 1) within 5 days of admission	46	54
Haase-Fielitz 200940	Patients undergoing cardiac surgery with CPB	Mean (SD): + 75.9 (4.8); – 67.6 (9.9)	On arrival in the ICU (6 hours)	150	AKIN (≥ 1) within 5 days of admission	23	77
Tuladhar 200967	Patients undergoing cardiac surgery	Median (IQR): + 70 (57–78); – 66 (41, 81)	2 hours post surgery	426	An increase in SCr in the postoperative period by > 0.5 mg/dl from baseline	9	41
Perry 201063	Patients undergoing CABG surgery	Mean (SD): + 65 (12); – 65 (10)	At CPB	353.5	AKIN (≥ 1) within 4 days postoperatively	75	804
Parikh 201160	Patients undergoing cardiac surgery (CABG or valve surgery)	Mean (SD): ± 71 (10)	Soon after arrival in the ICU	293	AKIN (≥ 2) within 4 days postoperatively	60	1159
Kidher 201447	Patients undergoing aortic valve replacement	Mean (SD): ± 71 (9)	3 hours post surgery	150	RIFLE (≥ R) within 2 days postoperatively	16	37
Park 201561	Patients undergoing cardiovascular surgery with CPB	Median (IQR): + 65 (50, 74); – 54 (40, 61)	On admission	168.5	RIFLE (≥ R) within 3 days postoperatively	5	72
Perrotti 201562	Patients undergoing cardiac surgery	Mean (SD): ± 77 (6)	15 minutes after interruption of extracorporeal circulation	178	RIFLE (≥ R) within 2 days postoperatively	52	114

TABLE 23 - Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using plasma

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Haase 200931	34	14	12	40	0.739 (0.597 to 0.844)	0.741 (0.611 to 0.839)	2.851 (1.760 to 4.618)	0.352 (0.211 to 0.587)	8.095 (3.303 to 19.84)
Haase-Fielitz 200940	18	17	5	60	0.783 (0.581 to 0.903)	0.779 (0.675 to 0.857)	3.545 (2.212 to 5.681)	0.279 (0.127 to 0.611)	12.706 (4.114 to 39.24)
Tuladhar 200967	7	14	2	27	0.778 (0.453 to 0.937)	0.659 (0.505 to 0.784)	2.278 (1.314 to 3.948)	0.337 (0.097 to 1.168)	6.750 (1.235 to 36.91)
Perry 201063	29	149	46	655	0.387 (0.285 to 0.500)	0.815 (0.786 to 0.840)	2.086 (1.515 to 2.873)	0.753 (0.627 to 0.904)	2.771 (1.685 to 4.558)
Parikh 201160	28	220	32	939	0.467 (0.346 to 0.591)	0.810 (0.787 to 0.832)	2.458 (1.830 to 3.304)	0.658 (0.519 to 0.835)	3.735 (2.203 to 6.332)
Kidher 201447	13	4	3	33	0.812 (0.570 to 0.934)	0.892 (0.753 to 0.957)	7.516 (2.892 to 19.53)	0.210 (0.075 to 0.587)	35.750 (7.013 to 182.23)
Park 201561	4	8	1	64	0.800 (0.376 to 0.964)	0.889 (0.796 to 0.943)	7.200 (3.278 to 15.81)	0.225 (0.039 to 1.301)	32.000 (3.172 to 322.8)
Perrotti 201562	28	33	24	81	0.538 (0.405 to 0.667)	0.711 (0.621 to 0.786)	1.860 (1.269 to 2.726)	0.650 (0.474 to 0.891)	2.864 (1.452 to 5.647)

Figure 9 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the cardiac surgery setting using patient plasma samples. The pooled sensitivity estimate was 0.62 (95% CI 0.49 to 0.74) and the pooled specificity estimate was 0.78 (95% CI 0.75 to 0.81). Figure 10 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction region suggests limited heterogeneity when considering specificity, with far greater heterogeneity when considering sensitivity, confirming the observations that can be made from the forest plot (see Figure 9).

FIGURE 9.

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the cardiac surgery health-care setting using plasma.

FIGURE 10.

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the cardiac surgery health-care setting using plasma.

Cardiac surgery: serum

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 24. Two studies were included,38,68 with a total of 239 patients [53 patients (22.2%) with a diagnosis of AKI and 186 patients (77.8%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies, with one study38 using a less stringent version of the AKIN stage 1 classification without justification. Similarly, the threshold used to define a positive test was not consistent, being 0.62 ng/ml in one study38 and 133.7 ng/ml in the other study. 68 Diagnostic accuracy summaries for the included studies are shown in Table 25.

TABLE 24 - Characteristics of studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using serum

+, with a diagnosis of AKI; –, without a diagnosis of AKI; CABG, coronary artery bypass graft; PCI, percutaneous coronary intervention; SCr, serum creatinine; SD, standard deviation; STEMI, ST elevation myocardial infarction.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Ghonemy 201438	Patients undergoing CABG surgery or valve replacement	Range: + 39–56; – 32–53	3 hours postoperatively	0.62	An increase in SCr either by 25% of the baseline or by 0.3 mg/dl above the baseline level within 24 hours postoperatively	17	33
Tung 201568	Patients with STEMI receiving PCI	Mean (SD): + 68.14 (12.6); – 61.33 (13.9)	At presentation	133.7	AKIN (≥ 1) within 48 hours of admission	36	153

TABLE 25 - Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using serum

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Ghonemy 201438	16	2	1	31	0.941 (0.730 to 0.990)	0.939 (0.804 to 0.983)	15.529 (4.032 to 59.81)	0.063 (0.009 to 0.420)	248.0 (20.87 to 2947)
Tung 201568	25	35	11	118	0.694 (0.531 to 0.820)	0.771 (0.699 to 0.831)	3.036 (2.112 to 4.363)	0.396 (0.240 to 0.653)	7.66 (3.432 to 17.12)

Figure 11 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the cardiac surgery setting using patient serum samples. The pooled sensitivity estimate was 0.84 (95% CI 0.43 to 0.97) and the pooled specificity estimate was 0.87 (95% CI 0.59 to 0.97). Figure 12 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. There appears to be considerable heterogeneity in both sensitivity and specificity.

FIGURE 11.

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the cardiac surgery health-care setting using serum.

FIGURE 12.

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the cardiac surgery health-care setting using serum.

Cardiac surgery: urine

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 26. Thirteen studies were included,41,50,52,53,56,58,60,64,65,67,69,70,72 with a total of 3226 patients [444 patients (13.8%) with a diagnosis of AKI and 2782 patients (86.2%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was largely consistent across the studies, with one study67 using a non-standard definition and one study41 using only the serum creatinine assessment tool of the AKIN criteria. However, there was heterogeneity in the time to outcome assessment across the studies. The threshold used to define a positive test was not consistent across all of the studies, with some studies using raw concentration values and concentrations normalised by units of urine creatinine. Diagnostic accuracy summaries for the included studies are shown in Table 27.

TABLE 26 - Characteristics of studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using urine

+, with a diagnosis of AKI; –, without a diagnosis of AKI; AMI, acute myocardial infarction; CABG, coronary artery bypass graft; CT, computerised tomography; EPO, erythropoietin; IQR, interquartile range (quartile 1, quartile 3); SCr, serum creatinine; SD, standard deviation.

Threshold measured in ng/ml unless stated otherwise.

Only absolute creatinine level used from stage 1 and all of stage 2 criterion.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)a	Outcome	Outcome
						+	–
Wagener 200872	Patients undergoing cardiovascular surgery	Mean (SD): + 69.5 (13.1); – 61.7 (14.9)	Immediately post surgery	23.5	AKIN (≥ 1) within 48 hours postoperatively	85	341
Han 200941	Patients undergoing cardiovascular surgery	Mean (SD): + 68.3 (2.3); – 60.4 (1.98)	Immediately post surgery	456.0 ng/mg of creatinine	Modified AKIN stages 1 and 2 within 72 hours postoperativelyb	36	54
Liangos 200950	Patients undergoing on-pump cardiovascular surgery	Mean (SD): + 73 (9); – 67 (12)	2 hours post surgery	166 ng/mg of creatinine	RIFLE (≥ R) within 72 hours postoperatively	13	90
Tuladhar 200967	Patients undergoing cardiac surgery	Median (IQR): + 70 (57, 78); – 66 (41, 81)	2 hours post surgery	393 ng/mmol of creatinine	An increase in SCr in the postoperative period by > 0.5 mg/dl from baseline	9	41
McIlroy 201053	Hospitalised patients undergoing contrast-enhanced CT	–	Immediately post surgery	8	AKIN (≥ 1) within 48 hours postoperatively	8	45
Parikh 201160	Patients undergoing cardiac surgery (CABG or valve surgery)	Mean (SD): ± 71 (10)	Soon after arrival in the ICU	102	AKIN (≥ 2) within 4 days postoperatively	60	1159
Oh 201258	Patients undergoing CABG surgery	Median (IQR): placebo: + 73 (69, 77.5), – 68 (62, 72); EPO: + 70 (62, 75), – 62 (56.5, 72.5)	Immediately post surgery	5	AKIN (≥ 1) within 72 hours postoperatively	21	50
Sargentini 201265	Patients undergoing cardiovascular surgery	Mean (SD): + 74 (6); – 67 (11)	4 hours post ICU admission	55.2 mg/g of creatinine	AKIN (≥ 1) within 48 hours postoperatively	15	37
Liu 201352	Patients undergoing cardiovascular surgery	Mean (SD): ± 63.0 (11.3)	Immediately postoperatively	131.1	AKIN (≥ 1) within 48 hours postoperatively excluding urine output	26	83
Munir 201356	Patients undergoing cardiovascular surgery	Median (IQR): + 56 (47–64); – 51 (45–61)	4 hours post CPB	109	AKIN (≥ 1) within 48 hours postoperatively	11	77
Prowle 201564	Patients undergoing cardiovascular surgery	Median (IQR): ± 70 (61–76)	Immediately postoperatively	195	RIFLE (≥ R) within 5 days postoperatively	25	68
Tziakas 201569	Patients admitted with acute, spontaneous (type 1) AMI undergoing cardiovascular surgery	Mean (SD): ± 62 (13)	During admission	Unclear	AKIN (≥ 1) within 48 hours postoperatively	118	687
Varela 201570	Patients undergoing cardiovascular surgery	Mean (SD): ± 68 (11)	6 hours	Unclear	AKIN (≥ 1) within 48 hours postoperatively	16	50

TABLE 27 - Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the cardiac surgery health-care setting using urine

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Wagener 200872	26	65	59	276	0.306 (0.218 to 0.410)	0.809 (0.764 to 0.848)	1.605 (1.089 to 2.365)	0.858 (0.738 to 0.997)	1.871 (1.096 to 3.194)
Han 200941	26	33	10	21	0.722 (0.560 to 0.842)	0.389 (0.270 to 0.522)	1.182 (0.881 to 1.585)	0.714 (0.383 to 1.333)	1.655 (0.665 to 4.117)
Liangos 200950	9	80	4	10	0.692 (0.424 to 0.873)	0.111 (0.061 to 0.193)	0.779 (0.538 to 1.127)	2.769 (1.016 to 7.551)	0.281 (0.073 to 1.084)
Tuladhar 200967	8	9	1	32	0.889 (0.565 to 0.980)	0.780 (0.633 to 0.880)	4.049 (2.175 to 7.540)	0.142 (0.022 to 0.910)	28.444 (3.131 to 258.4)
McIlroy 201053	2	41	6	4	0.250 (0.071 to 0.591)	0.089 (0.035 to 0.207)	0.274 (0.082 to 0.914)	8.437 (3.051 to 23.34)	0.033 (0.005 to 0.218)
Parikh 201160	30	209	30	950	0.500 (0.377 to 0.623)	0.820 (0.796 to 0.841)	2.773 (2.093 to 3.673)	0.610 (0.473 to 0.787)	4.545 (2.682 to 7.705)
Oh 201258	19	26	2	24	0.905 (0.711 to 0.973)	0.480 (0.348 to 0.615)	1.740 (1.289 to 2.35)	0.198 (0.051 to 0.765)	8.769 (1.844 to 41.693)
Sargentini 201265	8	10	7	27	0.533 (0.301 to 0.752)	0.730 (0.570 to 0.846)	1.973 (0.970 to 4.020)	0.640 (0.360 to 1.137)	3.086 (0.887 to 10.74)
Liu 201352	20	15	6	68	0.769 (0.579 to 0.890)	0.819 (0.723 to 0.887)	4.256 (2.571 to 7.047)	0.282 (0.139 to 0.572)	15.111 (5.183 to 44.055)
Munir 201356	6	1	5	76	0.545 (0.280 to 0.787)	0.987 (0.930 to 0.998)	42.000 (5.57 to 316. 8)	0.461 (0.241 to 0.880)	91.200 (9.123 to 911.7)
Prowle 201564	19	27	6	41	0.760 (0.566 to 0.885)	0.603 (0.484 to 0.711)	1.914 (1.327 to 2.761)	0.398 (0.193 to 0.821)	4.809 (1.702 to 13.584)
Tziakas 201569	14	17	2	33	0.875 (0.640 to 0.965)	0.660 (0.522 to 0.776)	2.574 (1.677 to 3.949)	0.189 (0.051 to 0.703)	13.588 (2.763 to 66.830)
Varela 201570	89	378	30	309	0.748 (0.663 to 0.817)	0.450 (0.413 to 0.487)	1.359 (1.200 to 1.539)	0.560 (0.407 to 0.772)	2.425 (1.562 to 3.766)

Figure 13 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the cardiac surgery setting using patient urine samples. The pooled sensitivity estimate was 0.66 (95% CI 0.54 to 0.76) and the pooled specificity estimate was 0.62 (95% CI 0.41 to 0.79). Figure 14 shows an estimate of the SROC with the 95% confidence region and 95% prediction region. The prediction interval covers almost all of the SROC space, suggesting that there is considerable heterogeneity between the studies included in this meta-analysis.

FIGURE 13.

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the cardiac surgery health-care setting using urine.

FIGURE 14.

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the cardiac surgery health-care setting using urine.

Cystatin C

Critical care unit: plasma

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 28. Three studies were included,32,48,49 with a total of 362 patients [140 patients (38.7%) with a diagnosis of AKI and 222 patients (61.3%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies. All of the studies can be considered to have used a similar end point, but there was heterogeneity in the time up to which the outcome assessment occurred (7 days post study entry up to during the hospital stay). Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 1040 ng/ml48 to 1500 ng/ml. 32 Diagnostic accuracy summaries for the included studies are shown in Table 29.

TABLE 28 - Characteristics of studies included in meta-analysis for cystatin C in the critical care unit health-care setting using plasma

+, with a diagnosis of AKI; –, without a diagnosis of AKI; IQR, interquartile range (quartile 1, quartile 3); SD, standard deviation.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Kokkoris 201248	Patients admitted to the critical care unit	Median (IQR): + 63 (50.3, 80.8); – 49 (35.0, 66.3)	Within 13 hours of admission	1040	RIFLE (≥ R) within 7 days of admission	36	64
Aydogdu 201332	Patients admitted to the ICU (without previous history of renal disease)	Mean (SD): + 70 (13) (sepsis); – 66 (10) (no sepsis); – 67 (15) (sepsis)	Daily throughout hospital stay	1500	RIFLE (≥ R) during the hospital stay	63	88
Legrand 201549	Patients admitted to the critical care unit with oliguria (diuresis < 0.5 ml/hour/kg for > 6 consecutive hours)	Median (IQR): + 55 (41, 70); – 55 (41, 70)	At the time of oliguria diagnosis	1375	KDIGO (≥ 1) within 7 days of admission	41	70

TABLE 29 - Diagnostic accuracy summaries for studies included in meta-analysis for cystatin C in the critical care unit health-care setting using plasma

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Kokkoris 201248	22	12	14	52	0.611 (0.449 to 0.752)	0.812 (0.700 to 0.889)	3.259 (1.838 to 5.779)	0.479 (0.313 to 0.733)	6.810 (2.719 to 17.055)
Aydogdu 201332	46	28	17	60	0.730 (0.610 to 0.824)	0.682 (0.579 to 0.770)	2.295 (1.632 to 3.226)	0.396 (0.257 to 0.609)	5.798 (2.838 to 11.848)
Legrand 201549	34	19	7	51	0.829 (0.687 to 0.915)	0.729 (0.615 to 0.819)	3.055 (2.031 to 4.595)	0.234 (0.118 to 0.467)	13.038 (4.946 to 34.364)

Figure 15 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for cystatin C in the critical care unit using patient plasma samples. The pooled sensitivity estimate was 0.72 (95% CI 0.59 to 0.82) and the pooled specificity estimate was 0.74 (95% CI 0.65 to 0.81). Figure 16 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. Examining the forest plot and prediction region suggests that there is greater heterogeneity in sensitivity than in specificity.

FIGURE 15.

Forest plot for studies included in the meta-analysis and pooled estimates for cystatin C in the critical care unit health-care setting using plasma.

FIGURE 16.

Summary receiver operating characteristic curve for studies included in the meta-analysis for cystatin C in the critical care unit health-care setting using plasma.

Critical care unit: serum

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 30. Four studies were included,34,42,46,71 with a total of 372 patients [110 patients (29.6%) with a diagnosis of AKI and 262 patients (70.4%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies, with one study using a definition that was less serious than the least serious stage of the RIFLE, AKIN or KDIGO classification system46 and one study basing the outcome on continuous urine collection. 71 Similarly, the threshold used to define a positive test was not consistent across all of the studies, ranging from absolute values of 1200 ng/ml46 to 1800 ng/ml34 and with other patient-specific relative thresholds used. Diagnostic accuracy summaries for the included studies are shown in Table 31.

TABLE 30 - Characteristics of studies included in meta-analysis for cystatin C in the critical care unit health-care setting using serum

+, with a diagnosis of AKI; –, without a diagnosis of AKI; CAG, coronary angiography; SCr, serum creatinine; SD, standard deviation; SE, standard error of the mean.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Herget-Rosenthal 200442	Patients predisposed to acute renal failure in the critical care unit	Mean (SD): + 70 (8); – 63 (11)	On admission	–	RIFLE (≥ R) within 48 hours	24	61
Villa 200571	Risk of renal failure	Range 21–86	In the morning	Elevated (no specific information)	Renal dysfunction (SCr < 80 ml/minute/1.73m2 – based on 24-hour urine sample)	25	25
Kato 200846	All patients scheduled for elective CAG returning to the critical care unit	Range 43–86	Prior to CAG	1200	An increase of > 25% from the baseline SCr value or an absolute increase of at least 0.5 mg/dl within 48 hours	18	69
Chen 201234	Patients admitted to the critical care unit	Mean (SE): 66 (1)	On admission	1800	AKIN (≥ 1) within 48 hours of admission	43	107

TABLE 31 - Diagnostic accuracy summaries for studies included in meta-analysis for cystatin C in the critical care unit health-care setting using serum

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Herget-Rosenthal 200442	13	3	11	58	0.542 (0.351 to 0.721)	0.951 (0.865 to 0.983)	11.014 (3.442 to 35.25)	0.482 (0.311 to 0.747)	22.848 (5.572 to 93.70)
Villa 200571	19	2	6	23	0.760 (0.566 to 0.885)	0.920 (0.750 to 0.978)	9.500 (2.469 to 36.55)	0.261 (0.129 to 0.529)	36.417 (6.575 to 201.7)
Kato 200846	17	10	1	59	0.944 (0.742 to 0.990)	0.855 (0.753 to 0.919)	6.517 (3.634 to 11.69)	0.065 (0.010 to 0.438)	100.300 (11.976 to 840.0)
Chen 201234	33	10	10	97	0.767 (0.623 to 0.868)	0.907 (0.836 to 0.948)	8.212 (4.450 to 15.15)	0.257 (0.149 to 0.443)	32.010 (12.239 to 83.72)

Figure 17 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for cystatin C in the critical care unit using patient serum samples. The pooled sensitivity estimate was 0.76 (95% CI 0.57 to 0.88) and the pooled specificity estimate was 0.91 (95% CI 0.85 to 0.95). Figure 18 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region, which again shows greater heterogeneity in sensitivity than in specificity.

FIGURE 17.

Forest plot for studies included in the meta-analysis and pooled estimates for cystatin C in the critical care unit health-care setting using serum.

FIGURE 18.

Summary receiver operating characteristic curve for studies included in the meta-analysis for cystatin C in the critical care unit health-care setting using serum.

Critical care unit: urine

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 32. Three studies were included,32,34,175 with a total of 745 patients [231 patients (31.0%) with a diagnosis of AKI and 514 patients (69.0%) without a diagnosis of AKI]. The definition of AKI used was fairly consistent across the studies, but the time period within which the outcome was assessed varied from 48 hours to the entire length of the hospital stay. Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 106 ng/ml32 to 200 ng/ml. 34 Diagnostic accuracy summaries for the included studies are shown in Table 33.

TABLE 32 - Characteristics of studies included in meta-analysis for cystatin C in the critical care unit health-care setting using urine

+, with a diagnosis of AKI; –, without a diagnosis of AKI; SD, standard deviation; SE, standard error of the mean.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Nejat 2010175	Patients admitted to the critical care unit	Mean (SD): + 62 (15); – 58 (18)	On admission	120	AKIN (≥ 1) within 7 days of admission	125	319
Chen 201234	Patients admitted to the critical care unit	Mean (SE): 66 (1)	On admission	200	AKIN (≥ 1) within 48 hours of admission	43	107
Aydogdu 201332	Patients admitted to the ICU (without a previous history of renal disease)	Mean (SD): + 70 (13) (sepsis); – 66 (10) (no sepsis); – 67 (15) (sepsis)	Daily throughout the hospital stay	106	RIFLE (≥ R) during the hospital stay	63	88

TABLE 33 - Diagnostic accuracy summaries for studies included in meta-analysis for cystatin C in the critical care unit health-care setting using urine

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Nejat 2010175	84	115	41	204	0.672 (0.586 to 0.748)	0.639 (0.585 to 0.690)	1.864 (1.540 to 2.256)	0.513 (0.394 to 0.668)	3.634 (2.346 to 5.631)
Chen 201234	20	17	23	90	0.465 (0.325 to 0.611)	0.841 (0.760 to 0.898)	2.927 (1.704 to 5.029)	0.636 (0.476 to 0.850)	4.604 (2.085 to 10.17)
Aydogdu 201332	54	18	9	70	0.857 (0.750 to 0.923)	0.795 (0.700 to 0.867)	4.190 (2.742 to 6.404)	0.180 (0.097 to 0.332)	23.333 (9.723 to 55.99)

Figure 19 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for cystatin C in the critical care unit using patient urine samples. The pooled sensitivity estimate was 0.68 (95% CI 0.43 to 0.86) and the pooled specificity estimate was 0.76 (95% CI 0.62 to 0.86). Figure 20 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. Heterogeneity appears to be considerable and greater in terms of sensitivity than specificity.

FIGURE 19.

Forest plot for studies included in the meta-analysis and pooled estimates for cystatin C in the critical care unit health-care setting using urine.

FIGURE 20.

Summary receiver operating characteristic curve for studies included in the meta-analysis for cystatin C in the critical care unit health-care setting using urine.

Cardiac surgery: serum

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 34. Five studies were included,31,38,39,64,68 with a total of 532 patients [147 patients (27.6%) with a diagnosis of AKI and 385 patients (72.4%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies, with one study38 using a definition that was less serious than the least serious stage of the RIFLE, AKIN or KDIGO classification system17 and one study basing the outcome on continuous urine collection (from 48 hours up to 4 days post study entry). Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 0.0265 ng/ml (26.5 pg/ml) to 1100 ng/ml. Diagnostic accuracy summaries for the included studies are shown in Table 35.

TABLE 34 - Characteristics of studies included in meta-analysis for cystatin C in the cardiac surgery health-care setting using serum

+, with a diagnosis of AKI; –, without a diagnosis of AKI; CABG, coronary artery bypass graft; IQR, interquartile range (quartile 1, quartile 3); PCI, percutaneous coronary intervention; SCr, serum creatinine; SD, standard deviation; STEMI, ST elevation myocardial infarction.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Haase 200931	Patients undergoing cardiac surgery in a tertiary hospital	Mean (SD): + 74.2 (6.9); – 68.3 (10.3)	On arrival in the ICU (6 hours)	1.1	AKIN (≥ 1) within 5 days of admission	46	54
Haase-Fielitz 200939	Patients undergoing cardiac surgery with CPB	Mean (SD): + 75.9 (4.8); – 67.6 (9.9)	On arrival in the ICU (6 hours)	1100	AKIN (≥ 1) within 5 days of admission	23	77
Ghonemy 201438	Patients undergoing CABG surgery or valve replacement	Range: + 39–56; – 32–53	3 hours postoperatively	0.0265	An increase in SCr either by 25% of the baseline value or by 0.3 mg/dl above the baseline level within 24 hours postoperatively	17	33
Prowle 201564	Patients undergoing cardiovascular surgery	Median (IQR): ± 70 (61–76)	Immediately postoperatively	1.24	RIFLE (≥ R) within 5 days postoperatively	25	68
Tung 201568	Patients with STEMI receiving PCI	Mean (SD): + 68.14 (12.6); – 61.33 (13.9)	At presentation	1.6	AKIN (≥ 1) within 48 hours of admission	36	153

TABLE 35 - Diagnostic accuracy summaries for studies included in meta-analysis for cystatin C in the cardiac surgery health-care setting using serum

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Haase 200931	34	18	12	36	0.739 (0.597 to 0.844)	0.667 (0.534 to 0.778)	2.217 (1.465 to 3.356)	0.391 (0.232 to 0.659)	5.667 (2.379 to 13.50)
Haase-Fielitz 200939	18	11	5	66	0.783 (0.581 to 0.903)	0.857 (0.762 to 0.918)	5.478 (3.043 to 9.863)	0.254 (0.116 to 0.554)	21.600 (6.646 to 70.20)
Ghonemy 201438	9	9	8	24	0.529 (0.310 to 0.738)	0.727 (0.558 to 0.849)	1.941 (0.950 to 3.968)	0.647 (0.375 to 1.117)	3.000 (0.884 to 10.18)
Prowle 201564	19	25	6	43	0.760 (0.566 to 0.885)	0.632 (0.514 to 0.737)	2.067 (1.411 to 3.028)	0.380 (0.185 to 0.780)	5.447 (1.922 to 15.44)
Tung 201568	28	47	8	106	0.778 (0.619 to 0.883)	0.693 (0.616 to 0.760)	2.532 (1.885 to 3.401)	0.321 (0.173 to 0.596)	7.894 (3.349 to 18.61)

Figure 21 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for cystatin C in the cardiac surgery setting using patient serum samples. The pooled sensitivity estimate was 0.73 (95% CI 0.65 to 0.80) and the pooled specificity estimate was 0.72 (95% CI 0.63 to 0.79). Figure 22 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The studies show limited evidence of heterogeneity, with a greater degree of heterogeneity with respect to specificity than sensitivity.

FIGURE 21.

Forest plot for studies included in the meta-analysis and pooled estimates for cystatin C in the cardiac surgery health-care setting using serum.

FIGURE 22.

Summary receiver operating characteristic curve for studies included in the meta-analysis for cystatin C in the cardiac surgery health-care setting using serum.

Cardiac surgery: urine

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 36. Two studies were included,50,69 with a total of 908 patients [131 patients (14.4%) with a diagnosis of AKI and 777 patients (85.6%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was similar in both studies, although the time frame over which the outcome was assessed varied by 24 hours. The threshold used to define a positive test was not consistent across the studies, with one study69 not clearly reporting the threshold used. Diagnostic accuracy summaries for the included studies are shown in Table 37.

TABLE 36 - Characteristics of studies included in meta-analysis for cystatin C in the cardiac surgery health-care setting using urine

+, with a diagnosis of AKI; –, without a diagnosis of AKI; AMI, acute myocardial infarction; SD, standard deviation.

First author and year	Patient group	Age (years)	Timing of sample	Threshold (ng/ml)	Outcome	Outcome
						+	–
Liangos 200950	Patients undergoing on-pump cardiovascular surgery	Mean (SD): + 73 (9); – 67 (12)	2 hours post surgery	192 ng/mg of creatinine	RIFLE (≥ R) within 72 hours postoperatively	13	90
Tziakas 201569	Patients admitted with spontaneous (type 1) AMI undergoing cardiovascular surgery	Mean (SD): ± 62 (13)	During admission	Unclear	AKIN (≥ 1) within 48 hours postoperatively	118	687

TABLE 37 - Diagnostic accuracy summaries for studies included in meta-analysis for cystatin C in the cardiac surgery health-care setting using urine

First author and year	TP, n	FP, n	FN, n	TN, n	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)	DOR (95% CI)
Liangos 200950	5	13	8	77	0.385 (0.177 to 0.645)	0.856 (0.768 to 0.914)	2.663 (1.136 to 6.241)	0.719 (0.464 to 1.115)	3.702 (1.047 to 13.083)
Tziakas 201569	76	309	42	378	0.644 (0.554 to 0.725)	0.550 (0.513 to 0.587)	1.432 (1.223 to 1.676)	0.647 (0.503 to 0.832)	2.214 (1.475 to 3.321)

Figure 23 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for cystatin C in the cardiac surgery setting using patient serum samples. The pooled sensitivity estimate was 0.52 (95% CI 0.27 to 0.76) and the pooled specificity estimate was 0.72 (95% CI 0.36 to 0.92). Figure 24 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. Examining the forest plot and prediction region in the SROC curve suggests that there is considerable heterogeneity between the studies in terms of sensitivity and specificity.

FIGURE 23.

Forest plot for studies included in the meta-analysis and pooled estimate for cystatin C in the cardiac surgery health-care setting using urine.

FIGURE 24.

Summary receiver operating characteristic curve for studies included in the meta-analysis for cystatin C in the cardiac surgery health-care setting using urine.

Population and perspective

The base-case primary evaluation focused on the assessment of AKI biomarkers for a population of adult all-comers (aged ≥ 18 years) admitted to the hospital critical care unit. A secondary analysis was also conducted to assess outcomes in a subgroup of patients admitted to critical care post cardiac surgery. These patients are known to be at particular risk of post-surgery renal failure and form a relatively homogeneous group who are managed in a controlled environment; they therefore represent a subgroup for whom there is a strong potential for management adjustments to lead to measurable changes in outcomes.

All analyses were conducted from a NHS and PSS perspective, as per the NICE reference case. 312 This aims to incorporate costs relating to NHS primary care, secondary care, community care and social care.

Testing strategies

Tests evaluated in the meta-analysis in Chapter 3 (Nephrocheck, cystatin C and NGAL) were included in the economic evaluation. Cystatin C and NGAL are currently available across three alternative media (plasma, urine and serum); each of these was considered separately in the analysis, which, together with the Nephrocheck test, resulted in a total of seven testing strategies.

All tests were assumed to be used once on entry to the critical care unit in addition to standard care, consisting of daily serum creatinine and urine output testing. Although these new tests could also be used sequentially or in a monitoring context, the majority of evidence on the diagnostic accuracy of these tests relates to single-time usage and this was the focus of the current systematic review. Assessment of sequential and monitoring testing strategies was therefore deemed beyond the scope of this analysis but remains a key area for future research.

Costs and health outcomes

All costs are reported in 2015 prices (Great British pounds) and were inflated when necessary using an online converter. 315

Health outcomes were measured in terms of QALYs. QALYs provide a generic measure of patient overall health and are a composite measure of patient survival weighted by quality of life (utility) over time, for example 1 year in full health is equivalent to 1 QALY, whereas 1 year at half-full health is equivalent to 0.5 QALYs. Expression of health benefit in terms of QALYs allows decision-makers to make a direct comparison between the cost-effectiveness of interventions across different disease areas and indications; NICE312 currently recommends the use of QALYs in cost-effectiveness analyses.

Literature review of acute kidney injury economic models

A literature search was conducted in February 2015 and updated in March 2016 to identify economic models of AKI to help inform the model structure and parameters.

The following databases were searched: MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, EMBASE, Cochrane Central register of Controlled Trials (CENTRAL), Database of Abstracts of Reviews of Effects (DARE) and NHS EED (NHS Economic Evaluation Database). Search strategies included concepts for AKI and economic models (see Appendix 8 for the full database details and an example search strategy). All search results were screened by one health economist using a two-stage process: initial abstract screening to identify potentially relevant references followed by full-text screening to determine final inclusion.

The initial inclusion criteria aimed to identify any models including AKI health states, regardless of setting. For the updated search only economic models assessing AKI biomarker testing strategies were included, as these were deemed to be of most use to the project at that stage.

The original search identified 235 references (179 after removal of duplicates), which increased to 296 (48 after removal of duplicates) references in the update search. In total, 11 studies were included in the review,187,205,316–324 two of which were economic evaluations of AKI biomarkers187,205 (Figure 27).

FIGURE 27.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for the AKI economic decision model search.

Details of the included models are presented in Table 41. The majority of studies not looking at biomarkers were concerned with assessing different dialysis modalities or contrast media to prevent or treat renal injury. These studies incorporated a range of health states, including within-hospital states such as dialysis/no dialysis, AKI/no AKI, AKI severity levels and hospital death, and post-discharge states, such as alive with/without dialysis, CKD, end-stage renal disease (ESRD), transplant and death.

Two studies assessed the cost-effectiveness of biomarkers for the early diagnosis of AKI in a critical care setting and are of greatest relevance to this study.

Focus group

A focus group was held on 26 February 2015 with two clinicians (nephrologists) and two patient representatives with experience of chronic kidney failure. The primary aim was to understand the care pathway for patients diagnosed with AKI in a critical care setting, with secondary aims of understanding the impact of AKI and its treatments on patient outcomes and quality of life. The session was co-ordinated by two qualitative research officers (NW and KVC) and two health economists (DM and AS). The group discussion followed a topic guide, which was focused on developing a diagrammatic representation of the patient pathway for someone experiencing AKI in critical care. Interviews were audio recorded and transcribed verbatim by a third party and framework analysis was undertaken to systematically sift, chart and sort material by key themes.

The findings were outlined according to four subsections, briefly outlined below:

1. People at risk of developing AKI. There are many triggers for AKI and there are multiple groups at risk, including patients with long-term conditions (CKD/heart failure), patients with comorbidities and the elderly. In the hospital patients on any ward may develop AKI and may have previously had normal renal function.

2. Diagnosing AKI. Multiple tests and assessments can be used to diagnose AKI in hospital. Some hospitals now use electronic alert systems but there is currently a lack of standardised practice in this area.

3. Treating AKI. After diagnosis, treatment generally consists of a non-specific care bundle including fluid assessment, medication review and additional investigations and monitoring. Treatment procedures may vary depending on the cause of the AKI, individual patient characteristics and hospital procedures.

4. Long-term health consequences of AKI. Post treatment some patients completely recover whereas others may go on to develop CKD and require dialysis or renal transplantation.

The themes identified in the focus group generally support NICE-identified pathways for AKI. 1,325–327 Particular issues identified that were pertinent to the development of the decision model included (1) the multiple aetiologies of AKI and the associated heterogeneity of the AKI population, (2) the non-specific nature of AKI treatment and (3) the need to capture downstream risks of CKD.

Model structure

The model structure was developed using the findings from the literature review and focus group, NICE guidelines1,325–327 and expert feedback. A simplified representation of the model is presented in Figures 28 and 29.

FIGURE 28.

Model structure: initial decision tree. M, Markov model.

FIGURE 29.

Model structure: main Markov model.

The model is split into an initial decision tree (see Figure 28) and a subsequent modified Markov model (see Figure 29). The decision tree separates patients into cohorts depending on their AKI status and test results; patients then enter the main Markov model, which describes patients’ health-care pathway over the duration of their hospital stay and post-discharge follow-up. In the Markov model patients occupy different health states and may move between those states over time, with movement triggered by events such as AKI onset, AKI progression, development of CKD and mortality. Each health state is associated with a specific cost and quality of life (utility) value such that over time patients accumulate costs and health benefits over model cycles of a defined length of time. The model is run separately for each of the specified cohorts within each of the arms (standard care and testing arms) to obtain an average cost and average QALYs (quality of life + survival), from which cost-effectiveness is calculated.

Initial decision tree model

The initial decision tree separates patients into cohorts depending on their AKI status and (for the intervention arms) their additional test results. In the standard care arm, patients are split into ‘AKI’ and ‘no AKI’ cohorts. Patients in the AKI cohort either arrive in the critical care unit (also hereafter referred to as the ICU) with pre-existing AKI or are destined to develop AKI at some point during their stay; patients in the no AKI cohort are those who maintain normal renal function throughout their critical care stay. Standard care testing is assumed to be perfect, such that all patients in the baseline model are correctly identified as either having or not having AKI.

Patients in the testing arms of the model are assumed to receive an additional test on admission to the critical care unit alongside standard care testing. It is expected that patients arriving in critical care with known moderate or severe AKI (KDIGO stages ≥ 2) would not receive additional testing; these patients follow the baseline AKI cohort pathway. All other patients are assumed to be tested and are separated into four cohorts according to the accuracy of the test results, that is, TPs, FNs, TNs and FPs.

Main Markov model

The Markov model consists of two periods: a hospital period, to assess patients’ short-term outcomes, and a follow-up period, to assess patients’ long-term outcomes post hospital discharge.

Impact of the tests

All patients in the testing arms without known pre-existing AKI (KDIGO stage ≥ 2) are assumed to be tested and, therefore, receive an additional cost of testing. Other baseline risks and costs in the model are adjusted according to the test result, as described in the following sections.

Primary searches

To help inform model parameter estimation, a literature review was conducted in July and August 2015 across the following databases: MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, EMBASE and The Cochrane Library. Six search strategies were devised to identify evidence on (1) AKI costs, (2) AKI utilities, (3) AKI risks, (4) CKD costs, (5) CKD utilities and (6) CKD risks. Date limits and UK filters were used in some searches to target studies with recent, UK-based data (i.e. of most relevance to the model). Full details are reported in Appendix 9.

For AKI searches, studies were included if they reported on relevant outcomes for adult patients with or recovering from AKI, or on dialysis, and had a hospital critical care or post-critical care setting. For CKD searches, studies were included if they reported on relevant outcomes for adult patients with a primary condition of CKD, who were on chronic dialysis or who had undergone a renal transplant, across any setting. Relevant outcomes were defined as follows:

• cost searches: UK patient or health-care costs

• utility searches: direct utilities reported on multiattribute utility indexes or through direct valuation exercises [i.e. EuroQol-5 Dimensions (EQ-5D),329 Short Form questionnaire-6 Dimensions (SF-6D)330 or time trade-off (TTO)/standard gamble (SG) utilities]

• AKI risk searches: AKI incidence or progression, mortality, dialysis dependence, progression to CKD or ICU/hospital length of stay

• CKD risk searches: incidence of CKD, ESRD or transplants post AKI, progression or recovery rates for these conditions, dialysis dependence, transplant success and mortality.

All screening and data extraction was conducted by one health economist (AS). Citations were initially screened by title and abstract, followed by full-text screening to determine inclusion. In cases in which the reviewer was unsure about study inclusion a second reviewer (DM or PH) was consulted. All papers screened at the full-text stage were hand-searched for additional relevant references. Search results were stored in six EndNote version X12 libraries [Clarivate Analytics (formerly Thomson Reuters), Philadelphia, PA, USA] and data extraction was conducted using a standard extraction form in Microsoft Excel^® 2010 (Microsoft Corporation, Redmond, WA, USA).

Across the six searches, 1153 potentially relevant citations were identified after removal of duplicates (Figure 30). In total, 129 references were included in the review: four for AKI costs,323,331–333 none for AKI utilities, 40 for AKI risks,323,331–369 33 for CKD costs,323,370–401 eight for CKD utilities402–409 and 44 for CKD risks. 370,373,410–451

FIGURE 30.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for the model parameter searches. Reasons for exclusion: A = no relevant outcome reported; B = non-UK study (when this was applied as an additional exclusion criteria); C = inappropriate study type (e.g. editorial); D = date (when date restrictions were applied).

For the AKI utilities search, no UK studies or international evidence syntheses were identified according to the original scope of the search. Secondary screening and citation tracking of the search results was therefore conducted to identify any relevant non-UK studies, with no date restrictions, leading to four non-UK-relevant papers being included. 317,452–454

Tables 42–44 provide a summary of the included studies. The majority of evidence identified related to studies reporting outcomes for patients with CKD. Of these studies, the following were identified as being of particular use for model parameter estimation: (1) a comprehensive worldwide meta-analysis reporting CKD, dialysis and transplant health state utilities;402 (2) a large (n = 7246) UK randomised multicentre clinical trial reporting CKD, dialysis and transplant costs, as well as the probabilities of transition between CKD, dialysis and death;370 and (3) the UK Renal Registry annual report, which reports key data on annual transitions between ESRD, dialysis, transplant and mortality states410 (note that model parameters were derived from the subsequently published 2015 version of this report). 455

Limited evidence on costs and utilities for AKI was identified: no UK utility studies were identified (compared with nine in the CKD search) and only three UK cost studies were identified (compared with 33 for CKD), none of which reported disaggregated or per-patient critical care costs that could be used to inform the model health state costs. For AKI risks, a worldwide systematic review334 was identified that reported an ICU KDIGO AKI incidence of 31.7% (95% CI 28.6 to 35.0) in an all-comer population and 24.3% (20.4 to 28.8) in a post-cardiac surgery population. 456 This was deemed to be the most reliable study to inform the estimates of AKI incidence. It was noted, however, that other clinical studies reported incidences both above and below the reported 95% CIs;334–342 this parameter was therefore included in the sensitivity analysis.

Additional searches

Two additional searches were conducted in an attempt to identify data on (1) cost and quality of life outcomes for patients treated in the ICU (AKI and no AKI cohorts) and (2) the impact of early AKI intervention.

Impact of early acute kidney injury intervention

A key parameter in the model concerns the impact of early AKI intervention on patient health outcomes, that is, the ‘treatment effect’ parameter linked to tests resulting from early identification of AKI. Two previous economic evaluations of AKI biomarkers187,205 identified in the model review assumed a uniform 25% reduction in subsequent AKI risks as a result of early diagnosis. However, in both studies no evidence base was cited to support this key parameter estimate.

A literature search was conducted in March 2015 to identify reviews of early treatment/preventative strategies for AKI in the ICU across the following databases: MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, EMBASE and The Cochrane Library. The search aimed to identify UK reviews including early treatment/preventative strategies for adults with AKI in the ICU, published in the last 5 years. Search strategies included the search concepts acute kidney injury, critical care, early treatment and treatment effectiveness (see Appendix 10 for full details).

The database searches identified 689 records, with 496 remaining after removal of duplicates (Figure 31). In total, 37 relevant reviews and papers were initially identified. Within these 37 studies, four key recurring interventions were identified: early RRT, early nephrologist involvement, AKI e-alert systems and intravenous administration of alkaline phosphatase. Based on a review of the literature findings and expert consultation, evidence on the impact of early RRT was deemed to be currently contentious and this intervention was therefore not explored further. For the remaining interventions citation tracking was conducted to identify UK primary ICU studies. In total, eight primary studies463–470 were included for data extraction and are summarised in Table 46.

FIGURE 31.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for the AKI early intervention search.

TABLE 46 - Summary of early AKI intervention studies in the ICU in the last 5 years

OR, odds ratio.

First author and year	Country	n	AKI status	Methods	Key results
Early nephrologist involvement
Flores-Gama 2013464	Mexico	1096	AKI (RIFLE/AKIN) within 7 days of cardiac surgery in the cardiac ICU	Retrospective time series analysis of impact of nephrology on-demand vs. nephrology on-site (i.e. integrated into the daily ICU team) on renal recovery	Patients treated with nephrology on-site had a lower incidence of AKI (25.7% vs. 31.9%; adjusted OR 0.71) and in-hospital mortality for severe AKI (34.1% vs. 55.9%; adjusted OR 0.33) and higher renal recovery at hospital discharge (61.0% vs. 35.3%; adjusted OR 3.57) than patients treated with nephrology on-demand. No differences in ICU stay or mechanical ventilation
Costa e Silva 2013465	Brazil	366	AKI (increase of ≥ 50% in baseline serum creatinine level according to the RIFLE risk stage) in the ICU and length of ICU stay > 48 hours	Prospective observational study to assess the impact of early (< 48 hours from AKI diagnosis day) vs. delayed (≥ 48 hours after AKI diagnosis) nephrology consultation	Of 53.6% who had a nephrologist consultation, those with a delayed consultation had a higher rate of hospital mortality (adjusted OR 3.39), increased dialysis dependence (adjusted OR 3.25), a longer ICU stay (19 vs. 13.5 days), a longer time on mechanical ventilation support (16.5 vs. 10 days) and a longer time from diagnosis to dialysis (7 vs. 2 days)
Ponce 2011466	Brazil	148	AKI (AKIN) in the ICU	Prospective observational study of the impact of delayed nephrologist consultation (≥ 48 hours after AKI diagnosis)	Of patients who received a consultation, 29 received an early consultation and 48 received a late consultation. Delayed consultation was associated with increased ICU mortality (65.4% vs. 88.2%, adjusted OR 1.32)
Mehta 2002467	USA	215	AKI in the ICU and hospital. AKI defined according to blood urea nitrogen and serum creatinine criteria	Prospective observational study to assess the impact of delayed (≥ 48 hours) nephrology consultation	Delayed consultation was associated with increased in-hospital mortality (adjusted OR 2.0), length of hospital stay (median 19 vs. 16 days) and length of ICU stay (17 vs. 6 days)
Electronic e-alerts
Wilson 2015463	USA	1393	Patients in hospital (including ICU) with stage 1 or above KDIGO AKI	Single-blind, parallel-group RCT of a text-based e-alert system. Patients were stratified by medical vs. surgical admission and ICU vs. non-ICU location	The e-alert system did not affect patient clinical outcomes (change in creatinine level, dialysis and death at 7 days post randomisation)
Colpaert 2012468	Belgium	951	AKI (RIFLE) in the ICU	Prospective time series analysis of a telephone alert system. Three consecutive study phases: a 1.5-month pre-alert control phase, a 3-month intervention phase and a 1.5-month post-alert control phase	More patients in the alert group received intervention within 60 minutes of alert (28.7% vs. 7.9% and 10.4% in the pre- and post-alert control groups respectively) and received fluid therapy, diuretics and vasopressors (p < 0.001) and more patients in the alert group returned to a baseline kidney function within 8 hours of a ‘risk’ alert (p = 0.048). There was no impact on ICU length of stay, use of RRT or mortality
Alkaline phosphatase
Pickkers 2012469	The Netherlands	36	Patients with severe sepsis or septic shock and AKI (AKIN) in the ICU	Prospective RCT to assess the impact of intravenous infusion of alkaline phosphatase within 48 hours of AKI onset	Creatinine clearance (baseline to day 28) was significantly higher in the treated group (from a mean of 50 ± 27 to 108 ± 73 ml/minute vs. from a mean of 40 ± 37 to 65 ± 30 ml/minute for placebo). Reductions in RRT requirement and duration were not significant
Heemskerk 2009470	The Netherlands	36	Patients with bacterial infection, more than two systemic inflammatory response syndrome criteria and < 12 hours end-organ dysfunction onset in the ICU	A multicentre RCT to assess treatment with an initial bolus intravenous injection of alkaline phosphatase followed by continuous infusion over the following 23 hours and 50 minutes	Over 28 days, mortality was 24% in the alkaline phosphatase group vs. 36% in the placebo group; RRT use was 24% in the alkaline phosphatase group vs. 36% in the placebo group. In patients with AKI, serum creatinine levels tended to decrease over the 2 days after the start of alkaline phosphatase treatment (p = 0.12), whereas there was a non-significant increase in the placebo group (p = 0.49)

Evidence on the impact of e-alert systems was mixed, with the largest identified study finding no impact. 463 Alkaline phosphatase appears to be a potentially effective treatment for AKI but as of yet there are limited data to support use of this intervention in practice. The evidence on the impact of early nephrologist consultation was deemed to be of most relevance for this study as this can reasonably be assumed to represent a proxy for the non-specific bundle of early AKI treatments that patients would access as a result of biomarker-led early diagnosis. All of the four identified studies in this area reported some impact of early consultation on patient mortality and the largest study (n = 1096) reported a significant impact of early consultation on the incidence of AKI, with an adjusted odds ratio of 0.71 (95% CI 0.53 to 0.95; p = 0.02). 464