Systematic review and validation of prediction rules for identifying children with serious infections in emergency departments and urgent-access primary care

Article history

The research reported in this issue of the journal was commissioned by the HTA programme as project number 07/37/05. The contractual start date was in March 2009. The draft report began editorial review in December 2010 and was accepted for publication in May 2011. As the funder, by devising a commissioning brief, the HTA programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

none

Copyright statement

© Queen’s Printer and Controller of HMSO 2012. This work was produced by Thompson et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (http://www.publicationethics.org/). This journal may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2012 Queen’s Printer and Controller of HMSO

Importance

Acute illness is one of the most common problems encountered in children attending emergency departments (EDs) as well as by primary-care services in the UK. Between 27% and 47% of patients who present to EDs in the UK do so for medical illness, rather than trauma. 1 In the case of children, the most common medical reasons for attending ED are breathing difficulty (31%), febrile illness (20%), diarrhoea/vomiting (16%), abdominal pain (6%), seizure (5%) and rash (5%). 1 Acute infections are responsible for four consultations per person-year in children aged < 1 year, and between one and two in children aged 1–15 years in the UK. 2 Children < 5 years of age also constitute a substantial part of the workload of urgent-access primary-care services. Indeed, the patient group which presents most commonly to out-of-hours clinics is children with acute infections. 3,4 Similarly, acute illness in children is also a major component of the work of NHS Direct, in which 22% of all telephone calls are related to children < 5 years of age. 5 Perhaps most importantly for the NHS, parents of unwell children are often concerned about their child and his/her risk of serious illness and need access to accurate and appropriate diagnostic clinical services. 6

Diagnostic difficulties in children with acute infections

One of the key tasks in both hospital EDs and primary-care settings is therefore to distinguish children who may have serious infections (e.g. meningitis, bacteraemia) or complications from infection (e.g. hypoxia from bronchiolitis, dehydration from gastroenteritis) from the vast majority with self-limiting or minor infections who can safely be managed at home. This task is challenging. With increasing ED attendance rates in the UK, hospital admission of children is becoming more common despite a falling incidence of serious infection. 7,8 At the same time, approximately half of children with meningococcal disease are missed at first consultation with a doctor, which results in a poorer health outcome. 9 Infections were responsible for 20% of childhood deaths reviewed in the recent Confidential Enquiry into Maternal and Child Health report Why children die,10 with the greatest number in the 1- to 4-year age group. One reason why recognition of serious infection in children is difficult is the low prevalence of serious illness. In a primary-care setting, < 1% of children assessed have a serious infection,8,11 while in the ED setting this can be as high as 25% of children referred by their GP for fever without source. 12 In addition, infections are dynamic illnesses, and children with serious illness may present at an early stage when severity is not apparent and deteriorate rapidly. Finally, assessment of children (particularly pre-verbal children) may be undertaken by staff with limited paediatric training or under high pressure because of large patient volumes. 13,14 Recent reductions in working hours may further reduce opportunities for trainees to gain experience of children with serious infections.

Clinical tests

Clinicians use several different combinations of clinical tools to identify children with serious infections in primary and emergency care. This primarily involves assessing symptoms reported spontaneously or elicited from parents and patients, overall or global assessment of severity of illness, as well as measurement of vital signs and findings from physical examination. 15 There has been no systematic assessment of which of these clinical features are most useful for identifying children with serious infection in ED or primary-care settings. Clinical prediction rules are a simple pragmatic technology that can potentially be used by clinical staff to assist them in assessment and clinical management. A widely implemented example which has been shown to reduce both resource use and missed diagnoses in EDs is the Ottawa Ankle Rule for ordering a radiograph. 16 There has been no systematic assessment of prediction rules for children with serious infection and, in particular, how well these rules have been validated or implemented in different clinical settings. These are necessary steps in the development of prediction rules, before widespread introduction in clinical care. 17

Additional testing

Some clinicians, particularly those working in EDs, have access to further diagnostic tests. These include urine dipsticks, blood tests for white blood cell counts (WBCs) and inflammatory markers, as well as imaging. Apart from urine dipsticks, most of these tests are carried out in hospital or ED settings, although several are now available as point-of-care tests which give an immediate result and could potentially be used in out-of-hours or other primary-care settings. EDs have the capability to use other tests in assessing children, for example additional biochemical and haematological tests, as well as microbiological cultures of blood, cerebrospinal fluid (CSF) and sputum, and a variety of imagining including plain radiographys, ultrasound, computerised tomography and magnetic resonance imaging. However, even the use of blood tests in identifying children with serious infections is not straightforward. In addition to the difficulties and discomfort of obtaining blood samples, particularly in younger children, the role of the test in the overall diagnostic pathway is important so that clinicians do not rely entirely on a test that does not have perfect discrimination. 18 Previous systematic reviews have assessed the value of selected inflammatory markers in either children only19 or children and adults20–22 for the diagnosis of various outcomes such as distinguishing viral from bacterial pneumonia23 or parenchymal involvement in children admitted with a urinary tract infection. 24 However, none has evaluated and compared all available laboratory tests for children with suspected serious infections in ambulatory care. In addition, most studies which evaluate blood tests neglect the available clinical information, which makes it difficult to determine their role in the clinical pathway, and their incremental value over clinical features. 18,25 In this monograph we will focus on urine and blood tests, which can potentially be used as initial diagnostic tests in most ED settings, rather than the more extensive list of all possible laboratory and imaging investigations.

Assessing the level of urgency

There are several triage systems currently in use in EDs in the UK. These are primarily designed to assess level of urgency for care, rather than as diagnostic tests for serious infections. The Manchester Triage System assigns the patient to one of five categories based on the maximum time that he/she can wait for full assessment. 26,27 It provides only modest sensitivity (63%) to detect emergency or very urgent cases, and is a generic instrument designed to deal with emergencies including trauma. 28,29 Other triage systems used internationally include the Emergency Severity Index, the Paediatric Canadian Triage and Acuity Scale, the Paediatric Risk of Admission Score and the Paediatric Emergency Assessment Tool. 30–34 A number of more specific ‘scoring systems’ for children presenting to EDs with medical illness have been developed. None has shown sufficient ability to rule out serious infection in children to be widely adopted in a NHS context. 35–38

National Institute for Health and Clinical Excellence guidance

The National Institute for Health and Clinical Excellence (NICE) guideline for the management of children <  5 years of age with a fever was published in 2007. 39 It is an important starting point because its recommendations are based on a literature review utilising stakeholders to identify key documents. One of the main outputs from this guideline was a ‘traffic light’ system of clinical features that are designed to be used to assess the risk of serious infection in children. This system assigned clinical features to green, amber and red categories based on the risk of serious infection, and provides clinical guidance for actions needed according to these categories. The current project aims to contribute to the guidelines for the management of children, in a wider context than the NICE guideline, namely by including all children rather than only those < 5 years of age or with fever, but will also be used to update the NICE feverish child guideline. Moreover, we will identify clinical prediction rules relevant to children with acute infection and validate their performance in different clinical settings.

Literature search

We searched the literature in four electronic databases, MEDLINE, EMBASE, the Database of Abstracts of Reviews of Effects and the Cumulative Index to Nursing and Allied Health Literature, using search terms that included terms related to five components: serious infections, children, clinical history and examination, laboratory tests and ambulatory care settings (see Appendix 1). The searches were conducted in October 2008 with an update to the search in June 2009. We searched the Medion database (www.mediondatabase.nl) for systematic reviews using the ‘signs and symptoms’ component of our search strategy. Reference lists of included studies were searched for additional articles. We also checked reference lists of relevant NICE guidelines published prior to 200839,41 and asked clinical experts to report any studies that had been omitted. We did not limit the search based on language or time.

Selection process

After a pilot selection on a sample of 20 studies, two independent reviewers (AVDB and THH), identified articles based on title and abstract for full-text review. Any discrepancies between the reviewers were resolved by a third independent reviewer (MJT). We used six criteria to determine inclusion and exclusion: study design, participants, setting, outcome, diagnostic tests and reporting of data (Table 1).

Quality assessment

We assessed quality using the quality assessment of diagnostic accuracy studies (QUADAS) instrument. 42 This was performed by one reviewer (AVDB) and checked by a second (THH), with discrepancies resolved by discussion involving additional researchers when necessary. In addition, we contacted study authors if additional clarification was necessary. Two items from the QUADAS criteria were used as study exclusion criteria:

• spectrum bias: this was considered present in case–control studies which used healthy controls or in studies in which participants were selected based on the performance of the reference standard

• validity of the reference standard: this was assessed by a clinical review committee consisting of a minimum of three researchers.

When there were insufficient data to be able to confidently assess whether or not a quality criterion had been met, we assessed it as not being met. The overall quality of included studies was rated from high (A) to low (D) using the following criteria:

• A: fulfilling all QUADAS criteria

• D: lacking total verification with the reference standard or with interpretation of the index feature unblinded to the results of the reference standard

• C: lacking an independent reference standard, with interpretation of the reference standard unblinded to the results of the index feature or with an unduly long time period between recording of the index feature and outcome

• B: all other studies.

Data extraction and analysis

Data were extracted by one reviewer (AVDB) and checked by a second (THH), and any errors identified were corrected. Key characteristics of included studies were extracted, including year, study design, setting or country, number of participants, proportion with serious illness, quality rating, age range, and inclusion and exclusion criteria. We reconstructed 2 × 2 tables based on information in the article or from additional information obtained from the authors. Where empty cells were present in the 2 × 2 tables, we added 0.5 to each cell. We calculated likelihood ratios (LRs) for the presence (LR+) or absence (LR–) of each feature along with the pre- and post-test probabilities of the outcome. Confidence intervals (CIs) were calculated on the basis of the standard error of a proportion using Stata version 9.2 (StataCorp LP, College Station, TX, USA).

Diagnostic features were categorised based on their diagnostic value as either red flags or as rule-out features for serious illness using the values of LR+ and LR–. Red flag (or rule-in) features were defined as those with a LR+ > 5.0. Rule-out features were defined as those with a LR– <0.2. 43 In cases in which studies reported more than one result on the same feature using different cut-off points, we presented the result with the highest LR+ or lowest LR–. We included features if at least one study reported a LR+ > 5 or a LR– < 0.2. Setting was used to categorise studies, as a proxy for prevalence of serious infection: < 5%, low-prevalence setting; 5–20%, intermediate-prevalence setting; and > 20%, high-prevalence setting.

The results of the LR+, LR–, pre- and post-test probability and prevalence were plotted on dumbbell plots. We report both the pre-test and post-test probabilities of serious infection for each study in dumbbell plots. We grouped studies based on (a) global clinical assessment features, (b) circulatory and respiratory features, (c) miscellaneous features, (d) inflammatory markers, (e) WBCs and (f) clinical prediction rules. The diagnostic value of temperature was explored using a plot of post-test values against pre-test prevalence on a log scale using R (www.r-project.org), using different cut-offs for temperature. Diagnostic meta-analysis of diagnostic markers across studies using multiple cut-points was carried out using the bivariate method of Dukic and Gatsonis,44 implemented in R.

Numbers of included studies

The electronic search of databases identified 1560 articles at the initial search date (October 2007) and a further 300 at the June 2009 update (Figure 1). After reviewing titles and abstracts, 176 of these articles were selected for full-text review. An additional 79 articles were identified based on reference lists of included studies, systematic reviews and NICE guidelines, and from content experts (‘snowballing’). Following the review of the full text of these articles, we included 104 for quality assessment, of which 35 were selected for inclusion in the review. The most common reason for exclusion was spectrum bias.

FIGURE 1.

Flow of literature search.

One study77 was excluded during the analysis stage because the 2 × 2 tables could not be reconstructed.

Setting of included studies

The majority of the studies were performed in the USA (16), with a further five from the UK, four from the Netherlands, three from Switzerland, two from Canada, and one each from Belgium, Italy, Australia, Denmark and Spain. Full details of the 35 included studies are shown in Table 2. There was only a single trial from general practice (which also recruited non-referred patients from ambulatory paediatric care and the ED in Belgium). 11 All other studies were performed in EDs, with four of these in paediatric departments (one a paediatric assessment unit). 29,59,60,75

Age and prevalence of serious infection

The age of eligible children also varied among the studies, with an upper age limit ranging from 3 months to 18 years. The median prevalence of serious infection was 15.8% (interquartile range 8.66–22.06%), ranging from 0.8% in the general practice study11 to 55.3% in one study at a paediatric assessment unit. 29

Outcomes reported

The majority (20/35) of studies reported on a composite outcome of serious infection, which included sepsis, bacteraemia, meningitis, pneumonia and urinary tract infection (and in some cases additional infections such as cellulitis, osteomyelitis and abscess). 11,29,36,45–61 All but two18,20 of these included children based on the presence of fever and six cases45,50,52–54,60 were based on the absence of localising signs or focus of infection. A further six studies reported outcomes on bacteraemia,62–67 and five of them62–64,66,67 included children with fever. Three studies used an outcome of meningitis,70–72 of which two included children with first episode of fever and seizures,70,71 and one based on signs of meningeal irritation. 72 Two studies used an outcome of gastroenteritis causing dehydration,68,69 one of which included children with clinical gastroenteritis68 and the other children with diarrhoea with or without vomiting and evidence of dehydration. 69 Two studies used an outcome of pneumonia,73,74 and included febrile children and children with clinical signs of respiratory infection. Finally, two studies75,76 used meningococcal infection as an outcome, both of which included haemorrhagic or non-blanching rash as inclusion criteria.

Quality of included studies

All but three of the studies used prospective designs; the remainder were retrospective. The majority (24) used consecutive recruitment. The quality of the included studies was modest (Figure 2). As representativeness of spectrum and valid reference standard were used as inclusion criteria, all studies met these quality criteria. The majority of studies were scored as yes or unclear for the criteria of index text interpreted blinded from reference standard, cut-offs were defined before study, definitions of positive results provided, treatment paradox absent, sample size exceeding 50, pre-specified objectives, and consecutive inclusion. The quality criteria filled by the fewest studies were blind reading of the reference standards which was explicitly mentioned in four studies, and scored as unclear in 21 studies.

FIGURE 2.

Summary of QUADAS features of studies included in systematic review.

Global assessment

Parental concern that the illness is different from previous illnesses (LR+ 14) and the clinician’s gut feeling that something is wrong (LR+ 23) were important red flags in a single study from a low-prevalence setting (Figure 3). 11 Overall assessment of severity based on clinical impression or the child appearing ill from six further studies50,56,63,67,69,76 (all but one in intermediate-prevalence setting) provided weaker LRs+ and LRs–.

FIGURE 3.

Potential red flag features for serious illness (LR+ > 5 in at least one study): global assessment and behavioural features. Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%). a, Meningococcal infection; b, gastroenteritis causing dehydration only; c, meningitis only.

Child behaviour

Change in the child’s crying pattern, drowsiness, moaning or inconsolability all had a LR+ > 5.0 and thus potential red flags, but again all were from the sole low-prevalence study. 11 In an additional three studies from intermediate or high prevalence settings,50,71,72 these features provided weak LRs+ and LRs– and, paradoxically, changed crying reduced the probability of serious disease in a high-prevalence setting.

Circulatory and respiratory clinical features

The presence of cyanosis had a LR+ ranging from 2.66 to 52.2 in three studies,11,29,72 suggesting a rule-in value (Figure 4). Poor peripheral circulation was noted in six studies, with a LR+ ranging from 2.39 to 38.8. 11,29,50,68,69,72 There did not appear to be a clear relationship between the prevalence of infection in the study setting and LR+. Rapid breathing and shortness of breath provided greatest LR+ in the single low-prevalence study,11 with a LR+ of 9.3–9.7. In the four other studies that assessed these features, the LR+ was weaker, ranging from 1.11 to 3.6. 29,50,73,74 The presence of crackles on auscultation and diminished breath sounds again provided a LR+ > 5 in the low-prevalence setting (LR+ 6.0–9.3),11 but little value in the single study conducted in an intermediate-prevalence setting. 73

FIGURE 4.

Potential red flag features for serious illness (LR+ > 5 in at least one study): circulatory and respiratory features. Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%). a, Meningococcal infection; b, gastroenteritis causing dehydration only; c, meningitis only; d, pneumonia only; e, capillary refill > 2 seconds; f, digitally measured capillary refill.

Neurological signs or petechial rash

The presence of meningeal irritation, petechial rash, decreased consciousness and seizures all had a LR+ > 5 in the majority of the eight studies which assessed these features (Figure 5). 11,60,70–72,75,76 This effect appeared to be relatively consistent across all settings. Meningeal irritation had a LR+ ranging from 2.57 to 275 in four studies. 11,71,72,75 The presence of a petechial rash had a LR+ of 6.18–83.7 in six different studies. 11,60,71,72,75,76 Seizures were a useful red flag in the study11 in a low-prevalence setting (LR+ 20.7),11 whereas in three studies in intermediate- or high-prevalence settings it was less useful (LR+ 1.68–5.9). 70–72 Loss of consciousness was assessed in only two studies, in which it had a LR+ of 19.8–155. 11,71

FIGURE 5.

Potential red flag features for serious illness (LR+ > 5 in at least one study): neurological and miscellaneous features. Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%). a, Meningococcal infection; b, gastroenteritis causing dehydration only; c, meningitis only; d, pneumonia only; e, diameter > 2 mm; f, during examination.

Fever

Data from 11 studies11,29,49,50,59,64–66,72,74,76 were available to plot on a graph of pre-test probability (i.e. prevalence) versus poste-test probability (Figure 6) using cut-offs with the highest LR+. The highest rule-in value was obtained in the setting with the lowest prevalence, where a temperature of 40 °C or more increased the likelihood of disease from 0.8% to 5.0%. 11 By contrast, the absence of high temperature (< 38.5 °C to 38.9 °C) had greatest rule-out value in a study with prevalence of serious infection of 29.1%. 59 However, this rule-out potential was not seen in any of the other five studies29,49,50,65,72 with prevalence > 20% and temperature had no rule-in value in these high prevalence settings.

FIGURE 6.

Probability of serious illness in children with temperature above (closed symbols) or below (open symbols) threshold in 11 studies11,29,49,50,59,64–66,72,74,76 carried out in health-care settings with different pre-test probabilities of serious infection and using different temperature thresholds. The distance of the symbol from the 45 °-line indicates the diagnostic value of temperature measurement in the study (applying the specified threshold). The figure is plotted on a log scale to achieve visual separation of the studies carried out in low-prevalence settings (Teele et al. ,66 the estimated post-test probability if the temperature was < 38.5 °C was 0%, which cannot be plotted on a log scale, so there is no lower symbol).

When we repeated the analysis using additional cut-offs (data not presented in Figure 6), results were very similar to the results presented in Figure 2. In the low-prevalence study,11 temperature ≥ 38 °C had a LR+ of 1.5 and a LR– of 0.38 and temperature ≥ 39 °C has a LR+ of 2.3 and a LR– of 0.59.

Clinical prediction rules

We identified clinical prediction rules which had been applied to a composite outcome of serious infection, pneumonia, meningitis and dehydration from gastroenteritis (see Appendix 2). The Yale Observation Scale (YOS) was assessed in seven studies (Figure 7). It provided a LR– < 0.2 in two studies,36,60 whereas in five other studies it varied from 0.68 to 0.94,45,48,52,53,57 associated with post-test probabilities ranging from 10% to 28%. The YOS was derived in the McCarthy et al. 1982 study36 at a time prior to routine vaccination against Haemophilus influenzae and pneumococcus, possibly explaining its better results. This would not explain the similar low LR– of the Thayyil et al. study,60 which was performed on a similar patient population in 2003. For meta-analysis, there was significant heterogeneity present (p = 0.002), which remained (p = 0.026) after exclusion of the McCarthy et al. study,36 but disappeared (p = 0.093) after exclusion of both studies. The summary sensitivity of the five remaining studies45,48,52,53,57 was 32.5% (95% CI 21.7% to 45.5%) and specificity was 78.9% (95% CI 73.9% to 83.1%), theoretically corresponding to a LR+ of 1.54 and a LR– of 0.86. An additional study58 assessed the YOS or the presence of any abnormal finding on history or clinical examination in an intermediate population; again it provided a rule-out value with a LR– of 0.17.

FIGURE 7.

Clinical prediction rules with the potential to rule in or rule out serious infection (LR+ > 0.5 or LR– < 0.2 in at least one study). Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%). a, Cut-off used: > 8, study 48; > 9, study 45; > 10, studies 36, 52, 53, 57, 60. b, If yes to any of five sequential questions: (1) clinician instinct that something is wrong; (2) dyspnoea; (3) temperature > 39.5 °C; (4) diarrhoea; (5) age 15–29 months.

The clinical prediction rule that performed best for ruling out serious infection (LR– 0.04) involved five sequential questions, but was assessed in only a single low-prevalence study. 11 Two prediction rules were identified which potentially ruled out pneumonia, with LR– 0.07, both involved the absence of shortness of breath and either parental or clinician concern. Again, these were assessed in only a single low-prevalence study. 11 We identified two prediction rules for meningitis derived in intermediate settings. One provided a rule-out value (LR– 0.05) if there was no abnormal neurological finding and care was not sought within 48 hours, whereas the other provided rule-in ability (LR+ 395) if any of petechiae, nuchal rigidity or coma was present. Finally, a single rule was identified for dehydration from gastroenteritis, which provided modest LR+ (6.10) and LR– (0.24) from a single high-prevalence study. 68

Features of limited help in ruling in or ruling out serious infections

Features that had a LR+ < 5.0 and/or a LR– > 0.2 (i.e. less helpful in either ruling in or ruling out serious infection) for any serious infection are listed in Table 3. The NICE traffic light system and the Manchester Triage System score were of little value in a single study with a high-prevalence setting. Symptoms that are common in children, such as cough, headache, tummy ache, vomiting, diarrhoea, poor feeding and coryza, had little diagnostic value. Two features, namely failure to smile (LR+ 4.2) and changed breathing pattern (LR+ 4.4), were just below the arbitrary LR+ cut-off of 5, but both had a weak LR–, suggesting no rule-out value (LR– 0.6–0.7). The behavioural features of a reactive child (i.e. moving, reaching for objects, looking around the room), lack of irritability and lack of sleepiness did not provide a rule-in or rule-out value based on three studies. 11,49,59 Although cyanosis and poor peripheral perfusion (which causes mottling and pallor) are red flag symptoms as described above, the report of abnormal skin colour (described as cyanotic, pallor or flushed/mottled) in three studies of low and high prevalence did not provide a useful LR. 11,49,50

The clinical features that were of limited rule-in or rule-out value for specific infections are listed in Table 4. Several clinical features in a single study of dehydration from gastroenteritis (low urine output, sunken eyes, dry mucous membranes, tachycardia, abnormal respiration) all had modest LRs (LR+ 1.8–3.8). The presence of paralysis or paresis had only a modest LR+ for meningitis (3.48).

Diagnostic value of laboratory tests for composite outcome of serious infection

Inflammatory markers

Seven studies identified in the systematic review45,49,52–54,57,60 provided data on the diagnostic value of six inflammatory markers: procalcitonin (PCT), C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), interleukin-6, interleukin-8 and interleukin-1 receptor antagonist (Figure 8). None of the studies was performed in low-prevalence settings. The three studies45,54,60 which reported the results of PCT all used the same cut-off (0.5 ng/ml) and demonstrated a LR+ of 1.75–2.96, with a LR– of 0.08–0.35. The five studies of CRP45,49,54,57,60 had cut-offs ranging from 9.8 to 50 mg/l, and provided a LR+ of 2.53–3.79 and a LR– of 0.25–0.61. Pooling of CRP results was possible and using bivariate meta-analysis of CRP yielded a pooled LR+ of 3.15 (95% CI 2.67 to 3.71) and a pooled LR– of 0.33 (95% CI 0.22 to 0.49) across all cut-offs.

FIGURE 8.

Diagnostic value of inflammatory markers for serious infection. IL-1RA, interleukin-1 receptor antagonist; IL-6, interleukin-6; IL-8, interleukin-8. Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%).

The summary receiver operator characteristics curve plots of CRP and PCT (Figure 9) confirm that the markers have similar diagnostic accuracy as the shapes of the curves are very similar and the CIs are overlapping. We plotted the LRs of CRP and PCT identified in the systematic review by cut-off value and disease prevalence to identify the optimal cut-off points to apply. To rule in serious infection, cut-off levels of 2 ng/ml for PCT or 80 mg/l for CRP both provide good diagnostic value (a LR+ of ≥ 4) but poor rule-out value. To rule out effectively, cut-offs of 0.5 ng/ml for PCT or 20 mg/l for CRP may be a better choice (a LR– of ≤ 0.2).

FIGURE 9.

Summary receiver operator characteristic curves for CRP and PCT.

The performance of ESR in a single study49 (cut-off of 50 mm/hour) provided LR+ 2.49 and LR– 0.34. The two studies investigating the three interleukins demonstrated somewhat inferior diagnostic values to CRP or PCT. 52,53

White blood cell counts

A total of nine studies45,49,54,57,59–61 provided data on either total WBC (seven studies45,49,54,57,59–61), absolute neutrophil count (two studies45,61), band count (three studies49,52,53) or left shift (single study54); all were from intermediate- or high-prevalence settings (Figure 10). All four WBC indices demonstrated little diagnostic value in ruling out serious infection: the minimum LR– is 0.61 with the 95% CI in most studies crossing 1.0. The LRs+ were also weaker than the inflammatory markers, with values ranging from 0.87 to 3.05. There was little evidence to determine whether or not any of the WBC markers offered superior diagnostic value. The results of one study61 appeared to show paradoxical results for WBC: the likelihood of disease was lower in children with a WBC ≥ 15,000/mm³.

FIGURE 10.

Diagnostic value of WBCs for serious infection. ANC, absolute neutrophil count. Setting: low, low prevalence of serious illness (< 5%); intermediate, intermediate prevalence of serious illness (5–20%); high, high prevalence of serious illness (> 20%).

Added value of combinations of blood tests and clinical features

We identified eight studies46,47,50–55,60 which reported the diagnostic value of prediction rules consisting of a single blood test or combinations of blood tests, with or without clinical features (Figure 11). The clinical features, laboratory tests and cut-offs used are provided in Appendix 3.

FIGURE 11.

Prediction rules combining CRP, WBC and PCT with clinical features. a, In patients negative on a clinical prediction rule; b, in patients positive on a clinical prediction rule. ANC, absolute neutrophil count. Full details of cut-off points applied are given in Annex 3 (Van Den Bruel et al. Diagnostic value of laboratory tests in identifying serious infections in febrile children: a systematic review. BMJ 2011;342:D3082).

Bleeker et al. 50 determined the added value of laboratory markers in patients testing high or low risk for a prediction rule of clinical symptoms. The clinical rule included duration of fever, history of vomiting, ill appearance, chest-wall retractions and/or tachypnoea, and decreased capillary refill. Their results showed that in children testing high risk on their clinical prediction rule, a combination of WBC, CRP and urinalysis lowered the probability from 42% to 15% when negative, but the probability was not increased substantially when positive (54%), indicating a greater ability to rule out than rule in serious infection. In contrast, the use of WBC, CRP and urine analysis in children who tested low risk on the clinical prediction rule lowered the probability of serious infection (from 12% to 4%) if negative and increased the probability to 31% if positive.

The laboratory clinical prediction rule reported by Thayyil et al. 60 provided the highest LR+ (10.67) based on the results of PCT, CRP and WBC. However, this offered little rule-out value with a LR– of 0.52. The best performing prediction rule by Galetto-Lacour et al. 54 reported a LR+ of 5 and a LR– of 0.07 and involved urine dipstick as well as measuring a score based on the levels of PCT and CRP. This rule had good rule-in and rule-out potential, a high score increasing the likelihood of a serious infection from 27% to 64%, and a below threshold score decreasing it to 2%. The inclusion criteria for both the studies of Thayyil et al. 60 and Galetto-Lacour et al. 54 included the presence of fever.

We also found that combinations of inflammatory markers offered little additional diagnostic value to the individual tests, with all the CIs of the combinations largely overlapping with those of the individual tests (Figure 12). In general, combinations in which all tests had to be positive increased the ability to rule in serious infection, whereas combinations in which at least one test had to be positive increased the ability to rule out serious infection.

FIGURE 12.

Likelihood ratios of individual and combinations of inflammatory markers and WBCs.

Although clinical prediction rules which involve a series of clinical features as well as more invasive investigations (i.e. cerebrospinal fluid parameters and chest radiography) are not within the scope of this review, the results of the clinical prediction rules identified are presented in Appendix 3. These rules were derived for use in infants < 3 months of age presenting to the ED with fever. The dumbbell plots show they provide little diagnostic value in ruling in serious infection (LR+ 1.27–1.70), but provide LR– ranging from 0.04 to 0.15 (pooled LR– 0.06; 95% CI 0.018 to 0.19), indicating good rule-out value.

Diagnostic value of laboratory tests for meningitis and/or bacteraemia

Only one study evaluating CRP for the diagnosis of meningitis and/or bacteraemia was available that showed CRP is able to rule out meningococcal infection if normal (LR– 0.05). 76 Two studies on meningococcal infection75,76 evaluated the value of coagulation markers, i.e. activated partial thromboplastin time (aPTT), international normalised ratio (INR) and platelets, and found good diagnostic value at ruling in the disease (LR+ 2.05–13.08), except for platelets (LR+ 3.20). Normal coagulation markers do not exclude the possibility of meningococcal infection.

Six studies evaluated the value of WBC, all with bacteraemia as outcome. 62–64,66,67,76 None of the studies reported a LR+ > 3.0 or a LR– < 0.3, suggesting that WBCs were of very little use in diagnosing or excluding bacteraemia. Indeed, after pooling these studies, the summary sensitivity of abnormal WBC was 62.71% (95% CI 52.60% to 71.81%), summary specificity was 69.27% (95% CI 62.71% to 75.13%), summary was LR+ 2.04 (95% CI 1.51 to 2.75) and summary LR– was 0.54 (95% CI 0.40 to 0.73). Absolute neutrophil count was reported in two studies, one on bacteraemia62 and one on meningococcal infection. 76 Both reported low LRs (LR+ 1.65–2.57 and LR– 0.40–0.60).

Five studies reported prediction rules for meningitis and bacteraemia which combined clinical features with inflammatory markers. 62,66,67,72,75 The rules by Teele et al. ,66 Crocker et al. 62 and Waskerwitz67 did not increase or decrease the probability of bacteraemia significantly (LR+ 1.21–2.92). Two prediction rules, those of Nielsen et al. 75 and Oostenbrink et al. ,72 combined clinical features with CRP, and both were able to rule out meningococcal disease and meningitis respectively (LR– 0.01–0.05). The Nielsen rule also increased the likelihood of meningococcal infection at higher cut-offs (LR+ 17.33). The Nielsen rule consists of the presence and characteristics of haemorrhages, general condition nuchal rigidity, band count and CRP. The Oostenbrink rule consists of duration of the main complaint, vomiting, meningeal irritation, cyanosis, petechiae, disturbed consciousness and CRP.

Identification of data sets

We identified existing data on children attending ED, urgent-access primary care or primary care from two sources. Firstly, we identified data sets included in the systematic review. We approached authors of data sets published within the last 10 years and invited them to contribute data and expertise to the multiple external validation study. Secondly, we contacted content experts in this clinical area and asked them to identify other existing data sets. The criteria we used to include data sets were modified from those used as inclusion criteria for the systematic review (Table 5).

Identification of clinical prediction rules

The systematic review identified clinical prediction rules which have been applied to a composite outcome of serious infection, pneumonia, meningitis and dehydration from gastroenteritis (see Appendix 2): the YOS, a five-stage decision tree, two prediction rules for pneumonia, two prediction rules for meningitis and a rule for dehydration from gastroenteritis.

To determine which data sets could be used to validate the prediction rules, we constructed a matrix of all variables recorded in the available data sets. Consensus through group discussion was reached on which clinical prediction rules we could validate based on the available data sets.

Exploring heterogeneity

Given that the analysis was performed on an individual patient data level, it was essential to tabulate heterogeneity and mark up similarities between studies. This heterogeneity may be related to patient characteristics, setting, geographical characteristics, test characteristics or study design.

Characteristics of included data sets

Key characteristics of each of the data sets were extracted, including inclusion and exclusion criteria, setting, age of participants and individual variables (see Table 6). The variables used in each data set were translated to English if necessary, and clarifications for variables were obtained from the researcher responsible for each data set. Coding of all key variables was checked not only on statistical grounds but also on clinical grounds, with the responsible researcher (i.e. meaning of the variable label ‘unknown’ in relation to missing observations). A merged data set was then created and checked for internal and external validity. The translation, the synopsis, the recoding and the data checking were carried out by one member of the study team (JV) and the results of each step were discussed with every responsible researcher when necessary.

Where variables were not entirely comparable, proxies were identified from other variables contained in the data sets and fitted to perform the analysis. If too few variables were present in one data set to allow cross-validation, that data set was excluded from the specific analysis (e.g. fewer than three variables present for a composite clinical prediction rule of six variables was deemed unsuitable). In addition to approximations on the variables, approximations to sum scores had to be used as well. For example, the YOS assigns points at three levels for each variable, depending on the severity of abnormality of that variable. For example, a normal reaction to parent stimulation is given one point, a moderately abnormal reaction is given three points and a severely abnormal reaction given five points. Some data sets had only dichotomous results for the variables included in the score, for which one point was awarded to a normal variable and five to an abnormal variable. Consequentially, the total sum score differs from the original sum score, and different cut-offs for defining an abnormal YOS were explored (i.e. cut-offs of 10 and 8).

Statistical analysis

The accuracy of the clinical prediction rules identified in the systematic review was assessed in each of the data sets in which this was possible. The outcome used to validate the prediction rule was as similar as possible to that of the original prediction rule derivation. For each prediction rule, we used dumbbell plots to display results of sensitivity, specificity, LR+, LR– and pre- and post-test probabilities. This analysis was performed for all children in each data set, as well as the subgroup analyses of only children who had been referred from primary care to the ED, and for children who had not been referred. When a data set had been used to derive a prediction rule, we did not use this data set to validate that same prediction rule. Because there were considerable missing data on key variables (see Appendix 6), several specialised methods (i.e. multiple imputation techniques by the Markov chain Monte Carlo method or by chained equation) were evaluated but none was feasible, because the condition of observations assumed to be missing at random was almost certainly violated, and also because in some cases the problem was that variables were missing completely. All analyses were performed with Stata (version 11).

Description of included data sets

We identified seven data sets (Table 6) for the multiple external validation. 11,28,29,60,72,79 Two had been collected in the UK,29,79 four were from the Netherlands and one was from Belgium,11 providing data on a total of 11,045 children. Two data sets had been collected from primary care,11 the remainder from ED settings. Four of the data sets were included in the systematic review (Van den Bruel et al. ,11 Thompson et al. ,29 Bleeker et al. 50 and Oostenbrink et al. 72). Inclusion criteria for the data sets were children with fever (three), acute illness (two), or acute infection (one) and referred children with meningeal signs (one). Six of the data sets excluded children with various types of comorbidity, and in one case children who required immediate resuscitation. The mean age of children ranged from 0.94 to 5.0 years, and the prevalence of serious infection ranged from 0.78% to 44.71%.

Clinical predictor variables included in data sets

The variables recorded in each data set varied (see Appendix 4). Most data sets included basic demographic characteristics such as age, duration and severity of illness, or referral status. Presenting symptoms were recorded in almost all data sets, apart from two gastrointestinal symptoms. Temperature was recorded in all data sets, followed by heart rate (five data sets), respiratory rate (five data sets), capillary refill time (five data sets) and oxygen saturations (four data sets). Investigations in all data sets were performed at the discretion of the clinical team and none had performed a complete set of investigations on all children.

A matrix was created to determine which of the data sets could be used to validate the clinical prediction rules. Consensus was reached for four clinical prediction rules: YOS, a five-stage decision tree, a pneumonia rule and a meningitis rule. Appendix 5 presents the variables used for external validation of the clinical prediction rules, either original ones or approximations of the predictor variables.

For the YOS, three data sets had variables which were identical to the original Yale scoring (Berger et al. , Brent et al. 79 and Thompson et al. 29). Two data sets (Van den Bruel et al. 11 and Oostenbrink et al. 72) required recoding of dichotomous variables to the YOS, in which cases we assigned a score of 1 for a normal value and 5 for an abnormal value. We considered using the intermediate value of 3 instead of 5 to indicate a normal value, but this did not provide any differences in distribution of the outcome variables. Different cut-offs (8 and 10) were used to validate the YOS, which was consistent with the cut-offs identified by the systematic review. Four data sets were available to validate the YOS using a cut-off of 10 (Van den Bruel et al. ,11 Berger et al. , Thompson et al. 29 and Brent et al. 79), and an additional data set when using a cut-off of 8 (Oostenbrink et al. 72).

One of the data sets had been used to derive the five-stage decision tree (Van den Bruel et al. 11), and an additional six data sets were suitable for validation (Berger et al. , Roukema et al. ,28 Brent et al. ,79 Bleeker et al. ,50 Thompson et al. 29 and Oostenbrink et al. 72). One variable in the derivation study (‘physician’s gut feeling that something is wrong’) was not recorded in any of the validation data sets. We, therefore, used a proxy variable of ‘clinical impression’ in the validation data sets. In addition, we compared the diagnostic characteristics of the rule in the derivation data set using both the original and the proxy variable. Four data sets had all variables present (with approximations, such as nasal flaring for dyspnoea); two data sets (Bleeker et al. 50 and Oostenbrink et al. 72) had up to two missing variables (diarrhoea and/or dyspnoea).

One data set was used to derive the pneumonia rule (Van den Bruel et al. 11) and five were available to validate it (Berger et al. , Oostenbrink et al. ,72 Roukema et al. ,28 Thompson et al. 29 and Brent et al. 79). As with the five-stage decision tree, this rule used clinical impression in the validation data sets as a proxy for the physician’s gut feeling that something is wrong. The same variables and approximations for clinical impression and dyspnoea were used as for the five-stage decision tree.

Four data sets were available to validate the meningitis rule (Van den Bruel et al. ,11 Thompson et al. ,29 Brent et al. 79 and Oostenbrink et al. 72). The variable petechiae was available in all data sets, nuchal rigidity in one data set (Oostenbrink et al. 72) with the proxies meningeal irritation, neck stiffness and AVPU (alert, responds to voice, pain, unresponsive) score of 3 in three of the data sets, whereas coma was approximated with unconsciousness and AVPU score of 4 in all four data sets.

Results of external validation

The results of the validation of the four clinical prediction rules for all children, those who were not referred and those who were referred are provided in Figures 13–15, respectively.

FIGURE 13.

Results of multiple external validation of clinical prediction rules with potential to rule or rule out serious infection in all children. a, Setting: low frequency (< 5%), intermediate frequency (5–20%) and high frequency (> 20%) of the selected outcome measure for serious infection in all children. b, If yes to any of the five sequential questions: (1) clinical instinct that something is wrong, (2) dyspnoea, (3) temperature > 39.5 °C, (4) diarrhoea and (5) age 15–29 months. c, Italic denotes a derivation study. d, ‘clinical instinct that something is wrong’ replaced by ‘clinical impression’. e, If yes to any of (1) shortness of breath and (2) clinicians concern. f, If yes to any of (1) petechiae, (2) nuchal rigidity and (3) coma.

FIGURE 14.

Results of multiple external validation of clinical prediction rules in non-referred children. a, Setting: low frequency (< 5%), intermediate frequency (5–20%) and high frequency (> 20%) of the selected outcome measure for serious infection in all children. b, If yes to any of the five sequential questions: (1) clinical instinct that something is wrong, (2) dyspnoea, (3) temperature > 39.5 °C, (4) diarrhoea and (5) age 15–29 months. c, Italic denotes a derivation study. d, ‘clinical instinct that something is wrong’ replaced by ‘clinical impression’. e, If yes to any of (1) shortness of breath and (2) clinicians concern. f, If yes to any of (1) petechiae, (2) nuchal rigidity and (3) coma.

FIGURE 15.

Results of multiple external validation of clinical prediction rules in referred children. a, Setting: low frequency (< 5%), intermediate frequency (5–20%) and high frequency (> 20%) of the selected outcome measure for serious infection in all children. b, If yes to any of the five sequential questions: (1) clinical instinct that something is wrong, (2) dyspnoea, (3) temperature > 39.5 °C, (4) diarrhoea and (5) age 15–29 months. c, If yes to any of (1) shortness of breath and (2) clinicians concern. d, If yes to any of (1) petechiae, (2) nuchal rigidity and (3) coma.

A normal YOS score, i.e. below a threshold of 8 or 10, provided little or no rule-out value, i.e. none of the LRs– were smaller than 0.46 (Figure 13). In one low-prevalence setting,11 the score was moderately to highly specific with a LR+ of 3.81–7.49, depending on the score cut-off selected. In two studies (intermediate and high prevalence) it provided a marginally useful rule-in value (LR+ 3.35–3.63). 72,79 Subgroup analysis of the children who had not been referred (Figure 14) indicated similar results, again with a marginally useful LR+ (3.16–3.81) depending on the cut-off selected, and again with no useful LR–. In children who had been referred (Figure 15), only two data sets were available to validate results, and again indicated a marginally useful LR+ in one study (LR+ 3.42),79 and no useful LR–.

We first compared the performance of the five-stage decision tree in the data set in which it had been derived (Van den Bruel et al. 11), substituting one variable used to derive this rule (physician’s gut feeling that something is wrong) with a proxy variable (clinical impression). Using this proxy variable, both the LR+ and LR– were slightly lower, although the CIs of the latter were overlapping. Using the five-stage decision tree with the proxy variable of clinical impression, we found that the rule had no rule-in value in any of the validation data sets, regardless of whether all children, referred children or non-referred children were considered. However, we found a marginally useful rule-out value in four data sets (LR– 0.13–0.35),28,50,79 but with wide CIs. When the rule was validated on the subgroups of children who had not been referred (see Figure 14) and had been referred (see Figure 15), results were similar: marginally useful LR– of 0.20–0.24 in non-referred children in two data sets,28 and LR– of 0.35 in referred children in one data set. 50

None of the five data sets28,29,50,79 used to validate the pneumonia rule demonstrated clinically useful LR+, ranging from 1.03 (95% CI 1.02 to 1.03) to 2.46 (95% CI 1.57 to 3.88); but in one data set (Berger et al. ) the LR– was 0.22, but with a wide CI (95% CI 0.08 to 0.65) (see Figure 13). However, when only referred children were considered (see Figure 15), we found a marginally useful LR+ of 3.43 (95% CI 1.89 to 6.23) in one data set (Thompson et al. 29). Subgroup analysis of non-referred children (see Figure 14) did not demonstrate useful LRs+ or LRs–.

Finally, validation of the meningitis rule demonstrated highly clinically useful LR+ in three data sets from low- intermediate- and high-prevalence settings, with LRs+ ranging from 9.96 (95% CI 7.97 to 12.4) to 38.9 (95 % CI 14.6 to 104), albeit with wide CIs (see Figure 13). 11,29,79 However, in a fourth data set (high prevalence), the LR+ was only 1.87 (95% CI 1.69 to 2.07). 72 These results did not change substantially when referred or non-referred subgroups were analysed (see Figures 14 and 15). The meningitis rule did provide a useful LR– in one data set from a high-prevalence setting (Oostenbrink et al. 72), with a LR– of 0.084 (95% CI 0.04 to 0.16). However, clinically useful LRs– were not found in any of the other data sets, regardless of referral status (see Figures 14 and 15).

Studies identified by systematic review

This systematic review of the existing literature on the diagnostic value of clinical features and laboratory tests in children presenting to primary care or EDs included a total of 36 studies. The majority of the studies were performed in the USA or Canada (18), with a further five from the UK, 11 from other European countries, and one from Australia. All but one of these studies were based in ED settings – we identified only a single study from primary care, performed in general practice in Belgium. As a result, the generalisability of our findings for primary-care settings in the UK may be limited, but we believe that the findings are generalisable to ED settings. The quality of the included studies was modest. However, the clinical context of this systematic review limited the likelihood that studies could meet ideal diagnostic study criteria. For example, verification of disease status for serious infections in all children would ideally involve an extensive diagnostic work-up in all children presenting with possible infection, which is neither feasible nor ethical. Blind interpretation of the reference standard from the index test was found in only four studies, which again reflects the fact that many clinical features are also used in the definition of the outcome.

Prevalence of serious infection varied widely in the included studies, from 0.78% in the primary-care study11 to 55.3% in one study29 from a paediatric assessment unit. The implication of this is that the pre- and post-test probability of the same clinical features and predictors may vary markedly between different studies. We therefore used various methods to reflect this when summarising the results. In particular, we developed ‘dumbbell’ plots, which provided a novel way of presenting the sensitivity, specificity, LR+, LR–, prevalence, and pre- and post-test probabilities in a simple visual format.

Serious infection is not a discrete clinical condition; the majority (20/35) of studies used a composite outcome of serious infection which included sepsis, bacteraemia, meningitis, pneumonia and urinary tract infection (and in some cases additional infections such as cellulitis, osteomyelitis and abscess). Other studies used outcomes of single clinical entities such as meningitis, gastroenteritis causing dehydration, pneumonia and meningococcal infection. In the majority of studies, children were included based on the presence of fever and, in some cases, clinical features. For example, the two studies which used meningococcal infection as an outcome included only children with haemorrhagic or non-blanching rash.

Predictive value of clinical features for serious infection

We identified a total of 30 studies11,29,36,45,48–50,52,53,56–76 in the systematic review which provided the diagnostic value of clinical features for serious infections as a composite outcome or for a specific serious infection. We grouped clinical features into those related to overall or global assessment, circulatory and respiratory features, meningeal irritation, temperature and common clinical symptoms. Overall and global assessment of severity of illness were useful diagnostic features for ruling in serious infection in the single study from a low-prevalence setting. 11 The diagnostic value of these overall or behavioural features was not consistently found in the studies from intermediate- or high-prevalence settings. The definition of overall assessment varied, but included parental concern that the illness was different from previous illnesses, clinician’s gut feeling that something was wrong, as well as changes in the child’s behaviour such as a change in crying pattern, drowsiness, moaning or inconsolability. Lack of consistent definition of the components that constitute overall or global assessment limits the value of this feature for clinical practice. It is likely that it includes multiple components of the clinical assessment which are internalised (by parent or clinician) in the course of parenting or clinical training. GPs’ gut feeling has also been described as the most discriminating test result in patients presenting with chest pain. 80 The usefulness of overall assessment is consistent with the results of a survey of GPs in the UK, which highlighted overall assessment as the most important part of the clinical assessment of pre-school children. 15 In that study, GPs reported that playing, smiling and moving around the room were the most important aspects of their overall assessment. It is unclear why overall assessment would be less valuable in higher prevalence settings; one reason could be the referral paradox, whereby the presence of children who have an abnormal overall assessment of serious illness is inflated in settings where more children are referred because of the presence of these features. Additional reasons may be the immediate availability of a battery of technological tests and the fact that initial assessment of unwell children in these settings tends to be performed by junior clinicians. Finally, we do not know the extent to which these features can be learned or taught.

The most useful circulatory predictors of serious infection were the presence of cyanosis and poor peripheral circulation. However, objective definitions of these were lacking, and clinical features that are commonly used to assess poor circulation (e.g. capillary refill time and pale skin colour) did not provide useful LRs in three studies of low and high prevalence. 11,49,50 Rapid breathing appeared to provide useful LR+ only in the low-prevalence study,11 it was weaker in the four other studies that assessed rapid breathing. 29,50,73,74 The presence of meningeal irritation, petechial rash, decreased consciousness and seizures was consistently identified as a red flag for bacterial meningitis across all settings. However, we know from other studies that these features occur late in the pre-hospital course of bacterial meningitis, and thus may have red flag value only later in the course of this illness. 9

The presence of fever provided diagnostic value in settings where the prevalence of serious infection was low or intermediate. Using cut-offs of both 40 °C and 38.5–38.9 °C, three studies showed substantial increased post-test probability, suggesting value as a rule-in or red flag feature. However, in the five studies with higher prevalence, temperature provided no rule-in ability, i.e. the post-test probability of serious infection was similar to the pre-test probability irrespective of whether it was above or below the cut-off threshold. Not surprisingly, symptoms that are common in children, such as cough, headache, tummy ache, vomiting, diarrhoea, poor feeding and coryza, had little diagnostic value. The NICE traffic light system and the Manchester Triage System score were of little value in a single study from a high-prevalence setting. 29 The authors are currently collecting data from other high-prevalence settings to validate these prediction rules.

Predictive value of laboratory tests rules for serious infection

The systematic review identified 14 studies45–47,49–55,57,59–61 which reported the diagnostic value of laboratory tests for the diagnosis of serious infections. However none of the studies was performed in primary-care settings. CRP is used far more widely in hospital clinical practice and EDs in the UK than PCT, reflecting the fact that this is an older test, performed in highly automated analysers. PCT is not routinely used in the UK in children, as it is currently more costly and more difficult to perform in many laboratories. Our findings indicate that both CRP and PCT provide comparable diagnostic accuracy. Pooling of the five studies45,49,54,57,60 which had assessed CRP for a composite outcome of serious infection yielded a LR+ of 3.15 (95% CI 2.67 to 3.71) and a LR– of 0.33 (95% CI 0.22 to 0.49). As all five studies were performed in intermediate- or high-prevalence settings, these moderate LR values provide relatively large changes in post-test probability. Our findings are similar to an earlier review of the value of CRP in children with fever which used slightly different methods, but found LR+ 3.64 and LR– 0.29 compared with LR+ 3.15 and LR– 0.33 in our study. 19 We were not able to pool the results of the three studies that had assessed PCT,45,54,60 which demonstrated LR+ ranging from 1.75 to 2.96 and LR– of 0.08 to 0.35. The optimal cut-off point to use for CRP and PCT depends on whether the clinical goal is to rule in or rule out serious infection. Based on our results, cut-off levels of 2 ng/ml for PCT and 80 mg/l for CRP provide good diagnostic value (a LR+ of ≥ 4) to rule-in serious infection. To rule out effectively, cut-off levels of 0.5 ng/ml for PCT or 20 mg/l for CRP may be a better choice (providing a LR– of ≤ 0.2).

White blood cell indicators (total WBC, absolute neutrophil count, band count or left shift) provided substantially lower diagnostic value than CRP or PCT for ruling in serious infection, and no value for ruling out. This also confirms the findings of a previous review which showed that WBCs did not provide additional diagnostic value over and above CRP. The performance of ESR in a single study49 (cut-off of 50 mm/hour) provided LR+ 2.49 and LR– 0.34. The two studies52,53 investigating the three interleukins demonstrated somewhat inferior diagnostic values to CRP or PCT.

We found no evidence to determine the roles of any of the inflammatory markers at different time points in the course of a serious infection. As PCT levels rise earlier in the course of bacterial infection than CRP, it is possible that PCT would have a comparatively greater diagnostic value in the early course of serious bacterial infections than CRP or WBCs. The use of combinations of inflammatory markers provides some evidence to support this hypothesis. In general, a combination of inflammatory markers in which all tests have to be positive increases the ability to rule in, whereas a combination in which at least one has to be positive increases the ability to rule out. The best performing prediction rule by Galetto-Lacour et al. 54 reported a LR+ of 5 and a LR– of 0.07 and involved urine dipstick as well as measuring a score based on the levels of PCT and CRP in febrile children. This rule had good rule-in and rule-out potential, a high score substantially increasing the likelihood of a serious infection from 27% to 64%, and a below-threshold score decreasing it to 2%. Using composite outcomes of serious infection to assess inflammatory markers may also mask important differences in the rule-in or rule-out values of these markers in different types of serious infection. However, we found no studies which allowed us to assess the value of individual or combinations of inflammatory markers for individual infections in ED or ambulatory settings.

In clinical practice, laboratory tests are usually requested based on the results of a clinical assessment, rather than in isolation. This is implicit in the inclusion criteria for the studies which assessed laboratory tests, given that fever (or other clinical features, or indeed referral) was used as an inclusion criterion. However, only a single study assessed the results of laboratory markers in patients testing positive or negative on a symptoms-only prediction rule. 50 This study found that in children testing positive on the symptoms-only prediction rule, a combination of WBC, CRP and urinalysis lowered the probability of serious infection from 42% to 15% when negative, but did not raise the probability substantially when positive (54%), indicating a greater ability to rule-out than to rule-in serious infection. In contrast, in children who tested negative on the symptoms-only prediction rule, the use of WBC, CRP and urinanalysis moderately lowered the probability of serious infection (12% to 4%) if all were negative and moderately increased the probability to 31% if all were positive.

Predictive value of clinical prediction rules for serious infection

Given that most of the clinical prediction rules identified had limited evidence of validation, we attempted to use existing data sets to perform cross-validation. We identified a total of six existing data sets11,28,29,50,72,79 including 11,045 children presenting to primary care or EDs with suspected acute infection. Some of the data sets were used to derive the prediction rules we had identified in the systematic review, and thus could not be used to validate the same rule.

The best performing clinical prediction rule (‘five-stage rule’) for ruling out serious infection identified in the systematic review was derived in a low-prevalence setting and was based on the presence of five features: clinician’s impression something is wrong, dyspnoea, temperature, diarrhoea and age. 11 We did not identify any published studies which had validated this rule. Our attempts to validate this rule produced highly variable results; it had no rule-in value in any of the validation data sets, regardless of whether or not children were referred, but had marginally useful rule-out value in four data sets (LR– 0.13–0.35) with wide CIs. 28,50,79 Although this rule appears to offer some diagnostic value in primary care, the lack of consistency within and between settings limits the adoption of this prediction rule in routine care.

The YOS was developed in the early 1980s, in a tertiary US hospital, and assesses the presence of a series of clinical features in children < 3 years of age grouped as follows: colour, breathing difficulty, response to social overture, hydration, level of activity and state variation. We had sufficient data to calculate a summary sensitivity of 32.5% (95% CI 21.7% to 45.5%) and a summary specificity of 78.9% (95% CI 73.9% to 83.1%). When we attempted to validate the YOS, we found little or no rule-out value, i.e. none of the LRs– was < 0.46. However, in both low- and high-prevalence settings,11,72,79 the YOS provided moderate to high specificity and a LR+ of 3.81–7.49 depending on the cut-off selected. The YOS is the best known and most studied clinical prediction rule, and the overall findings and heterogeneity were disappointing. The methodology used to derive clinical prediction rules has advanced considerably since the YOS was developed. Indeed, some key methodological limitations of the YOS were lack of weighting for items (all are assigned equal weight), crude methods used to identify and combine clinical predictors, lack of information on interobserver interpretation of items and the scoring system used, as well as limited clinical generalisability as it was developed in US tertiary paediatrics hospital and was limited to children < 3 years of age.

The systematic review identified two prediction rules which had a LR– sufficiently low to rule out pneumonia: both involved the absence of shortness of breath and absence of either parental or a clinician’s concern. These were derived in only a single low-prevalence study11 and, therefore despite good LRs+ (11.3 and 13.8), were not sufficient on their own to significantly raise post-test probabilities in this setting. We did not identify any published prediction rules for pneumonia which had been derived or validated in intermediate or higher prevalence settings. When we attempted to validate the pneumonia prediction rules in the new data sets, none of them demonstrated clinically useful LR+, but in one low-prevalence setting (Berger et al. ) the LR– was 0.22, suggesting a useful rule-out value.

We identified two prediction rules for bacterial meningitis in the systematic review, both of which had been derived in intermediate settings. 70,71 One provided rule-out value (no abnormal neurological finding, and the parent had not sought care within 48 hours), whereas the other provided rule-in ability (any of petechia, nuchal rigidity or coma present). Clinically useful LRs+ were found when we validated the meningitis rule in three data sets from low-prevalence settings11,29,79 (LR+ 9.96–38.9), but not in the data set72 from a high-prevalence setting (LR 1.87). In contrast to the derivation studies, we could not confirm useful LR– in the low-prevalence studies, but we did find an extremely small LR– (0.084) from one high-prevalence study,72 suggesting rule-out value.

Finally, the systematic review identified a single rule for dehydration from gastroenteritis, which provided a modest LR+ and LR– from a single high-prevalence study. 68 We did not find any published validation studies for this rule, and were not able to validate it in the data sets identified.

Limitations of the systematic review

We searched the literature using standard systematic review methods. However, given that the search filters and strategies for diagnostic systematic reviews are less well developed than those for reviews of interventions, we used several additional methods to attempt to identify all relevant studies, such as searches of relevant guidelines, reference lists and contact with content experts.

The inclusion criteria for our review were explicit. We excluded studies of hospitalised children and those from developing (or transitional) countries. Studies from developing countries have a far higher incidence of serious infection in children, and a higher prevalence of comorbidities than is found in the UK or elsewhere in Europe. In addition, our review focused on serious infection, rather than other illnesses, so we may have missed potentially useful studies which addressed other non-infectious serious illnesses in children. For example, we excluded studies of the Baby Check score, which was devised in the UK in order to help parents and clinicians detect all serious illness in infants, and which has been shown in a small series of 87 children from UK general practice to have a sensitivity of 100% and a specificity 67%. 81 In addition, we eliminated studies which used a case–control design as these are recognised to substantially overestimate diagnostic odds ratios. 82

The studies included in the systematic review were highly heterogeneous in terms of setting, prevalence of serious infection, definition of outcome quality, age range and inclusion of clinical predictors. We found only a single study based on primary care, and although this was a large well-conducted study, we are cautious in basing our conclusions for primary care on the results of a single study.

In the analysis and summaries of the predictive value of clinical features, we selected a LR+ of 5.0 and a LR– of 0.2 as cut-off values to indicate clinically useful LRs. We recognise that this is arbitrary, and where possible have commented on the LRs that are close to these cut-offs. We also recognise that the effect of a set LR will obviously be dependent on the pre-test probability. Therefore, in presenting the results we include plots of pre- and post-test probabilities, so that the impact of the LR can be assessed.

Limitations of validation of existing clinical prediction rules

The steps which are necessary for multivariable diagnostic research have been described by Toll et al. 83 Because good calibration (agreement between predicted probabilities and observed outcome frequencies) and good discrimination (ability to distinguish between patients with and without the outcome) in a derivation study are no guarantee of prediction in new patients, it is important to conduct validation studies of clinical prediction rules. 84,85 Indeed, most prediction rules demonstrate worse performance when validated in new populations. 84,86 Therefore, additional aims of this project were not only to identify published validation studies, but also to identify data sets which could be used to validate the clinical prediction rules that we identified in the systematic review.

The YOS was the only clinical prediction rule identified in the systematic review for which validation studies had been published. We therefore attempted to validate this and the other prediction rules that we identified by using existing data sets of children presenting to primary and ED settings with acute infection or illness. These seven data sets provided data on 11,045 children from three European countries presenting to either primary care or EDs with suspected acute infection. However, our ability to use these data sets to validate existing rules was limited for several reasons. In some cases, a data set had been used to derive a prediction rule and therefore could not be used to then validate it. In other cases, we were limited by heterogeneity between the derivation and validation populations, and issues related to external generalisability.

Heterogeneity between the derivation and validation populations may occur when definitions of predictors, outcome variables and measurement methods are different, and especially if measurement or interpretation is subjective. The seven data sets used for external validation required intensive work to translate variables into English, understand the exact meaning of variables (including cultural differences in measuring and interpreting physical signs and symptoms), and assess which clinical predictors were shared across data sets and how outcomes were defined. Even across these data sets, we were limited not only by the number of shared predictors, but also by how they were defined. The results from the multiple external validation, presented in Chapter 8, demonstrate the degree of approximation which had to be used to validate the original clinical prediction rule. A further cause of heterogeneity is when case mix differs between derivation and validation populations, especially if the outcome measure is predicted by a variable responsible for a difference in case mix. Case mix tends to be related to the setting in which a study is performed. In both our systematic literature review and our validation study, we included studies performed in a variety of ambulatory settings. Prevalence of serious disease was used as a proxy to account for the effect of setting. Furthermore, clinicians working in a similar setting in different countries may differ in the types of patients who present to them, the proportion of referred or self-referred patients and their role in dealing with emergencies. Differences in case mix can lead not only to different prevalence of serious diseases, but also to different measures of diagnostic accuracy. Sensitivity and specificity of the same test will change when used in different settings87 in directions that are not always predictable. To some extent, these changes are influenced by the presence/absence of referral and the reasons why a referral decision has been made. Finally, heterogeneity may have been caused when there were fewer cases in the validation study as a result of random variation, which makes the populations appear different.

Two components for external validity need to be assessed to ensure generalisability: reproducibility and transportability. Reproducibility should be assessed in the derivation study by statistical techniques such as bootstrapping to evaluate the degree of overfitting of a model. This is not part of this report and is extensively described elsewhere. Transportability can be caused by underfitting, which occurs when important independent predictors of the outcome are left out of the prediction rule. Five types of transportability are described in the literature: historical, geographic, methodological, spectrum and interval. In our validation studies, historical transportability was not an issue because all data sets were performed in more or less the same time period. Geographical transportability could have influenced the validation, although all validation studies were conducted in Europe, North America or Australia. Methodological transportability could be an important issue in these validation studies, as it may well be the most common problem when systems fail to generalise. Spectrum transportability was difficult as well, because the spectrum (i.e. being more or less advanced in the disease process) could be influenced by differences in referral status and settings. As described by Justice et al. ,86 in reality the calibration of most clinical prediction rules will be compromised when these systems are tested in a sample of patients with very different levels of disease severity, as was the case in our validation studies, owing to calibration to the overall outcome prevalence in the development study. Follow-up period transportability probably did not influence our validation results, because there were no variables or outcomes related to longer follow-up periods.

Overall clinical implications

The 35 studies identified in this systematic review were highly heterogeneous, and we are therefore limited in drawing firm conclusions about our findings. Indeed, our findings illustrate the diagnostic gap between the predictive value achievable by considering the clinical features and the threshold of risk of serious infection. 78 This gap is currently filled by clinical ‘gut feeling’ and diagnostic safety-netting, which are still not well defined in primary or ED settings. 88 Moreover, teaching ‘gut feeling’ to trainee clinicians is problematic, particularly given the decrease in opportunities in clinical training to encounter children with serious infections. New video-based interactive educational resources, such as Spotting the Sick Child (www.spottingthesickchild.com), may help improve the ability of clinical staff to recognise children with acute infection. 86

A single abnormal clinical finding is insufficient on its own to substantially raise or lower the risk of serious infection. We did identify several clinical features that were highly specific for serious infection, and can be considered ‘red flags’:

• parental concern that the illness is different from previous illnesses

• clinician’s gut feeling that something is wrong

• change in the child’s crying pattern

• drowsiness

• moaning or inconsolability

• fever (depending on cut-off selected and type of setting)

• cyanosis

• poor peripheral circulation

• rapid breathing and shortness of breath

• crackles on auscultation

• diminished breath sounds

• meningeal irritation

• petechial rash

• decreased consciousness

• seizures

• loss of consciousness.

When present, these features should prompt a more thorough or repeat assessment of the child. However, as these red flags will occur infrequently (owing to their low sensitivity) even in children with a serious infection, they provide limited value as screening tests. It may be equally useful to identify a set of frequently occurring clinical features, which when all are absent would effectively ‘rule out’ most serious infections – these might be termed ‘green flags’. However, existing evidence is not sufficient to be confident in defining these.

Clinical prediction rules should be ideally suited to the complicated task of identifying children with serious infection. Although we identified two rules for overall serious infection and five for specific conditions, only one (YOS score) had any published validation studies. By using existing data sets to validate these rules we were able to draw some clinical conclusions without the need to collect new data from children. Firstly, clinical prediction rules offer different diagnostic value, depending particularly (but not only) on the prevalence of serious infection. Secondly, the five-stage decision tree and YOS score provide moderate specificity and offer some rule-in value in primary and ED settings, and one rule for meningitis provided high specificity and rule-in value. We are aware of strategies to adjust prediction rules to other settings, but methodological analysis of this type was outwith the scope of this project. 89–91

Measuring CRP or PCT is a helpful step in the diagnostic work-up of children in an ED setting where the prevalence of serious disease is relatively high. We did not find a difference in the diagnostic performance of CRP or PCT, but these were superior to WBCs. The LR+ and LR– of CRP and PCT are not very high, confirming the importance of assessing results in the light of clinical findings. Moreover, cut-off values need to be selected depending on whether these will be used as rule-in or rule-out tests, which may vary depending on setting in particular. There is insufficient evidence to confirm the utility of CRP or PCT in primary care and out-of-hours assessment centres.

Research implications

The authors have several recommendations for further research in this area. These recommendations are based on the analysis and findings of the systematic review and external validation of prediction rules carried out for this report.

Contribution of authors

M Thompson conceived the study, collected and contributed original data, performed the literature search and analyses, drafted the manuscript and was responsible for submitting the final report.

A Van den Bruel conceived the study, performed the literature search and analyses, collected and contributed original data and drafted the manuscript.

J Verbakel performed data analysis and interpretation and drafted the manuscript.

M Lakhanpaul collected and contributed original data, assisted with the drafting of the manuscript and performed data analysis and interpretation.

T Haj-Hassan performed the literature search and analyses and assisted with the drafting of the manuscript.

R Stevens conducted analyses, interpreted results and contributed to the drafting of the manuscript.

H Moll collected and contributed original data, interpreted results and contributed to the drafting of the manuscript.

F Buntinx assisted with the drafting of the manuscript and supervised J Verbakel and his analyses.

M Berger collected and contributed original data, interpreted results and contributed to the drafting of the manuscript.

B Aertgeerts commented on drafts of the manuscript and supervised JV and his analyses.

R Oostenbrink collected and contributed original data, performed data analysis and interpretation and contributed to the drafting of the manuscript.

D Mant assisted with the interpretation of the analyses and the drafting of the manuscript.

All authors had full access to the data.

Publications

Van den Bruel A, Thompson MJ, Haj-Hassan T, Stevens R, Moll H, Lakhanpaul M, Mant D. Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review. BMJ 2011;8:342:d3082.

Van den Bruel A, Haj-Hassan T, Thompson M, Buntinx F, Mant D. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet 2010;375:834–45.

Oostenbrink R, Thompson M, Steyerberg EW, ERNIE members. Barriers to translating diagnostic research in febrilechildren to clinical practice: a systematic review. Arch Dis Child 2012;in press.

Disclaimers

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

1. Project title: Systematic review and validation of clinical prediction rules for identifying children with serious infections in emergency departments and urgent-access primary care

2. How the project has changed since the outline proposal was submitted

The Board had several comments on the outline proposal which we have addressed.

3. Planned investigation

4. Project timetable and milestones

The following will be the key milestones for the study:

5. Expertise

The research team that has been assembled for this project brings together methodological expertise in systematic reviewing, diagnostic test systematic reviewing, individual patient data meta analysis, as well as considerable clinical expertise in both emergency departments and primary care settings. Moreover it draws on this expertise not only from the UK, but also from Belgium and the Netherlands.

Dr Thompson is a Clinical Lecturer in Primary Care and half-time Principal in General Practice who also works regularly in an out of hours GP surgery. He has performed several research studies examining clinical predictors of serious infections in primary care and paediatric assessment units. These have included prospective studies of predictive value of vital signs, severity of illness scores and inflammatory markers in children a paediatric assessment unit. He has also published on the early signs of meningococcal disease in children. His systematic review experience includes the treatment of common upper respiratory tract infections with steroids, and he is also currently a member of two National Institute for Health & Clinical Excellence (NICE) guideline development groups (prescribing antibiotics for upper respiratory tract infections, and diagnosis and treatment of meningitis). In addition to his clinical experience and expertise in diagnostic studies he will be responsible for coordinating the proposed study and will supervise the staff funded by this grant.

Professor David Mant is the head of the Department of Primary Health Care at the University of Oxford and has an international reputation in primary care research. His research has included numerous seminal studies on childhood infections and cardiovascular disease. He was the PI of the MRC-funded Oxford Childhood infection study from 2001–6, and sits on national committees such as the Standing Advisory Committee on Antibiotic Resistance (paediatric sub-group) 2005–7 and National Expert Panel on New and Emerging Infections (2003–7). He will contribute extensive methodological input on the study design and meta-analysis, and will provide direct support to Dr Thompson.

Dr Glasziou has extensive expertise in conducting systematic reviews and individual patient data meta-analyses. He has published several textbooks on systematic reviewing, and authored numerous systematic reviews. He is a member of the Cochrane Collaborations diagnostics sub-group and currently Professor of Evidence Based Medicine at the University of Oxford. He is also a practising GP in Oxford. He will contribute extensively to the methodology of the systematic review and IPD meta-analysis.

Dr Van den Bruel has done research in the area of serious infections in children for the last 6 years. She has performed several studies, including one in which a clinical prediction rule for the exclusion of serious infections was developed. In addition, she has been working at the Belgian HTA agency for the last 4 years, where she is responsible for the evaluation of diagnostic tests and the methodology of systematic reviews. Previously she worked as a GP for seven years. In addition to sharing her dataset of children with acute infections, she will contribute expertise in systematic reviews of diagnostic studies.

Dr Moll is Head of the Paediatric Emergency Department of the Sophia’s Children’s Hospital –ErasmusMC in Rotterdam. Her research has focussed on emergency department triage, and the development and validation of prediction rules for acute paediatric infections. In particular she has performed studies on meningitis, fever/serious bacterial infections, RSV and pneumonia in the emergency department setting. She will contribute not only her dataset to this study, but also her clinical experience in emergency paediatrics, and research experience in diagnostic studies and validation of prediction rules.

Professor Buntinx has been working as a GP in Belgium for 32 years and as a researcher and Professor at the Departments of General Practice of the University of Maastricht (Netherlands) and Leuven (Belgium) since 1989. From the start of his research career, he has been focusing on the methodology and execution of diagnostic studies, including multivariate analyses and diagnostic meta-analysis. He has published some 175 papers in international peer reviewed journals and almost as many in Dutch language peer reviewed journals. He currently serves as the research director of the Department of General Practice in Leuven and as the founding president of the Belgian Centre of Evidence-based Medicine. With Prof. Knottnerus he also is the co-editor of the new edition of ‘The evidence base of clinical diagnosis’ (Blackwells, in press). In 2006, he wad elected member of the Belgian Royal Academy of Medicine. He will contribute to the methodological input on diagnostic studies and systematic review of diagnostic studies.

Professor Bert Aertgeerts is a GP in Belgium. He has done research on screening for alcohol abuse and dependence in different settings, and has conducted several systematic diagnostic reviews on various clinical topics. He was also responsible for the European First Aid Manual, led by Stijn Vandevelde from the Red Cross Flanders. He is the director (2001) of the Centre of Evidence-Based Medicine (Belgian Branch of the Dutch Cochrane Collaboration) and is currently head of the department of General Practice at the Katholieke Universiteit Leuven. In 2006, he was elected member of the Belgian Royal Academy of medicine. He will contribute to the systematic review methodology, particularly in relation to diagnostic studies.

Professor Geert-Jan Dinant is vice-chair of the Department of General Practice at the University of Maastricht in the Netherlands. He has extensive research experience in performing diagnostic studies in primary care on pneumonia and osteoporosis and has authored textbooks on evidence based clinical diagnosis. He will bring methodological expertise in diagnostic studies to the research team.

Dr Shelly Segal is a Consultant Paediatrician with special interest in Infectious disease and is the Clinical lead in the Paediatric Emergency department at the John Radcliffe Hospital in Oxford. In addition to her paediatric emergency experience, she has performed several studies on the genetic susceptibility to infectious diseases in children, particularly invasive pneumoccal disease. She will contribute her paediatric emergency clinical experience to the research team.

Dr Monica Lakhanpaul is a Consultant Paediatrician and Co-Director for National Collaborating Centre for Women’s Health and Children’s Health of the Royal College of Paediatrics and Child Health which has been responsible for undertaking several NICE reviews, in particular the recent guideline on management of the Feverish child. She is also clinical lead for developing nurse-led urgent care services in Leicester. She has recently completed a prospective study of children attending A&E in Nottingham. In addition to sharing dataset for the IPD meta-analysis, she will contribute clinical paediatric experience, and experience of literature review to the project team.

Dr Rafael Perera is a University Lecturer in Statistics and Director of Research Methodologies at the Centre for Evidence-Based Medicine. He has extensive experience conducting systematic reviews and IPD meta-analysis and has published numerous systematic reviews. He will contribute extensively to the data analysis for this study.

6. Service users

The research team has extensive current experience as front-line service clinicians in the provision of clinical care to children in emergency and urgent primary care settings. Dr Segal is in charge of paediatric emergency medicine at the John Radcliffe Hospital in Oxford, Dr Moll is Head of Paediatric Emergency Department at Sophia’s Children Hospital in Rotterdam, and Dr Lakhanpaul is a Community Paediatrician in Leicester. Professors Mant, Glasziou, Dinant, Aertgeerts, Buntinx, and Drs Thompson and Van den Bruel are all general practitioners who have worked in general practice in England, the Netherlands, and Belgium. Dr Thompson also works in an out of hours GP centre in Oxford. We will also gather the input of parents/carers input in order to assess the likely impact of this rule in the real world setting, and to ensure that the predictors we identify (e.g. vital sign measurements, possibly blood tests) are acceptable to most parents/carers. We will therefore assemble a group of parents who have had personal experience with children in emergency care or urgent access primary care and obtain their input on the final prediction rules.

7. Justification of support required

The main support required for this project are salary support for a data manager, salary support for a statistician, and reimbursement for meetings of the research team. The data manager will be responsible for the retrieval of articles identified in the systematic review, obtaining and assembling the datasets required for the individual patient data meta analysis, as well as organising meetings of the research team. The data manager may be expected to undertake simple descriptive analysis of the datasets. Salary support has been requested for Dr Perera to undertake statistical work for this study. Dr Perera will be responsible for the summarising of the studies identified in the systematic review. He will also undertake the individual patient data meta-analysis. Dr Van den Bruel will be reimbursed for undertaking one part of the analysis, and will invoice the University of Oxford for a specified component of this work. In order to take advantage of the considerable expertise of the collaborators who have agreed to work on this project, reimbursement for attending four research team meetings during the study period will be provided to Professor Buntinx, Professor Aertgeerts, Professor Dinant, Dr Van den Bruel, Dr Moll, Dr Lakhanpaul and Dr Segal. Nominal salary support has been requested for Dr Lakhanpaul, Professor Mant, Professor Glasziou, Dr Segal and Dr Thompson. No salary support has been requested for Professor Buntinx, Professor Aertgeerts, Professor Dinant, or Dr Moll.

In order to minimise impact on the environment the Dutch and Belgian collaborators on this study will attend meetings in England by rail where possible rather than flying. This project also seeks to reduce the need and costs associated with further prospective studies of predictors of serious infection in children, by taking advantage of literature that has already been published on this topic, and by using individual patient data meta-analysis of studies that have already been carried out.

8. References

9. Flow diagram

Health Technology Assessment programme

1. Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

2. Professor of Dermato-Epidemiology, Centre of Evidence-Based Dermatology, University of Nottingham