Systematic review and individual patient data meta-analysis of diagnosis of heart failure, with modelling of implications of different diagnostic strategies in primary care

Article history

The research reported in this issue of the journal was commissioned by the HTA programme as project number 05/06/01. The contractual start date was in February 2006. The draft report began editorial review in October 2007 and was accepted for publication in January 2009. As the funder, by devising a commissioning brief, the HTA programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

David Mant has received BNP assay kits and technical support from Bayer HealthCare in conducting primary research on the use of BNP for diagnosis and guided treatment of ventricular dysfunction in primary care. He also received funding to present the results of this research at the 2006 World Cardiology Conference. Martin Cowie has provided consultancy advice to Roche Diagnostics, Biosite and Stirling Medical, companies interested in BNP testing. He has no shares in any device or assay company. Richard Hobbs has received research funding and research assays from Roche Diagnostics and occasional speaker fees and symposia expenses from Roche and Bayer Diagnostics.

Copyright statement

© 2009 Queen’s Printer and Controller of HMSO. This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2009 Queen’s Printer and Controller of HMSO

Introduction

Heart failure is a syndrome resulting from a structural or functional cardiac disorder. For a diagnosis of heart failure to be made there should be symptoms or signs such as breathlessness, effort intolerance or fluid retention together with objective evidence of cardiac dysfunction.

Heart failure is an increasingly important chronic syndrome, associated with poor prognosis and poor quality of life for patients, and responsible for high health-care costs. 1,2 Annual mortality in severe heart failure has been reported to be as high as 60%. 3 In the general population, in which all grades of heart failure are represented, 5-year mortality is around 42%,4 but when the diagnosis is established during a hospital admission 5-year mortality is between 50% and 75%. 5,6

Prevalence and incidence of heart failure

Early studies of heart failure prevalence used clinical diagnostic criteria known to be inaccurate,7 particularly early in the disease process. 8,9 More recent studies have included an objective assessment of left ventricular (LV) function, usually echocardiography,10,11 and indicate a prevalence of left ventricular systolic dysfunction (LVSD) of 2.9% in patients under 75 years10 and up to 7.5% in 75- to 84-year-olds. 11 However, limitations of these studies include not screening all adult age groups,10 with data particularly lacking in the elderly in whom heart failure is more common, not examining representative populations, or only examining heart failure due to LV systolic dysfunction. 11

In the largest recent prospective evaluation of heart failure in the community (ECHOES),12 LVSD [left ventricular ejection fraction (LVEF) < 40%] was found in 1.8% (95% CI 1.4–2.3) of the population aged over 45 years; borderline LV dysfunction (LVEF 40–50%) was found in a further 3.5%; definite heart failure was found in 2.3% (95% CI 1.9–2.8) of the population (with LVEF < 40% in 41% of cases); and using an LVEF cut-off of < 50% rather than 40%, 3.1% (95% CI 2.6–3.7%) of people aged 45 years or over had heart failure.

Estimates on heart failure incidence are less available and vary from 0.913 to 2.26 cases per 1000 population per annum in women aged 45–74 years and from 1.613 to 4.66 cases per 1000 population per annum in men aged 45–74 years. Incidence rises rapidly in the elderly, with 1% of men per year developing heart failure after 75 years and almost 2% per year after 85 years.

Burden of heart failure on patients

Mortality rates in heart failure are high. Annual mortality in the placebo arms of recent trials, with many patients on angiotensin-converting enzyme (ACE) inhibitors, has ranged from 7%14 in mild heart failure [New York Heart Association (NYHA) class II], to 11%15 to 13%12 in moderate cases (NYHA III), to 20%,5 23%16 or 28%17 in severe heart failure. By comparison, the Framingham cohort showed an overall 1-year heart failure mortality rate of 17%, a 2-year mortality rate of 30% and a 10-year mortality rate of 78%. 18 The National Health and Nutrition Examination Survey (NHANES) study, conducted from 1971–86 in the USA, revealed 10-year mortality rates of 43% in patients who self-reported heart failure and 38% in patients who had heart failure defined by a clinical score. 19

Mortality data from more recent epidemiological studies provide more reliable case definitions but mainly report on LVSD heart failure only, younger patients only20 or patients presenting to hospital, usually with incident symptomatic heart failure. 21,22 In these last studies mortality is particularly high with 50% 2-year mortality rates, probably representing late presentations; these rates equate to the prognosis of newly diagnosed colorectal cancer in men or ovarian cancer in women. A more accurate estimate of prognosis of prevalent heart failure, across all ages and stages, is available from follow-up of the ECHOES cohort. 4 The 5-year survival rate of the general population was 93%, compared with 58% in those with a prevalent diagnosis of LVSD and 58% in those with prevalent definite heart failure. The median survival time of definite heart failure was 7 years 7 months. Those with a diagnosis of heart failure had the lowest survival compared with the general population and survival improved significantly with increasing ejection fraction. However, amongst patients with ‘borderline’ ejection fraction levels of between 40% and 50%, mortality rates were still over 1.5 times higher than in those with ‘normal’ ejection fractions over 50%. Those with multiple causes of heart failure had the poorest survival. The ECHOES mortality data provide recent confirmation of the poor prognosis of patients suffering from heart failure across the community, providing a mortality risk estimate of 8–9% per year. 4 Importantly, outcomes in heart failure are improving, which is presumed to be because of better initiation and maintenance of evidence-based therapies. 23

Morbidity in heart failure is considerable, whether measured by symptom severity, quality of life24 or need for consultation, treatment or hospital admission. Studies with comparative normative data are few and suggest that heart failure worsens quality of life more than other chronic diseases25 (although heart failure diagnosis in this study was not determined on the basis of objective tests) and that women may suffer worse impairment. 26 Other studies have shown that heart failure is associated with depressive illness27 and, further, that this is then linked to a worse prognosis. 28 Those with heart failure had significant impairment of all of the measured aspects of physical and mental health, not only physical functioning. Significantly worse impairment was found in those with more severe heart failure by NYHA class. 24 Patients with asymptomatic LV dysfunction and patients rendered asymptomatic by treatment had similar scores to those of the random population sample. Those with heart failure reported more severe impairment of quality of life than those giving a history of chronic lung disease or arthritis, and a similar level to patients with depression.

Burden of heart failure on health-care systems

Chronic heart failure remains one of the most costly conditions to manage in many health systems. This is principally because the syndrome is common, it frequently results in hospital admission (which is the disproportionate driver of health-care expenditure), admissions are prolonged (averaging 11 days in Europe) and readmission is frequent (nearly 25% of patients are readmitted within 12 weeks of discharge). 29 In the UK, 4.9% of admissions to one hospital were for heart failure, extrapolating to up to 120,000 admissions per year nationally,30 and these continue to rise. 31,32

As a consequence, heart failure accounts for at least 2% of total health-care expenditure, namely €26 million per million population in the UK, €37 million per million in Germany, €39 million per million in France and €70 million per million in the USA. 33 The average cost per hospital admission in Europe is €10,000. 33 The burden of heart failure is expected to rise as prevalence rises, which is presumed to be the result of improved survival of patients post myocardial infarction, better treatment of heart failure and an ageing population. 34

Management of heart failure

Angiotensin-converting enzyme inhibitors improve both morbidity and mortality in all grades of symptomatic heart failure due to LVSD,35 and they can delay or prevent progression to symptomatic heart failure in patients with asymptomatic LVSD. 36,37 Beta-blocker therapy in heart failure due to LVSD has also been demonstrated to improve prognosis and reduce admission rates,38 although these agents have to be introduced slowly and may be associated with slight worsening of symptoms initially in a proportion of patients. ACE inhibitors and beta-blockers35,39 have been shown to improve exercise tolerance and symptoms (as assessed by NYHA functional class) in patients with heart failure due to LVSD, as well as significantly prolonging survival and reducing hospitalisation rates. These drugs have also been shown to improve global quality of life in sufferers,40,41 as have other interventions producing symptom gains, such as exercise training42 and intensive nurse-led discharge and outreach programmes. 43 Aldosterone blockers reduce hospitalisation and mortality in severely symptomatic (NYHA grade II and IV) patients16 or in post-myocardial infarction heart failure or LVSD. 44 Care is needed with these agents in the elderly community as they may be associatedwith increased mortality if not used carefully in routine practice. 45 Recent data have demonstrated the general utility of angiotensin receptor blockers (ARBs) in patients intolerant of ACE inhibitors or in addition to ACE inhibitors and beta-blockers in those with impaired LV function. 46

Despite this extensive evidence base for treatments that improve heart failure prognosis and symptoms, heart failure remains suboptimally diagnosed and treatedin many countries,47–49 due at least in part to many patients with suspected heart failure not receiving a formal assessment of LV function. 48,49

Diagnostic issues in heart failure

Heart failure is a complex syndrome that can result from any structural or functional cardiac disorder which impairs the ability of the heart to function as a pump to support a physiological circulation. 50 The evaluation of a patient with suspected heart failure therefore entails more than determining whether or not the syndrome is present – it also requires an identification of the underlying abnormality of the heart.

The commonest cause of heart failure is LVSD, present in around half of cases, but other causes include valve disease, atrial fibrillation and isolated diastolic dysfunction of the left ventricle. In many patients, particularly the elderly, several cardiac abnormalities may be found concurrently, such as systolic impairment with atrial fibrillation and mild valve disease. The bulk of the evidence base for treating heart failure, summarised above, is derived from randomised controlled trials (RCTs) on people with underlying LVSD.

An essential element for treatment success is the reliable and precise diagnosis of heart failure. The major issue in the diagnosis of the disease relates to the criteria definitions. Guidelines for the evaluation and management of heart failure are established in both the USA [American College of Cardiology (ACC)/American Heart Association (AHA) and consensus recommendations51] and Europe [European Society of Cardiology (ESC)23]. These state that the diagnosis of heart failure is justified when there are typical signs and symptoms of heart failure and myocardial dysfunction, confirmed by the objective evidence of cardiac dysfunction at rest. In case of diagnostic uncertainty, a clinical response to treatment directed at heart failure is helpful in establishing the diagnosis. Simple and reliable diagnostic procedures are very important for primary care physicians, who are responsible for the early diagnosis of heart failure and the implementation of adequate therapy.

However, current diagnosis of heart failure in primary care is often inaccurate. In one recent UK study,52 only 34% of patients with an existing clinical label of heart failure in routine general practice records had this diagnosis confirmed following echocardiography and blinded review by a panel of three specialist clinicians. A recent review by the Healthcare Commission53 on progress towards implementation of the National Service Framework (NSF) for Coronary Heart Disease found that only one in five patients with a diagnosis of heart failure had had an echocardiogram, and that the average wait for this investigation was 67 days.

This picture of general practice diagnosing heart failure on mainly clinical grounds, with only a minority of patients receiving confirmatory tests before the diagnosis is confirmed, is replicated across much of Europe. 48,54 Primary care physicians often have variable or delayed access to tests such as echocardiography. As a consequence, doctors rely on alternatives such as the electrocardiograph (ECG) or chest X-ray (CXR), with both tests perceived as useful and actually used in most cases of heart failure in the IMPROVEMENT study. 54 A normal ECG recording will, in most cases, exclude LVSD;55,56 however, changes may be subtle and the lack of ECG interpretation skills may still require referral for specialist opinion. Chest X-rays are often cited as useful in diagnosis, but a normal result does not exclude heart failure. 57,58 Furthermore, symptoms and signs may indicate the possibility of heart failure but are not reliable for establishing the diagnosis. 59 It is therefore unsurprising that studies exploring the validity of a clinical diagnosis of heart failure in primary care report high rates of misdiagnosis when patients are assessed against objective criteria (rates of 25–50% accuracy reported in different series). 60–62 Furthermore, underdiagnosis of heart failure is not confined to the primary care physician,63 with only 31% of patients being offered echocardiography by hospital physicians following referral with possible heart failure in one study. 64

In this context, the potential role of natriuretic peptides in diagnosing heart failure on the basis of a simple and inexpensive blood test has emerged. Numerous studies have confirmed the stability and feasibility of natriuretic peptide testing, although there are relatively few data testing the peptides in the clinical setting where they would be most used, i.e. in adults presenting with persisting breathlessness in the community. However, current evidence suggests that selecting natriuretic peptide cut-off values to ensure a high negative predictive value, which is important in a primary care setting, reduces the specificity of the test. For example, both NT-proBNP and BNP assays set at cut-offs to achieve a sensitivity of 100% showed a specificity of 70%, a positive predictive value of 7%, a negative predictive value of 100% and an area under the receiver operatingcharacteristic (ROC) curve (AUC) of 0.92 (95% CI 0.82–1.0) for diagnosing heart failure in the general population. 52 The performance of the assays was similar whatever the cause of heart failure and similar negative predictive values were also shown for diagnosing LVSD. 23

These data indicate that a normal level of natriuretic peptides virtually guarantees that heart failure is not present, but that confirmatory echocardiography is needed in patients with elevated peptides to confirm the diagnosis. The cost-effectiveness of natriuretic peptides versus standard diagnostic triage is not established. However, they may also have an important role in guiding therapy, at least in specialist and emergency room settings. 65–67

Current UK guidance

The current National Institute for Health and Clinical Excellence (NICE) guideline50 recommends that patients with suspected heart failure should have an ECG and/or natriuretic peptide test performed, the latter ‘where available’. If both are normal then heart failure is unlikely and an alternative diagnosis to explain the symptoms should be considered. If either one is abnormal then the patient should have a Doppler echocardiogram. This guidance was based on the high sensitivity of BNP and ECG, and the result of a health economic analysis that demonstrated that the cost per life-year gained through echocardiography is dependent upon the proportion of patients referred for echocardiography in whom the diagnosis of heart failure is confirmed.

Clinical experience in the UK suggests that the pretest probability of heart failure being present in patients referred for echocardiography varies markedly, with some centres performing many echocardiograms but with few showing any abnormality. Such inefficient use of the limited resource of echocardiography is problematic, and the NICE committee therefore wished to produce simple guidelines as to which patients should be referred by their GP for further investigation. However, the amount of data available to the committee was limited, and therefore the Health Technology Assessment call that funded this work was timely.

This study will therefore help address important unanswered questions and thus refine the national guideline in a number of areas. What is the optimal decision cut-off point for plasma BNP (or its co-secreted NT-proBNP) in terms of referral for echocardiography? Is performing an ECG and carrying out a BNP test better than carrying out only one of these investigations? What is the diagnostic value added to clinical examination by adding either a BNP test and/or ECG interpretation to the diagnostic process, when there is guidance to the general practitioner as to which clinical features are most important in distinguishing heart failure from other causes of symptoms such as breathlessness?

Current evidence

There have been five recent systematic reviews relevant to the diagnosis of heart failure, four of which have involved the applicants. 50,68–70 Two50,69 of these reviews covered all symptoms, signs and diagnostic tests, and two68,70 were specifically concerned with BNP.The fifth is a review of the accuracy of 12-lead ECG. 71 The following points emerge from this evidence base.

Individual symptoms (such as breathlessness, fatigue, exercise intolerance and fluid retention) and signs (such as resting tachycardia, raised jugular venous pressure (JVP), displaced apex beat, third heart sound) are generally weak predictors of heart failure and have poor reliability, with little agreement between clinicians on their presence. A number of clinical scoring systems have been developed to diagnose heart failure, but these are not highly specific. 67,72 However, recent as yet unpublished work led by Hoes in Utrecht suggests that use of a clinical scoring system based on a combination of symptoms and signs may be a reasonable predictor of heart failure (AUC: 0.82) (A Hoes, Utrecht, 2005, personal communication).

Both ECG and BNP have high sensitivity for heart failure and so are good tests for ruling out the diagnosis. UK-based studies restricted to the use of ECG in primary care, however, give a more mixed picture on the value of ECG, with sensitivity in one study69 as low as 73%. This may relate to both differences in population characteristics and the skill of the practitioner interpreting the ECG. Although the CXR may show evidence of heart failure (e.g. cardiomegaly, pulmonary vascular congestion), it is not a good independent predictor of the syndrome and is of most value in identifying alternative causes of symptoms.

Echocardiography is the ‘gold standard’ investigation for LVSD and valve disease. Indirect measures of diastolic dysfunction can be made on echocardiography, but the interpretation of the findings may be difficult, particularly in the elderly and in patients with atrial fibrillation (up to 30% of new cases of heart failure in most series). Most often, ‘diastolic’ (or ‘non-systolic’) heart failure is a diagnosis of exclusion, i.e. symptoms and signs for which other causes have been exhaustively excluded and for which there is a response to therapy for heart failure.

Although these reviews are reasonably contemporary, only one has addressed the specific population of patients presenting with suspected heart failure in primary care, and this review was restricted to only UK studies (of which there were four). 69

Complexities of the evidence base

The complexities of the evidence base that this study therefore seeks to address are discussed in the following sections.

Summary

Heart failure is a common disorder, especially in the elderly, with major and increasing significance for patients and health-care systems. There is a need for better identification of patients and more intensive attempts to introduce and maintain the large evidence base for therapies. The most clinically effective and cost-effective diagnostic algorithms are currently not determined.

Inclusion and exclusion criteria

Studies were included if they estimated the diagnostic accuracy or reliability of symptoms, signs or investigations for detecting heart failure. Although the main focus of the review was on the diagnostic accuracy for suspected cases of heart failure in primary care, we also included studies from all patient settings, including emergency department, hospital and outpatient settings, as well as population cohort or screening studies and we grouped data by setting. Studies varied in whether they included patients with previously diagnosed heart failure or not; both groups of studies were included in the review. No language restriction was applied.

Studies were eligible if they compared a symptom, sign, ECG, CXR or BNP with an adequate reference standard comprising either a clinical or an echocardiographic diagnosis of heart failure. More specifically, adequate reference standards were considered to be prospective planned evaluation of: (1) a clinical diagnosis, including all information, for example using ESC criteria; (2) echocardiographic criteria for LVSD (such as assessment of LVEF or global assessment of ventricular function); or (3) echocardiographic criteria for heart failure with preserved systolic function. We excluded studies that (1) included children; (2) used an inappropriate index test, for example urinary natriuretic peptides; (3) used a reference standard that was inappropriate for the purposes of this review, such as measures of diastolic function alone or pulmonary capillary wedge pressure; (4) used a retrospective study design (e.g. a reference standard using a hospital discharge diagnosis of heart failure); (5) used a case–control design; or (6) that provided results such that 2 × 2 data could not be extracted. Although studies that used echocardiographic criteria for LVSD were included in our principle results tables (see Appendix 4), the meta-analysis was restricted to studies that used a diagnosis of heart failure as the reference standard.

Search strategy

MEDLINE and CINAHL were searched from inception to 7 July 2006, including citations in progress. Given that our previous searches on this topic did not find any additional studies in EMBASE, we did not search EMBASE during this review. 75 The search combined terms for the condition of interest (e.g. heart failure; systolic dysfunction) with terms for the index tests of interest. No language restriction or methodological filters were applied as our previous study found that such filters reduced the sensitivity of the search strategy. 75 Details of the search strategy are shown in Appendix 1.

To identify studies missed by the search we checked the reference lists of all primary studies that met the inclusion criteria and any review articles we found in this area. In addition, ‘grey literature’ databases and conference proceedings of relevant societies (ACC; AHA; ESC; British Cardiac Society; Heart Failure Society of America; Royal College of Physicians; International Academy of Cardiology; International Heart Failure Society; and the Cardiac Society of Australia and New Zealand) were searched. Finally, authors of relevant studies were contacted to clarify any questions regarding overlapping studies or to provide 2 × 2 data where possible.

Data extraction

For the ECG, CXR and BNP studies, two reviewers screened the titles and abstracts for relevant studies. However, given the size of the search results only one reviewer carried out the initial screening for relevant studies on symptoms and signs. Potentially relevant studies were obtained in hard copy and assessed by two reviewers against the inclusion criteria for the review. When there was disagreement over a study it was discussed with a third reviewer.

Data were extracted by both reviewers on potential sources of bias, demographic details of included subjects, operator and test characteristics (e.g. who assessed the symptoms and signs; who read the ECG; type of BNP assay), reference standard characteristics and test performance results (2 × 2 tables comparing test with reference standard; test reproducibility data when provided). Quality was assessed using the QUADAS criteria. 76

Data analysis

The data synthesis was performed using methods recommended by the working group of the Cochrane Collaboration on systematic reviews of diagnostic test accuracy. We grouped the studies by the index test, including the type of assay (BNP and NT-proBNP), and by the type of reference standard (clinical diagnosis of heart failure, echocardiographic criteria of LVSD, echocardiographic criteria of LVSD plus heart failure with preserved systolic function). The studies were then further sorted by the clinical setting (primary care, screening studies, emergency departments, outpatient secondary care settings and inpatients). From the 2 × 2 tables we calculated sensitivity, specificity, negative and positive predictive values, and likelihood ratios.

The sensitivity and specificity of each of the index tests were plotted in ROC space. The data were then pooled using a bivariate random-effects meta-analysis to calculate summary estimates of the sensitivity, specificity, diagnostic odds ratio (DOR) and positive and negative likelihood ratios for each of the index tests with software codes kindly provided by Roger Harbord. 77 The statistical software package stata9 (StataCorp) was used for these analyses. Tests of heterogeneity were not used as such tests may be misleading for systematic reviews of diagnostic test accuracy and are not recommended by the Cochrane diagnostic test accuracy group (Jon Deeks, University of Birmingham, 2007, personal communication).

To understand any influence of setting and prevalence we plotted the predictive values (post-test probabilities) against the prevalence of heart failure (pretest probability).

For studies that contained a direct within-study comparison we pooled the data to compare the diagnostic accuracy of BNP versus NT-proBNP, and BNP (or NT-proBNP) versus ECG. The two tests were compared in a hierarchical summary ROC analysis using software codes kindly provided by Petra Macaskill. 78 This analysis was also used to determine a relative DOR in those studies that directly compared two tests for heart failure.

Symptoms and signs for the diagnosis of clinical heart failure

Fifteen studies – five in general practice, five in patients referred from primary to secondary care, and five in acute care – examined the diagnostic accuracy of symptoms and signs of heart failure compared with an adequate reference standard of a clinically defined diagnosis of heart failure.

Of the general practice studies, the largest single study79 recruited 5260 patients who attended a practice in Portugal; if patients scored 3 or more on the ‘Boston’ score they were further assessed by echocardiography. The diagnostic accuracy of symptoms and signs was assessed in these 1058 patients; 200 patients could not be assessed by echocardiography or had uninterpretable echocardiograms (and therefore were classified as not having heart failure). The other four studies were conducted in a random selection of patients from general practice registers: three80–82 in a random selection of general practice patients and one83 in patients with COPD not previously known to have heart failure. The ECHOES study by Hobbs et al. 82 was divided into substudies of (1) patients with symptoms and signs of heart failure; (2) patients who were over the age of 45 years; (3) patients who were considered at risk of heart failure; (4) patients who had previously been diagnosed with heart failure; and (5) patients who were currently taking diuretics. Unless otherwise stated the data used in the analyses below are from the ECHOES substudy conducted in patients who had symptoms or signs of heart failure as this is the patient group most relevant to this assessment.

Five studies were conducted in general practice patients with suspected heart failure who were referred for further assessment at an open access heart failure clinic or as part of the study design. 84–88 The other five studies were conducted in patients presenting with dyspnoea in accident and emergency departments. 89–93 The studies conducted by Dao et al. 94 and Morrison et al. 92 involved overlapping cohorts of patients: we have used the Morrison study as it had more participants.

Some of the data in this section were obtained from the study authors as part of this assessment and have not been published previously. 80–88

Figure 5 shows the numbers of patients with and without heart failure in each of the included studies. The outlier study, with the highest number of patients with heart failure and the highest proportion of patients with heart failure, was the Fonseca study,79 set in Portugal.

FIGURE 5.

Studies assessing the accuracy of symptoms and signs for the clinical diagnosis of heart failure.

The quality of the included studies is shown in Tables 37–46 in Appendix 3. The studies were of variable quality, although most of the quality criteria either were met or were unclear from the study report.

Table 1 gives a summary of the overall results for the symptoms and signs assessed in this review. There was considerable variation across the studies, which is illustrated in the figures in the following sections. These differences may be due to differing definitions or elicitation of the symptoms or signs, or to differences in the patient groups studied. In particular, it is likely that those presenting to accident and emergency will be at the more severe end of the heart failure spectrum.

History of myocardial infarction

Ten studies81–85,87–91 estimated the accuracy of a previous history of myocardial infarction for the clinical diagnosis of heart failure (Figure 6). The summary estimates of diagnostic accuracy are:

• sensitivity: 0.26 (95% CI 0.19–0.37)

• specificity: 0.89 (95% CI 0.85–0.91)

• DOR: 2.87 (95% CI 1.71–4.82)

• positive likelihood ratio: 2.37 (95% CI 1.58–3.54)

• negative likelihood ratio: 0.82 (95% CI 0.73–0.93).

FIGURE 6.

History of myocardial infarction for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Note that these studies used the patient’s self-report of the history of myocardial infarction and the diagnosis was not verified. Also, most of these studies were conducted before 2003 and were therefore likely to be using a definition of myocardial infarction that did not include new criteria which include serum troponin elevation.

Dyspnoea for the diagnosis of clinically defined heart failure

Dyspnoea is an important presenting symptom of heart failure. Several of the studies used dyspnoea as an inclusion criterion for the study and therefore it was not possible to estimate the diagnostic accuracy of this symptom from these studies. Five studies79,80,82,84,92 estimated the diagnostic accuracy of this symptom, with the results showing considerable heterogeneity (Figure 7):

• sensitivity: 0.83 (95% CI 0.62–0.94)

• specificity: 0.54 (95% CI 0.40–0.67)

• DOR: 5.71 (95% CI 1.78–18.31)

• positive likelihood ratio: 1.79 (95% CI 1.30–2.47)

• negative likelihood ratio: 0.31 (95% CI 0.12–0.79).

FIGURE 7.

Dyspnoea for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Because this symptom is one of the few symptoms or signs with a relatively high sensitivity, this feature may be a potential method for identifying patients who have heart failure. For this reason, in Table 2 and Figure 8 we have included the data from the original studies, including more specific methods for eliciting this symptom as defined in the individual studies. As might be anticipated, the more restrictive definitions of breathlessness (e.g. dyspnoea on exertion) led to higher specificity but lower sensitivity.

TABLE 2 - Diagnostic accuracy of dyspnoea for the diagnosis of clinically defined heart failure

Study	n	Setting	Prevalence of clinically defined heart failure (%)	Symptom	Sensitivity (%)	Specificity (%)	Positive predictive value (%)	Negative predictive value (%)
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea	91	72	13	99
Hobbs et al., 200482	273	Patients with symptoms and signs of heart failure	6	Dyspnoea	100	45	11	100
	304	Patients aged over 45 years	2	Dyspnoea	100	77	8	100
	124	Patients at high risk of heart failure	4	Dyspnoea	100	61	10	100
	71	Patients taking diuretics	11	Dyspnoea	100	59	24	100
	103	Patients previously diagnosed with heart failure	34	Dyspnoea	97	63	4	56
Alehagen et al., 200380	458	Patients presenting with dyspnoea, fatigue or peripheral oedema	15	Dyspnoea	66	64	24	92
Cowie et al., 199784	122	Patients referred from general practice to an open access heart failure clinic, not previously diagnosed with heart failure	29	Dyspnoea	86	30	33	84
Morrison et al., 200292	276	Patients presenting to an emergency department with acute dyspnoea	49	Dyspnoea	52	54	45	61
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea at rest	11	99	48	96
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea on exertion	79	84	18	99
Morrison et al., 200292	276	Patients presenting to an emergency department with acute dyspnoea	49	Dyspnoea on exertion	85	32	47	75
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea when walking on the flat	36	99	54	97
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea when walking fast or slightly uphill	77	82	16	99
Fonseca et al., 200479	1058	Patients attending general practice	10	Dyspnoea when walking uphill	88	77	15	99

FIGURE 8.

Methods for eliciting the symptom of dyspnoea: receiver operating characteristic (ROC) plot of studies. HF, heart failure.

The symbols in the ROC plot in Figure 8 illustrate the considerable heterogeneity in the estimated sensitivity and specificity of dyspnoea for the diagnosis of clinically defined heart failure. The lines between points in the ROC space illustrate where different measures of dyspnoea have been used in the same study. Dyspnoea on exertion has a higher sensitivity but lower specificity than dyspnoea at rest or generally defined dyspnoea. However, there is considerable variation between studies in the estimation of the diagnostic accuracy of dyspnoea.

Orthopnoea and paroxysmal nocturnal dyspnoea for the diagnosis of clinically defined heart failure

Six studies79,83,89–92 estimated the diagnostic accuracy of orthopnoea for the diagnosis of clinically defined heart failure (Figure 9). These showed low sensitivity and varying specificity:

• sensitivity: 0.44 (95% CI 0.33–0.56)

• specificity: 0.89 (95% CI 0.69–0.96)

• DOR: 6.23 (95% CI 2.30–16.92)

• positive likelihood ratio: 3.91 (95% CI 1.51–10.11)

• negative likelihood ratio: 0.63 (95% CI 0.53–0.74).

FIGURE 9.

Orthopnoea for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Paroxysmal nocturnal dyspnoea was evaluated in three studies and showed similar sensitivity and specificity to that of orthopnoea (Fonseca et al. 79: sensitivity 29%, specificity 98%; Morrison et al. 92: sensitivity 34%, specificity 86%; Mueller et al. 93: sensitivity 47%, specificity 73%).

Oedema (as a symptom or sign) for the diagnosis of clinically defined heart failure

Twelve studies79,82–86,88–93 estimated the accuracy of oedema (as either a symptom or a sign) for the diagnosis of clinically defined heart failure. Again, this clinical feature shows low sensitivity and varying specificity (Figure 10). In the study by Wright et al. ,87 which enrolled patients with either dyspnoea or oedema of recent onset, only 5% of patients who were diagnosed as having heart failure had oedema with no symptoms of dyspnoea:

• sensitivity: 0.53 (95% CI 0.44–0.62)

• specificity: 0.72 (95% CI 0.62–0.80)

• DOR: 2.91 (95% CI 1.89–4.49)

• positive likelihood ratio: 1.89 (95% CI 1.42–2.51)

• negative likelihood ratio: 0.65 (95% CI 0.54–0.78).

FIGURE 10.

Oedema for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Elevated jugular venous pressure for the diagnosis of clinically defined heart failure

Seven studies79,83,89–93 estimated the accuracy of elevated JVP for the diagnosis of clinically defined heart failure. One study90 defined elevated JVP as JVP > 6 cm; in the other studies, elevated JVP was not further defined. This symptom also showed poor sensitivity with relatively poor specificity (Figure 11):

• sensitivity: 0.52 (95% CI 0.41–0.63)

• specificity: 0.70 (95% CI 0.56–0.80)

• DOR: 2.52 (95% CI 1.51–4.22)

• positive likelihood ratio: 1.73 (95% CI 1.23–2.43)

• negative likelihood ratio: 0.68 (95% CI 0.56–0.84).

FIGURE 11.

Elevated jugular venous pressure for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Added heart sounds for the diagnosis of clinically defined heart failure

Six studies79,89–93 estimated the accuracy of added heart sounds (third heart sound – S3 or gallop rhythm) for the diagnosis of clinically defined heart failure (Figure 12). This sign has very low sensitivity but high specificity:

• sensitivity: 0.11 (95% CI 0.04–0.24)

• specificity: 0.99 (95% CI 0.97–1.00)

• DOR: 13.4 (95% CI 6.58–27.3)

• positive likelihood ratio: 12.1 (95% CI 5.74–25.4)

• negative likelihood ratio: 0.90 (95% CI 0.82–0.99).

FIGURE 12.

Added heart sounds for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

This means that if the sign is present it helps to rule the disease in but if absent it does not rule the disease out.

Lung crepitations for the diagnosis of clinically defined heart failure

Eleven studies79,82–85,87,88,90–93 estimated the accuracy of the presence of lung crepitations for the diagnosis of clinically defined heart failure (Figure 13). Lung crepitations have poor sensitivity and moderate specificity:

• sensitivity: 0.51 (95% CI 0.44–0.58)

• specificity: 0.81 (95% CI 0.71–0.88)

• DOR: 4.34 (95% CI 2.91–6.47)

• positive likelihood ratio: 2.64 (95% CI 1.86–3.74)

• negative likelihood ratio: 0.61 (95% CI 0.55–0.68).

FIGURE 13.

Lung crepitations for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Investigations for the diagnosis of clinically defined heart failure

Electrocardiogram for the diagnosis of clinically defined heart failure

Eleven studies79–88,90 estimated the accuracy of an abnormal ECG for the diagnosis of clinically defined heart failure.

Figure 14 shows the numbers of patients with and without heart failure in each of the studies that assessed the diagnostic accuracy of ECG for the clinical diagnosis of heart failure. The largest study, which also had a much higher proportion of patients with heart failure than the other studies (the outlier), was the Fonseca study. 79

FIGURE 14.

Studies assessing the accuracy of electrocardiography for the clinical diagnosis of heart failure.

In most of the studies the ECG criteria for defining an abnormality used to determine the presence of heart failure were quite broad. For example, in the study by Rutten et al. ,83 the criteria used were abnormal Q waves, complete or incomplete left bundle branch block, LV hypertrophy, atrial fibrillation, ST and/or T wave abnormalities and sinus tachycardia. Using broad criteria for ECG abnormality achieves a relatively high sensitivity but only moderate specificity (Figure 15):

• sensitivity: 0.89 (95% CI 0.77–0.95)

• specificity: 0.56 (95% CI 0.46–0.66)

• DOR: 4.80 (95% CI 4.36–25.7)

• positive likelihood ratio: 2.03 (95% CI 1.62–2.53)

• negative likelihood ratio: 0.19 (95% CI 0.09–0.42).

FIGURE 15.

Electrocardiography for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

A completely normal ECG can help to rule out the diagnosis of heart failure, but the presence of any abnormality does not help to rule the diagnosis in.

It should also be remembered that in these studies the diagnostic accuracy of the ECG was obtained either from an ECG read by a cardiologist or from the automatic reading of an ECG. In a study comparing the diagnostic accuracy of general practitioners and hospital physicians in detecting heart failure on an ECG the sensitivity and specificity of an ECG read by a general practitioner were 53% and 63%, respectively, and those read by a hospital physician were 95% and 47% respectively. 95 However, the mean sensitivity of 123 Scottish GPs reviewing 180 ECGs was higher at 94%. 96

The studies of diagnostic accuracy indicate how well a diagnostic test converts the pretest probability of a disease into the probability that a patient has the disease after the test. Figure 16 shows this graphically for the studies estimating the diagnostic accuracy of ECG for heart failure. The prevalence of heart failure in the patients who were enrolled in the study is shown on the x-axis as the pretest probability of disease. The probability that a patient has heart failure after an abnormal ECG is shown by the closed symbols, and the probability that a patient has heart failure after a normal ECG is shown by the open symbols. The summary estimates of how well the test is able to rule in or rule out the disease (as calculated by the positive and negative likelihood ratios) are shown by the curved lines. The further that these lines are from the line at 45° to the x-axis, the better the test is able to discriminate between those who have the disease and those who do not have the disease.

FIGURE 16.

Pretest/post-test graph of electrocardiography for the diagnosis of clinically defined heart failure. Note that the curved lines are back calculated from the overall likelihood ratios and not by a regression fit. Closed symbols represent post-test probability when test result is positive; open symbols represent post-test probability when test result is negative.

Chest X-ray for the diagnosis of clinically defined heart failure

Nine studies79,80,84,85,87,89–92 measured the accuracy of either any abnormality seen on CXR or an increase in the cardiothoracic ratio.

Five studies79,80,84,85,87 estimated the accuracy of any sign of heart failure on CXR to detect the diagnosis of clinically defined heart failure (Figures 17, 18 and 19). The estimates for the diagnostic accuracy of this test varied greatly:

• sensitivity: 0.68 (95% CI 0.40–0.88)

• specificity: 0.83 (95% CI 0.66–0.93)

• DOR: 10.7 (95% CI 4.45–25.5)

• positive likelihood ratio: 4.07 (95% CI 2.25–7.39)

• negative likelihood ratio: 0.38 (95% CI 0.18–0.78).

FIGURE 17.

Studies assessing the accuracy of chest X-ray for the clinical diagnosis of heart failure.

FIGURE 18.

Chest X-ray for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

FIGURE 19.

Pretest/post-test graph of chest X-ray for the diagnosis of clinically defined heart failure. Note that the curved lines are back calculated from the overall likelihood ratios and not by a regression fit. Closed symbols represent post-test probability when test result is positive; open symbols represent post-test probability when test result is negative.

An abnormal CXR is moderately helpful for ruling the diagnosis in, but a normal CXR is not able to rule out the diagnosis.

Six studies79,87,89–92 estimated the diagnostic accuracy of increased cardiothoracic ratio on CXR:

• sensitivity: 0.67 (95% CI 0.53–0.78)

• specificity: 0.76 (95% CI 0.65–0.84)

• DOR: 6.25 (95% CI 3.60–10.8)

• positive likelihood ratio: 2.73 (95% CI 1.94–3.86)

• negative likelihood ratio: 0.44 (95% CI 0.31–0.61).

B-type natriuretic peptides for the diagnosis of clinically defined heart failure

Twenty studies82,84,88,91–93,97–110 examined the accuracy of BNP for a diagnosis of clinically defined heart failure (Figure 20).The largest single study was the Breathing Not Properly Study104 (the outlier in Figure 20), which recruited 1586 patients in 11 emergency departments in the USA and Europe.

FIGURE 20.

Studies assessing the accuracy of BNP for the clinical diagnosis of heart failure.

The results of the studies show a consistently high sensitivity but varying specificity for the diagnosis of heart failure (Figures 21 and 22). An elevated BNP does not confirm the diagnosis of clinically defined heart failure but a normal level rules the diagnosis out:

• sensitivity: 0.93 (95% CI 0.91–0.95)

• specificity: 0.74 (95% CI 0.63–0.83)

• DOR: 39.5 (95% CI 21.44–72.6)

• positive likelihood ratio: 3.57 (95% CI 2.44–5.21)

• negative likelihood ratio: 0.09 (95% CI 0.06–0.13).

FIGURE 21.

BNP for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

FIGURE 22.

Pretest/post-test graph of BNP for the diagnosis of clinically defined heart failure. Note that the curved lines are back calculated from the overall likelihood ratios and not by a regression fit. Closed symbols represent post-test probability when test result is positive; open symbols represent post-test probability when test result is negative.

Four studies82,84,88,97 estimated the diagnostic accuracy of BNP for the diagnosis of clinical heart failure in patients in general practice or patients referred from general practice (Figure 23). The studies in general practice showed slightly lower sensitivity than, but similar specificity to, the studies overall:

• sensitivity: 0.84 (95% CI 0.72–0.92)

• specificity: 0.73 (95% CI 0.65–0.80)

• DOR: 14.3 (95% CI 5.45–37.8)

• positive likelihood ratio: 3.12 (95% CI 2.22–4.39)

• negative likelihood ratio: 0.22 (95% CI 0.11–0.42).

FIGURE 23.

BNP for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies set in general practice. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

N-terminal pro-B-type natriuretic peptides for the diagnosis of clinically defined heart failure

Sixteen studies80–83,86–89,93,109,102,104,111–114 examined the accuracy of NT-proBNP for the diagnosis of clinically defined heart failure (Figure 24).

FIGURE 24.

Studies assessing the accuracy of NT-proBNP for the clinical diagnosis of heart failure.

The results for NT-proBNP again show generally high sensitivity but varying specificity, with a somewhat lower specificity than for BNP (Figures 25 and 26):

• sensitivity: 0.93 (95% CI 0.88–0.96)

• specificity: 0.65 (95% CI 0.56–0.74)

• DOR: 24.6 (95% CI 14.4–42.2)

• positive likelihood ratio: 2.70 (95% CI 2.12–3.43)

• negative likelihood ratio: 0.11 (95% CI 0.07–0.18).

FIGURE 25.

NT-proBNP for the diagnosis of clinically defined heart failure: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

FIGURE 26.

Pretest/post-test graph of NT-proBNP for the diagnosis of clinically defined heart failure. Note that the curved lines are back calculated from the overall likelihood ratios and not by a regression fit. Closed symbols represent post-test probability when test result is positive; open symbols represent post-test probability when test result is negative.

Eight of the studies80–83,86–88,111 that examined the accuracy of NT-proBNP for the diagnosis of clinically defined heart failure were conducted in general practice patients or patients referred from general practice. Results for the studies conducted in general practice patients were similar to the overall results for NT-proBNP, with slightly lower specificity than for the overall results (Figure 27):

• sensitivity: 0.90 (95% CI 0.81–0.96)

• specificity: 0.60 (95% CI 0.50–0.70)

• DOR: 14.3 (95% CI 7.73–26.5)

• positive likelihood ratio: 2.28 (95% CI 1.82–2.86)

• negative likelihood ratio: 0.16 (95% CI 0.09–0.30).

FIGURE 27.

NT-proBNP for the diagnosis of clinically defined heart failure in general practice: summary receiver operating characteristic (curve) (SROC) plot of studies. Each circle in the above plot represents the estimated sensitivity and specificity from each study plotted in a receiver operating characteristic (ROC) space. The size of each circle is proportional to the size of the study. HSROC, hierarchical summary receiver operating characteristic.

Comparison of BNP versus NTpro-BNP for the diagnosis of clinically defined heart failure

Six studies82,88,93,100,102,104 (n = 1623) compared the diagnostic accuracy of BNP with that of NT-proBNP for the clinical diagnosis of heart failure. There was no statistical difference in the diagnostic accuracy between the two tests, with a relative DOR of NT-proBNP/BNP of 1.20 (95% CI 0.30–4.80) (p = 0.77).

The performance of the individual assays is shown in Figure 28. There is no clear evidence of the superiority of one assay over another. In some studies an individual assay performs better than the overall group (i.e. the point representing the study falls outside the curves); however, the same assay performs worse than the overall group in other studies.

FIGURE 28.

Pretest/post-test graph of BNP and NT-proBNP by assay type for the diagnosis of clinically defined heart failure. Note that the curved lines are back calculated from the overall likelihood ratios and not by a regression fit. Closed symbols represent post-test probability when test result is positive; open symbols represent post-test probability when test result is negative.

Studies included in the individual patient data analysis

Studies from the systematic review were deemed suitable for inclusion in the IPD analysis if they (1) were based in primary care and (2) had a minimum of 100 recently symptomatic patients. This limit on sample size was made to both reduce publication bias and limit the inclusion of smaller studies of lower methodological quality.

Model development

Validation: area under the curve

Tables 7 and 8 present the results of the internal and external validation. BNP – both alone and in combination – showed excellent discrimination in the derivation and external data sets with the AUC ranging between 0.83 and 0.96. The clinical model and ECG models provided ‘acceptable’ discrimination with AUCs from 0.66–0.83 (Table 7). Similar results were found for models that included NT-proBNP, in which AUC ranged from 0.82 to 0.91 (Table 8).

Validation: calibration

Figure 29 shows the calibration plots for the derivation data set. The post-test probabilities fall reasonably close to the diagonal line for all models indicating that the models are well calibrated. Figures 30 and 31 show calibration plots for the Cowie et al. 84 and Hobbs et al. 82 data sets respectively. The Hobbs data were closer to the line than the Cowie data with wide confidence intervals reflecting the low prevalence. All models are well calibrated at the low end of the probability scale. The BNP and BNP + clinical models underestimate the top end, whereas the other models overestimate the top end. Figure 32 shows calibration plots for the Cost97 data. The ECG and clinical + ECG models are well-calibrated models. The clinical alone model overestimates the top end whereas all other models underestimate across the range of probabilities. The clinical + ECG model is the best calibrated model of the Fox et al. 85 data (Figure 33) whereas the clinical model is the best calibrated model from the Wright et al. 87 data (Figure 34). Calibration plots of the corresponding NT-proBNP models show similar goodness of fit and are provided in Figures 35 and 36.

FIGURE 29.

Calibration plots using Zaphiriou88 data set-derived models. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 1: BNP only. (b) Model 2: clinical only. (c) Model 3: ECG only. (d) Model 4: BNP + clinical. (e) BNP + ECG. (f) Model 6: clinical + ECG. (g) Model 7: BNP + clinical + ECG.

FIGURE 30.

Calibration plots of Zaphiriou-derived88 models applied to Cowie84 data. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 1: BNP only. (b) Model 2: clinical only. (c) Model 3: ECG only. (d) Model 4: BNP + clinical. (e) Model 5: BNP + ECG. (f) Model 6: clinical + ECG. (g) Model 7: BNP + clinical + ECG.

FIGURE 31.

Calibration plots of Zaphiriou-derived88 models applied to Hobbs82 data. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 1: BNP only. (b) Model 2: clinical only. (c) Model 3: ECG only. (d) Model 4: BNP + clinical. (e) Model 5: BNP + ECG. (f) Model 6: clinical + ECG. (g) BNP + clinical + ECG.

FIGURE 32.

Calibration plots of Zaphiriou-derived88 models applied to Cost97 data. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 1: BNP only. (b) Model 2: clinical only. (c) Model 3: ECG only. (d) Model 4: BNP + clinical. (e) Model 5: BNP + ECG. (f) Model 6: clinical + ECG. (g) BNP + clinical + ECG.

FIGURE 33.

Calibration plots of Zaphiriou-derived88 models applied to Fox85 data. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 2: clinical only. (b) model 3: ECG only. (c) Model 6: clinical + ECG.

FIGURE 34.

Calibration plots of Zaphiriou-derived88 models applied to Wright87 data. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Model 2: clinical only. (b) model 3: ECG only. (c) Model 6: clinical + ECG.

FIGURE 35.

Calibration plots of model 1 (NT- proBNP only) by data set. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Zaphiriou88 data. (b) Hobbs82 data. (c) Wright87 data. (d) Alehagen80 data. (e) Lim86 data. (f) Galasko81 data.

FIGURE 36.

Calibration plots of model 5 (NT-proBNP + clinical) by data set. The circles represent the observed proportion in the data set that had heart failure, and the vertical lines the 95% confidence interval around this proportion. (a) Zaphiriou88 data. (b) Hobbs82 data. (c) Wright87 data.

Parsimonious model

Table 9 shows the deviances calculated for each logistic model developed from the derivation data set. The BNP model improved by adding either clinical features (model 1 versus model 4: χ²(5) = 27.9, p < 0.0001) or ECG (model 1 versus model 5: χ²(1) = 4.9, p = 0.03). However, there was no gain by the addition of ECG to BNP + clinical (model 4 versus model 7: χ²(1) = 2.5, p = 0.11), whereas BNP + ECG did improve when clinical features were added (model 5 versus model 7: χ²(5) = 35.5, p < 0.0001). Hence, BNP + clinical was identified as the parsimonious model.

These model comparisons were also undertaken with NT-proBNP substituted for BNP. Similar results were found, with NT-proBNP + clinical shown to be the most parsimonious model (Table 10).

Simplifying the heart failure prediction model

For the BNP + clinical model to be used in practice it was advantageous to simplify it further by splitting the model into a two-stage process:

1. clinical

2. nomogram of clinical score with BNP.

Those with a high probability of heart failure from the clinical score alone could then be referred directly for echocardiography; the remainder would undergo a BNP test and then use a nomogram that would give a probability estimate of heart failure dependent on BNP result and clinical score.

As the age term in the clinical model alone (see Table 6) was not significant, this was removed from further analyses. The model was then rerun for the clinical part only, with the resultant model being:

where p = probability of heart failure.

The coefficients for this model are different from those shown in Table 6 because of the omission of age. This model was then simplified further into a scoring system by creating new weights that were related to the parameter estimates in the model. The following weights were assigned (the four features can be remembered as MICE: male infarction crepitations oedema):

• male: 2 points

• history of myocardial infarction: 6 points

• crepitations: 5 points

• ankle oedema: 3 points

Thus, any individual presenting with symptoms of heart failure could be given a clinical score between 0 (female with no history of myocardial infarction, no ankle oedema, no basal crepitations) and 16 (all features present).

Performance characteristics of the simple clinical rule

Table 12 gives the performance characteristics of the simple clinical rule, likelihood ratios of a positive test and post-test probability of heart failure associated with a pretest probability of 30%. A plot of the ROC curve demonstrated that the optimal cut-point on performance characteristics would be 5 (Figure 37).

TABLE 12 - Performance characteristics of the simple clinical rule to predict heart failure in individuals presenting with symptoms suggestive of heart failure

LR +, likelihood ratio of a positive test.

Note: Because of division by zero, calculations cannot be made for the last three rows.

Cut-point ≥	Sensitivity (%)	Specificity (%)	LR +	Pretest probability	Post-test probability
0	100	0	1	30	30
2	96.2	19.8	1.20	30	34
3	92.3	32.7	1.37	30	37
5	79.8	62.9	2.15	30	48
6	60.6	76.7	2.60	30	53
7	59.6	76.7	2.56	30	52
8	53.8	80.7	2.79	30	54
9	35.6	91.1	4.00	30	63
10	30.8	94.1	5.22	30	69
11	19.2	98.5	12.8	30	85
13	8.7	100	> 20	30	> 90
14	6.7	100	> 20	30	> 90
16	2.9	100	> 20	30	> 90

FIGURE 37.

Receiver operating characteristic curve of the simple clinical rule predicting heart failure among patients with symptoms suggestive of heart failure. Diagonal segments are produced by ties.

This suggested the simple clinical rule shown in Box 1.

BOX 1 - Simple clinical rule

In a patient presenting with symptoms in whom heart failure is suspected, refer straight for echocardiography if the patient has any one of:

• history of myocardial infarction

• basal crepitations

• is a male with ankle oedema

then refer straight for echocardiography

Otherwise, carry out a BNP (or NT-proBNP) test and refer to echocardiography depending on the results of the BNP or NT-proBNP test

The interpretation of the BNP result would depend upon the clinical score, as shown in the nomogram (Figure 38). For example, to obtain the post-test probability estimate of heart failure for a female with no clinical features (score of zero) with a BNP of 100 pg/ml, the clinician would draw a perpendicular line from the BNP value on the x-axis up to the appropriate curve and read the corresponding probability off the y-axis (10%). The nomogram for NT-proBNP and clinical features is presented in Figure 39.

FIGURE 38.

Nomogram estimating the probability of heart failure in patients with symptoms using the clinical score and BNP.

FIGURE 39.

Nomogram estimating the probability of heart failure in patients with symptoms using the clinical score and NT-proBNP.

Effect modifiers

Further evaluation of interactions between the plasma concentration of natriuretic peptides and patient characteristics in the prediction of heart failure was performed on all available data. A breakdown of patient characteristics by data set is presented in Table 14. Results of the logistic models, adjusting for clinical score, are given in Table 15.

There was no evidence that age, atrial fibrillation, diabetes or COPD had an effect on the performance of BNP or NT-proBNP. A marginal effect of obesity on BNP88 and diuretics on NT-proBNP87 was found, although this effect was not seen in the remaining data sets. The relationship between BNP and the risk of heart failure varied with gender of the individual for the Cost97 data only (p = 0.03). Several significant interactions were found for the Hobbs et al. 82 data; the relationship between NT-proBNP and risk of heart failure varied according to whether patients had IHD (p = 0.04) or were on beta-blockers (p = 0.03) or ACE inhibitors (p = 0.003). Similarly there was evidence from these data that ACE inhibitors had a modifying effect on BNP (p = 0.004); however, these effects were not replicated in the other data sets.

Summary of results of individual patient data analysis in terms of the objectives

1. Can a clinical scoring system based on symptoms and signs usefully predict the presence of heart failure? We found that a simple clinical scoring system based on previous myocardial infarction, basal crepitations and ankle oedema did usefully predict the presence of heart failure in terms of determining whether or not an individual should be referred immediately for echocardiography or should have a BNP test, with the decision to proceed to echocardiography depending upon the results of that test.

2. To rule out heart failure in primary care, what is the optimum decision cut-off point for plasma natriuretic peptides (BNP)? Figures 38 and 39 show the post-test probability of heart failure for a given BNP result and a given clinical score. The determination of the optimum decision cut-point depends upon the value placed upon making a correct diagnosis of heart failure. This is explored further in the decision modelling.

3. Does the diagnostic performance of plasma natriuretic peptides vary according to patient characteristics (including age, gender, presence of IHD, presence of COPD, diabetes mellitus, obesity, atrial fibrillation, existing pharmacological therapy at time of diagnostic test)? We found no consistent evidence of any significant interactions between the performance of plasma natriuretic peptides and patient characteristics.

4. How accurate is the combination of plasma natriuretic peptides and ECG at diagnosing heart failure? We found that adding ECG to clinical features + BNP did not result in improved accuracy of diagnosis.

Approach taken for the decision analysis

The approach we took needed to take into account the fact that BNP is a continuous variable, with no predefined positive or negative value. Therefore we needed to compare a large number of possible strategies, corresponding to the many possible BNP cut-points. To do this we first needed to define how much it would be worth spending to diagnose a case of heart failure – we refer to this as ‘willingness to pay’ (WTP). Once we had established a WTP, we could calculate what would be an appropriate cut-point for BNP. This cut-point will vary according to the pretest probability (which is given by the MICE score) and the test performance of BNP (which we have reported in the IPD analysis). Having done this we could reduce the decision analysis to a manageable number of alternatives: do nothing; perform BNP and echocardiography depending upon the result of the BNP test; or proceed straight to echocardiography. Therefore a key first step was to calculate plausible extremes for the WTP. This is described in the following section.

Estimating the value of diagnosing heart failure

To inform our estimation of the cost of a missed diagnosis we updated the systematic reviews carried out by NICE for the 2003 guideline on diagnosis and management of heart failure. 50 This updated review is shown in Appendix 6.

There is good evidence that treatment of heart failure with beta-blockers124 and ACE inhibitors125 can reduce mortality and the risk of hospital admission from heart failure. Therefore, once a diagnosis has been made, it can be assumed that the diagnosis will precipitate treatment that will reduce the risk of death and the risk of hospital admission. Conversely, it can be assumed that if a diagnosis is not made then the condition will worsen and result in an acute admission to hospital (when a diagnosis will be made), sudden death or worsening symptoms such that the diagnosis is reviewed and the correct diagnosis made. For the purposes of the calculations below we have assumed that the diagnosis will be delayed by 6 months (unless there is a hospital admission or the patient dies) in patients with genuine heart failure if they are not referred for echocardiography. The potential costs of missed diagnoses are avoidable hospital admissions and reduced life expectancy and quality of life. The size of these costs will depend upon the proportion of people who would be treated with beta-blockers and ACE inhibitors once a diagnosis has been made, as these are the two treatments for heart failure that have been shown to improve outcome and are considered indicated for the vast majority of patients with heart failure.

What is the likely survival and QALY gain from early detection of heart failure?

The 1-year survival rate from diagnosis of heart failure in patients in the Framingham Heart Study18 was 57% for men and 64% for women. The mean age of these patients was 70 years and there was no temporal change in survival over the 40 years that patients with heart failure were identified (1948–88). As ACE inhibitors were not available for most of that period, and beta-blockers were not used to treat heart failure, the Framingham survival data can be taken to be the prognosis in the minimally treated population, i.e. symptomatic treatment with diuretics alone, without use of drugs known to improve the prognosis.

Systematic reviews of the beta-blocker and ACE inhibitor trials124,125 suggest that the relative risk of death is reduced by 33% and 20%, respectively, by these agents. From the data available from these systematic reviews and the Framingham study18 it is possible to generate survival curves (untreated) for men and women separately and odds ratios for mortality for those taking ACE inhibitors alone (0.8) and ACE inhibitors combined with beta-blockers (0.5). If we assume that these odds ratios are stable over time and applicable separately to men and women, it is possible to calculate survival probabilities as shown in Table 16.

TABLE 16 - Estimated survival probabilities for people with heart failure treated with angiotensin-converting enzyme inhibitors and beta-blockers and on no therapy

ACEI, angiotensin-converting enzyme inhibitor; BB, beta-blocker.

Time (year)	Men			Women
	Untreated	ACEI only	ACEI + BB	Untreated	ACEI only	ACEI + BB
0.25	0.73	0.77	0.84	0.72	0.76	0.84
1	0.57	0.62	0.73	0.64	0.69	0.78
2	0.46	0.52	0.63	0.56	0.61	0.72
5	0.25	0.29	0.40	0.38	0.43	0.55
10	0.11	0.13	0.20	0.21	0.25	0.35

We then used these data to generate survival curves by linear interpolation. A patient with a 6-month delay to diagnosis is assumed to have the same probability of survival as someone who is untreated for that 6-month period. After 6 months it is assumed that their probability of survival is the same as someone who is on treatment. For example, the estimated survival curves for men diagnosed early (and therefore treated early), diagnosed late (and therefore treated late) and untreated are shown in Figure 40. In this illustrative example the treatment entails ACE inhibitors. Similar curves could be drawn for the effect of the combination of ACE inhibitors and beta-blockers, and for women.

FIGURE 40.

Estimated survival from earlier diagnosis of heart failure if treated with angiotensin-converting enzyme inhibitors.

The survival benefit due to early treatment is the area between the upper and middle curves. The longer the time horizon used to estimate the quality-adjusted life-year (QALY) gain from early diagnosis, the greater the benefit. Table 17, derived from Figure 40 (and similar graphs not shown), reports the estimated survival gain up to a time horizon of 3, 5 or 10 years for men and women separately treated either with an ACE inhibitor alone or with an ACE inhibitor plus beta-blocker. Assuming equal numbers of men and women, the average increase in life expectancy for a person diagnosed early is the mean of the increases in life expectancy given in Table 17 weighted for the proportions of people in each treatment category (51% treatment with ACE inhibitors alone; 34% ACE inhibitors plus beta-blockers; 15% no treatment). This gives an overall estimated survival gain of 0.163–0.390 years depending upon the time horizon (Table 17).

TABLE 17 - Estimated increases in survival and quality-adjusted life-years (QALYs) as a result of early diagnosis of heart failure by treatment

ACEI, angiotensin-converting enzyme inhibitor; BB, beta-blocker.

Time horizon (years)	Life-years gained (years)					QALY gain
	Men		Women		Overall
	ACEI	ACEI + BB	ACEI	ACEI + BB		Overall
3	0.104	0.307	0.113	0.326	0.163	0.106
5	0.150	0.459	0.174	0.511	0.247	0.161
10	0.218	0.697	0.278	0.854	0.390	0.254

In terms of QALY gain, a patient with significant heart failure (NYHA class III or IV) has a mean EuroQol 5 dimensions (EQ-5D) score of 0.6. 129 Patients with all categories of heart failure are likely to have a higher overall mean EQ-5D score, for example 0.65. Using this estimate of 0.65, the overall life-years gained shown in Table 17 can be converted to estimates of overall QALY gain: 0.106, 0.161 and 0.254 for the 3-, 5- and 10-year time horizons respectively.

Methods for heart failure modelling

The cost-effectiveness modelling is based on the decision tree shown in Figure 41. Three strategies are compared: ‘do nothing’, in which no further investigation is made; ‘BNP’, in which patients are given a BNP test and those whose BNP score exceeds a given threshold are then sent on for echocardiography; and ‘echo all’, in which all patients are referred immediately for echocardiography. The model is run for a patient group defined by a clinical score (the MICE score – see Chapter 8, Simplifying the heart failure prediction model), to produce a preferred option for that score. Running the model for a range of clinical scores then gives a policy of a minimum score for immediate referral for echocardiography, and possibly a minimum score below which BNP testing is not cost-effective.

FIGURE 41.

Decision tree for heart failure diagnosis.

The outputs of the model are in terms of investigation costs and cases detected. For convenience these are calculated per 1000 patients with any particular clinical score. This produces an incremental cost-effectiveness ratio (ICER) between any two strategies, which is the additional cost per additional case detected in comparing the more effective strategy with the less effective. The ICER is then compared against a threshold that represents the WTP for each additional case found. The WTP is estimated from two viewpoints. In the principal analysis only the NHS costs averted by early detection are considered (including hospital admissions), whereas a sensitivity analysis also includes valuation of the estimated QALYs gained from early detection.

Methods of sensitivity analysis

The parameters that go into the model are as follows:

• cost of investigation (namely echocardiography and BNP testing)

• WTP

• test performance of the BNP test

• pretest probability of heart failure.

A probabilistic sensitivity analysis cannot be performed here as the analysis is dependent upon a quantitative variable, BNP, whose cut-off in turn depends upon model parameters. In effect, the true decision problem has a very large number of options, and methods of probabilistic sensitivity analysis are not readily applicable to such problems. Therefore, we have used the approach below to explore the effect of varying all of the model parameters.

The pretest probability of heart failure is already varied in the base-case analysis through the use of the MICE score.

For cost of investigation sensitivity analysis we used an ‘extreme case’ approach, looking at the extremes in cost that would be most favourable to echocardiography (namely low-cost echocardiography and expensive BNP) and least favourable to echocardiography (high-cost echocardiography and cheap BNP).

With regard to WTP, the base-case analysis used a low value in that it ignored potential benefits in terms of increased life expectancy and quality of life and only took into account potential benefits in terms of reduced hospitalisation. Therefore, the sensitivity analysis explored the impact of increasing this WTP, taking into account increased life expectancy due to earlier treatment. We did not take into account improved quality of life as a result of symptom improvement as we felt that this effect would be relatively minor over the short time scale (maximum of 6 months) that we anticipated that treatment could be delayed relative to the effects of emergency admissions avoided and improved survival.

We varied test performance by repeating the analysis using the test performance characteristics for NT-proBNP, which is the principal alternative to BNP that is available.

The actual parameters that we changed were as follows:

• cost of echocardiography: £50–150

• cost of BNP testing: £10–20

• WTP to detect a case of heart failure: up to £5370 per case detected

• blood test used: NT-proBNP.

Results of heart failure modelling

Impact of changing the costs of investigation

In the baseline analysis the cost of echocardiography was set at £100 and BNP testing at £15. We conducted sensitivity analyses looking at the impact of changing these costs. Two cases are considered: one most favourable to echocardiography, in which the echocardiography cost is lowered and the BNP cost raised; and the other least favourable to echocardiography, in which the echocardiography cost is raised and the BNP cost lowered.

Impact of changing time horizon (i.e. changing QALY gain)

For the base-case analysis that incorporated QALY gain, referral for echocardiography was the preferred strategy for all patients except those with a MICE score of 0. As the time horizon extends, these benefits of echocardiography become greater.

We tested the robustness of this conclusion by repeating the sensitivity analysis on costs but this time using the WTP derived from incorporating QALYs. Because echocardiography came out as the preferred option for all MICE scores other than 0 in the base-case analysis incorporating QALYs, we have not reported the impact of lowering the cost of echocardiography as this would inevitably reach the same conclusion. When raising the cost of echocardiography and lowering the cost of BNP has led to a change in the conclusion, we have also looked at the longer time horizons. The results of this analysis are shown in Table 24.

The conclusion that echocardiography is the preferred initial investigation if the impact of early diagnosis on QALYs is taken into account is robust to increasing the cost of echocardiography (and lowering the cost of BNP) at all MICE scores except 0, 2 and 3. At a score of 0, BNP first is the preferred strategy regardless of the time horizon used, whereas the optimal decision changes back to echocardiography at a score of 3 if a 5-year or longer time horizon is used, and at a score of 2 if a 10-year time horizon is used.

Impact of using NT-proBNP

Finally, we repeated all of the analyses using test performance data for NT-proBNP rather than for BNP. The results were the same as for BNP.

Symptoms and signs of heart failure

The systematic review identified a number of symptoms and signs that were potentially helpful in the diagnosis of heart failure. Only one of these (dyspnoea) had a sensitivity over 80%. A number were reasonably specific, including history of myocardial infarction (89%), orthopnoea (89%), cardiomegaly (85%), added heart sounds (99%), lung crepitations (81%) and hepatomegaly (97%). In primary care the most potentially useful symptoms/signs in this context would be those with high sensitivity, as this might enable the clinician to rule out heart failure, if the symptom/sign was absent, without the need to refer for further investigation. Dyspnoea is the only clinical feature that comes close to this category with a sensitivity of 87%. As observed in the IPD analysis, in practice this symptom is present in the majority of patients in whom heart failure is suspected, with a frequency as high as 95% in one of the data sets. 88 Nevertheless, a sensitivity of 87% is not high enough to rule out heart failure if dyspnoea is absent.

Symptoms and signs with high specificity are useful for making a positive diagnosis, but their absence does not mean that the diagnosis can be excluded. Therefore, in the primary care context, clinical features of high specificity are of less value. A second factor is that the highly specific signs – added heart sounds and hepatomegaly – are not picked up reliably, even by specialists. For example, in a study of three clinicians examining 80 patients after myocardial infarction,115 the agreement as measured by the kappa co-efficient was low, ranging from 0–0.16 for hepatomegaly to 0.14–0.37 for a gallop rhythm.

In practice, clinicians do not interpret symptoms and signs in isolation, but rather in the context of the overall clinical picture. The IPD analysis validated a model for diagnosing heart failure based on clinical features that had been derived from the UK BNP study. 88 A clinical model based upon the combination of gender, age, past history of myocardial infarction, presence of ankle oedema and presence of basal crepitations was found to have reasonable predictive validity, with the AUC ranging from 0.66 to 0.79 in the five data sets in which it could be validated. Although breathlessness had been identified from the systematic review as the symptom with the highest sensitivity, it was not included in the clinical model. This was because its prevalence was very high (95%) in the derivation data set, reflecting how often breathlessness is the presenting symptom of heart failure. Nevertheless, the model worked equally well in populations with a lower prevalence of breathlessness. The Cost97 and Wright et al. 87 data sets had the lowest proportions of people with breathlessness (24% and 46% respectively; see Table 5) and had AUCs of 0.73 and 0.79, which are similar to the AUC of 0.76 achieved in the Zaphiriou et al. 88 derivation data set (Table 7). We explored whether adding breathlessness back into the full model (i.e. including BNP or NT-proBNP) would improve its overall accuracy, but we found that this was not the case, with the odds ratio for heart failure in the presence of breathlessness not being significantly greater than 1 once adjusted for the other clinical features and BNP score in the three data sets for which we could provide robust estimates (see Table 11). This may reflect the close correlation between factors already in the model, such as basal crepitations, and this symptom, so that, although in univariate analysis (as demonstrated in the systematic review) it was an important predictive symptom, it was less so when its diagnostic value was adjusted for these other factors.

Investigations for heart failure

The systematic review confirmed that ECG and BNP (or NT-proBNP) have high sensitivity for heart failure, and that CXR abnormalities are reasonably specific for heart failure. A problem with ECG reading in primary care is that the high sensitivity obtained when an ECG is read by a cardiologist or by automated reading may be lost if it is read by a GP. 95 However, it is clear that many GPs can detect relevant abnormalities accurately96 and so the key issue may be one of quality assurance of the ECG reading. 74

There are a number of different BNP assays available but we found no evidence of superiority of any one assay (BNP or NT-proBNP) over another.

The IPD analysis explored the value of ECG and BNP in addition to clinical features in the diagnosis of heart failure. We found that the best results were obtained when BNP testing was combined with clinical features. BNP plus clinical features performed better than ECG plus clinical features. Therefore, if BNP is available then ECG is not necessary as a screening investigation for heart failure.

Strengths and weaknesses of the systematic review

The report has synthesised all available published data on the diagnosis and investigation of heart failure, including data on symptoms, signs, ECG, CXR and the natriuretic peptides. The main focus of the report has been on the diagnostic accuracy of these tests for the diagnosis of heart failure, using clinical criteria such as the ESC criteria, rather than for the diagnosis of LVSD. The results of studies that investigated the diagnostic accuracy of these tests for the diagnosis of LVSD are shown in Appendix 4. This approach was taken as many patients who present with heart failure requiring further investigation and management in the primary care setting will have preserved systolic function. However, the lack of an objective and universally agreed definition for the reference standard and variability in the way that the reference standard is applied introduce uncertainty into the estimate of the diagnostic accuracy of the tests and increase the heterogeneity of the results.

As the main purpose of the report was to provide assistance for the diagnosis of heart failure in the primary care setting, we have included a prespecified subgroup analysis of those studies that examined the diagnostic accuracy of ECG, CXR and natriuretic peptides for the diagnosis of heart failure in general practice, those referred from general practice, and in accident and emergency and hospital and outpatient settings. There were no differences in the diagnostic accuracy of each of the investigations observed between the clinical settings.

There was considerable heterogeneity in the estimates of sensitivity and specificity for many of the individual clinical features. This is likely to reflect the poor reliability of some of the signs and the varying definitions of the symptoms/signs used in the studies. For example, with the symptom of breathlessness, the more restrictive definitions led to higher specificity and lower sensitivity. Statistical tests of heterogeneity were not used as they may be misleading in the context of systematic reviews of the accuracy of diagnostic tests and they are not supported by the Cochrane diagnostic test accuracy working group.

Development of a simple clinical tool

For a tool to be useful in clinical practice it needs to be straightforward to apply. A potential use of the clinical tool would be to discriminate between patients who had a sufficiently high probability of heart failure that they should be referred for echocardiography and those who should have further investigation before proceeding (or not) to echocardiography. A simplification of the model developed by the IPD analysis led to a simple rule:

• in a patient presenting with symptoms in whom heart failure is suspected, refer straight for echocardiography if the patient has any one of:

  1. – history of myocardial infarction

  2. – basal crepitations

  3. – ankle oedema in a male patient

• otherwise carry out a BNP test and refer to echocardiography depending on the results of the test.

Cost-effectiveness of this clinical rule

We tested the cost-effectiveness of this clinical rule by determining the optimum decision points at which to perform a BNP test and/or refer for echocardiography by using a decision analysis based upon willingness to pay (WTP). We used two approaches to WTP. One was highly conservative and assumed that the diagnostic strategy should be cost neutral, with costs of diagnosis offset by savings in terms of admissions avoided as a result of diagnosis. The second approach also took into account the impact that earlier diagnosis would have on survival, using an assumed WTP of £20,000 per QALY gained. This is the threshold that is likely to be considered cost-effective within the NHS, as this is the threshold adopted by NICE. A summary of the results is shown in Table 25. For comparison, the simple clinical tool that was developed purely on the basis of its performance characteristics (i.e. not taking cost-effectiveness into account) is shown at the foot of the table.

Key points to make are:

1. The conclusions of the modelling are sensitive to the assumptions made. If the aim was a cost-neutral strategy for the NHS there were some scenarios in which it was appropriate at low MICE scores (i.e. scores of 3 or less, which correspond to a pretest probability of up to 20%) not to investigate further.

2. However, for a cost-effectiveness analysis it is important to take into account the likely impact of early diagnosis on survival. If this is taken into account then the analysis suggests that virtually all patients with suspected heart failure should be referred straight for echocardiography.

3. When BNP (or NT-proBNP) testing is used it is important to take into account the clinical features (i.e. the MICE score) in interpreting the result, as the appropriate cut-off points vary by MICE score.

The simple decision rule that was derived in Chapter 8 sits fairly centrally within the bounds of the modelling in that it falls between the highly conservative analysis based upon a WTP that is cost neutral to the NHS and a WTP that is based upon £20,000 per QALY.

Strengths and weaknesses of the individual patient data analysis

The quality of an IPD analysis is determined both by the willingness of authors to make data sets available and by the definitions of key fields used in the analysis. Our group had authorship of a number of the original data sets and the majority of our colleagues made their data available. We have attempted to eliminate any bias in characterisation of fields by applying, when data were available, a common definition of heart failure, ensuring that peptides were converted to standard units (pg/ml) when necessary and using a consistent classification of an abnormal ECG across data sets when possible.

The clinical rule was developed on a breathless population and therefore designed for use on patients presenting with shortness of breath, which is the most common presentation of heart failure. However the external validation provides evidence of its discriminatory ability across populations of varying prevalence of breathlessness and therefore it could be used in patients presenting with any symptom of heart failure. Included clinical variables, although elicited by cardiologists in the original studies, are either ‘yes/no’ clinical history items (gender and previous medical history of myocardial infarction) or straightforwardly ascertained examination points (oedema and crepitations) and so should be transferable to a primary care setting. Nomograms, although novel for BNP interpretation, are commonly used in primary care, for example in the cardiovascular risk charts printed in the back of the BNF. The nomogram could be used by the clinician to estimate the post-test probability of heart failure for a given BNP result. Alternatively, simply the ‘cut-off’ values for BNP for referral for echocardiography as derived from the decision analysis could be used (for instance by incorporation into standard laboratory results), without need of the nomogram.

There is no ‘gold standard’ test for all cases of heart failure, particularly in cases of heart failure with preserved ejection fraction. We used the ESC criteria for heart failure, namely appropriate symptoms plus objective evidence of cardiac dysfunction. It is reassuring that the validation and calibration results for the clinical rule across different data sets was reasonably good, despite the subjective nature of the ESC criteria. We could have restricted ourselves to cases with a low ejection fraction, for which echocardiography is an agreed reference standard. However, from the perspective of diagnosis in primary care this would have limited the utility of the approach, as the general practitioner needs to ensure that a diagnosis is made in all patients with suspected heart failure and thus needs to know who should be referred for further investigation (regardless of whether the underlying diagnosis is low ejection fraction heart failure).

The calibration plots indicate that, with the exception of the Cost97 data, the clinical rule gives reasonably accurate estimates of the probability of heart failure at the lower end of the probability scale. It is in this area of the scale, where primary care patients typically present, that GPs would benefit the most with help in determining whether a patient might have heart failure and should undergo further tests. The variability across the BNP calibration plots could be due to differences between the tests, in particular laboratory versus near patient tests and differing coefficients of variation. The use of the lower prevalence studies to validate the clinical rule also increases the face validity of the results as studies relying on GP referral to secondary care are inevitably adding a ‘filter’ as opposed to the undifferentiated person with breathlessness who might be considered the typical diagnostic issue for heart failure in primary care.

We were unable to pool individual-level data sets because of evidence of heterogeneity between them. The characteristics that were found to alter the performance of the measurement of plasma concentration of natriuretic peptides were from studies of non-incident participants and may therefore reflect the case selection. The few significant effect modifiers that we identified are likely to reflect spurious effects given the multiple statistical tests that we performed. Indeed, interactions were no longer significant when the Bonferroni adjustment was applied to the probabilities. Therefore, we found no evidence that the test performance of BNP or NT-proBNP is significantly influenced by factors such as age, gender or co-existent disease.

There is a lack of methodology published in the area of IPD meta-analysis in diagnostic testing and further research is therefore warranted in this area.

Strengths and weaknesses of the decision analysis

Not surprisingly there have been no clinical trials to determine the clinical impact of early diagnosis of heart failure and so our estimate of the likely benefit of diagnosis has to be to some extent speculative. In particular, it is difficult to estimate for how long the diagnosis of heart failure will be delayed if it is not made at presentation. In the decision analysis we assumed that if a patient was not referred for echocardiography then the diagnosis would be made after an average delay of 6 months if the patient did not die or was not admitted to hospital before that time interval. There are no data on which to base such an assumption. If the time delay is shorter, the WTP to diagnose a case of heart failure would be reduced. However, in our cost-effectiveness analysis we adopted a conservative time horizon of 3 years. In other words, we did not take into account survival benefits that would be anticipated more than 3 years after diagnosis, although we estimated (see Figure 40) that there would be some residual benefit beyond this time. Therefore, despite the inherent limitations of the modelling, it is unlikely that the general conclusion that echocardiography would be the preferred option for investigation would be overturned. The likelihood is that benefits from early diagnosis are greater than we assumed.

We did not take into account waiting lists for echocardiography, and the model assumed that there was sufficient capacity. In reality in the NHS this is not the case. Recent data from the Healthcare Commission128 show that 72% of patients referred for echocardiography receive the investigation within 13 weeks. If we built in a 13-week delay to echocardiography this would nullify much of the benefit of early diagnosis.

Costs of BNP tests vary by manufacturer. Nevertheless, we found that the conclusions of our decision analysis were robust to significant changes in the costs of BNP tests and echocardiography. Even in the circumstance most adverse to echocardiography, the cost-effectiveness analyses incorporating quality of life showed that the threshold for referral straight to echocardiography only increased to a clinical score of 5, but the BNP cut-point for referral in these circumstances was low, with a cut-point varying from 17 to 58 pg/ml.

The decision analysis was based upon the assumption that all patients with a diagnosis of heart failure should proceed to echocardiography to inform the diagnosis and provide information on the underlying cause of the heart failure. It is recognised that alternative ‘reference standard’ investigations might become available/be used, but at the current time our use of echocardiography reflects standard practice, as reflected in the NICE guideline. 50 Furthermore, it is recognised that BNP analysis is increasingly being used as a test with diagnostic value in its own right, independent of the results of echocardiography. For example, heart failure with preserved ejection fraction may have normal echocardiography findings but abnormal BNP. However, this does not remove the need for echocardiography in a patient with abnormal BNP and so does not affect the overall conclusions of this review.

Other recent systematic reviews

Our systematic review findings are broadly consistent with those of other systematic reviews in this area that have been recently published. Khunti and colleagues71 reviewed four studies that had evaluated the diagnostic accuracy of ECG in the specific context of referral from primary care to echocardiography. They found that sensitivity in these studies varied from 73% to 91% and concluded therefore that the ECG was an inadequate screening tool. Davenport and colleagues131 reviewed the diagnostic accuracy of natriuretic peptides and ECG in the diagnosis of LVSD and found similar diagnostic accuracy between ECG, BNP and NT-proBNP and no value from combining BNP with ECG. Although we found no value from combining BNP with ECG, we did find evidence that BNP was superior to ECG in both the systematic review and the IPD analysis. Davenport identified two studies132,133 that provided data evaluating the combination of BNP and ECG. One of these132 was excluded from our review because the index test was deemed inappropriate (the ECG abnormality was simply prolonged QRS duration) and the other133 because the reference standard was assessment of LVSD and not heart failure. We found four studies that had data on both ECG and BNP, but only one of these had published the data. 90 The remaining three82,84,88 provided us with the relevant data so that we could perform the calculations. It is likely that BNP is a more accurate test for heart failure than it is for LVSD. Indeed, a recent systematic review134 of BNP studies concluded that, although BNP is useful for excluding heart failure, it is more limited for ruling out systolic dysfunction, with an AUC of 0.93 for heart failure but only 0.75 for systolic dysfunction. Clerico and colleagues,135 like our review, found no evidence of any significant differences in test performance between BNP and NT-proBNP. Other recent reviews of BNP have confirmed its value as a ‘rule out’ test for heart failure. 136–138

Interpretation of the research findings in the context of the NHS

The current NICE guideline for heart failure recommends that, in patients with suspected heart failure, a 12-lead ECG and/or a BNP or NT-proBNP test should be performed to exclude heart failure, and only those patients with a positive ECG or BNP should proceed to echocardiography. The systematic review and the IPD analysis have demonstrated that, when taken in combination with clinical features, BNP (or NT-proBNP) is superior to ECG, and performing ECG adds nothing if a BNP test has been performed. The IPD analysis has further demonstrated that a simple clinical score (the MICE score – male 2 points, infarction 6 points, crepitations 5 points, oedema 3 points) can usefully predict the presence of heart failure. On the basis of the test performance of BNP it was possible to provide a rational strategy by which patients should proceed straight to echocardiography, namely if their MICE score was 5 or more, they should be referred straight for echocardiography, otherwise they should have a BNP test first with referral for echocardiography dependent upon the results of the BNP test. Thus, the analysis we have performed points to the need for important changes to the NICE recommendations. First, BNP (or NT-proBNP) should be recommended over ECG and, second, some patients should be referred straight for echocardiography without undergoing any preliminary investigation. Therefore, BNP (or NT-proBNP) testing should be available in primary care.

The third part of our research, the decision analysis, sought to refine these conclusions by considering cost-effectiveness. Our base-case analysis took into account the cost to the NHS. We estimated that missing a case of heart failure would on average cost the NHS £270 in terms of avoidable hospital admissions, and therefore assumed that the NHS would be willing to pay at least this amount of money to diagnose a new case of heart failure. We then estimated how much the NHS would be willing to pay if we also took into account the impact of improved survival from earlier diagnosis, valuing an additional QALY at £20,000. From these analyses the simple decision rule that we developed is likely to be considered cost-effective as it sits fairly centrally within the bounds of these two analyses. However, the analysis also suggested that the preferred option may be to refer virtually all patients with suspected heart failure straight for echocardiography without undergoing preliminary BNP testing, given that this strategy falls within a WTP threshold that takes into account likely improved survival resulting from earlier diagnosis.

The reality of availability of echocardiography services in the NHS means that referral of all patients straight to echocardiography may not be an immediately viable option. Furthermore, it is not likely to be an option in the near future because of the implications for both training and service provision. In this context the strategy of using the MICE score to determine who should be referred straight to echocardiography and who should be referred after a BNP test has been performed is an attractive option in that it is more cost-effective than current recommended practice and will make less demand on already overstretched echocardiography services than referring all patients straight for echocardiography. If this strategy is adopted, then the question remains of what should be the appropriate cut-points for BNP (or NT-proBNP). It is clear from our analysis that the cut-points should take into account the underlying risk of heart failure (i.e. the MICE score). Given that the rationale for using the MICE score in this instance would be to make optimal use of a scarce resource, it follows that a rational cut-point could be determined by the proportion of patients referred for echocardiography who turn out to have heart failure (i.e. the post-BNP probability). Table 26 shows what these cut-points would be for different post-test probabilities for BNP and NT-proBNP using the methodology described in Chapter 9 (see Methods for heart failure modelling) and adjusting the WTP to obtain the desired post-test probability. Thus, applying the clinical decision rule, for a MICE score of 0, the appropriate cut-point for BNP might lie between 210 pg/ml (in which case one in five people referred to echocardiography would have heart failure) and 360 pg/ml (in which case three in ten people referred would have heart failure).

Implications for health care

• The analysis that we have performed points to the need for important changes to the NICE recommendations. First, BNP (or NT-proBNP) should be recommended over ECG and, second, some patients should be referred straight for echocardiography without undergoing any preliminary investigation.

• Therefore, natriuretic peptide testing should be available in primary care.

• If there is sufficient local capacity, the evidence synthesised here suggests that many patients with symptoms indicating possible heart failure should be referred straight for echocardiography.

• In the presence of a limited supply of echocardiography, the authors suggest the following:

  1. – patients with symptoms suggestive of heart failure such as breathlessness should be referred straight for echocardiography only if they have a history of myocardial infarction or if they have basal crepitations on examination or if they are male with ankle oedema

  2. – otherwise, they should have a BNP test performed and the decision to refer for echocardiography should depend upon the BNP result interpreted in the light of their gender and the presence/absence of ankle oedema.

• There is no need to perform an ECG as part of the assessment of whether or not heart failure is present (although it is recognised that there may be other indications for performing an ECG).

Recommendations for research

• Evaluation of the diagnostic value of repeated BNP measurements for the diagnosis of heart failure.

• Evaluation of the diagnostic accuracy of automated ECG readings in the diagnosis of heart failure.

• Evaluation of the usability of the clinical rule described above in clinical practice.

• Development of methods to conduct IPD meta-analysis for diagnostic tests.

Contribution of authors

Jonathan Mant (Professor of Primary Care Stroke Research) was the lead investigator for the study, supported the conduct of the IPD analysis and the economic analysis, and was responsible for editing the final draft. Jenny Doust (Associate Professor of General Practice) led the systematic review. Andrea Roalfe (Senior Lecturer in Statistics) conducted the IPD analysis. Pelham Barton (Senior Lecturer in Mathematical Modelling) conducted the economic modelling. Martin Cowie (Professor of Cardiology) provided expert cardiological input to the systematic review, the IPD analysis and the economic analysis. Paul Glasziou (Professor of General Practice) supported the conduct of the systematic review and provided expert methodological and primary care input to the IPD analysis. David Mant (Professor of General Practice) provided expert primary care cardiological input to the systematic review and the IPD analysis. Richard McManus (Clinical Senior Lecturer in General Practice) provided expert primary care input to the systematic review, the IPD analysis and the economic analysis. Roger Holder (Head of Statistics) provided statistical expertise for the IPD analysis. Jon Deeks (Professor of Statistics) provided statistical expertise for the IPD analysis. Kate Fletcher (Programme Manager) project managed the research and prepared the final document for publication. Michelle Qume (Data Manager) cleaned the data for the IPD analysis and carried out the validation checks. Sundip Sohanpal (Research Associate) carried out the update of the NICE systematic reviews to inform the economic model. Sharon Sanders (Research Associate) carried out the data extraction for the systematic review. Richard Hobbs (Professor of General Practice) provided expert primary care cardiological input to the systematic review and the IPD analysis.

Disclaimers

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.