A Mortality Risk Model to Adjust for Case Mix in UK Paediatric Cardiac Surgery

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its proceeding programmes as project number 09/2001/13. The contractual start date was in July 2010. The final report began editorial review in October 2011 and was accepted for publication in June 2012. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors' report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

The authors declare (1) no financial support for the submitted work from anyone other than their employer and NIHR as listed above; (2) no financial relationships with commercial entities that might have an interest in the submitted work; (3) no spouses, partners or children with relationships with commercial entities that might have an interest in the submitted work; and (4) no non-financial interests that may be relevant to the submitted work.

Copyright statement

© Queen's Printer and Controller of HMSO 2013. This work was produced by Pagel et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Congenital heart disease

Congenital heart disease (CHD) is a relatively common disorder in childhood, affecting approximately 8–9 per 1000 live-born infants annually in the UK. 1 The crude prevalence of CHD was estimated in 2008 to be 3.05 per 1000 patients registered with UK general practitioners. 2 CHD often involves serious abnormalities and is an important cause of childhood mortality, morbidity and disability. 3,4 Around one-third of deaths due to CHD occur before 14 years of age. Infancy is the highest risk period, with 21% of CHD deaths occurring in the first year of life,5,6 but the proportion of deaths in adults > 45 years is increasing. 7 In 2006, CHD accounted for 3% of child deaths. 5

Currently, around 9000 procedures (including both cardiac surgery and interventional cardiac catheterisation) are performed for CHD in the UK each year, the majority of them in children. 8 Recent studies document greater numbers of children surviving to lead adult lives with repaired or palliated CHD, who experienced an increasing level of hospital admissions between 1994 and 20077 and used more primary care facilities than matched control subjects. 2 Reoperations for CHD, long-term survival for individual cardiac diagnoses and quality of life are viewed as important issues by parents and professionals. 9

The Safe and Sustainable review

This work has been conducted against the backdrop of a major review of paediatric cardiac surgery services in the UK. At the time of writing there are 14 specialist paediatric cardiac surgery centres in the UK, which perform a variable total number of procedures and different individual procedures. 8 The Safe and Sustainable review of paediatric cardiac surgery10 set out to define criteria that characterise a high-quality paediatric cardiac surgery service, review existing units against those criteria and make recommendations concerning the future delivery of the service in England and Wales relating to the current 11 NHS centres in that region. Following a consultation process, its recommendations include a reduction in the number of units providing this care. For some, the process adopted by the Safe and Sustainable team presents a blueprint for how other specialist services should be assessed and, if necessary, reconfigured. For others, the review has meant further discord, uncertainty and disruption for a specialty that has, arguably, been the subject of more external scrutiny than any other. 11–16

The Safe and Sustainable review documents published in 201010 ranked centres according to health-care delivery, organisation, staff and research metrics including volume of cases and quality of post discharge care. Outcome measures were not included in the review data because the advisory panel concluded that validated methods for adjustment of case complexity were not available, and therefore the raw outcome data available from the Central Cardiac Audit Database (CCAD), albeit complete and validated, were too challenging to interpret for inclusion.

On the basis of the data obtained in the review process, the Joint Commission of Primary Care Trusts proposed four options for service reconfiguration, with closure of four or five surgical centres, leaving six or seven larger ones, on the grounds that this change would make the service safer and more sustainable.

Among its recommendations, the Safe and Sustainable review board concluded that CCAD should ‘review its process for reporting outcomes such that these were timely and meaningful’. 10 Our project is very timely with respect to this recommendation and, based on the interest shown by surgical units in our findings (see Chapter 9), the review would seem to have contributed to a receptive context for this piece of research.

Monitoring risk-adjusted outcomes in cardiac surgery

In the wake of the Bristol Royal Infirmary Inquiry, which followed one centre experiencing an abnormally high death rate in children after cardiac surgery,17 there has been an emerging culture of audit and quality improvement in the NHS and particular interest in monitoring outcomes and centre performance within paediatric cardiac surgery. 12,13

It is generally recognised that it is important and valuable to monitor outcomes in cardiac surgery and that to do so fairly and effectively one needs to risk stratify the case load of each unit. 18 Risk stratification of adult cardiac surgery patients is an essential part of the audit process, which reduces the prospect of unfair assessment of outcomes attributed to a surgeon or team whose mortality rate is relatively high simply because it reflects patients who were inherently higher risk. There is evidence that, since outcome monitoring in adult cardiac surgery became mandatory and routine, outcomes have improved, and there has been no consequent negative effect in terms of centres turning away high-risk cases, as was originally feared. 19 There were consequent benefits to patients and their families in terms of the quality of care and the improved information they received. Analytical methods for monitoring outcomes are well advanced for adult cardiac surgery and the use of graphical techniques such as the variable life-adjusted display (commonly know by the acronym VLAD) to display risk-adjusted outcome charts is now a common part of the quality assurance process. 20–22

Our research objectives

At present, no process for routinely monitoring risk-adjusted outcomes in paediatric cardiac surgery exists. Achieving this for paediatric congenital heart surgery is clearly desirable. The challenge is due to the great diversity of the patient population in terms of the diagnoses, indications for surgery, operations performed, age at operation and other factors23 as well as the logistics of co-ordinating such an endeavour across many disparate cardiac centres in a geographical area.

Importantly, as outlined in Chapter 2, the literature and our own previous work focus on outcomes and risks of paediatric surgery according to the procedure(s) that patients have undergone, augmented by patient-specific information such as age and weight. Up to now, no use has been made of information concerning the nature of the heart defect or the diagnosis. There are some recent examples showing that consideration of the comorbidities that the patient has may augment information about procedural risk, and this is also an important factor to consider in respect of case complexity. 24 Although in many surgical specialties there is a one-to-one mapping between diagnosis and surgical intervention, this is not the case in paediatric congenital heart surgery. Some procedures are performed for a number of diagnoses, which undermines the extent to which the procedures performed by a surgeon or within a unit accurately reflect case mix. Similarly, the same heart defect may be managed using different surgical interventions, with the choice of intervention reflecting other aspects of the patient's condition (age, weight, comorbidities and severity of symptoms), but also with scope for there to be differences in surgical strategy between units based on local experience and, potentially, on a different balance being struck between long-term objectives and short-term risks. For these reasons, we were keen to explore, in our project, the scope for using diagnostic information in risk adjustment.

Our objectives for this project were:

1. to test an existing risk model based on the Risk Adjustment for Congenital Heart Surgery-1 (RACHS-1) score and patient age, derived from outcomes at one centre (Great Ormond Street Hospital, London, UK), in the CCAD data from all centres across the UK

2. to understand the contribution that diagnostic information can make to risk estimation and monitoring of outcomes, establish whether or not information concerning comorbidities can contribute to improved methods of risk estimation and, if indicated and possible, revise the existing risk model such that it is suitable for use at other centres and by CCAD

3. to examine the implications of reporting mortality outcomes by diagnosis as well as by procedure category

4. to disseminate our findings and any risk models and monitoring tools developed to UK centres and to CCAD, so that it can consider how best to share the information with stakeholders, including through its ‘public portal’ web pages.

Subjective risk stratification schemes

With respect to risk-stratifying patients according to the operation performed, a classification scheme called RACHS-125 was developed by a committee of experts and published in 2002, by which 79 different types of paediatric cardiac operation are grouped into six categories ranked in order of increasing risk, as perceived by clinicians. The RACHS-1 scheme appears to be useful as a basis for forecasting risk,23 has been validated in a range of contexts and, as of October 2011, has been cited in 337 scholarly articles. One limitation of this scheme is that not all of the great range of operations described for CHD are incorporated and, therefore, a significant proportion of procedures do not have a risk category attributed. The scheme considers operative information only, with some minimal information on age ranges for a small number of procedures. A further limitation is that this scheme was developed before large amounts of registry data were available and, therefore, was based on clinician expert opinion rather than empirical information.

A further consensus-based scoring system developed by clinicians to describe perceived surgical risk is known as Aristotle. 26 The Aristotle system is based on three components: perceived risk of mortality, morbidity and perception of technical difficulty. Estimates of these three factors were made by a panel of experts for 145 different procedures. There are two versions of the Aristotle score, the more detailed of which requires collection of 248 variables: this high level of data collection is perceived as a barrier to the score being implemented. The Aristotle system has been widely used for grading the complexity of operations for the purposes of audit, but has been less widely taken up than RACHS-1.

Our research team has previously reported the development of a model for the risk of perioperative mortality using data pertaining to 1083 congenital open-heart operations performed between April 2000 and March 2003 at one institution: Great Ormond Street Hospital, London, UK. 23 This model incorporated only information about procedure type, classified according to the RACHS-1 scheme,25 and patient age. We reported the use of this model for monitoring outcomes for cases performed between April 2003 and May 200427 and used these analyses as the basis for a comparison between the RACHS-1 and Aristotle complexity schemes. 28

Empirical risk stratification schemes

Since the start of the millennium, there has been a worldwide focus and effort in the field of paediatric cardiac surgery (as for other specialties) to collect audit data for the purposes of quality assurance and benchmarking. 29–31 This era has seen the evolution of CCAD in the UK, the Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database in North America and the European Association for Cardio-Thoracic Surgery (EACTS) Congenital Heart Surgery Database, which accepts data submissions from various countries in Europe and even a few from North Africa. These are multi-institutional databases serving geographical regions and, as such, must conform to the laws and culture of the regions they serve. As mentioned earlier, of all of these databases CCAD is unique because data submission is mandatory and universal within a clearly defined region, data are validated at each centre annually and outcome is independently verified.

One key factor that has allowed for the evolution of these databases is the underpinning work performed on congenital cardiac diagnostic and procedural coding and the development of universally applicable codes to describe congenital cardiac case mix,32–35 which has been relatively recent and indeed is an ongoing endeavour. Accrual of standardised data in these large multi-institutional databases has led to a shift from the use of consensus-based risk stratification tools to risk stratification based on the available empirical data.

In 2009 an empirical risk-adjustment tool was published using information from the large STS (43,934 episodes) and EACTS (33,360 episodes) databases. 24 Procedures were assigned a numerical score, calculated using a Bayesian model, and performance of this model was validated in a test set. One of the aims of this study was to group procedures with similar estimated mortality risks into relatively homogeneous groups, which could then in turn be used to adjust for case mix when analysing outcomes and benchmarking. The risk model performed well and compared favourably with previous schemes used for risk stratification based on consensus such as RACHS-1 and Aristotle.

The outcome measure or end point for the model was hospital mortality, because consistent verification of outcome outside the hospital environment was not available to these databases. One problem with this approach is that patients may die shortly after discharge home and such deaths may be related to the operation. 16 A further problem is that patients have extremely variable lengths of hospital stay, with outlying patients staying weeks or months having a greater mortality risk than those with shorter lengths of stay. 36 Use of hospital discharge (or eventual death) as the end point for such analyses makes it very challenging to include such long-stay patients, who are higher risk, in a VLAD-type contemporaneous analysis of risk-adjusted outcomes. A final problem with this approach is that patients may be discharged from one institution to another, sometimes still on life support, some of whom may die at the second institution; these outcomes would be excluded from the analysis.

A further consideration, which was alluded to by the authors, is that participation in the databases is voluntary, with not all institutions in the same regions taking part. The participating centres are subject to a variable level of validation, with some centres undergoing thorough validation and others undergoing none. This is in part due to the legal obstacles associated with patient confidentiality, which are a barrier to independent verification of data from an outside body. This issue has been averted in the CCAD data that have been used for our study because parent consent for participation in national audit is sought by the centres, which allows universal submission of data and validation to occur.

The coverage of procedure types included a higher proportion in the 2009 study than in the previous consensus-based RACHS-1 score, which did not incorporate all procedure types. However, this newer empirical risk-adjustment scheme did not consider diagnostic information and noted that the approach to augment procedure-specific information with information about other patient-specific factors, such as comorbidities, was as yet undetermined.

Criteria for the assessment of risk models

In the literature described in the previous sections, the main considerations in judging or comparing the attributes of different risk stratification schemes and risk models are the coverage, that is, the proportion of cases that can be incorporated into the scheme or model, and the discrimination, that is, the extent to which the scheme or model is successful in distinguishing groups of patients with higher and lower postoperative mortality. The standard measure of discrimination adopted in this literature and in this project is the area under the receiver operating characteristic curve (AUC). This measure gives the probability that a patient who died, chosen at random, would have had a higher risk score than a randomly selected survivor.

For models that estimate a percentage risk of postoperative death, another useful measure of model performance is the Hosmer–Lemeshow chi-squared statistic. This gives an indication of how statistically significant any deviations are between predicted and observed mortality rates in groups of patient defined by deciles of predicted risk.

In addition to these measures, we have used the graphical tool MADCAP (mean adjusted deaths compared against predicted chart) to assess the discrimination and accuracy of the candidate models developed in this project. MADCAP enables visual identification of patterns of systematic over- or underestimation of risk on the part of a risk model. 37

Origins and process

Since 2000, quality assurance in paediatric cardiac surgery in the UK has been underpinned by CCAD;8 this provides aggregated national and institution-specific data. Mandatory data submissions to CCAD are requested every 3 months from hospitals performing cardiac surgery in the UK, including details about the patients and the operations performed. Patient identifiers including NHS number are provided to CCAD, which periodically requests information on patients' survival status from the Central Register of NHS patients. This process is approved by the National Information Governance Board for Health and Social Care, with consent requested from parents for participation in the national audit of outcomes. There is an extensive quality assurance process incorporating a rolling programme of professionally led site visits.

As it has evolved to date, CCAD receives data on about 9000 procedures annually. The 30-day surgical survival rates for 38 clinically recognisable ‘specific procedures’ are available online via a public portal (see Specific procedure algorithm for more details). The data are currently displayed as funnel plots of survival rate against case volume8 for each of the individual specific procedures for a 3-year period, which permit institutions with different case loads to benchmark their outcomes in relation to others and to the national average.

There is currently no case mix or other risk adjustment used in the presentation of the aggregated data for each institution, but the data set has several features that make it amenable for the development of a risk model. With respect to the data sets used to develop other risk models for this surgery (see Chapter 2), the advantages of the CCAD data set are that data submission is mandatory for all UK centres performing this work, the data are subject to a quality assurance process, the survival status of patients is consistently defined and independently established and there is detailed diagnostic and comorbidity information available to potentially augment procedural factors.

Data fields supplied to the research team

The data fields supplied to us for each record are shown in Figure 1.

FIGURE 1.

The data fields supplied by CCAD. Peri- and postoperative variables were not considered in the model (these are struck through).

Appropriately, for reasons of confidentiality, we were not supplied with raw data concerning patient dates of birth and exact dates of procedure, but rather the calculated age at operation (in days) and the month and year of procedure. This information was of equivalent value in the development of the risk model, but the agreed absence of date of birth and date of procedure data did present some challenges when attempting to identify duplicate records and confirm the matching of records pertaining to the same patient (see Chapter 4, Separation of development and test sets and Removal of duplicate records).

Specific procedure algorithm

The concept of benchmark operations has previously been used as an approach to deal with adjustment for case complexity in paediatric cardiac surgery, in the absence of other methods being available. This approach is still valued as a complement to other methods that have been developed. Institutional results for certain more prevalent and recognisable benchmark procedures were published shortly after the Bristol Inquiry in the form of reports (e.g. arterial switch operation, complete repair of tetralogy of Fallot). One of the reasons is the stated great diversity and complexity of CHD, which may lead to variations of the same procedure being perceived as differing in complexity and therefore risk. It was viewed as crucial to build consensus within the professional community around the classification of individual operations. A significant amount of effort went into defining each one so that comparisons of performance would both be fair and be seen to be fair.

As CCAD was being set up by the professional organisations involved in the care of children with cardiac disease in the late 1990s, the approach used for comparison of institutional performance was through a core group of benchmark operations. Over the first few years of CCAD data collection, the steering committee of CCAD, which consists of a mixture of experienced paediatric cardiac surgeons and paediatric cardiologists, worked on extending the reach of the audit in terms of presentation of results from an increasingly large pool of recognisable operations. This process necessitated the development of the specific procedure algorithm to ensure consistency and transparency with respect to procedure definitions.

The specific procedure algorithm is a means to link the individual International Paediatric Congenital and Cardiac (IPCC) codes submitted to CCAD as part of a patient's procedure record in order to define a recognisable operation. Several individual IPCC codes (up to eight) may be submitted to describe each operation, which when considered in combination give the most accurate account of what the operation involved.

The algorithm defines the series of IPCC codes that may be included to identify an individual operation, and in a proportion of operations in which the definition may be more complicated also defines the list of IPCC codes that must be excluded. The algorithm contains a hierarchy that ranks the recognisable procedures in order, with the most complex at the top (grouped first) and the least complex at the bottom (grouped last).

The algorithm has been refined and improved by the CCAD steering committee year on year, such that the definitions of each operation are tight and consistent.

The CCAD public portal was launched in 2005 with publication of paediatric cardiac surgery results by recognisable procedures in the form of funnel plots. Following public display of these results, the level of engagement from the UK institutions caring for children with heart disease increased, as professionals from the centres took a great interest in the results as they appeared on the public site. This engagement and comment has led to further refinements and improvements in the specific procedure definitions, which are themselves provided on the website as part of the supporting materials. The algorithm has been periodically improved and updated based on feedback and comment from professionals in the community.

Compatibility of the Central Cardiac Audit Database data set and the Risk Adjustment for Congenital Heart Surgery-1 scoring system

One of our stated aims was to explore the utility of a previously developed risk score based on RACHS-1 and age. 23 Because we did not receive the definitive CCAD data set for this project until December 2010, we undertook this task using an older version of the CCAD data set supplied previously for pilot work.

We wrote an algorithm that could map the procedure codes as they appear in the database to RACHS-1 categories. The CCAD surgical codes are European short codes (referred to as IPCC short codes), of which there are over 1000. These IPCC short codes appear in different combinations in each database record, which, when considered together, provide information about the operation that the patient underwent.

The RACHS-1 groupings of procedures represent lists of clinical descriptions of actual operations, each of which may be described by several different individual IPCC codes or combinations of IPCC codes. Some of these IPCC short codes can correspond to more than one RACHS-1 category and, hence, could not be assigned unambiguously to a RACHS-1 category. It is also a known feature of RACHS-1 that not all surgical procedures are encompassed by the scheme; some unusual or rarer complex operations do not appear at all.

In 64% of CCAD records, all recorded surgical procedures could be classified and given an unambiguous score under the RACHS-1 scheme. Another 27% of records had at least one recorded surgical procedure that was not classifiable with RACHS-1, 7% of records had at least one recorded surgical procedure that could not be assigned to an unambiguous RACHS-1 category and 2% of records had at least one unclassifiable procedure as well as one procedure that could not be assigned to an unambiguous RACHS-1 category.

Given that over one-third of CCAD records could not be assigned to an unambiguous RACHS-1 category, we decided not to pursue using RACHS-1 in an updated risk model. Instead, we chose to adopt the CCAD classification of ‘specific procedure’ discussed in the previous section.

Separation of development and test sets

We received the CCAD data set for use in this project in December 2010. After removing all records in which the patient was > 16 years at the time of the procedure and all records for patients who underwent only catheter procedures, we split the data set into development (70% of patients) and test (30% of patients) data sets using random allocation stratified by year and institution of first procedure for a patient. From previous work11 we were aware of the possibility of improving outcomes over time, which motivated stratifying the randomisation by year of procedure. We wanted to have as much data as possible for developing a model while maintaining a sufficient sample for meaningful evaluation of model performance. Given the size of the CCAD data set, we felt that a 70/30 split was suitable. All further analysis was performed on the development set, which contained 34,385 records, corresponding to 22,449 unique patients. The quarantined test set contained 14,316 records, corresponding to 9354 unique patients, and played no part in risk model development. It is necessary to observe such a quarantine process to avoid the bias that inevitably results if the same data that are used to develop a risk model are used to test it, which almost always leads to overoptimistic test findings.

Given that we had previously worked with some of these data (see Chapter 3, Compatibility of the Central Cardiac Audit Database data set and the Risk Adjustment for Congenital Heart Surgery-1 scoring system), we wanted to ensure that those records previously analysed formed part of the development set. This required us to match cases across two versions of the data based on the patients' pseudonymised NHS numbers, the pseudonymised institution IDs and the patients' pseudonymised hospital numbers. We encountered some problems in doing this, which were resolved through analysis and discussions between the analytical team and David Cunningham at CCAD.

We note that the necessary pseudonymisation process had the negative effect of essentially blinding the analytical team to missing or anomalous data in the patient and hospital ID fields. This meant that we had to conduct extensive face validity exercises when trying to identify genuine groups of records pertaining to the same patient. This included analysis to check internal consistency of patient histories in terms of the dates of operations and ages at each operation and also some manual checks of the clinical face validity of sequences of operations. The close collaboration between clinical and analytical teams was essential here, as was having the involvement of CCAD, which could, when necessary, go back to its own records for clarification.

Removal of duplicate records

Some records appeared to be duplicates based on systematic comparison of pseudonymised patient NHS number or hospital number, year and month of operation, age, specific procedure, procedure type, procedure 1 (the first procedure code specified), bypass time and catheter procedure time variables. There is a plausible mechanism for record duplication: if an institution tries to update an individual record within CCAD by changing either the patient's hospital ID number or the procedure date (both of which are key fields), a second, duplicate, record will be created instead of the original record being overwritten. Two members of the clinical team (KB and NM) went through the list of possible duplicates identified by the comparison algorithm developed by the analysts and marked those records that were considered to be likely duplicates. We then used a formal protocol (see Appendix 1) to determine which record of each pair to retain, based on completeness and plausibility of the data contained in the records. A total (across the test and development sets) of 244 duplicate records were removed from the analysis.

Data cleaning

As discussed in Chapter 3, the CCAD data set is widely recognised as being of an extremely high quality (arguably the best of its type in the world), but given its sheer size and the complexity of paediatric cardiac diagnoses and surgery it is perhaps inevitable that there would be some errors and anomalies in the data. This is particularly the case for information that has previously not been widely used, such as diagnostic and comorbidity information. The analytical team spent considerable amounts of time understanding the data set and, where necessary, amending it. At key steps in this process David Cunningham, at CCAD, was consulted to confirm the analyst's interpretation of the data. The cleaning processes observed for key fields are set out below. Note that cleaning in the test set was performed after model development was completed.

Defining comorbidity

International Paediatric Congenital and Cardiac codes defining comorbid conditions were grouped into four categories. Comorbid conditions appearing in any of the comorbidity or diagnosis fields were classed as such:

• Prematurity. This is defined by CCAD as being gestational age < 37 weeks at birth.

• Down syndrome or trisomy 21.

• Congenital comorbidity of all types other than Down syndrome. This essentially includes all genetic syndromes, clinical constellations of features that constitute a recognised syndrome and congenital structural defects of organs other than the heart. 38

• Acquired comorbidities. This is a group of codes increasingly used over time in CCAD to define acquired non-cardiac abnormalities of other organ systems that were present preoperatively. These include conditions generally arising secondary to heart disease affecting other organ systems. 39 Examples include renal failure and necrotising enterocolitis.

Defining episodes of surgical management for analysis

There was extensive discussion among the project team about how to assign outcomes to patients undergoing multiple operations. The problem faced is that, for a patient who has a reoperation within 30 days of their first operation, there is genuine ambiguity as to which operation to attribute a death to that occurs within 30 days of both operations. In the analysis of these cases there is scope for double-counting deaths (and for that matter survivals) or under-reporting surgical activity if reoperation or reoperated patients are ignored. Another alternative would be to use as an outcome measure for the first operation the vital status of the patient 30 days after the last operation in a sequence of reoperations within 30 days. Given the focus of this project on developing a risk model to facilitate timely monitoring of short-term outcomes, the final decision was to have a ’30-day episode’ as the unit of analysis. A 30-day episode starts with the first surgical procedure on a patient. This episode is then assigned an outcome of alive or dead according to the vital status of the patient 30 days after this first surgical procedure. Any further surgical procedures that the same patient underwent within 30 days of this first procedure were not included in model development. The next procedure recorded for the same patient > 30 days after the first surgical procedure was treated as the start of a new 30-day episode.

Examples of how the 30-day episode was allocated are shown in Figure 2.

FIGURE 2.

Examples of how the 30-day episode was allocated.

The 30-day episode unit of analysis was allocated using the same protocol in both the development set and the test set.

One drawback of this approach is that not all surgical activity contributes to the risk model development. This does not affect the utility of the risk model as long as, when used for monitoring, it is applied to 30-day episodes as defined above. That said, the level of reintervention within 30 days is rightly seen as important outcome information. For this reason we retained the information on the number of further procedures (both surgical and catheter) recorded for each patient within each 30-day episode and devised ways of presenting this additional information alongside risk-adjusted 30-day mortality.

Field completeness and descriptive analyses

As stated above, the completeness of procedural data was very high. The completeness of diagnosis data fields improved over the period covered by the data set, as can be seen in Figure 3 (development set).

FIGURE 3.

The completeness of diagnosis data fields over time (episode level within the development set).

There is also an apparent improvement in recording of comorbidity data (Figure 4), although, whereas diagnostic information should be available for all episodes, an empty comorbidity field can correctly signify ‘no comorbidity’ rather than missing data.

FIGURE 4.

The completeness of comorbidity data fields over time (episode level within the development set).

The completeness of the diagnosis and comorbidity data fields varied across institutions, as depicted in Figures 5 and 6 respectively. Table 1 shows how case load varies across these institutions in terms of the number of episodes in the development set.

FIGURE 5.

The completeness of diagnosis data fields by institution (episode level within the development set). Institutions J, L and N have case loads ≤ 3 within the entire development set and so are not included.

FIGURE 6.

The completeness of comorbidity data fields by institution (episode level within the development set). Institutions J, L and N have case loads ≤ 3 within the entire development set and so are not included.

Data completeness for the other fields, as recorded in the entire development set, is given in Table 2.

There was a greater proportion of male patient episodes than female in the development set (Table 3). The age and weight distributions at episode level are shown in Figures 7 and 8, respectively, and a scatterplot of weight against age is given in Figure 9.

FIGURE 7.

The distribution of patient age at episode level in the development set (n = 27,212).

FIGURE 8.

The distribution of patient weight at episode level in the development set (n = 27,212).

FIGURE 9.

Weight vs age scatterplot for episodes in the development set (n = 27,212). Black markers are for 30-day status either alive or unknown; grey markers are for 30-day status dead.

Figure 10 shows the number of episodes with missing 30-day status and/or patient age that were removed from the development and test sets, and the number of episodes with missing or anomalous patient weight that were excluded from the test set and replaced in the development set by the mean weight for age. We note that 90% of records with missing status occurred before 2002. We examined the data in the development set to look for differences between records with valid 30-day status and records missing 30-day status and did not find any significant differences.

FIGURE 10.

Number of episodes with missing 30-day status and/or patient age and number of episodes with missing or anomalous patient weight. Episodes with missing 30-day status and/or patient age were removed from the development set, whereas episodes with missing or anomalous patient weight were replaced by the mean weight for age (see text). a, Includes episodes with missing diagnostic information (n = 948).

Preamble on diagnosis classification

Typically, paediatric cardiac disease has multiple components so that a single patient's CCAD record often has several diagnosis codes. IPCC codes32,40 are used to list each item contributing to a patient's heart defect, as well as any associated conditions. Allocation of a ‘primary’ or ‘underpinning’ diagnosis is challenging and no consensus exists regarding an optimal method. Previous authors have identified the primary diagnosis as the lesion with the greatest anatomic severity,41 but for some situations such an allocation may be inappropriate. On the other hand, choosing the lesion requiring the earliest intervention42 also challenges clinical intuition in some scenarios. An alternative approach is to use the underlying physiology of the disease to group diagnoses,1,43 as in the Extracorporeal Life Support Organisation Registry database;44 the limitation here is that physiological groupings may encompass a wide range of conditions with correspondingly different prognoses. In principle, diagnostic information could also be categorised in relation to risk, much as short-term surgical data is in systems such as RACHS-1,25 but actuarial risk rather than short-term risk data would be required to make this pertinent.

Recent epidemiological studies of CHD in the northern region of the UK by Wren et al. 45–47 specified possible ‘primary diagnoses’ and placed them in an explicitly hierarchical list, broadly by clinical severity. An individual patient was then allocated to the highest grouping consistent with their anatomical features, informed if necessary by the lesion precipitating their earliest intervention. Limitations of this categorisation include its failure to consistently distinguish single or biventricular status, and its exclusion of certain diagnoses that still did not fit into the scheme. Our study used the approach of hierarchical categorisation as the starting point, modifying it for the different context of CCAD. Motivated by the possibility that a large number of combinations of diagnostic and procedure categories, some with very small patient numbers, could undermine robust statistical modelling, we also elected to develop a second scheme consisting of larger and less refined groupings based on a categorisation described by Clancy et al.,48 according to the number of functioning ventricles and the presence of aortic obstruction. We anticipated that both schemes would allow patients to be prospectively categorised by clinicians submitting CCAD data, and that there was the potential for either or both schemes to add information to our proposed risk-adjustment model. The diagnosis schemes described in the following sections were developed in close collaboration with Dr Rodney Franklyn and Dr Catherine Bull, both experts in paediatric cardiology.

Diagnosis classification scheme 1 (mapping International Paediatric Congenital and Cardiac codes to primary diagnosis)

In this section we first describe the methods used to map each of the individual IPCC codes into one of 27 diagnostic groupings. We then describe how the several diagnostic groupings that may appear in any particular record were combined to determine a single ‘primary cardiac diagnosis’ for that record. We used the entire development set to devise the scheme.

Diagnosis classification scheme 2 (mapping International Paediatric Congenital and Cardiac codes to diagnostic groups defined by ventricle number and aortic obstruction)

We describe first the methods used to categorise individual IPCC codes before describing how the codes relevant to any particular record were combined. We used the entire development set to devise the scheme.

At the level of a record

If any of the IPCC codes within a record were identified as a congenital or an acquired diagnosis, or indicated a UVH or the presence of an arch/systemic arterial obstruction, this attribute was carried forward to the entire record to be used as a candidate variable in model development. However, a record was identified with a BVH only if there was at least one IPCC code within the record that indicated a BVH and none that indicated a UVH.

Any given record within the CCAD data set will have one of a large number of possible combinations of these five potential diagnostic attributes, some of which are common and some of which are much rarer. A given combination of diagnostic attributes would not necessarily represent a clinically intuitive patient group and, so, for the purposes of defining subgroups of patient in which to assess the performance of the risk model, we used the methodology set out in Figure 11 to assign records to one of seven clinically intuitive diagnostic groups. Records with an acquired code but no congenital codes (e.g. ‘cardiomyopathy’) fell into the ‘acquired only’ group.

Motivated by the intended prospective use of this mapping scheme, in which the attributes would almost always be known preoperatively, we retrospectively assigned attributes to a minority of records in which the recorded procedure unambiguously defined them. The CCAD specific procedure field was used to assign diagnostic information to a record in which the procedure was deemed to be an unambiguous indication of UVH [Norwood procedure (stage 1); Fontan procedure; and bidirectional cavopulmonary shunt] or the presence of an arch or systemic arterial obstruction [Norwood procedure (stage 1); isolated coarctation repair; interrupted aortic arch repair; truncus and interruption repair; aortic valvotomy; subvalvar aortic stenosis repair; and supravalvar aortic stenosis repair].

FIGURE 11.

Flow diagram depicting the method for allocating a record to a diagnosis group on the basis of diagnosis attributes assigned to diagnosis codes within that record.

The resultant diagnostic classification schemes

Summary of work on diagnosis classification

This process has been successful in identifying diagnostic categories likely to add value to procedural information in developing a risk model for 30-day postoperative mortality. This grouping of patient records in CCAD based on diagnostic information enables us to comment for the first time on the frequency or proportions of patients with particular diagnostic features. For example, we can now say that the proportion of patients in CCAD (which captures all UK children who have undergone an intervention) since 2004 who have a UVH is 9.3%.

We selected terms to label the primary diagnosis groupings, such as ‘ventricular septal defect’ and ‘hypoplastic left heart syndrome’, that will be recognisable to both laypersons who may have an interest in this topic and professionals. This choice was in keeping with the philosophy of CCAD, which shares information about outcomes with the public; we considered that more technical medically focused schemes for describing and allocating primary diagnoses would be less suitable for this purpose.

In addition to contributing to the development of the risk model, this work on identifying a primary cardiac diagnosis will also inform subsequent analyses of long-term outcome by diagnosis in CCAD and may provide valuable information in terms of national audit of outcomes.

Initial exploratory analyses

First, descriptive analyses were performed to characterise the make-up of the development set population. The development set was then used as the basis for a number of exploratory analyses to learn more about preoperative factors reported to CCAD that are associated with outcome. We also investigated whether or not there were attributes of the data that would affect the development and use of risk scores. Mortality rates based on 30-day status [with 95% confidence intervals (CIs)] were calculated for the following preoperative risk factors, selected as candidate model predictors on the basis of clinical relevance and availability within the data set:

• patient information: sex, age, weight, weight-for-age z-score, ethnicity, sternotomy sequence, antenatal diagnosis

• procedural information: procedure type, specific procedure, episode number for patient

• comorbidity information: premature (from diagnosis and comorbidity fields), Down syndrome (from diagnosis and comorbidity fields), congenital non-Down syndrome comorbidity (from diagnosis and comorbidity fields), acquired comorbidity (from diagnosis and comorbidity fields)

• diagnostic information (see Chapter 5 for details): either scheme 1 diagnostic category or congenital attribute, UVH attribute, BVH attribute, congenital not assigned attribute, arch obstruction attribute, acquired attribute, comorbid diagnosis attribute, not cardiac diagnosis attribute.

For brevity and clarity, mortality rates for only the most relevant groupings of candidate factors are shown in Table 7 and Figures 12–18. In the development set overall (n = 26,519), the episode-level 30-day mortality rate was 3.2% (95% CI 3.0% to 3.4%).

FIGURE 12.

Mortality rates (95% CIs) based on 30-day status for age bands within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

FIGURE 13.

Mortality rates (95% CIs) based on 30-day status for weight bands within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

FIGURE 14.

Mortality rates (95% CIs) based on 30-day status for specific procedures within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

FIGURE 15.

Mortality rates (95% CIs) based on 30-day status for scheme 1 diagnosis groups within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%). ASD, atrial septal defect; AV, atrioventricular; DORV, double-outlet right ventricle; IVS, intact ventricular septum; PA, pulmonary atresia; TAPVC, totally anomalous pulmonary venous connection; VA, ventriculoarterial.

FIGURE 16.

Mortality rates (95% CIs) based on 30-day status for ethnicity within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

FIGURE 17.

Mortality rates (95% CIs) based on 30-day status for sternotomy sequence within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

FIGURE 18.

Mortality rates (95% CIs) based on 30-day status for antenatal diagnosis within the development set (n = 26,519). Dotted line indicates overall average mortality rate (3.2%).

In discussion with all project team members, potential preprocedural risk factors removed at this stage were:

• Antenatal diagnosis: in addition to concern about the level of missing data, it was felt that, although potentially acting as a surrogate marker for severity of congenital defect, the relationship between antenatal diagnosis and outcome was not straightforward and might not be stable with changing patterns of provision of antenatal diagnostic services.

• Preprocedure seizures: data quality was considered to be poor for this field.

• Preprocedure Paediatric Cerebral Performance Category (PCPC): data quality was considered to be poor for this field.

• Sternotomy sequence number: although this field would seem to offer information about the position in a sequence of operations for the patient, it was felt by the clinicians that data quality for this field was not sufficient to use it within the risk model. Using the number of times that the patient has previously appeared in the data set was explored, but this measure has no meaning for patients with sequences that started before the data set or whose care was not delivered entirely within the UK.

• Ethnicity: it was considered that, with the high level of missing data and a lack of a clear understanding of the mechanisms for genuine clinical differences in mortality risk in different ethnic groups, it would not be appropriate to include ethnicity in the model development process.

• Deprivation: the team considered that to adjust for deprivation could potentially contribute to complacency over possible social gradients in outcomes and that it would be better to exclude measures of deprivation from the risk model. Analysis of risk-adjusted outcomes by quintile of deprivation will be the focus of future work.

We note that episodes with a 30-day status of ‘unknown’ (2.5% of episodes in the development set) were not included in the denominator of univariate mortality rates, nor included within model development.

Model building process

The process adopted for constructing the risk model consisted of the iterative use of the following steps.

Diagnostic information

We used classification and regression tree analysis to investigate groupings of diagnostic scheme 1 categories with similar mortality rates, resulting in the low-, medium- and high-risk groupings depicted in Figure 19 (which was developed on 70% of the development set and validated on the remaining 30%).

FIGURE 19.

Grouping of diagnostic categories into low, medium and high risk. ASD, atrial septal defect; DORV, double-outlet right ventricle; IVS, intact ventricular septum; PA, pulmonary atresia; TAPVC, totally anomalous pulmonary venous connection.

For diagnosis scheme 2, only the UVH attribute consistently remained significant in the multivariate logistic regression analyses.

Collapsing comorbidity information

The sparseness of comorbidity information within the CCAD data set results in low volumes of episodes associated with the four comorbidity categories, even if the comorbidity information derived from both the comorbidity and the diagnosis data fields is pooled. The stability of the model improved notably when, rather than including indicator predictor variables for all four comorbidity groups, we instead grouped the non-Down syndrome comorbidities to produce a single binary indicator for the presence (or absence) of non-Down syndrome comorbidity information for an episode. We note that univariate analysis indicated that the presence of an acquired, premature or congenital non-Down syndrome comorbidity had an adverse effect on 30-day outcome, whereas Down syndrome seemed to have a protective effect (although this was not statistically significant).

An additional consideration in favour of including this single comorbidity indicator in the model is the potential for its presence to drive up data quality prospectively (the comorbidity field has poor completeness, particularly pre 2005).

Age and weight

The association between age and mortality is non-linear and defied transformation. We therefore pragmatically chose to include both continuous age and age band [neonate, infant, child (> 1 year)] in the model. The last age, i.e. child (> 1 year), has strong clinical face validity. We explored the possibility of developing a separate model for each age band, but decided that there were insufficient events among children to support the degrees of freedom of specific procedures (even with the low-volume specific procedure group).

We also include continuous age within the model and hence have a stepped age/risk relationship with the same gradient in each of the age bands. We investigated a piecewise linear age/risk relationship with different slopes in each band but this led to more instability in the parameter values across 50% runs within the development set and a higher Hosmer–Lemeshow chi-squared statistic and so was not adopted.

Continuous weight and weight-for-age z-score were also considered as potential risk factors in the model, with the weight-for-age z-score using the development set as a reference population (split into 23 age bands, narrower at young age). In multivariate analyses, weight and weight-for-age z-score were not typically both significant, depending on which other variables were in the model. Of the two variables, continuous weight was thought to be the easier to implement in terms of prospective use, as z-score would require a reference population.

Low-volume specific procedures

As far as possible we wanted to maintain the specific procedures; however, to develop a robust model we needed to reduce the number of predictor variables associated with procedural information. We therefore decided to group the nine specific procedures with the lowest volumes (all of which have < 70 episodes over the entire development set) into a ‘low-volume’ specific procedure group. This low-volume group consists of the following specific procedures: aortic root replacement (not Ross); aortopulmonary window repair; AVSD and tetralogy repair; cor triatriatum repair; multiple VSD closure; Senning or Mustard procedure; tetralogy with absent pulmonary valve repair; tricuspid valve replacement; and truncus and interruption repair.

Year of surgery

We looked at the performance of our initial risk model in each year from 2000 to 2010 and found a clear, but not unexpected,11 trend of underestimating overall risk in each year to 2005 (by less in later epochs) and increasingly overestimating in the subsequent years. This trend suggests that any model calibrated on data from this entire period would already be out of date, and also highlights the importance for timely recalibration of any risk model prospectively. We considered calibrating the model on a subset of later years in the development set but found that the loss in volume of data led to model instability. To include year of surgery as a variable would be to assume that improvement over time is inevitable and ignores ceiling and threshold effects. We therefore decided to include a variable in the model to indicate whether an episode occurred pre 2007 or from 2007 onwards. Although there is no clinical mechanism for such a risk factor, it enables the entire development set to be used in the model calibration while also ensuring that the calibration is more up to date and fit for purpose prospectively. The inclusion of this indicator variable also highlights the need for a rolling programme of recalibration for a model in routine use.

After finalising the model we reran it in the development set without the variable indicating whether an episode occurred pre 2007 or from 2007 onwards. Figure 20 shows the difference between observed and predicted deaths from this reparameterised model evaluated in the development set, for each year, weighted by the number of episodes in that year. This indicates a trend of improving risk-adjusted outcomes over time.

FIGURE 20.

Difference between observed and predicted deaths for each year, weighted by the number of episodes in each year, evaluated in the development set using a reparameterised model that excludes the 2007 indicator variable.

Final parameter selection and model performance in the development set

Based on the analysis and considerations set out above, the final model, decided on jointly by the clinical and analytical teams, was a logistic regression model with the following variables:

• age (both as a continuous measure and as neonate/infant/child bands)

• weight (as a continuous measure)

• specific procedure (one of 27 CCAD groups, ‘no specific procedure’ or ‘low-volume specific procedure’)

• procedure type (bypass or non-bypass)

• broad diagnosis group (low-, medium- or high-risk group)

• UVH attribute (indicator variable)

• presence/absence of a recorded non-Down syndrome comorbidity

• indicator variable for whether an episode occurred pre 2007 or from 2007 onwards.

The ‘enter’ regression model was used to parameterise this model across the entire development data set. The AUC for the model in the development set was 0.78 (95% CI 0.76 to 0.79) and the Hosmer–Lemeshow chi-squared statistic was 9.2 (p = 0.325), indicating no statistically significant differences between observed and expected number of deaths when calculated in deciles of predicted risk (see Harrell50 for more information on these measures). We note that data completeness was reasonable for all variables included in the final model.

Although we gave considerable thought to the interaction of age and weight, we did not perform a detailed study of interactions between specific procedures and diagnostic classifications. Consideration of such interactions might have improved model performance in the development set, but we wanted to avoid introducing too many additional degrees of freedom. The characteristics of the model were considered acceptable to the project team, noting that model choice was not a purely statistical consideration and that consideration of uptake by CCAD and centres supported the use of as many specific procedure groups as possible and the opportunity presented by the risk model to drive improvements in completeness and data quality concerning diagnosis and comorbidity entered into our thinking. Details of the model are provided in Appendix 4.

Global performance in the test set

The performance of the model described in Chapter 7 (see Final parameter selection and model performance in the development set) across the spectrum of predicted risk is shown in Figure 22. The AUC for this model was calculated as 0.76 (95% CI 0.74 to 0.79) within the test set, compared with 0.78 (95% CI 0.76 to 0.79) in the development set. The Hosmer–Lemeshow chi-squared statistic was 25.1 (p = 0.002), compared with 9.2 in the development set. The poor Hosmer–Lemeshow chi-squared statistic in the test set is due to an excess of deaths in the first (6 observed vs 1.6 predicted) and fourth (22 observed vs 13 predicted) deciles of risk. We note that there were three deaths from Rastelli procedures in the test set (out of 62 procedures), all in the first decile of risk, compared with no deaths from 125 Rastelli procedures in the development set. Overall, there were 335 observed deaths in the test set compared with 329 predicted.

FIGURE 22.

Cumulative deaths in the test set plotted against episode number with episodes ordered by increasing risk, as predicted using the model set out in Chapter 7, Final parameter selection and model performance in the development set (n = 10,597). Note that some data were corrected subsequent to this analysis (see text below and Figure 23).

In exploring the performance of this risk model and features of the case mix across different institutions, pre and post 2007, we identified an error in the data provided to us. Essentially, the algorithm employed by CCAD to determine the ‘specific procedure’ classification for each record had not been implemented for a tranche of records deriving from one centre and covering the period 2007 onwards, leaving 1474 episodes (4% of episodes over both sets) with an erroneous classification of ‘not a specific procedure’. This arose as the centre concerned had resubmitted these data en bloc in 2010 and CCAD had not rerun the algorithm.

Given that we had already ‘opened’ the test set of data at this point, we could not revisit any of the analysis that led to the choice of factors to include in the logistic regression model. However, we decided that, if CCAD were able to provide us with corrected data (which they were), we would recalibrate the model on the corrected development data set. In addition to updating the specific procedure information for affected episodes, we reran the process of using specific procedure to identify episodes in which the patient had a functionally UVH but in which this was not recorded in the diagnostic fields [see Chapter 5, Diagnosis classification scheme 2 (mapping International Paediatric Congenital and Cardiac codes to diagnostic groups defined by ventricle number and aortic obstruction). The specific procedure field was also corrected for those episodes in the test data set that were affected.

From this point of the report, the performance of the model in the test set of data is presented following the recalibration of the model on the corrected development data. As the intent of this project is to provide units with a way of prospectively monitoring risk-adjusted outcomes, there is a focus on features of model performance in test data from 2007 (see Chapter 7, Year of surgery).

Figure 23 shows the performance of the recalibrated model across the spectrum of predicted risk when assessed among the corrected test data from all years. Figure 24 shows the receiver operating characteristic curve for this model in the test set across all years, for which the AUC is 0.77 (95% CI 0.74 to 0.79), similar to the value calculated for the development set (0.78, 95% CI 0.76 to 0.79). This confirms the good discrimination between high- and low-risk cases indicated by the appearance of the MADCAP curve, with a shallow climb of the stepped line indicating cumulative observed deaths at low predicted risk and a steeper climb at high predicted risk. The Hosmer–Lemeshow chi-squared statistic is 22.7, indicating that the discrepancies between observed and predicted mortality in deciles of predicted risk are statistically significant (p = 0.004). These discrepancies are evident in the portions of the MADCAP chart where the stepped line indicating cumulative observed deaths climbs at a higher or lower rate than the smooth line indicating cumulative predicted deaths. The extent to which these features of model performance affect the utility of the model in prospective use is discussed in Institutional case mix and model performance and Chapter 9, Model limitations and cautions. The overall number of observed deaths was 335 compared with 329.3 predicted.

FIGURE 23.

Cumulative deaths among the corrected test data from all years plotted against episode number with episodes ordered by increasing risk as predicted using the model set out in Chapter 7, Final parameter selection and model performance in the development set (n = 10,597).

In terms of the added value of including age, diagnosis, comorbidity and use of bypass, it is interesting to note that the performance of a model based solely on a specific procedure (AUC 0.72, 95% CI 0.69 to 0.75) is worse than the performance of a model based on all factors other than a specific procedure (AUC 0.74, 95% CI 0.72 to 0.77) (see Figure 24).

FIGURE 24.

Receiver operating characteristic curve for three models evaluated in the test set: the final risk model (AUC = 0.77); a model based solely on a specific procedure (AUC = 0.72); and a model based on all factors in the final model except a specific procedure (AUC = 0.74) (n = 10,597).

Features of the test set compared with the development set

Figures 25–28 and Table 8 show a comparison between the development and the test data sets in terms of the episode-level mortality rates associated with the factors included in the risk model. These show broad similarity in the univariate associations between individual factors and 30-day mortality in the two data sets as would be expected given the random selection process used. That said, there are some features worth noting, particularly the disparity in 30-day mortality rates associated with some individual specific procedure classifications and the higher 30-day mortality rate among neonates.

FIGURE 25.

Observed mortality rate by age band in the development set (black, n = 26,447) and the test set (grey, n = 10,597).

FIGURE 26.

Observed mortality rate by weight category in the development set (black, n = 26,447) and the test set (grey, n = 10,597).

FIGURE 27.

Observed mortality for specific procedures in the development set (black, n = 26,447) and the test set (grey, n = 10,597). ASD, atrial septal defect.

FIGURE 28.

Observed mortality by diagnosis risk group in the development set (black, n = 26,447) and the test set (grey, n = 10,597).

Global performance of model from 2007 onwards

The risk model is intended for future use in routine monitoring. Given this and given the observed improvement in outcomes over time and our adjustment for this fact in the model (see Chapter 7, Final parameter selection and model performance in the development set), it is the performance of the model in the test set episodes that occurred during or after 2007 that is most informative concerning its fitness for purpose.

The MADCAP chart for these test episodes is shown in Figure 29.

FIGURE 29.

The performance of the model in the test set for all episodes occurring after 1 January 2007 (n = 3905).

Although the model has a higher value for the AUC among data from 2007 onwards than overall (0.81 vs 0.78), and so shows better discrimination among these episodes, the model underestimates risk at the very high-risk end of the spectrum of predicted risk (very right-hand side of Figure 29) and is, as a result, less accurate overall in these more recent data than in the full development set. The overall number of observed deaths is 128 compared with 113 predicted. We considered the potential impact of Copas shrinkage on the performance of the model in the test set post 2007. We observe an underprediction of mortality at high risk and an overprediction at low risk, which is the opposite of that expected from Copas. Additionally, the inaccuracies observed are present only in the recent (post 2007) cohort rather than across the entire test set. Thus, we believe that the differing performance of the model in the test and development sets is driven by other effects, such as the markedly higher mortality rate among neonates in the randomly selected validation set (see Figure 25).

To explore the performance of the model in recent data in more detail, we plotted MADCAP charts in a number of important subgroups. We show here those we consider most relevant. Figure 30 shows the performance of the model in neonates, infants and children, which shows that the model generally underestimates risk in the neonatal group. It is worth remembering here that the mortality rate in neonates was markedly higher in the test set than in the development set (see Figure 25). This artefact of the random selection of development and test sets underlines the value of having a quarantined test set to mimic as closely as possible prospective use.

FIGURE 30.

The performance of the model in age band subgroups in the test set for data after 2007 (n = 832 for neonates; n = 1480 for infants; n = 1593 for children).

The performance of the model within four mutually exclusive diagnostic groups is shown in Figure 31. There is underestimation of risk in higher-risk UVH patients, likely to correspond to neonates. That aside, the generally good performance across these groups is encouraging.

FIGURE 31.

The performance of the model in diagnostic group subgroups in the test set for data after 2007 (n = 359 for UVH without arch/systemic obstruction; n = 306 for UVH with arch/systemic obstruction; n = 2422 for BVH without arch/systemic obstruction; and n = 640 for BVH with arch/systemic obstruction).

Figure 32 shows the performance by procedure type: bypass or non-bypass. The model performs well in both subgroups, but shows more discrimination in the bypass procedures.

FIGURE 32.

The performance of the model in procedure type subgroups in the test set for data after 2007 (n = 2963 for bypass and n = 942 for non-bypass).

Institutional case mix and model performance

In addition to understanding the performance of the model across the spectrum of predicted risk and within key subgroups of patients, to assess the fitness for purpose of the model it is important to understand the distribution of case mix and, in particular, differences in case mix between institutions. Essentially, it is important to understand whether or not and under what circumstances differences in case mix and the differential performance of the risk model in different subgroups would combine to give an artefactual impression of better or worse risk-adjusted outcomes at one unit compared with another.

Figure 33 shows the cumulative distributions of predicted risk within the development and test sets after 1 January 2007. The distribution of predicted risk in the two data sets is very similar and shows that just over 30% of episodes have a predicted risk of 30-day mortality of ≤ 1% and 80% have a predicted risk of ≤ 4%, but that around 5% of episodes have a predicted risk of > 10%.

FIGURE 33.

Comparison of the case mix as determined by predicted risk between the development (n = 26,447) and the test (n = 10,597) sets.

The differences in case mix between centres, in terms of predicted 30-day mortality risk, using the model across the development and test sets after 1 January 2007, are shown in Figure 34. Note that the institution numbers given in this plot are not in the same order as the institution letters in Chapter 4. Each line shows the case mix for a single institution. The steeper lines indicate institutions with a higher proportion of relatively low-risk episodes, as estimated by the model. Many of the institutions have quite similar profiles of predicted risk, but there are some marked differences. For instance, at institution 5, 12% of episodes have a predicted risk of > 10%, whereas at institution 7 this proportion is < 1%.

FIGURE 34.

Comparison of the case mix as determined by predicted risk between the different institutions. There are four institutions within the data set that had a very low number of procedures (non-specialist centres that happened to perform a few procedures which were eligible for paediatric CCAD submission). Given the low numbers, these institutions are not included.

Given that the model underestimated risk at the very high-risk end of the recent (since 2007) test data, risk adjustment using the model as currently parameterised will potentially give an unfair assessment of outcomes at those centres with a high proportion of high-risk cases. This is an important caveat to the interpretation of risk-adjusted outcomes within and across institutions that will need to be discussed with CCAD and the institutions as this work is taken forward.

Proposed monitoring tool

Figure 35 shows an example of the graphical monitoring tool that we envisage the risk model feeding into. The figure shows a standard VLAD chart of the difference between the expected number of deaths over a series of episodes, determined by summing the estimated risk for each episode, and the observed number of deaths. The trace rises with each survival (more so for high-risk episodes) and falls with each death (more so for low-risk episodes). The data shown here are national data for consecutive cases during the first 5 months of 2009. It should be noted that, as we had access only to year and month of operation, the date of cases within each month has been generated arbitrarily.

Shown along with the running tally of ‘(expected – observed) deaths’ are episodes in which there was a surgical or catheter intervention within 30 days. This is added to discern between periods in which equivalent 30-day survival is achieved but with a higher reintervention rate and so arguably a higher level of postoperative morbidity.

FIGURE 35.

An example VLAD chart for 5 months of national data in 2009.

Figure 36 shows a ‘mock-up’ of how the monitoring tool would be used at an institutional level to review risk-adjusted mortality and reinterventions on a regular basis.

FIGURE 36.

An example VLAD chart for how the monitoring tool might be used within institutions on a rolling quarterly basis using a ‘mocked-up’ institution assuming a case load of 600 cases per year.

Model overview and attributes

We have developed a risk model for use in monitoring 30-day mortality in paediatric cardiac surgery that incorporated diagnostic information in addition to procedure, age, weight and comorbidity. The model shows reasonable accuracy and good discrimination between groups of patients with high and low mortality, with an AUC of 0.77 when evaluated across the entire validation data set and an AUC of 0.81 for post-2007 data. This compares well with other published risk-adjustment tools from the same field of practice. 24 As we have shown, supplementing procedural information with diagnostic, age, weight and comorbidity characteristics increased the discriminatory performance of the risk model: the AUC across the entire validation data set was 0.72 when only specific procedure was included. The risk model we have developed is an empirical (such as the STS-EACTS score) rather than a consensus-based risk-adjustment tool (such as RACHS-1 or Aristotle). This offers the advantage of reflecting the data rather than subjective personal opinion. Developed using the UK national database, the model can be applied objectively to data already routinely collected by cardiac paediatric units. We believe that the risk model is fit for the purpose of routine monitoring of outcomes in paediatric cardiac surgery, and we have gone some way to disseminating our findings by meeting with key stakeholders to inform them of the model and its potential use for quality assurance.

It has been previously observed that procedure categories developed for use in risk adjustment may have incomplete coverage, leaving some operations excluded from outcome analyses. 23–25 The empirically based tool for analysing mortality published by STS and EACTS in 200924 increased the coverage of records by including 148 types of operation and using a Bayesian model to adjust for small denominators. As discussed in the methods, the specific procedure categories reported by CCAD online8 are well established and accepted within the UK for benchmarking: these cover 83% of records in the data set. We therefore consider that these are the most appropriate markers of procedural complexity for a risk model designed for routine use in a UK context. Indeed, all choices of model inputs were made on the basis of their being both readily available and easily understood by clinicians ‘on the ground’. We gathered additional information for use in mortality predictions by ascertaining cardiac diagnosis, which could be allocated to 97.1% of those records classed as ‘not a specific procedure’, the most common diagnosis (11.6%) in these records being ‘acquired’. Therefore, the coverage of the data set in terms of attributed clinical information was very high, comparing well with previous attempts at risk adjustment in this context.

Model limitations and cautions

The necessary pseudonymisation process had the negative effect of essentially blinding the analytical team to missing or anomalous data in the patient and hospital ID fields. This meant that we had to conduct extensive face validity exercises when trying to identify genuine groups of records pertaining to the same patient. This included analysis to check the internal consistency of patient histories in terms of the dates of operations and ages at each operation and also some manual checks of the clinical face validity of sequences of operations. We cannot be certain that the study team identified all incorrectly entered IDs. The close collaboration between the clinical and the analytical teams was essential here, as was having the involvement of CCAD, which could, when necessary, go back to CCAD’S own records for clarification. We note that using any audit database for research purposes requires some cleaning of the data and that collaboration with the CCAD team was essential for understanding the data and performing appropriate cleaning. As part of this ongoing collaboration, the analytical team shared the data cleaning procedures used with the CCAD team.

An important yet obvious caveat of this work is that the model pertains to short-term 30-day outcomes (not long-term outcomes) and is designed for the purpose of routine monitoring for quality assurance rather than bedside-type predictions for individual patients. The unit of analysis is a 30-day episode so that neither deaths nor survivors are counted twice, and this is a caution to be aware of when calculating mortality percentages. We note that 4.5% of episodes contain at least one surgical reintervention and 1.8% contain at least one catheter reintervention within 30 days in the development set.

The risk model is intended for future use in routine monitoring of risk-adjusted outcomes within UK paediatric cardiac centres. It is the performance of the model during or after 2007 that is most informative concerning its fitness for this purpose. In this period, the model was found to underestimate risk at the very high-risk end, in particular among neonates. This indicates that risk adjustment based on the current parameterisation of the model will potentially give an unduly negative impression of outcomes at those centres with a high proportion of high-risk cases. This is an important caveat to interpretation of risk-adjusted outcomes within and between centres that will need to be considered as the work is taken forward. Options that will be considered in our future work with CCAD and the centres include recalibration of the model over both development and test sets and then testing on data accrued since our project started and (in the longer term) reparameterisation using these new data to, potentially, improve stratification at high risk (for instance, specific comorbidities). Importantly, in our work with centres we will ensure that the case mix distribution is available to individual centres alongside risk-adjusted outcomes so that centres with a higher-risk case mix are openly identified and thus all are prewarned as to this issue.

It is important to understand how differences in case mix and differential performance of the risk model in different subgroups could combine to give an artefactual impression of better or worse risk-adjusted outcomes at one centre compared with another. This issue is of particular importance given the level of scrutiny to which these types of outcome data are exposed. Although the UK is currently the only country that displays unit-specific outcomes of procedures online,8 there has been considerable debate of this issue in the professional journals, with the suggestion that programme-level reporting of unit-specific outcomes across a range of domains may evolve internationally over the coming years. 51,52 Centre effects were purposefully left out of the model. Our principal aim was to develop, for prospective use, a risk-adjustment system to account for patient-level factors, rather than create the most accurate statistical model of historical 30-day mortality. Importantly, it would not be possible from analysis of data alone to tease out whether any centre-to-centre differences in mortality rate are due to differences in case mix not captured by the data set, or to genuine, historical differences in the care processes delivered at surgical units. We are currently exploring the robustness and utility of the model across centres by working directly with centres and CCAD in a pilot study. Given some of the sensitivities that surround centre and ‘volume’ effects, we feel that routine monitoring of outcomes should start locally within individual clinical teams.

There is a need for a rolling programme of reparameterisation for a model in routine use to account for anticipated improvements in outcomes over time11 and indeed any other evolving trends. Alongside a reparameterisation, a growing volume of records over time may support a model with a greater number of variables, for example including more diagnostic categories, and it is hoped that the completeness of comorbidity data will increase with time as clinicians perceive the relevance of this information to risk adjustment. Data quality improvements over time for antenatal diagnosis and deprivation score may allow these factors to be reconsidered.

Implications for clinical practice

The interactions and engagement between our research group, clinical units and CCAD are an important component of taking the risk model forward for use in routine monitoring for the purposes of quality improvement. The active engagement and genuine collaboration of the analytical team with CCAD and clinical co-applicants and collaborators was essential in the development of this risk model and will be equally essential in any implementation of this model within current practice.

Implications and recommendations for research

The immediate research application of the risk-adjustment model is in the production of VLAD charts. There exists a strong level of engagement and intent to participate in this pilot project from Evelina Children's Hospital and Yorkhill Hospital in Glasgow. The pilot project will hopefully enable VLAD charts to be developed that are useful and meaningful to clinicians. Given the importance of case mix severity in monitoring outcome in paediatric cardiac surgery, it would also be advantageous to explore methods to describe and display information about centre-specific case mix complexity alongside VLAD plots.

The National Institute for Cardiovascular Outcomes Research (NICOR) and CCAD assisted with analytical work underpinning this project over the 5 months between November 2011 and March 2012. This pilot project of risk-adjusted VLAD incorporated data from Great Ormond Street Children's Hospital, Evelina Children's Hospital and Yorkhill Hospital in Glasgow. Although not part of the planned pilot study, future collaboration with qualitative researchers to identify barriers and facilitators of using risk-monitoring tools in clinical practice could be a fruitful area of research.

As a second step, the risk model provides the means to describe case mix severity both nationally and locally at the level of the centres. This information, which has not previously been available, will permit some exploration of trends in case mix severity over time and local practice patterns in terms of, for example, referral of higher-risk cases to particular centres. This type of future research analysis may be of importance when planning health services and would certainly have been of interest to the Safe and Sustainable review had it been available at that time.

A longer-term aim of our study was groundwork that could inform subsequent analyses of long-term outcome by diagnosis in CCAD as this may provide valuable information in terms of national audit of outcomes; the latter is a larger project beyond the scope of the current study. Such future analyses would carry the following advantages: diagnosis-based outcomes provide more relevant information to patients and carers; many CHD patients have multiple procedures during their lives, each of which carries an individual risk; diagnosis-based analyses allow study of the variation in the strategies adopted in different centres for the surgical management of a particular diagnosis; and diagnosis-based analyses provide an appropriate way of assessing a service given that the aim of paediatric cardiac surgery is to contribute to good long-term outcomes for a group of patients with a CHD diagnosis.

We selected terms to label the primary diagnosis groupings, such as ‘ventricular septal defect’ and ‘hypoplastic left heart syndrome’, that will be recognisable both to lay persons who may have an interest in this topic and to professionals. This choice was in keeping with the philosophy of CCAD, which shares information about outcomes with the public; we considered that more technical medically focused schemes for describing and allocating primary diagnoses would be less suitable for this purpose.

The groundwork we have carried out has the potential for future exploitation in terms of identifying the incidence of particular cardiac diagnoses in CCAD, which is in effect a national registry of patients who have undergone procedures. Future research in this area may provide useful information to commissioners and other CHD stakeholders. For example, we can now say that the proportion of patients with a UVH in CCAD, which captures all UK children who have undergone an intervention since 2000, is 9.3%. A limitation of this information is the inclusion criterion children undergoing surgical or catheter intervention only, which excludes those children who were unfit for intervention from the outset.

Contribution of authors

Christina Pagel (Senior Research Fellow, Operational Research) cleaned the data, developed and tested the risk model and contributed to writing the report.

Kate Brown (Consultant, Cardiac Intensive Care) contributed a clinical perspective to development of the risk model and contributed to writing the report.

Sonya Crowe (Research Associate, Operational Research) cleaned the data, developed and tested the risk model and contributed to writing the report.

Martin Utley (Professor of Operational Research) supervised the data cleaning and the development and testing of the risk model and contributed to writing the report.

David Cunningham (Senior Strategist, NICOR) provided expertise with respect to the data set (particularly Chapters 3 and 4) and contributed to writing the report.

Victor T Tsang (Consultant Cardiothoracic Surgeon) contributed a clinical perspective to development of the risk model and contributed to writing the report.

Disclaimers

The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PHR editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the PHR programme or the Department of Health.

Publications

1. Brown KL, Crowe S, Pagel C, Bull C, Muthialu N, Gibbs J, et al. Use of diagnostic information submitted to the United Kingdom Central Cardiac Audit Database: development of categorisation and allocation algorithms [published online ahead of print 2 October 2012]. Cardiol Young 2012. doi:http://dx.doi.org/10.1017/S1047951112001369. URL: http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8704423 (accessed 16 December 2012).

2. Crowe S, Brown KL, Pagel C, Muthialu N, Cunningham D, Gibbs J, et al. Development of a diagnosis- and procedure-based risk model for 30-day outcome after pediatric cardiac surgery [published online ahead of print 23 July 2012]. J Thoracic Cardiovasc Surg 2012. URL: www.jtcvsonline.org/article/S0022-5223%2812%2900701-5/abstract (accessed 16 December 2012).