Unicompartmental compared with total knee replacement for patients with multimorbidities: a cohort study using propensity score stratification and inverse probability weighting

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 15/80/40. The contractual start date was in June 2017. The draft report began editorial review in January 2020 and was accepted for publication in July 2020. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© Queen’s Printer and Controller of HMSO 2021. This work was produced by Prats-Uribe et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2021 Queen’s Printer and Controller of HMSO

Background

Surgical randomised controlled trials (RCTs) generate gold standard evidence on the causal effects of surgery. Recent evidence suggests that they are both safe and useful for informing clinical practice in surgical specialties. 1 However, such studies remain uncommon owing to, for example, resource intensity, time required from design to completion, ethics considerations, need for surgeon equipoise and other feasibility issues. 2,3

Non-randomised studies that rely on routinely collected data offer an efficient alternative for the comparative assessment of established surgical interventions and/or implantable medical devices available in the NHS. When conducted well, so-called ‘real-world’ evidence studies offer results that are potentially generalisable to the whole population of NHS patients, regardless of comorbidities, socioeconomic status, sex or age, including patients who would have been excluded from RCTs. However, observational studies are limited by confounding indication and related channelling bias owing to non-random allocation of treatment alternatives. Although analytical and study design methods have been used in drug safety studies to minimise confounding, there are few data on their performance in comparative effectiveness and safety research of surgery and medical devices.

Large-scale analyses of medicines are used to inform regulatory and clinical decision-making, and both the US Food and Drug Administration (FDA) and the European Medicines Agency1 have recently published guidelines on the use of routinely collected data for regulatory purposes. Collaborations such as the Observational Health Data Sciences and Informatics (OHDSI) [www.ohdsi.org (accessed 20 December 2019)] and the European Health Data and Evidence Network (EHDEN) [www.ehden.eu (accessed 20 December 2019)] are accelerating the creation of multinational networks and tools for curating and analysing real-world data at scale. The combined existence of data and best practices for analyses are leading to high-impact publications that will influence clinical guidelines by replacing ‘expert opinion’ for which RCT evidence is lacking. 2

Well-designed randomisation in clinical trials eliminates systematic bias. In surgical RCTs evaluating implantable devices or alternative surgical procedures, randomisation can account for patient characteristics and surgeon characteristics and expertise. High levels of adherence in RCTs reduce performance bias and attrition bias. Well-designed, well-conducted RCTs, thus, have excellent internal validity.

Randomised controlled trials give detailed evidence about the potential effects of new interventions under ideal circumstances, and are considered the gold standard for casual inference and health technology assessment. However, the main criticism of RCTs is that their rigid eligibility criteria can mean that trial participants are not representative of the full target population. The more restrictive the trial, the more limited the trial’s external validity.

An example of ongoing debate in surgery is the choice of total knee replacement (TKR) or unicompartmental knee replacement (UKR), also known as partial knee replacement, for severe knee osteoarthritis. In response, the National Institute for Health Research (NIHR) funded a surgical RCT called TOPKAT (Total or Partial Knee Arthroplasty Trial) [Health Technology Assessment (HTA) 08/14/08]. 4 This recently concluded multicentre RCT successfully recruited, randomised and followed up participants for 5 years. The trial results were reported in The Lancet5 and the full report has now been published in the NIHR HTA journal. 4 In brief, TOPKAT demonstrated that UKR had a small benefit in a patient-reported outcome over TKR of < 2 points in the Oxford Knee Score (OKS) in the short term (1 year), but no difference in the longer term (5 years). UKR was more cost-effective than TKR over the 5 years of follow-up.

TOPKAT was a relatively pragmatic trial that excluded only patients with an unusually high American Society of Anesthesiologists (ASA) grade of ≥ 3, owing to severe comorbidity and potentially limited lifespan. The National Joint Registry (NJR) report suggests that only about 17% of people receiving knee replacement surgery have an ASA grade of ≥ 33 and, therefore, would have been ineligible for TOPKAT. TOPKAT has, alongside some observational studies, been identified by NIHR Signals as potentially relevant for informing future National Institute for Health and Care Excellence (NICE) guidelines and NHS practices. 6

There is an opportunity to complement the results from TOPKAT with good-quality data on the performance of these two surgical approaches for multimorbid patients requiring knee surgery, which TOPKAT cannot provide. Observational data from the NJR can potentially provide insights into the impact of different types of knee replacement for all NHS patients. A recent Lancet paper used one of the most widely extended methods [propensity score (PS)] to minimise bias. 7–10 The authors acknowledged that unmeasured confounders (such as unrecorded conditions, disease severity or drug use) could at least partially explain the study findings, as PS can account only for measured confounders. Such unresolved bias can sometimes be minimised with alternative pharmacoepidemiological analytical methods, such as instrumental variables (IVs)11 or high-dimensional PSs. 12

The FDA and colleagues from a number of academic institutions are replicating previous drug RCTs using observational methods to demonstrate their usefulness for drug and vaccine safety and comparative effectiveness research. 6,13 However, to our knowledge, these methods have not yet been used to replicate the results of surgical or implantable device RCTs. There is a need for a better understanding of the performance of these methods in comparative effectiveness and safety studies to evaluate surgical and implantable medical device innovations using routinely collected data. The existence of a national, multicentre surgical RCT comparing two common surgical techniques in TOPKAT, and the availability of good-quality national data on these treatments and the primary study outcome from the NJR, offers a unique opportunity to study the validity of analytical methods for researching surgical and medical device innovations using observational data.

Evidence explaining why this research is needed now

The recent multicentre RCT TOPKAT provided high-quality evidence on the clinical effectiveness and cost-effectiveness of UKR compared with TKR for medial compartmental knee osteoarthritis. However, the results might not be generalisable to patients with an ASA grade of ≥ 3, equivalent to severe or very severe systemic disease, as they were ineligible for participation in the trial. 3 Recent NJR documents have reported3 that differences in patient-reported outcome measures (PROMs) exist according to ASA grade, and that there are known associations between comorbidities and postoperative complications and mortality. 14

Patients with an ASA grade of ≥ 3 currently represent only 17% of those undergoing knee replacement surgery. However, this proportion will probably increase as our population ages, and will probably account for a high proportion of the NHS expenditure on knee replacement surgery and related hospital admissions, given their baseline medical history and risk factors. The difficulties in recruiting older people and patients with severe and/or multiple comorbidities for surgical RCTs are well known. Alternative solutions are needed to generate evidence for this group of people. 3 This study follows previously published NIHR themed calls on evaluating interventions and services for older people with multimorbidity or complex health needs.

TOPKAT’s finding that UKR was not associated with an excess risk of revision contradicted all previous observational research, including previous publications using NJR data,9,10 a BMJ meta-analysis15 and a recent multinational analysis led by OHDSI and EHDEN collaborators. 16 This discrepancy could have been driven by residual confounding in the observational studies or heterogeneity in the population of patients, surgeons or hospitals in the RCT and population-based cohort analyses. Evidence is required on the mechanisms behind this difference in results to inform future related research and health-care delivery in the NHS.

A new European regulation for medical devices will be implemented in May 2021, which will require a more comprehensive evaluation of implantable devices, including orthopaedic prostheses. 17 There is, therefore, an urgent need for methodological guidelines on using real-world data to inform the post-marketing use, effectiveness and safety of medical devices.

Research aims and objectives

We undertook the Unicompartmental (vs. Total) knee replacement for patients with Multimorbidity Study (UTMoSt) in two stages:

• Stage 1 – we studied the validity of different methods previously used in drug and vaccine studies to minimise confounding for assessing the comparative effectiveness of alternative surgical procedures and implantable devices. We used knee replacement (UKR and TKR) in patients with an ASA grade of 1 or 2 (eligible for TOPKAT) as an example, and TOPKAT as a gold standard for comparison. Methods that gave results comparable to TOPKAT were deemed valid and were used in stage 2.

• Stage 2 – we used the methods that were able to replicate the RCT findings in stage 1 to compare the benefits (OKS), risks (revision surgery, complications and mortality), hospital costs and cost-effectiveness of UKR versus TKR among NJR participants with multiple and severe comorbidities (ASA grade of ≥ 3).

Structure of this report

Chapter 2 describes the data sources, defines the exposures (UKR and TKR) and potential confounders, and summarises the statistical methods used to minimise confounding.

Chapters 3–5 report the main findings of stage 1. Chapter 3 reports results from stage 1 based on PS analyses. Chapter 4 reports results from stage 1 based on IVs. Stage 1 conclusions and implications for stage 2 are discussed in Chapter 5.

Chapters 6–8 report stage 2. Chapter 6 describes the population, Chapter 7 reports the results of the safety-effectiveness analyses and Chapter 8 reports the health economics analysis.

We synthesise and discuss the results, study strengths and limitations, future research and implications of the study findings in Chapter 9.

Data sources

Data linkage

The PROMs data were matched to HES by NHS Digital (DARS-NIC-172121-G0Z1H-v0.11) using the probabilistic linkage methods. A rank score was created based on patient-identifiable fields, provider codes and operation codes and dates, for which the highest score of 1 was given if identical information in patient-identifiable fields was recorded in the databases. 27 NJR patients were matched to HES/PROMs in a deterministic fashion that required the same information in patient-identifiable fields. The data linkage was approved and conducted by NHS Digital (DARS-NIC-172121-G0Z1H-v0.11). Figure 1 shows a flow chart of data sources used in this study.

FIGURE 1.

Data source flow chart.

Methods

Ethics and scientific approval

No additional ethics approval was required as this study used pseudo-anonymised, routinely collected data from HES, NJR and PROMs. A NJR data request was approved by the NJR research subcommittee (reference number RSC2016/13). The HES PROMs and linkage to NJR data request was approved by NHS Digital (reference number DARS-NIC-172121-G0Z1H). The Confidentiality Advisory Group (CAG) approved the data linkage (reference 17/CAG/0174).

Study population and participant flow

The NJR database contained 868,785 records of TKR or UKR. Of these, 553,567 records had unique HES linkage data, which were required for inclusion. HES covers treatment centres and hospitals in England only, whereas the NJR contains patient records for the whole of the UK.

After removing 2099 duplicate records, 514,435 patients were reported to have had TKR and 39,132 patients were reported to have had UKR in the linked database. Additional exclusion criteria were applied, as follows:

• We included only the first and unilateral knee replacement procedures. We excluded 5141 patients undergoing coded ‘bilateral knee replacement surgery’ and 74,601 patients undergoing knee replacements on both knees on the same date (suggestive of bilateral knee replacement surgery).

• We excluded patients whose surgeries were carried out after 2016 or who died before postoperative OKS data collection.

• We excluded 5402 patients who received patellofemoral or lateral knee replacements.

• We excluded 4567 patients without IMD data and 52 patients with inconsistent age data in HES and NJR, with a difference of > 3 years.

The final data set included 425,284 TKR and 32,293 UKR patients for further analyses. Figure 2 summarises patient flow through the study.

To replicate TOPKAT’s findings in stage 1, we also applied the trial’s eligibility criteria:

• We excluded 75,074 patients (TKR, n = 72,183; UKR, n = 2891) who had a preoperative ASA score of > 2.

• We excluded another 79,571 TKR patients and 8376 UKR patients using TOPKAT’s clinical eligibility criteria listed in Table 1.

FIGURE 2.

Patient flow showing the common exclusion criteria used for the whole study.

After we applied the TOPKAT criteria, 273,530 TKR patients and 21,026 UKR patients were included in the stage 1 revision analysis. They formed the revision cohort. Of these, 1197 UKR and 125,834 TKR patients had postoperative OKS data and could be used to analyse the primary outcome: postoperative OKS. They formed the OKS cohort. Figure 3 summarises patient flow from the full cohort to the stage 1 revision and OKS cohorts.

FIGURE 3.

Patient flow showing the selection of patients from the full cohort for the stage 1 revision and OKS cohorts.

Table 3 shows the unadjusted patient-level characteristics in the revision and OKS cohorts before matching, stratification or other strategies were used to minimise confounding. The two cohorts were generally comparable. However, UKR patients in the OKS cohort appeared healthier than those in the revision cohort.

Overall, in the revision cohort there were noticeable differences between patients who received TKR and patients who received UKR in terms of sex (43% vs. 52% men, respectively), health status (11% vs. 21% rated as fit and healthy, respectively), comorbidity levels (69% vs. 73% with no reported comorbidity, respectively) and age [mean (SD): 70.2 (8.9) years vs. 64.3 (9.5) years, respectively]. TKR patients had a lower mean preoperative OKS [mean (SD): 19.3 (6.8)] than UKR patients [mean (SD): 21.3 (6.2)]. UKR patients were more likely to have comorbid osteoarthritis and other joint problems (18% vs. 13% in TKR) and cardiovascular disease (58% vs. 46%) than TKR patients.

Similar differences in sex, mean age, ASA grade and the PROMs for general health and mean preoperative OKS were observed between TKR and UKR patients in the OKS cohort. However, UKR patients were generally healthier than TKR patients in the OKS cohort. In addition, more UKR patients had no Charlson Comorbidity Index scores than TKR patients (76% vs. 69%, respectively). TKR patients were more likely than UKR patients in the OKS cohort to have a history of gastrointestinal disease (20% vs. 15%), osteoarthritis and other joint problems (19% vs. 12%), or cardiovascular disease (58% vs. 43%).

Covariate balance assessment

Covariate balance assessments were conducted for PS matching, PS stratification and IPW methods.

Propensity score matching

We PS matched 1197 UKR patients to 5652 TKR patients in the PS-matched postoperative OKS cohort. Before PS matching, TKR patients had much lower PS values than UKR patients. The wide range of ASMD values for the different characteristics shows the degree of mismatch (Figure 4a). Such differences disappeared in the PS-matched cohort, with estimated ASMD values for all baseline characteristics dropping to below 0.1 after matching (see Figure 4a). The TKR and UKR groups were, therefore, well-balanced after matching.

FIGURE 4.

The ASMD of each covariate included in the PS matching for the (a) postoperative OKS and (b) revision cohorts, before and after PS matching. 1, overall PS; 2, males; 3, age; 4, BMI; 5a, Rural Index – town and fringe; 5b, Rural Index – village; 5c, Rural Index – isolated; 6a, IMD – less deprived 10–20%; 6b, IMD – less deprived 21–30%; 6c, IMD – less deprived 31–40%; 6d, IMD – less deprived 41%–50%; 6e, IMD – more deprived 10–20%; 6f, IMD – more deprived 21–30%; 6g, IMD – more deprived 31–40%; 6h, IMD – more deprived 41–50%; 6i, IMD – most deprived; 7a, general health = 1; 7b, general health = 2; 7c, general health = 3; 7d, general health = 4; 7e, general health = 5; 8, preoperative quality-of-life measure (EQ-5D); 9, preoperative OKS; 10, ASA grade of 2, mild diseases; 11a, Charlson Comorbidity Index score = 1; 11b, Charlson Comorbidity Index score = 2; 11c, Charlson Comorbidity Index score = 3; 11d, Charlson Comorbidity Index score = 4; 12, gastrointestinal diseases; 13, osteoarthritis and other joint problems; 14, mental health; 15, respiratory diseases; 16, cardiovascular diseases; 17, thyroid problems; 18, foot, hip or spinal pain; 19, coxarthrosis; 20, neurological disorders; 21, other arthrosis; 22, polyarthrosis; and 23, spondylosis.

Baseline characteristics for the whole OKS cohort (before PS matching) and the matched cohort (after PS matching) are detailed in Appendix 1, Table 23. The characteristics of TKR patients were different in the matched and unmatched cohorts. After matching, TKR patients became more like the UKR patients: they were healthier and younger and a greater proportion were men.

We PS matched 21,026 UKR patients and 92,071 TKR patients from the revision cohort to form the PS-matched revision cohort, excluding 181,459 TKR patients in the process. The UKR and TKR patients in the matched revision cohort had generally similar baseline characteristics (see Appendix 1, Table 24). All patient-level covariates were well below the prespecified threshold of an ASMD of ≤ 0.1 after matching, suggesting that PS matching produced excellently balanced matched samples of TKR and UKR patients (see Figure 4b).

Propensity score stratification

As defined in Chapter 2, Propensity score stratification, two sets of 10 strata were created in the OKS and revision cohorts. One set was created by splitting the distribution of the estimated PS stratification in the whole cohort (PSS_whole) into 10, and the other set was created by splitting the distribution of the PS stratification in the UKR cohort (PSS_exp) into 10.

In the OKS cohort, PSS_exp stratification based on the PS distribution in the UKR cohort resulted in similar PS distributions for TKR and UKR patients in each stratum (see Figure 20a) and equal proportions of UKR and TKR patients between and within strata (see Figure 20b), suggesting good overall covariate balance. By contrast, PSS_whole stratification based on the PS distribution in the whole study population resulted in covariate imbalances in 6 out of the 10 strata: strata 1–6 were dominated by TKR patients and had < 1% UKR patients.

Figure 5a shows the covariate balance (mean ASMD) for each confounder across strata in the OKS cohort when using the PSS_whole method. Overall, the PS remained imbalanced, with an ASMD of > 0.1. In particular, BMI remained imbalanced between TKR and UKR patients, with an ASMD of 0.11. Covariate balance within strata was not always achieved, especially in strata 1–6. This is not surprising, as there were < 1% UKR patients included in these strata.

FIGURE 5.

The ASMD for each covariate included in the PS stratification for the (a) ASMD for the OKS cohort with full PSS; (b) ASMD for the OKS cohort with exposure PSS; (c) safety cohort with full PSS; and (d) safety cohort with exposure PSS. 1, Overall PS; 2, males; 3, age; 4, BMI; 5a, Rural Index – town and fringe; 5b, Rural Index – village; 5c, Rural Index – isolated; 6a, IMD – less deprived 10–20%; 6b, IMD – less deprived 21–30%; 6c, IMD – less deprived 31–40%; 6d, IMD – less deprived 41%–50%; 6e, IMD – more deprived 10–20%; 6f, IMD – more deprived 21–30%; 6g, IMD – more deprived 31–40%; 6h, IMD – more deprived 41–50%; 6i, IMD – most deprived; 7a, general health = 1; 7b, general health = 2; 7c, general health = 3; 7d, general health = 4; 7e, general health = 5; 8, preoperative quality-of-life measure (EQ-5D); 9, preoperative OKS; 10, ASA grade of 2, mild diseases; 11a, Charlson Comorbidity Index score = 1; 11b, Charlson Comorbidity Index score = 2; 11c, Charlson Comorbidity Index score = 3; 11d, Charlson Comorbidity Index score = 4; 12, gastrointestinal diseases; 13, osteoarthritis and other joint problems; 14, mental health; 15, respiratory diseases; 16, cardiovascular diseases; 17, thyroid problems; 18, foot, hip or spinal pain; 19, coxarthrosis; 20, neurological disorders; 21, other arthrosis; 22, polyarthrosis; and 23, spondylosis.

By contrast, PSS_exp stratification balanced all covariates, with an average ASMD of ≤ 0.1 across strata (see Figure 5b). This method also had better covariate balance within strata in most strata. In conclusion, in the OKS cohort, PSS_exp resulted in a balanced distribution of baseline characteristics between TKR and UKR patients. BMI remained imbalanced when using the PSS_whole method and was, therefore, included as a covariate adjustment when estimating the exposure effect [see Primary outcome (postoperative Oxford Knee Score) results and comparison with the TOPKAT findings].

In the revision cohort, in both methods, TKR and UKR patients have similar PSs (see Figure 21). The PSS_whole method performed better for the revision cohort than for the OKS cohort. Only stratum 1 in the revision cohort had < 1% UKR patients. The PSS_whole method achieved within-stratum covariate balance, except for sex; age; BMI; IMD; preoperative general health; EQ-5D; OKS; Charlson Comorbidity Index; mental health diseases; cardiovascular diseases; thyroid problems; foot, hip and spinal pain; and coxarthrosis in some strata. On average across the 10 strata, only the preoperative OKS had an imbalanced distribution after stratification, with a mean ASMD of 0.14 (see Figure 5c). It was, therefore, included in the exposure effect estimation.

By contrast, the PSS_exp method resulted in a mean ASMD of ≤ 0.1 across strata for all covariates, which indicates good average covariate balance (see Figure 5d). Within-stratum covariate balance was also achieved for all covariates except sex, BMI, general health, EQ-5D and OKS in some strata. In conclusion, the PSS_exp method resulted in better covariate balance than the PSS_whole method in the revision cohort, as was found with the OKS cohort.

Inverse probability weighting

In the OKS pseudo-population, the 1197 UKR patients had a stabilised weight ranging from 0.04 to 7.90 [interquartile range (IQR) 0.37–1.30], with a mean of 1. The TKR patients had a stabilised weight ranging from 0.99 to 1.35 (IQR 0.99–1.00), with a mean of 1.

The UKR and TKR patients in the OKS cohort had similar distributions in all covariates included in the PS except BMI, which had an ASMD just above 0.1 (Figure 6a). This imbalance was of limited clinical relevance: UKR patients had a mean BMI of 29.87 kg/m² and TKR patients had a mean BMI of 30.43 kg/m².

FIGURE 6.

The ASMD for each covariate included in the PS matching for the (a) postoperative OKS cohort and (b) the revision cohort, before and after IPW. 1, overall PS; 2, males; 3, age; 4, BMI; 5a, Rural Index – town and fringe; 5b, Rural Index – village; 5c, Rural Index – isolated; 6a, IMD – less deprived 10–20%; 6b, IMD – less deprived 21–30%; 6c, IMD – less deprived 31–40%; 6d, IMD – less deprived 41%–50%; 6e, IMD – more deprived 10–20%; 6f, IMD – more deprived 21–30%; 6g, IMD – more deprived 31–40%; 6h, IMD – more deprived 41–50%; 6i, IMD – most deprived; 7a, general health = 1; 7b, general health = 2; 7c, general health = 3; 7d, general health = 4; 7e, general health = 5; 8, preoperative quality-of-life measure (EQ-5D); 9, preoperative OKS; 10, ASA grade of 2, mild diseases; 11a, Charlson Comorbidity Index score = 1; 11b, Charlson Comorbidity Index score = 2; 11c, Charlson Comorbidity Index score = 3; 11d, Charlson Comorbidity Index score = 4; 12, gastrointestinal diseases; 13, osteoarthritis and other joint problems; 14, mental health; 15, respiratory diseases; 16, cardiovascular diseases; 17, thyroid problems; 18, foot, hip or spinal pain; 19, coxarthrosis; 20, neurological disorders; 21, other arthrosis; 22, polyarthrosis; and 23, spondylosis.

In the revision cohort, the 21,026 UKR patients had a weight ranging from 0.09 to 27.73, and the 273,530 TKR patients had a weight ranging from 0.93 to 12.30; both had a mean weight of 1. Both groups had a balanced distribution for all of the covariates, with an ASMD of ≤ 0.1 (see Figure 6b).

Inverse probability weighting minimised confounding to an acceptable degree based on the prespecified threshold (ASMD of ≤ 0.1) in both cohorts. Only BMI in the OKS cohort remained unbalanced, but this imbalance was of little clinical significance (< 0.6 kg/m² difference in means between UKR and TKR recipients) and was adjusted for in the final analyses.

Primary outcome (postoperative Oxford Knee Score) results and comparison with the TOPKAT findings

One of the key limiting factors for replicating RCTs with observational data is identifying a population with similar characteristics to the trial participants. Table 4 shows preoperative and postoperative OKSs collected in TOPKAT and estimated using each of the tested methods: crude scores (no adjustment for confounding), PS matching, IPW, PSS_whole, PSS_exp, and linear and non-linear PS adjustment. Most of the tested methods produced similar mean baseline OKSs for patients receiving TKR and UKR to those reported in TOPKAT:

• The crude/unadjusted mean preoperative OKS for patients receiving TKR (19.68) and UKR (21.88) differed by about 2 points at baseline. The mean preoperative OKS for TKR participants was similar to that in TOPKAT (mean preoperative OKS of 19.0). However, the mean preoperative OKS for UKR participants was about 3 points higher than that in TOPKAT (mean preoperative OKS of 18.8).

• PS stratification (both PSS_whole and PSS_exp) and adjustment (both linear and non-linear) included the whole population and resulted in the same mean preoperative OKS as the crude/unadjusted analysis.

• PS matching produced more similar mean preoperative OKSs for the TKR and the UKR patients than the crude/unadjusted analysis (TKR: 21.96; UKR: 21.88). Both groups were more different from the trial participants than in the crude analysis, with a mean preoperative OKS more than 2 points higher in both patient groups in the PS-matched cohort than in TOPKAT.

• The pseudo-population created by IPW had similar baseline preoperative OKSs for TKR (average 19.70) and UKR (average 20.41). The means differed from those in TOPKAT by < 1.5 points.

At the postoperative time point, approximately 6–8 months after the operation, TOPKAT and all of the tested methods showed a large improvement in OKS from baseline, in line with previous literature on knee replacement surgery. 23

All of the applied analytical methods obtained a treatment effect estimate that favoured UKR surgery over TKR surgery, as TOPKAT did. However, PS matching, IPW, PSS_whole and PS adjustment all obtained estimates with 95% CIs that included the null effect (0). Only PSS_exp found a statistically significant difference between UKR and TKR, with a point estimate (95% CI) of 0.76 (0.15 to 1.36). All of the tested methods yielded a treatment effect estimate at least 1 point lower than the 1-year effect observed in TOPKAT (Figure 7). Although none of the obtained estimates was completely covered by the 95% CI from TOPKAT, all of the estimates overlapped partially with it.

FIGURE 7.

Forest plot of the postoperative OKS effect size for TOPKAT and each of the tested PS methods, with heterogeneity measures (I², χ² and τ²). PSM, propensity score matching; PS_Alin, propensity score linear adjustment; PS_Anonlin, propensity score non-linear adjustment with PS⁰ and ln(PS)⁰.

All of the methods except PS adjustment had a chi-squared p-value of > 0.05, implying that any differences in the treatment effects collected in TOPKAT and calculated using the tested methods were likely attributable to chance.

Propensity score stratification yielded the treatment effect estimates closest to TOPKAT, with the smallest τ² value (PSS_whole: 0.23; PSS_exp: 0.48). PSS_exp was the only method with a small heterogeneity: I² < 40%. IPW showed moderate heterogeneity (I² = 48%; τ² = 0.43).

Propensity score matching resulted in a point estimate of 0.27, which was close to the lower 95% CI of the TOPKAT estimate, 0.20. It also had high heterogeneity (I² = 68%; τ² = 0.91). PS linear and non-linear adjustments resulted in OKS effect sizes that were even more different from TOPKAT. They also had the largest I² and τ² values, suggesting that PS matching and adjustment could not replicate the TOPKAT findings.

Five-year revision risks for unicompartmental knee replacement

Overall, 852 out of 21,026 UKR participants (4.1%) and 4090 out of 273,530 TKR participants (1.5%) in UTMoSt stage 1 underwent revision surgery within 5 years of the index procedure. In the PS-matched sample, 852 out of 21,026 (4.1%) UKR and 1383 out of 71,045 (1.5%) TKR patients underwent revision surgery.

All of the tested methods yielded a greater than twofold (statistically significant) increase in the risk of 5-year revision for UKR participants compared with TKR participants. By contrast, TOPKAT found no significant difference in risk between UKR and TKR participants. Potential reasons underlying these differences are discussed in detail in Chapter 5. However, as observed in Figure 8, all of the methods in UTMoSt yielded treatment effect estimates fully covered by the 95% CI observed in TOPKAT. PSS_whole and PSS_exp were the only methods with moderate to high heterogeneity (I² = 71% and I² = 63%, respectively). The other methods had no heterogeneity in their estimates of revision risk compared with TOPKAT, with I² = 0%. None of the tested methods had a significant chi-squared result when compared with TOPKAT, suggesting that any differences in treatment estimates were probably a result of chance. Even the smallest recorded chi-squared results (PSS_whole: p = 0.07; PSS_exp: p = 0.10) were still insignificant.

FIGURE 8.

Forest plot of the 5-year relative risk of revision for TOPKAT and each of the PS methods, with heterogeneity measures (I², χ² and τ²). PSM, propensity score matching; PS_Alin, propensity score linear adjustment; PS_Anonlin, propensity score non-linear adjustment with PS^0.5 and ln(PS)⁰; RR, relative risk.

Within this 5-year window, 496 out of 21,026 (2.4%) UKR participants and 14,004 out of 273,530 (5.1%) TKR participants died. As UKR appeared to be associated with a consistent reduction in mortality in all of our analytical methods (Table 5), this result suggests that further modelling to account for risk of death as a competing event may be warranted. However, such modelling was not carried out in TOPKAT. We, therefore, did not account for risk of death, so that we could compare our findings with those in TOPKAT as planned.

Sensitivity analyses

Oxford Knee Score cohort

We conducted a sensitivity analysis of patients whose surgery was performed by an ‘experienced’ lead surgeon who had performed at least 10 surgeries of the same type in the previous year (Table 6). We used the same volume-based definition of ‘experienced’ (i.e. number of surgeries performed) as that used to recruit participating surgeons in TOPKAT, although arguably volume does not accurately represent a surgeon’s true experience.

TABLE 6 - Number of participants and surgeons in the OKS and full cohorts, according to surgeon expertise in performing the index procedure

Surgeon expertise	OKS cohort				Full cohort
	Patients		Surgeons		Patients		Surgeons
	TKR	UKR	TKR	UKR	TKR	UKR	TKR	UKR
All, n	125,834	1197	3895	452	273,530	21,026	4597	1462
≥ 10 surgeries in the previous year, n (%)	114,871 (91.3)	602 (50.3)	2625 (67.4)	164 (36.3)	248,785 (91.0)	13,334 (63.4)	3001 (65.3)	474 (32.4)
≥ 30 surgeries in the previous year, n (%)	91,504 (72.7)	217 (18.1)	1556 (39.9)	43 (9.5)	195,898 (71.6)	5555 (26.4)	1730 (37.6)	128 (8.8)
≥ 50 surgeries in the previous year, n (%)	66,166 (52.6)	83 (6.9)	996 (25.6)	17 (3.8)	139,396 (51.0)	2550 (12.1)	1109 (24.1)	51 (3.5)

We found 2625 out of 3895 (67.4%) patients were operated on by TKR lead surgeons and 164 out of 452 (36.3%) patients were operated on by UKR lead surgeons in our cohort. The proposed sensitivity analysis that was restricted to patients operated on by experienced surgeons included 602 out of 1197 (50.2%) UKR patients and 114,871 out of 125,834 (91.3%) TKR patients from our OKS cohort. Their baseline characteristics are reported in Appendix 1, Table 25.

We applied IPW, PSS_whole and PSS_exp to the subcohort of patients in the OKS cohort who had been operated on by experienced surgeons. The resulting treatment effects were closer to that seen in TOPKAT than when using the full OKS cohort. The results from this sensitivity analysis (Figure 9) suggested that restricting the analysis to surgeons eligible for the trial would result in a treatment effect closer to that seen in TOPKAT than the treatment effect obtained for the full cohort.

FIGURE 9.

Forest plot of the postoperative OKS effect size for TOPKAT and each of the validated methods in the whole OKS cohort and in the sensitivity cohort of patients operated on by surgeons who had performed ≥ 10 surgeries of the same type in the previous year, with heterogeneity measures (I², χ² and τ²).

The treatment effect estimates obtained for the experienced surgeon cohort lay fully within the 95% CI of the TOPKAT estimate. Heterogeneity was dramatically lower in the experienced surgeon cohort than in the full OKS cohort, with the I² for all three methods dropping to 0% and τ² = 0. These results implied that surgeon experience contributed to the differences in treatment effect observed between the main OKS cohort analysis and TOPKAT.

Revision cohort

We also examined the association between UKR (vs. TKR) and 5-year revision and death risks stratified by surgeon experience. We defined three subcohorts of the revision cohort, based on whether the surgeon had performed ≥ 10, ≥ 30 or ≥ 50 surgeries of the same type as the index surgery in the previous year. This restricted the analysis to 248,785 out of 273,530 (91.0%), 195,898 out of 273,530 (71.6%) and 139,396 out of 273,530 (51.0%) TKR participants, and to 13,334 out of 21,026 (63.4%), 5555 out of 21,026 (26.4%) and 2550 out of 21,026 (12.1%) UKR participants, respectively (see Table 6). These cohorts included patients operated on by 3001 out of 4597 (65.3%) surgeons who had performed ≥ 10 TKR surgeries in the previous year, 1730 out of 4597 (37.6%) surgeons who had performed ≥ 30 TKR surgeries in the previous year and 1109 out of 4597 (24.1%) surgeons who had performed ≥ 50 TKR surgeries in the previous year. These cohorts also included 474 out of 1462 (32.4%) surgeons who had performed ≥ 10 UKR surgeries in the previous year, 128 out of 1462 (8.8%) surgeons who had performed ≥ 30 UKR surgeries in the previous year and 51 out of 1462 (3.5%) surgeons who had performed ≥ 50 UKR surgeries in the previous year. Baseline characteristics of the full revision cohort and the three subcohorts are reported in Appendix 1, Table 26.

Table 7 shows the number and percentage of UKR/TKR patients who underwent revision surgery or died within 5 years of their index operation in TOPKAT, the full revision cohort and the three experienced surgeon subcohorts. The proportion of TKR patients undergoing revision decreased with surgeon experience from 1.5% in the full cohort to 1.3% among patients operated on by the most experienced surgeons. The decrease in the proportion of patients undergoing revision was more striking for UKR patients, dropping from 4.1% in the full cohort to 3.3% among patients operated on by surgeons who had performed ≥ 10 UKR surgeries in the previous year, 2.5% of those operated on by surgeons who had performed ≥ 30 UKR surgeries in the previous year and 1.9% of those operated on by surgeons who had performed ≥ 50 UKR surgeries in the previous year.

TABLE 7 - Number (%) of participants undergoing revision surgery and dying in the 5 years after index surgery in TOPKAT, the full UTMoSt cohort (main) and the three subcohorts of participants operated on by experienced surgeons who had performed ≥ 10, ≥ 30 and ≥ 50 surgeries of the same type as the index surgery in the year before the index surgery

n/N refers to number of patients undergoing surgery/dying over the total number of patients for that group.

	5-year revision, n/N (%)		5-year mortality, n/N (%)
	UKR	TKR	UKR	TKR
TOPKAT	10/264 (3.8)	8/264 (3.0)	11/264 (4.2)	6/264 (2.3)
Main	852/21,026 (4.1)	4090/273,530 (1.5)	496/21,026 (2.4)	14,004/273,530 (5.1)
≥ 10 surgeries	435/13,334 (3.3)	3633/248,785 (1.5)	313/13,334 (2.3)	12,452/248,785 (5.0)
≥ 30 surgeries	137/5555 (2.5)	2670/195,898 (1.4)	122/5555 (2.2)	9472/195,898 (4.8)
≥ 50 surgeries	48/2550 (1.9)	1791/139,396 (1.3)	43/2550 (1.7)	6403/139,396 (4.6)

Mortality did not change substantially with surgeon volume in the TKR cohorts (4.7% in the full cohort vs. 4.6% for those operated on by the highest-volume surgeons). However, a monotonic decrease in mortality was seen among UKR patients, with mortality dropping from 2.4% in the full cohort to 2.3%, 2.2% and 1.7% in those operated on by surgeons who had performed ≥ 10, ≥ 30 and ≥ 50 UKR surgeries in the previous year, respectively.

When using PSS_whole to adjust for covariates, the 5-year relative risk of revision decreased from 3.07 (95% CI 2.80 to 3.36) in the main cohort to 1.49 (95% CI 1.05 to 2.10) in the highest-volume surgeon cohort. The effect of UKR (vs. TKR) on 5-year revision risk in the highest-volume surgeon cohort was much closer to that seen in TOPKAT (Figure 10) than in the other two surgeon groups. There was no heterogeneity between TOPKAT and the highest-volume surgeon group, with I² = 0%, χ² > 0.9 and τ² = 0. Similar trends were observed when using PSS_exp.

FIGURE 10.

Forest plot of the relative risk of revision surgery within 5 years of initial surgery for TOPKAT and each of the validated methods in the full revision cohort (main) and the sensitivity cohorts of patients operated on by surgeons who had performed ≥ 10, ≥ 30 and ≥ 50 surgeries of the same type in the previous year, with heterogeneity measures (I², χ² and τ²).

When using IPW, the risk, again, decreased with an increase in surgeon experience, but smaller differences were observed than those observed when using either PS stratification method. IPW yielded almost identical findings for the highest-volume surgeon cohort as those seen in TOPKAT, with relative risks of 1.39 (95% CI 0.93 to 2.07) for IPW and 1.40 (95% CI 0.50 to 4.00) for TOPKAT. Restriction to high-volume surgeons did not have a striking effect on the observed association between UKR (vs. TKR) and 5-year mortality following surgery (Figure 11).

FIGURE 11.

Forest plot of the estimated relative risk of death within 5 years of surgery, by index surgery type. Estimates were made using each of the validated methods with the full revision cohort (main) and the sensitivity subcohorts of patients operated on by surgeons who had performed ≥ 10, ≥ 30 and ≥ 50 surgeries of the same type in the previous year, with heterogeneity measures (I², χ² and τ²).

Patient characteristics

Instrumental variable creation

Possible instruments for analysis were generated and tested against the IV assumptions. We tested three types of preference-based instruments (surgeon-, hospital- and region-based preference) and volume-, area- and calendar-time-based instruments.

Calendar time

Calendar time is used in drug safety research as an instrument when there are clear changes in a medicine’s secular trends of use over time, such as when a new product is approved for use or when the conditions of use change dramatically over time. We attempted to identify such a change in the use/uptake of UKR surgery in our analytical data set. However, although the use of UKR surgery increased from 2011, the change was minimal, from an average prevalence of 6.64% before 2011 to 7.33% after 2011 (Figure 12). We, therefore, halted our attempt to build and test a calendar-time-based instrument.

FIGURE 12.

Secular trends in the prevalence (%) of UKR (vs. TKR) in the analytical data set per calendar year.

Results from the selected instrumental variables

Figure 13 shows the two-stage regression results for the association between UKR and postoperative OKS (primary outcome) for the five selected instruments compared with the TOPKAT results. All of the proposed instruments gave different results to TOPKAT. None of the estimates or their CIs overlapped with the main estimate obtained from TOPKAT or its upper or lower CI limits.

FIGURE 13.

Association between UKR (vs. TKR) and postoperative OKS recorded in TOPKAT and estimated with IV analysis using the five shortlisted IVs.

Quantitative estimates suggested that these results departed significantly from those obtained from TOPKAT, with tau² estimates ranging from 85.3 (consultant surgeon preference based on the previous 50 surgeries) to 190.88 (lead surgeon preference based on the previous 20 surgeries), I² ranging from 92.7% to 97.7% for the same instruments and all chi-squared test p-values < 0.001 (Table 14).

Conclusions from instrumental variable analysis

Only 5 of the 17 tested potential instruments passed our diagnostic tests and were eligible for two-stage regression analysis. Of the instruments that failed, four failed owing to residual confounding (one or more variables with an imbalance of > 0.1), six owing to instrument weakness (odds ratio of > 2 in the association between instrument and UKR exposure) and two (both area-based instruments) owing to both measures.

The surgeon preference-based instruments that failed owing to unresolved confounding did so because of an imbalance in socioeconomic status. Surgeon preference for UKR surgery may be geographically determined and associated to some degree with socioeconomic status. Previous studies have reported on the heterogeneity of UKR surgery use nationally and on the determinants of surgeon47 and patient48 choice. There is evidence for inequality in access to knee replacement generally, assessed as provision versus need. 49 However, to our knowledge, there are no data available on the potential heterogeneity in access to UKR nationally and/or globally and on its effect on patient outcomes. A recent study by Garriga et al. 50 reported geographical variation in outcomes of primary knee replacement, with greater surgical volume (by surgeon and hospital) associated with better patient outcomes, and UKR surgery associated with a lower risk of complications than TKR surgery.

All of the volume-based and regional/area-based instruments tested were not sufficiently strongly associated with the use of UKR. The area-based instruments also resulted in residual imbalances in preoperative OKS and socioeconomic status.

In conclusion, none of the tested IVs yielded results comparable with those obtained from TOPKAT. We, therefore, did not use IV analysis in UTMoSt stage 2. The reasons underlying this result could form the basis of future methodological research. They could include the violation of untested assumptions (e.g. a direct association between surgeon preference for UKR and postoperative OKS), the presence of residual confounding for unobserved variables or the non-normal distribution of OKSs.

Study participants identified from NHS routine practice and their eligibility for surgical randomised controlled trials

Applying the TOPKAT ASA grade inclusion criterion (no patients with an ASA grade of > 2) to our real-world set of knee replacement patients (NJR linked to HES and PROMs) excluded 75,074 out of 457,577 (16%) patients, which was expected from NJR annual reports. Another 87,947 (23% of the remaining 382,503) patients were excluded because of other TOPKAT exclusion criteria.

Although TOPKAT was a relatively pragmatic trial, about one-third of the patients who receive a knee replacement in the NHS would not have been eligible for the trial, in some cases because of indication for UKR, as about 50% of TKR patients were not eligible for UKR and, therefore, would not have been eligible for TOPKAT. Of particular concern is the use of RCT-based efficacy and safety data for patients with multiple comorbidities and/or complex health needs. Overall, the proportion excluded was lower than what has been observed when users of widely used medicines are compared with participants in pivotal RCTs. 53 However, it still raises concerns about the external validity of RCT findings when applied to the general population and patients with severe comorbidity.

Surgical RCTs are potentially limited by the participation of more academic and specialised surgeons, hospitals and treatment centres. Including only patients operated on in the NHS by surgeons who had performed ≥ 10 UKR surgeries in the previous year (following the TOPKAT published protocol)28 excluded almost half of the patients undergoing UKR surgery in the NHS from our data set (see Table 6). This surgeon-based exclusion preferentially affected the treatment under study: UKR. One-third of surgeons would not have been eligible for inclusion in TOPKAT based on their lack of previous experience with TKR, but up to two-thirds would have been excluded because of their lack of previous experience with UKR.

Results from propensity score analyses

Results from instrumental variable analysis

Strengths and limitations

To our knowledge, this study is the first attempt to investigate different statistical methods that account for confounding in trial replications. A trial duplication study59 run by the FDA and colleagues from a number of academic institutions focused on a framework of trial replication rather than method comparisons.

Our study team was blinded to TOPKAT findings during the analyses of PS-based methods and IVs, guarding against bias. We also used the same outcomes and outcome analysis methods as TOPKAT to ensure a practical comparison.

Our stage 1 analyses have a number of limitations. First, one of the key TOPKAT inclusion criteria was that patients had medial compartment osteoarthritis with exposed bone on both the femur and the tibia. However, the NJR database does not record the indication for a surgery. We unsuccessfully attempted to emulate this criterion by exploring osteoarthritis records in the HES data, but very few UKR or TKR patients had osteoarthritis recorded in the HES database. Such information is more likely to be recorded in the primary care consultation database, but linkage to primary care data is not routinely available. Despite our best efforts, our stage 1 populations (both revision and OKS cohorts) might, therefore, have differed from the TOPKAT participants. Compared with TOPKAT participants, TKR patients included in our stage 1 cohorts were generally older (mean age of 70.2 years and 70.4 years in the revision and OKS cohorts, respectively, vs. 65.2 years in TOPKAT) and had noticeably higher preoperative EQ-5D scores. These participant differences might have led to differences in the results obtained from UTMoSt stage 1 and TOPKAT. It is, therefore, reassuring that some of the proposed methods replicated TOPKAT successfully, but it is possible that some of the methods deemed invalid (e.g. PS matching) could have obtained more similar findings to TOPKAT in a more ideal scenario.

Second, clinical covariates included in the PS were based on HES inpatient data, implying that residual confounding might have been an issue owing to unmeasured variables. For example, osteoarthritis consulted in primary care would not have been identified in the HES inpatient database. Nevertheless, our linked observational data with IPW, PSS_exp and PSS_cohort yielded similar results to the gold standard TOPKAT estimates for the postoperative OKS. PS-based methods, therefore, have potential for trial replication or generalisation, and some of the included covariates might act as proxies for the unavailable confounders.

Third, there was a small difference in the timing of postoperative OKS collection in TOPKAT (1 year after randomisation) and UTMoSt (6–12 months after surgery). This difference might also have contributed to the differences in results in some of the analyses.

Finally, we estimated PSs using the most commonly used technique, logistic regression. Further research could explore alternative methods, such as machine learning, large-scale PSs or multilevel PSs. In PS matching, we used only calliper matching to form matched cohorts. Although this method has performed well when treatment effects are examined,35,36 PS matching with different algorithms may have led to results closer to the TOPKAT estimates.

We used only the IPW estimator to calculate treatment effect. Further studies could explore the augmented IPW estimator method in large samples. This approach requires PS estimates and separate outcome models for the exposed and unexposed cohorts. However, simulation studies have found that this method might not be suitable in small samples, such as our UKR OKS cohort. 60 Further studies could explore this method in large samples.

Conclusions and implications for UTMoSt stage 2

UTMoSt stage 1 demonstrated that surgical RCTs can be replicated using routine data recorded in actual NHS practice conditions. This is a pragmatic validation of methods routinely used in post-marketing drug safety observational research for surgical and medical device epidemiology, but does not imply that surgical RCTs are no longer needed. In fact, UTMoSt stage 1 was possible only because of the existence of a surgical RCT (TOPKAT) and good-quality routine data from the NJR linked to patient-reported outcomes and hospital inpatient records.

Some, but not all, of the methods tested obtained treatment effect estimates comparable to those obtained from TOPKAT for the trial’s primary outcome (postoperative OKS). When focused on the target population, only PS stratification based on the distribution of the PS in the exposed (UKR) arm (PSS_exp) successfully replicated TOPKAT according to all of the criteria used (chi-squared test, I² for heterogeneity, coverage, and statistical significance agreement). Two other methods (PSS_whole and IPW) replicated the trial according to two of the four criteria (chi-squared test and coverage), but did not pass the heterogeneity (prespecified, I² < 40%) or statistical significance agreement tests.

Our findings demonstrate challenges when replicating surgical trials, compared with ongoing international efforts to replicate RCT findings on the effects of medicines. The effects of surgeon expertise on outcome can affect the replicability of surgical RCTs, which are typically conducted in specialised treatment centres by experienced surgeons who can deliver both procedures and are in equipoise. Restricting the UTMoSt stage 1 analyses to patients operated on by surgeons with the volume required to participate in TOPKAT drove the obtained treatment estimates much closer to those seen in the trial, compared with using all eligible patients. Three tested methods (PSS_exp, PSS_whole and IPW) were able to replicate the TOPKAT findings in accordance with all four criteria when using the restricted population (chi-squared test p-values > 0.5, I² = 0%, τ² = 0.0, and statistical significance agreement and coverage). These three methods were selected for the UTMoSt stage 2 analysis, which is reported in Chapters 6–8.

Unfortunately, none of the proposed IVs passed the two prespecified tests for concordance with TOPKAT. The reasons underlying this failure warrant further investigation, for which we will seek funding from other streams focused on methodological research. Shortlisted instruments passed falsification tests for two of three key assumptions. We, therefore, speculate that the direct effect of surgeon expertise on the primary outcome (postoperative OKS) observed in the sensitivity analysis could explain to some degree the failure of preference-based instruments to replicate TOPKAT. None of the built instruments was selected for UTMoSt stage 2.

Target population

The target population for UTMoSt stage 2 was NJR participants undergoing primary UKR or TKR surgery who had severe comorbidities, defined as an ASA grade of 3 or 4 at the time of surgery.

Besides the general inclusion and exclusion criteria listed in Table 1, we also excluded patients who met any of the following exclusion criteria:

• NJR participants with no possible linkage to an episode in HES, as this was needed to study complications and costs

• NJR participants with no linked data available on pre or postoperative PROMs, as these were required for comparative effectiveness analyses

• NJR participants with previous cruciate ligament injury or inflammatory arthritis, as they would not have been eligible for UKR

• participants for whom revision costs could not be estimated because the linked HES episode did not provide valid/adequate information on a Healthcare Resource Group (HRG), as this was needed for the health economic analysis.

In stage 1, patients with a record of foot, hip or spinal pain in the 1 year before surgery were excluded. These patients were not excluded in stage 2, as this information was instead included as a PS covariate. We did not include the stage 1 exclusion criteria of prior knee surgery, patella dislocation or septic arthritis as PS covariates because too few patients had positive records.

Outcomes

UTMoSt stage 2 used the same primary outcome (postoperative OKS) as TOPKAT and UTMoSt stage 1. The secondary outcomes of interest were:

• 5-year risk of revision identified in the NJR or mortality identified in the HES data set

• 90-day risk of postoperative complications, including myocardial infarction, venous thromboembolism and prosthetic joint infection. These complications were identified using the primary diagnosis ICD-10 code in the HES data set. Code lists for these outcomes were prespecified based on previous research and are shown in Tables 27–29. 61

Participants were followed up from the index surgery date to the earliest of:

• end of enrolment in the database or 31 December 2016

• date of revision surgery (for secondary outcome analyses)

• date of a surgery for the other knee (for secondary outcome analyses)

• death

• end of 5 years of observation after the index surgery date.

We censored people at the surgery date in case of contralateral knee replacement, which would have made it difficult to attribute any surgical complications and costs to a specific knee.

Statistical analyses

Patient-level characteristics of the included TKR and UKR patients were compared using ASMDs with a cut-off value of 0.1. Any remaining imbalance (ASMD of > 0.1) in patient-level characteristics was accounted for by including the non-balanced covariate in the subsequent outcome analyses.

The following analyses were conducted for each of the proposed outcomes:

• Primary outcome – differences in postoperative OKS between UKR and TKR patients were estimated using multilevel linear regression (cluster 1: lead surgeons; cluster 2: patients).

• Secondary outcomes – postoperative complications. For each 90-day risk of an adverse event, we compared the cumulative incidence of all adverse events of interest between UKR and TKR patients. Relative risk and 95% CIs were estimated using Poisson models with robust standard errors. Mortality was not considered a competing risk in the 90-day risk because of low mortality rates over this period.

• Secondary outcomes – mortality and revision risk. Incidence rates and 95% CIs of revision and mortality for UKR and TKR patients were estimated using Poisson models, with the jack-knife method for CI calculations, and reported per 1000 person-years. Cause-specific hazard models were fitted to estimate risk of revision or mortality, censoring patients when they had a competing event (revision or mortality).

All outcome analyses were conducted in each of the imputed data sets and combined using Rubin’s rules.

The methods used to analyse hospital costs, health-related quality of life (HRQoL) and derived cost-effectiveness analyses are shown in Chapter 8.

Sensitivity analyses

No sensitivity analyses were conducted for postoperative OKS or any of the 90-day postoperative complications. As few patients had these outcomes, there was a lack of statistical power to detect a significant difference.

Sensitivity analyses were conducted for 5-year revision risk. Three predefined interactions were tested for using multiplicative terms in the above models. Stratified analyses by sex, age (younger or older than the median age in the study data sets) and ASA grade were reported if the p-value was < 0.1. To explore the impact of learning curves, revision analyses were restricted to surgeries undertaken by lead surgeons who had performed at least 10, 30 or 50 surgeries of the same type in the previous year.

Study population and participant flow

Of the 457,577 patients (UKR, n = 32,293; TKR, n = 425,284) available in the source-linked data, 383,522 patients (UKR, n = 29,403; TKR, n = 353,119) had an ASA grade of 1 or 2 and were, therefore, eligible for TOPKAT and UTMoSt stage 1, but excluded from stage 2 (Figure 14). The stage 2 exclusion criteria (see Chapter 6, Target population) excluded a further 15,117 patients. The resulting cohort for analysing safety outcomes (safety cohort) comprised 57,682 TKR patients and 2256 UKR patients. Of these patients, OKS postoperative data were available for 145 UKR and 23,344 TKR patients (OKS cohort).

FIGURE 14.

Stage 2-specific eligibility criteria and resulting patient selection.

Table 17 shows the unadjusted baseline characteristics for the safety and OKS cohorts. TKR patients in the OKS cohort had similar baseline characteristics to those in the safety cohort. UKR patients in the OKS cohort were healthier (34% vs. 38% had a Charlson Comorbidity Index score of 0) and more likely to live in the countryside (22% vs. 16% with a Rural Index of 3 or 4) than UKR patients in the safety cohort.

In the safety cohort, UKR patients were younger than TKR patients [mean age (SD): 69 (10) years vs. 73.5 (8.9) years, respectively] and were more likely to be men (57% vs. 44%, respectively) and live in the countryside (16% vs. 11%, respectively, with a Rural Index of 3 or 4) or the least deprived areas (26% vs. 18%, respectively). UKR patients were less likely than TKR patients to have a history of osteoarthritis and other joint problems (19% vs. 26%, respectively).

In the OKS cohort, there are similar noticeable differences in sex, Rural Index and socioeconomic status between UKR and TKR patients. UKR patients were also more likely than TKR patients to be healthy (Charlson Comorbidity Index score of 0) (34% vs. 39%, respectively). There was a vast difference in the number and proportion of UKR and TKR patients who responded to the postoperative OKS: 145 out of 2256 (6.4%) UKR patients versus 23,344 out of 57,682 (40.5%) TKR patients.

Covariate balance assessment

Oxford Knee Score cohort

We applied the three validated methods (IPW, PSS_whole and PSS_exp) to the OKS cohort to compare the treatment effect, as measured by the OKS, for UKR and TKR recipients.

Propensity score stratification based on the distribution of PSs in the whole cohort (PSS_whole) resulted in few UKR patients in some groups, leading to imbalanced PS distributions in some strata (see Figure 22a). Within-stratum covariate balance was not always achieved in all strata in each of the 10 imputed data sets. For example, across the 10 imputed data sets, all covariates had an ASMD of > 0.1 in stratum 1. This is not surprising because there were only two (0.09%) UKR patients in this stratum, which was defined by a low PS and, therefore, a low probability of UKR treatment. Strata 2–5 had between six and 17 UKR patients each, and the distribution of some covariates remained imbalanced when UKR and TKR patients were compared in some of the 10 imputed data sets. Within-stratum covariate balance improved in strata 6–10 because they were defined by a higher PS and probability of treatment; therefore, strata 6–10 included a larger number and higher proportion of UKR patients. However, the average ASMD across strata for each of the covariates was ≤ 0.1 (Figure 15a), indicating that good balance was achieved for all individual covariates across strata using this widely accepted, pre-defined threshold.

FIGURE 15.

The ASMD of each covariate included in the PS for the postoperative OKS cohort before and after balancing covariates by (a) PSS_whole, (b) PSS_exp and (c) IPW. 1, overall PS; 2, males; 3a, Rural Index – urban (≥ 10,000); 3, age; 4, BMI; 5a, Rural Index – town and fringe; 5b, Rural Index – village; 5c, Rural Index – isolated; 6a, IMD – less deprived 10–20%; 6b, IMD – less deprived 21–30%; 6c, IMD – less deprived 31–40%; 6d, IMD – less deprived 41%–50%; 6e, IMD – more deprived 10–20%; 6f, IMD – more deprived 21–30%; 6g, IMD – more deprived 31–40%; 6h, IMD – more deprived 41–50%; 6i, IMD – most deprived; 7a, general health = 1; 7b, IMD – general health = 2; 7c, general health = 3; 7d, general health = 4; 7e, general health = 5; 8, preoperative quality-of-life measure (EQ-5D); 9, preoperative OKS; 10, ASA grade of 2, mild diseases; 11a, Charlson Comorbidity Index score = 1; 11b, Charlson Comorbidity Index score = 2; 11c, Charlson Comorbidity Index score = 3; 11d, Charlson Comorbidity Index score = 4; 12, gastrointestinal diseases; 13, osteoarthritis and other joint problems; 14, mental health; 15, respiratory diseases; 16, cardiovascular diseases; 17, thyroid problems; 18, foot, hip or spinal pain; 19, coxarthrosis; 20, neurological disorders; 21, other arthrosis; 22, polyarthrosis; and 23, spondylosis.

The PSS_exp stratified based on the distribution of PSs in the exposure (UKR) cohort and most accurately replicated the TOPKAT findings in UTMoSt stage 1. It resulted in equal numbers of UKR patients in each stratum, and obtained better balance than PSS_whole in the PS distribution between UKR and TKR patients (see Figure 22b). It also led to better within-stratum covariate balance for each of the identified confounders, although most covariates had an ASMD of > 0.1 in at least one stratum. Overall, average covariate balance across the 10 strata was achieved (Figure 15b).

The IPW pseudo-population included 145 UKR patients with a stabilised weight ranging from 0.08 to 4.45 (IQR 0.38–1.28) and 23,344 TKR patients with a stabilised weight close to 1 (minimum, 25th percentile, 75th percentile, maximum: 0.99, 1.00, 1.00, 1.10). Four covariates remained imbalanced (ASMD of > 0.1) after IPW: respiratory disease, sex, socioeconomic deprivation and history of spondylosis (Figure 15c). UKR patients had a higher prevalence than TKR patients of respiratory disease (31% vs. 26%, respectively), male sex (53% vs. 46%, respectively) and residence in more deprived areas (40% vs. 29%, respectively). They were also less likely than TKR patients to have spondylosis (2% vs. 4%, respectively). These covariates were further (double) adjusted in the outcome analyses in Primary outcome analyses: postoperative Oxford Knee Score and Comparative safety analyses.

Safety cohort

We also applied the three validated methods to the safety cohort. PSS_whole resulted in similar overall PS distributions for UKR and TKR patients in each stratum (see Figure 23a). UKR patients had a higher predicted probability than TKR patients of receiving UKR based on their baseline characteristics, which was indicated through a higher PS. As a result, the lower PS quintiles (strata 1–3) each included < 1% of the UKR patients. Within-stratum covariate balance between UKR and TKR was much better than that in the OKS cohort, probably because the safety cohort had higher power. Overall, PSS_whole stratification controlled confounding to an acceptable degree based on ASMD (Figure 16a).

FIGURE 16.

The ASMD of each covariate included in the PS for the postoperative safety cohort before and after covariate balancing by (a) PSS_whole, (b) PSS_exp and (c) IPW. 1, overall PS; 2, males; 3a, Rural Index – urban (≥ 10,000); 3, age; 4, BMI; 5a, Rural Index – town and fringe; 5b, Rural Index – village; 5c, Rural Index – isolated; 6a, IMD – less deprived 10–20%; 6b, IMD – less deprived 21–30%; 6c, IMD – less deprived 31–40%; 6d, IMD – less deprived 41%–50%; 6e, IMD – more deprived 10–20%; 6f, IMD – more deprived 21–30%; 6g, IMD – more deprived 31–40%; 6h, IMD – more deprived 41–50%; 6i, IMD – most deprived; 7a, general health = 1; 7b, general health = 2; 7c, general health = 3; 7d, general health = 4; 7e, general health = 5; 8, preoperative quality-of-life measure (EQ-5D); 9, preoperative OKS; 10, ASA grade of 2, mild diseases; 11a, Charlson Comorbidity Index score = 1; 11b, Charlson Comorbidity Index score = 2; 11c, Charlson Comorbidity Index score = 3; 11d, Charlson Comorbidity Index score = 4; 12, gastrointestinal diseases; 13, osteoarthritis and other joint problems; 14, mental health; 15, respiratory diseases; 16, cardiovascular diseases; 17, thyroid problems; 18, foot, hip or spinal pain; 19, coxarthrosis; 20, neurological disorders; 21, other arthrosis; 22, polyarthrosis; and 23, spondylosis.

The PSS_exp stratification yielded a more equally distributed PS than PSS_whole (see Figure 23b) between UKR and TKR patients. The within-stratum covariate balance was also better with PSS_exp than with PSS_whole, with fewer variables with an ASMD of > 0.1. Overall, good covariate balance was achieved for all of the observed confounders, with average ASMDs of < 0.1 across all strata (see Figure 16b).

In the IPW pseudo-population, 2256 UKR patients were given a stabilised weight ranging from 0.09 to 9.45 and TKR patients were given weights of around 1. There were balanced distributions in all of the covariates (ASMD of ≤ 0.1) (see Figure 16c).

Primary outcome analyses: postoperative Oxford Knee Score

Table 18 shows the pre and postoperative OKS estimates from the stage 1 (ASA grade of 1 or 2) and stage 2 (ASA grade of 3 or 4) OKS cohort for the validated analyses using IPW, PSS_whole and PSS_exp:

• Stage 2 patients had a baseline (preoperative) OKS, on average, about 3 points lower than stage 1 patients for both UKR and TKR recipients and for any of the three methods. This difference was probably a result of stage 2 participants having higher comorbidity than stage 1 participants, or having surgery delayed because of high ASA grade.

• Stage 2 TKR and UKR patients in the IPW pseudo-population had similar mean (SD) preoperative OKSs [16.99 (7.56) versus 17.28 (8.37), respectively].

• PSS_whole and PSS_exp both included the whole cohort and, therefore, found a better mean (SD) preoperative OKS for stage 2 UKR patients [19.44 (8.55)] than TKR patients [16.97 (7.55)]. However, the variance in the mean preoperative OKS for UKR patients was larger than that for TKR patients, resulting in an agreeable degree of balance, as demonstrated in Figure 15a and Figure 15b.

Approximately 6–8 months after the operation, the mean OKS was more than twice the preoperative OKS in all participant groups, indicating a dramatic improvement because of surgery in both stage 1 and stage 2 patients. PSS_exp, the preferred method from stage 1, resulted in a statistically significant positive effect for UKR, with an estimated mean postoperative OKS difference of 1.83 (95% CI 0.10 to 3.56) points in favour of UKR. The other validated PS stratification method (PSS_whole) found very similar results, with an estimated mean difference in postoperative OKS of 1.82 (95% CI 0.10 to 3.56) points, again favouring UKR. IPW analyses found a non-significant difference in postoperative OKS between TKR and UKR, with a mean difference between groups of 1.00 (95% CI –1.28 to 3.27) points.

Comparative safety analyses

Long-term (5-year) complications

The cumulative risk of revision increased faster for UKR patients than for TKR patients over 5 years of follow-up (Figure 17). The incidence rates of revision were 13.09 (95% CI 10.64 to 16.09) after UKR and 4.88 (95% CI 4.56 to 5.22) after TKR, an almost threefold increase in revision risk for UKR compared with TKR (crude hazard ratio 2.70, 95% CI 2.16 to 3.37). Adjustment for confounding using the validated methods did not attenuate this risk, with a resulting cause-specific hazard ratio of 2.70 (95% CI 2.15 to 3.38) for PSS_whole and PSS_exp, and 2.60 (95% CI 1.94 to 3.47) for IPW (Table 20).

FIGURE 17.

Cumulative incidence functions of (a) risk of revision and (b) mortality, for UKR (UKR = 1) and TKR (UKR = 0) over 5 years of follow-up. Number at risk: number of patients with a particular surgery who did not experience any of the outcomes.

TABLE 20 - Long-term (5-year) complications after UKR or TKR

HR, hazard ratio; REF, reference.

Complication	Cumulative incidence (95% CI)	Crude HR (95% CI)	Method, HR (95% CI)
			PSSwhole	PSSexp	IPW
Revision surgery
TKR (n = 847)	4.88 (4.56 to 5.22)	REF	REF	REF	REF
UKR (n = 90)	13.09 (10.64 to 16.09)	2.70 (2.16 to 3.37)	2.70 (2.15 to 3.38)	2.70 (2.15 to 3.38)	2.60 (1.94 to 3.47)
All-cause mortality
TKR (n = 6401)	36.89 (36.00 to 37.81)	REF	REF	REF	REF
UKR (n = 164)	23.85 (20.46 to 27.79)	0.64 (0.55 to 0.75)	0.64 (0.55 to 0.75)	0.64 (0.55 to 0.75)	0.83 (0.67 to 1.03)

Participants receiving UKR surgery had lower 5-year mortality than TKR patients (see Figure 17b). UKR surgery, therefore, appeared to be associated with reduced all-cause mortality in the unadjusted analysis, with a crude hazard ratio of 0.64 (95% CI 0.55 to 0.75).

The observed decrease in mortality associated with UKR (vs. TKR) remained after adjustment for confounding using PSS_whole or PSS_exp, both resulting in a cause-specific hazard ratio of 0.64 (95% CI 0.55 to 0.75) (see Table 20). However, the observed effect on mortality was attenuated and became non-significant when using IPW, with a cause-specific hazard ratio of 0.83 (95% CI 0.67 to 1.03).

Sensitivity analyses

Analysis restricted to high-volume surgeons

We restricted the three validated analyses to surgeries performed by experienced surgeons to examine whether or not patients’ risk of long-term complications changed. As in Chapter 3, Revision cohort, we defined three subcohorts of the safety cohort based on the number of surgeries of the same type performed by the lead surgeon in the previous year: ≥ 10, ≥ 30 and ≥ 50 surgeries. Of the 57,682 TKR patients included in the total cohort, 51,118 (89%), 38,321 (66%) and 25,944 (45%) were included in these three lead surgeon subcohorts, respectively. Of the 2256 UKR patients included in the total cohort, a smaller proportion were included [1449 (64%), 610 (27%) and 242 (11%), respectively]. See Appendix 1, Table 30 for the baseline characteristics for these subcohorts.

Cause-specific risks of revision and mortality for UKR (vs. TKR) patients calculated with the three analytical methods are reported in Figure 18. The observed excess risk of revision seen in UKR patients was somewhat reduced when the analyses were restricted to those operated on by high-volume surgeons. The cause-specific hazard ratio of 2.60 (95% CI 1.94 to 3.47) in the main cohort decreased to 2.05 (95% CI 1.03 to 4.09) when restricting to high-volume surgeons with ≥ 30 surgeries in the past year and using IPW, and to 1.65 (95% CI 1.01 to 2.69) when using PS stratification. However, the CIs of these estimates overlapped, indicating no significant difference (see Figure 18a).

FIGURE 18.

Cause-specific hazard ratios for risk of (a) 5-year revision and (b) mortality for patients undergoing UKR (vs. TKR) in sensitivity analyses restricted to lead surgeons with ≥ 10, ≥ 30 or ≥ 50 surgeries of a particular type in the previous year.

Excess revision risks increased again when restricting to the highest-volume surgeons (≥ 50 surgeries of the same type in the previous year). However, this subanalysis was limited by low statistical power, resulting in wide CIs.

Restricting to high-volume surgeons did not have striking effects on the observed association with 5-year mortality following surgery (see Figure 18b). The overlapping CIs of these estimates suggested no clear trend in differential mortality between UKR and TKR with increasing surgeon volume.

Introduction

The economic burden to the health-care system associated with knee replacements is substantial, owing to the large number of replacements performed each year and the risk of complications and revision surgery. The relative burden, and, more specifically, the cost-effectiveness, of UKR compared with TKR is not known. The length of hospital stay is shorter for UKR than for TKR, which may lead to less intense health-care resource use. 9 A previous modelling study62 using data from the NJR for England, Wales, Northern Ireland and the Isle of Man found that UKR was expected to generate lower lifetime costs and better quality-of-life gains than TKR. Most of the evidence for the cost-effectiveness of UKR over TKR comes from modelling studies that extrapolate outcomes observed for a specific period over a lifetime based on assumptions. 63 Economic evaluations alongside RCTs typically require fewer assumptions, but examine cost-effectiveness over shorter time periods.

An economic evaluation was carried out alongside TOPKAT that examined the clinical effectiveness and cost-effectiveness of UKR compared with TKR for selected participants over 5 years. 5 TOPKAT included 232 patients who underwent a UKR surgery and 238 who underwent a TKR surgery; these patients were recruited from 27 sites across the UK. The authors concluded that UKR and TKR were both effective and had similar clinical outcomes, complication rates and revision rates, but that UKR resulted in lower health-care costs and better cost-effectiveness 5 years after surgery than TKR. 5,62

TOPKAT and its economic evaluation may have low external validity owing to the strict patient inclusion criteria, a limitation that affects most RCTs. Patients were eligible for TOPKAT if they were medically fit and had an ASA grade of 1 or 2. 28 Although patients suffering from multiple comorbidities (i.e. ASA grade of 3 or 4) routinely receive knee replacements, they are often ineligible for clinical trials. It remains unknown whether or not UKR is cost-effective over TKR for these patients with multiple comorbidities. We used routinely collected data and the methods assessed in the previous chapters to evaluate the cost-effectiveness of UKR compared with TKR for complex patients with an ASA grade of 3 or 4.

Methods

Results

Cost-effectiveness analysis

The results from the cost-effectiveness analysis using each of the three adjustment methods are presented in Table 22, and the uncertainty surrounding the ICER shown in Figure 19. As we used regression equations with difference as the outcome, the results give differences between UKR and TKR with no separate adjusted means for costs and QALYs.

FIGURE 19.

Cost-effectiveness plane for UKR compared with TKR in patients with an ASA grade of 3 or 4.

TABLE 22 - Cost-effectiveness analysis results, stage 2 UTMoSt

NE, north-east; NW, north-west; SE, south-east; SW, south-west.

Method	Difference in costs (£) (95% CI)	Difference in QALYs (95% CI)	ICER	Distribution of bootstrapped cost and QALYs on cost-effectiveness plane by quadrant (%)
				NW	NE	SW	SE
IPW	–334 (–362 to –306)	0.135 (–0.482 to 0.753)	UKR dominant	–	–	10	90
PSSwhole	–359 (–378 to –339)	0.330 (–0.305 to 0.967)	UKR dominant	–	–	–	100
PSSexp	–359 (–378 to –340)	0.330 (–0.309 to 0.970)	UKR dominant	–	–	–	100

Conducting the cost-effectiveness analysis using the IPW method to minimise the risk of confounding led to UKR costs that were £334 (95% CI £306 to £362) lower than TKR costs. The mean difference in QALYs gained was 0.147 (95% CI –0.507 to 0.803) favouring UKR; however, this difference was not statistically significant. UKR dominated TKR as it was less expensive and more effective. Using the IPW method, the probability of UKR being more cost-effective than TKR was 100% at a willingness-to-pay (WTP) threshold of £0 (see Figure 19). From the bootstrapped analysis, 90% of resulting cost-and-effect pairs were located in the south-east quadrant (UKR less expensive and more effective than TKR) and 10% were located in the south-west quadrant (UKR less expensive and less effective than TKR) of the cost-effectiveness plane (see Table 22).

Using PSS_whole to minimise the risk of confounding produced a mean difference in costs of £359 (95% CI £339 to £378), again favouring UKR over TKR. The improvement in QALYs was 0.330 (95% CI –0.305 to 0.967) greater for UKR patients than for TKR patients; however, again, this difference was not statistically significant. UKR, therefore, dominated TKR because it was less expensive and more effective. The probability of UKR being more cost-effective than TKR was 100% at any positive WTP threshold (see Figure 19). Table 22 shows that all of the bootstrapped cost-and-effect pairs were located in the south-east quadrant (UKR less expensive and more effective than TKR) of the cost-effectiveness plane.

Using PSS_exp to minimise the risk of confounding produced a mean difference in costs of £359 (95% CI £340 to £378) favouring UKR over TKR. The mean difference in QALYs gained was 0.330 (95% CI –0.309 to 0.970), also favouring UKR over TKR. Once again, this difference was not statistically significant. UKR, therefore, dominated TKR. The probability of UKR being more cost-effective than TKR was 100% at any positive WTP threshold (see Figure 19). All of the bootstrapped cost-and-effect pairs were located in the south-east quadrant (UKR less expensive and more effective than TKR) of the cost-effectiveness plane (see Table 22).

Discussion

The aim of this study was to assess the cost-effectiveness of UKR compared with TKR for complex patients with comorbidities (ASA grade of 3 or 4), who are generally ineligible for clinical trials, using routinely collected data. The three methods validated in stage 1 to replicate the TOPKAT results in routinely collected data were used to assess cost-effectiveness for this patient group during the 5 years after primary knee replacement. Under all three methods, UKR was associated with lower mean hospital costs and higher mean QALYs gained than TKR. Lower costs for patients with UKR were mainly a result of the lower costs of primary knee replacement. The mean difference in QALYs between the two groups favoured UKR by between 0.1 and 0.3 discounted QALYs over the 5 years and was not statistically significant.

We found UKR to be more effective and less costly than TKR at 5-year follow-up for patients with an ASA grade of 3 or 4. This finding agrees with previous findings using trial and routinely collected data. 5,60,62,63,71,72 Burn et al. 62 compared UKR and TKR lifetime QALYs gained and health-care costs in a modelling study using data from the UK. They found that UKR was related to more QALYs gained and lower costs than TKR for different sex and age groups; however, no separate analysis was undertaken for complex patients with multiple comorbidities. Similarly, TOPKAT found that UKR was more effective (mean QALY difference 0.240, 95% CI 0.046 to 0.434) and less expensive (mean cost difference –£910, 95% CI –£1503 to –£317) than TKR during 5-year follow-up. However, the trial was restricted to patients with an ASA grade of 1 or 2. 5 UKR, therefore, appears to be consistently associated with more QALYs gained and lower costs than TKR regardless of patient comorbidity. To our knowledge, this is the first study assessing cost-effectiveness specifically for patients with multiple comorbidities and applying previously assessed methods that minimise the risks of confounding by indication that generally affect observational studies.

The difference in costs between UKR and TKR was probably driven by the lower cost of primary UKR surgery to the health-care system. The difference in the cost of primary knee replacement may be because UKR is a less invasive surgery than TKR. 62 For instance, TOPKAT found that UKR had lower primary surgery costs owing to shorter hospital stays, fewer perioperative complications and lower implant device costs than TKR. 5 Although we also found that the costs for revision and complications were lower for UKR than for TKR, these events were rare.

The mean difference in QALYs gained favoured UKR over TKR, but this difference was not marked and not statistically significant. The uncertainty around the mean difference in QALYs between the two groups was also reflected in the cost-effectiveness plane. When using the IPW method, 10% of the cost and QALY pairs were located in the south-west quadrant, indicating that UKR is less expensive and less effective than TKR. TOPKAT found that the 0.2 difference in QALYs between the two groups was statistically significant, whereas Burn et al. 62 found that the mean difference was statistically significant for some patient subgroups but not others. 5 As we identified only 145 UKR patients meeting the inclusion criteria, it is possible that the lack of statistical significance in our findings was because of the sample size. However, we found a difference in QALYs that varied between 0.1 and 0.3 depending on the confounding minimisation method used, which was consistent with the TOPKAT findings. These findings suggest that patients’ complexity and comorbidities did not prevent them from experiencing the quality-of-life benefits of a UKR rather than a TKR. We found that UKR and TKR both resulted in substantial improvements in utility scores from preoperative measurements to measurements 6 months after surgery, indicating that both procedures were highly beneficial for most patients.

We used the three PS methods validated in UTMoSt stage 1 (IPW, PSS_whole and PSS_exp) as being able to account for confounding bias and mimic TOPKAT. All three methods found similar mean differences in costs between UKR and TKR and overlapping 95% CIs. PSS_whole and PSS_exp resulted in wider 95% CIs than IPW for mean difference in QALYs gained between the two groups. Overall, the choice of method to account for potential confounding did not substantially alter the study’s conclusions about the cost-effectiveness of the two surgeries.

Conclusion

Our findings showed that UKR was more cost-effective and led to lower additional costs and higher additional QALYs than TKR for patients with multiple comorbidities who were eligible for either surgery. The costs were lower mainly because UKR primary replacements cost less than TKR primary replacements. UKR and TKR were both associated with a substantial improvement in health-utility estimates from before surgery to 6 months after primary surgery, indicating that both procedures were beneficial for complex, highly comorbid patients.

Study conclusions: UTMoSt stage 1

To the best of our knowledge, UTMoSt stage 1 is one of the first studies in the world to attempt to mimic a surgical RCT using real-world data and different analytical methods to minimise confounding. A number of recent initiatives aim to prove the value of real-world data for regulatory and clinical decision-making. Funded and sponsored by the public [FDA (Silver Spring, MD, USA); US National Heart, Lung, and Blood Institute (Bethesda, MD, USA); and Harvard University (Cambridge, MA, USA)] and private institutions [Aetion Inc. (New York, NY, USA)], RCT DUPLICATE73 is perhaps the best known of these initiatives. Another initiative, the LEGEND study,86 conducted by the OHDSI collaboration, has reported preliminary results in the area of hypertension. In a multidatabase, multidrug, PS-based analysis, the authors found that well-performed pharmacoepidemiological analyses provided findings that were highly consistent with previously performed head-to-head trials (slides available for review86). To our knowledge, all of these initiatives attempt to replicate the findings of RCTs for the study of medicines, but none has attempted to mimic surgical or medical device trials.

Our stage 1 findings show that the replication of surgical RCTs shares some challenges with studies replicating drug or medicinal product RCTs, but also has its own challenges. In common with the replication of drug RCTs, we found that even in a relatively pragmatic, post-marketing RCT, such as TOPKAT, a good proportion of actual NHS patients would not have been eligible. About 37% of patients undergoing TKR and 35% of patients undergoing UKR in the NHS would not have been eligible for TOPKAT based on their clinical characteristics. There is clearly a space for real-world data to contribute useful information on the risk–benefit and cost-effectiveness of medical devices and alternative surgical approaches and procedures for the many patients (here around one in three) for whom no RCT-derived evidence is available. UTMoSt stage 2 delivered the best available information in the absence of RCT data for NHS patients who were not eligible for TOPKAT. The results are summarised in Study conclusions: UTMoSt stage 2.

More worryingly, the generalisability of surgical and medical device evaluation RCTs can be limited by inclusion criteria that dictate which surgeons and treatment centres are eligible to participate. Surgical RCTs tend to require surgeons with a certain level of expertise with both treatments under study and seek clinicians who are in equipoise when deciding what treatment is best for the eligible patients. TOPKAT in particular used an experience-based design, including surgeons who had carried out ≥ 10 procedures (UKR or TKR) in the previous year of the same type as the allocated treatment. 28

Restricting UTMoSt analyses to participants operated on by surgeons with such levels of experience excluded another 9% of possible TKR participants and one in three TKR surgeons from UTMoSt stage 1. Probably because of the lower uptake of UKR, the same criterion excluded almost half of the possible UKR participants and 64% of UKR surgeons from UTMoSt stage 1. TOPKAT surgeons had completed many surgeries in their careers before TOPKAT: a median of 100 (IQR 50–200) UKR procedures and 300 (IQR 260–400) TKR procedures. 87 Although these numbers are not directly equivalent to the previous-year surgery counts that we used, they do indicate that the surgeons participating in TOPKAT probably had greater expertise than strictly required for participation. For illustrative purposes, we reported the impact of restricting the analysis to operations performed by surgeons who had performed ≥ 30 and ≥ 50 surgeries of the same type in the previous year. These criteria excluded, respectively, > 80% and > 90% of the UKR patients operated on in the NHS who were initially eligible for UTMoSt stage 1. We discuss the impact of restricting to expert surgeons below.

Our stage 1 findings demonstrated that some (but not all) of the previously approved methods for the study of drug safety and post-marketing comparative effectiveness research can also be applied to the study of implantable devices and surgical procedures. Our results showed that some PS methods could reliably approximate the TOPKAT primary outcome results (postoperative OKS). Methods that estimated the ATEs (PS stratification and IPW) more closely approximated the TOPKAT findings than other tested methods. In the full cohort analysis, only PS stratification based on the distribution of the PS in the UKR (exposed) cohort, PSS_exp, passed all of the proposed diagnostics and was classified as able to replicate the TOPKAT findings. PSS_whole and IPW came close to passing these diagnostics.

The sensitivity analysis was restricted to patients operated on by surgeons who would have been eligible for TOPKAT, based on their experience with the index surgery, resulting in findings much closer to those seen in the RCT, with ATEs of 1.32 (95% CI 0.32 to 2.33) (IPW) and 1.37 (95% CI 0.54 to 2.20) (PS stratification) compared with 1.91 (95% CI 0.20, 3.62) in TOPKAT, all in favour of UKR over TKR. PSS_whole, PSS_exp and IPW were, therefore, deemed valid and taken forward into UTMoSt stage 2.

As well as the effect of surgeon experience on the PROM OKS, we found strong evidence of an interaction between surgeon experience and the association between type of surgery (UKR vs. TKR) and 5-year revision risk. Using the three validated methods, we demonstrated that the excess risk observed among patients undergoing UKR in the whole cohort decreased dramatically when UKR was performed by surgeons with more UKR experience. UKR patients in the whole cohort had more than double the risk of 5-year revision surgery than TKR patients. However, this increased risk dropped to around 40–50%, and was no longer significantly different from TKR patients’ risk when the analysis was restricted to patients operated on by surgeons who had performed ≥ 50 surgeries of the same type in the previous year. This finding came closest to replicating the TOPKAT findings, for which the estimated odds ratio for revision was 1.40 (95% CI 0.50 to 4.00) for UKR versus TKR, but was limited by low statistical power and, therefore, needs further research in other settings.

These findings have both methodological consequences for future research in the field and clinical relevance. Given the observed interaction between the surgeons’ experience and the obtained health outcomes (both OKS and revision risk), we suggest that UKR surgery be centralised in specialised treatment centres and provided by specialist surgeons. Our results suggest that patients will have the best possible results when their UKR surgeries are performed by surgeons who perform ≥ 50 UKR surgeries per year. This is equivalent to around one UKR surgery per working week. Conversely, UKR surgeries performed by surgeons who have performed ≤ 10 UKRs in the previous year may have suboptimal patient benefit and could even lead to higher risks of revision than TKR surgery.

Study conclusions: UTMoSt stage 2

UTMoSt stage 2 provided evidence on the comparative effectiveness, safety and cost-effectiveness of UKR compared with TKR for patients with multiple comorbidities, as measured by an ASA grade of 3 or 4. These patients would not have been eligible for TOPKAT, and it is unlikely that there will be a follow-up trial to include this subpopulation. To our knowledge, the results summarised here are the best-quality data available to date for the approximately 15–20% of patients who undergo knee replacement surgery in the NHS while having relatively poor health status.

The stage 2 analyses included a much smaller number of participants than stage 1: 2256 UKR patients and 57,682 TKR patients, of whom only 145 UKR patients and 23,344 TKR patients contributed primary outcome data. However, the safety analyses included almost 10 times more UKR patients and more than 200 times more TKR patients than TOPKAT.

Given that no RCT has included people comparable to the participants of UTMoSt stage 2, there are no gold standard data available for comparison. We, therefore, judged the performance of our analytical methods by their ability to minimise confounding for the variables available in our analytical data set. These analyses were still limited by the effect of potential unobserved confounders that were not recorded in any of the three linked data sources used for UTMoSt (NJR, HES and NHS PROMs database).

Of the tested methods, PS stratification based on the whole cohort most efficiently minimised confounding for the primary outcome analysis. PS stratification based on the UKR cohort and IPW led to unacceptable imbalances in some confounders and required double adjustment for the imbalanced variables. The safety analysis included a much larger population than the primary outcome analysis. The three methods successfully achieved balance for all of the known confounders available in the UTMoSt data set for this larger analysis.

When using the preferred method of PS stratification based on the UKR cohort, UKR had similar effectiveness (compared with TKR) to that observed in TOPKAT and UTMoSt stage 1, with an ATE of 1.83 (95% CI 0.10 to 3.56) OKS points in favour of UKR. Although statistically significant, this increased patient-reported benefit for UKR over TKR is not likely to be clinically relevant. After double adjustment for unresolved imbalances, PSS_whole yielded an estimate of 1.82 (95% CI 0.10 to 3.56) and an IPW estimate of 1.00 (95% CI –1.28 to 3.27). In summary, UKR had similar comparative effectiveness in patients with severe systemic disease and/or substantial functional limitations, as defined by an ASA grade of ≥ 3, as in the overall population. There were small, clinically irrelevant differences in postoperative OKS between UKR and TKR in these patients. Sensitivity analyses of effectiveness restricted to more experienced surgeons could not be performed because of limited statistical power.

Safety considerations, particularly short-term complications, are particularly important for patients with multimorbidity. As the two surgical approaches had no appreciable difference in benefit, any differences in risks would be highly relevant for decision-making. All three analyses suggested a strongly protective effect against postoperative venous thromboembolism for UKR patients, with a 60–67% relative reduction in risk compared with TKR patients. Venous thromboembolism is the most common postoperative complication of knee replacement and affected up to 8% of TKR patients and just below 3% of UKR patients in UTMoSt. Acute myocardial infarction and prosthetic joint infection were also analysed. In the first 90 days after surgery, almost 5% of TKR patients and just over 3.5% of UKR patients experienced myocardial infarction, and 1.9% of TKR patients and 1.8% of UKR patients experienced a prosthetic joint infection. However, no clear statistical difference for either complication could be demonstrated.

TOPKAT’s sample size did not give sufficient power to reliably study postoperative complications. However, our findings are consistent with a recent multinational collaboration study led by EHDEN (of which we are members) and OHDSI. 16 For example, Burn et al. 16 analysed over 32,000 UKR participants and more than 250,000 TKR participants after PS matching and found a 50% reduction in the risk of 90-day postoperative venous thromboembolism, but no significant reduction in the risk of infection. These results are also consistent with a meta-analysis published in BMJ in 2019,88 which found that UKR was associated with a 60% reduction in the risk of postoperative venous thromboembolism compared with TKR.

The UTMoSt stage 2 results suggested that the relative effects of UKR and TKR on short-term postoperative complications in patients with severe systemic disease were consistent with those seen in previous literature for the wider population. UKR seemed to be safer in the short term and resulted in a lower (about 40–50% reduced) risk of postoperative venous thromboembolism in the 90 days after knee replacement surgery. This is highly relevant for patients and clinicians, as such complications can have deleterious effects on patients with baseline comorbidity.

We also assessed the effects of UKR (vs. TKR) on long-term consequences, 5-year revision surgery and mortality. UTMoSt stage 2 showed that, as expected, patients with complex health needs were more likely to die than have revision surgery. The 5-year cumulative mortality rate was 24% for UKR patients and 37% for TKR patients, compared with cumulative revision rates of 13% and 5%, respectively. Survival analyses found that UKR was associated with an almost threefold higher revision risk than TKR in these patients with systemic comorbidity, but with > 30% reduction in all-cause mortality. However, these analyses were hampered by a lack of reliable information on post-revision mortality, as the analyses were censored at the earliest of both events, and on cause of death. More data are, therefore, required to elucidate how UKR (vs. TKR) reduced long-term mortality and how the observed excess revision risk for UKR patients affects subsequent (post-revision) risk of death.

TOPKAT found that UKR had a lower health-care cost (to the NHS) and better cost-effectiveness at 5 years post surgery,5 with an additional benefit of 0.24 QALYs and a cost-saving of £910 per procedure over TKR. In UTMoSt stage 2, we used an economic evaluation to compare the cost-effectiveness of UKR and TKR for patients with severe systemic disease, as defined by an ASA grade of ≥ 3, who would have been excluded from TOPKAT. We used the three validated PS methods to minimise confounding, then analysed cost-effectiveness using similar health economics methods to those used in TOPKAT. These analyses demonstrated that UKR had an average cost of £6246 and TKR had a slightly higher average cost of £6627. Although UKR was associated with an increased revision risk, the cost of a revision after UKR was just over £5100, which was substantially lower than the cost of over £9100 associated with a revision after TKR. After discounting, UKR had a mean gain in quality of life within 5 years of 2.24 QALYs, higher than the 1.87 QALYs for TKR. The UTMoSt cost-effectiveness analysis suggested that UKR dominated TKR for patients with substantial comorbidity (ASA grade of 3 or 4), as it was more beneficial and less expensive.

The analyses performed in UTMoSt stage 2 were limited by the potential for residual confounding and information bias related to the use of data routinely collected for clinical purposes rather than for research purposes. However, it is unlikely that better data will be obtained from randomised studies any time soon. The striking finding that UKR was dominant over TKR for patients with multiple comorbidity should guide future provision of care in the NHS.

Public and patient involvement

A patient representative for the National Rheumatoid Arthritis Society was a co-applicant for the grant application. She assessed the study’s topic and relevance. She was involved in co-investigator meetings for the study, during which the study progress and results were assessed and evaluated. She has not raised any concerns about the work. We also had a plan to include a patient and public involvement (PPI) representative in the Study Steering Committee. However, despite a substantial effort by Versus Arthritis, no PPI representative was recruited.

Implications for future research and clinical practice

The results of UTMoSt stage 1 have clinical and methodological implications. Despite challenges inherent in the nature of the data used for these analyses, our findings suggest that real-world evidence and some PS methods can reliably mimic surgical RCTs. This is of fundamental importance and is very timely, as coming changes in the regulation of medical devices will probably require comprehensive observational post-marketing surveillance.

Key recommendations arising from stage 1 for future research include:

• PS stratification and IPW are useful for minimising confounding when evaluating the comparative effectiveness and risks of alternative medical devices and surgical procedures. PS matching, PS adjustment and IV analyses should be used with caution as they failed to replicate the RCT results in our study.

• More methodological research is needed to produce guidance on which analytical methods are preferable in different scenarios and circumstances for the post-marketing surveillance of medical devices and surgical epidemiology. Real-world evidence studies will continue to emerge in the absence of equivalent RCTs. It is likely that observational post-marketing safety studies will grow in number with upcoming European and global regulations, creating an urgent need for better guidance on the best use of analytical methods when evaluating surgery and implantable medical devices.

• Future attempts to mimic surgical trials should take into account the eligibility criteria of both patients and surgeons contributing to RCTs, as the patients and clinicians participating in RCTs are not representative of routine NHS care.

The main clinical implication of UTMoSt stage 1 arises from the finding that the potential additional benefits of UKR in terms of reduced risk and increased benefit are achieved only when performed by surgeons with high volume in this surgical technique. Health-care services, including the NHS, should consider centralising the delivery of UKR in specialised centres or by specialised surgeons to maximise the potential cost-effectiveness gains described in TOPKAT and consistently demonstrated in a sub-analysis of UTMoSt stage 1 restricted to surgeons with higher volume of knee replacement surgery. 89

UTMoSt stage 2 also has implications for clinical care. Although most surgeons support the use of UKR for fit young patients, our findings suggest that UKR has similar benefits over TKR for patients with severe systemic disease. We found that patients with severe systemic disease had better patient-reported outcomes (probably not clinically relevant) and dramatically fewer safety events, particularly thromboembolic events, after UKR than after TKR, as seen in fit young patients. Despite an excess revision risk, mortality was also lower among UKR patients. This information should be clearly communicated to patients. Patients with a limited lifespan might prefer a procedure that provides similar benefits but that is potentially safer in the short term, despite the fact that it may lead to higher risk of revision surgery in the long term.

From a NHS perspective, UKR is the preferable option for this patient subgroup where suitable, as it provides better and less expensive care than TKR. From NJR data, we estimate that about 50% of patients undergoing knee replacement would be suitable for UKR, but that < 10% receive it. More strikingly, less than 4% (2890/75,055) of patients with an ASA grade of ≥ 3 underwent UKR in our data set. There is a clear need for NICE guidelines on the use of UKR for patients with multimorbidity in need of knee replacement surgery.

Contributions of authors

Albert Prats-Uribe (https://orcid.org/0000-0003-1202-9153) contributed to data management, cleaning and analysis, and led the drafting of the study report.

Spyros Kolovos (https://orcid.org/0000-0003-3201-1743) contributed to data management, cleaning and analysis, and led the drafting of the study report.

Klara Berencsi (https://orcid.org/0000-0002-2109-6369) contributed to data management, cleaning and analysis.

Andrew Carr (https://orcid.org/0000-0001-5940-1464) contributed to study design.

Andrew Judge (https://orcid.org/0000-0003-3015-0432) contributed to study design.

Alan Silman (https://orcid.org/0000-0001-8426-8925) contributed to study design.

Nigel Arden (https://orcid.org/0000-0002-3452-3382) contributed to study design.

Irene Petersen (https://orcid.org/0000-0002-0037-7524) contributed to study design.

Ian J Douglas (https://orcid.org/0000-0002-8970-1406) contributed to study design.

J Mark Wilkinson (https://orcid.org/0000-0001-5577-3674) contributed to study design.

David Murray (https://orcid.org/0000-0002-0839-3166) contributed to study design.

Jose M Valderas (https://orcid.org/0000-0002-9299-1555) contributed to study design.

David J Beard (https://orcid.org/0000-0001-7884-6389) contributed to study design.

Sarah E Lamb (https://orcid.org/0000-0003-4349-7195) contributed to study design.

M Sanni Ali (https://orcid.org/0000-0002-4192-7908) contributed to study design.

Rafael Pinedo-Villanueva (https://orcid.org/0000-0002-4723-5128) contributed to study design and data management, cleaning, and analysis, and led the drafting of the study report.

Victoria Y Strauss (https://orcid.org/0000-0002-5172-512X) contributed to data management, cleaning, and analysis and led the drafting of the study report.

Daniel Prieto-Alhambra (https://orcid.org/0000-0002-3950-6346) contributed to study design and data management, cleaning, and analysis, and led the drafting of the study report.

All authors provided feedback and critically reviewed the report, and approved the final version of the report for submission.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to anonymised data may be granted following review.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care.

Eligibility criteria code lists

Covariates included in the propensity score