Modelling approaches for histology-independent cancer drugs to inform NICE appraisals: a systematic review and decision-framework

Food and Drug Administration review of pembrolizumab

The FDA approved pembrolizumab on 23 May 2017 for the treatment of adult and paediatric patients with unresectable or metastatic MSI-H or dMMR solid tumours. The approval is for patients who have progressed following prior treatment and who have no satisfactory alternative treatment options, and for the treatment of unresectable or metastatic MSI-H or dMMR colorectal cancer (CRC) that has progressed following treatment with a fluoropyrimidine, oxaliplatin (Eloxatin, Sanofi-Aventis, Paris, France) and irinotecan (Campto, Pfizer, New York City, NY, USA). 1

The efficacy of pembrolizumab in patients with MSI-H or dMMR solid tumours was derived from five uncontrolled, open-label, multicohort, multicentre, single-arm studies. Patients received either 200 mg of pembrolizumab every 3 weeks or 10 mg/kg of pembrolizumab every 2 weeks. Treatment continued until unacceptable toxicity or disease progression (up to a maximum of 24 months of treatment).

A total of 149 patients with MSI-H or dMMR cancers were included across the five clinical trials. The median age of patients was 55 years; 98% of patients had metastatic disease and 2% of patients had locally advanced, unresectable disease. In total, 90 (60%) out of the 149 patients had CRC, with the remainder diagnosed with other tumour types. The median number of prior therapies for metastatic or unresectable disease was two.

The identification of MSI-H or dMMR tumour status was prospectively established for the majority of patients (n = 135/149) using local laboratory-developed polymerase chain reaction (PCR) tests for MSI-H status or immunohistochemistry (IHC) tests for dMMR. Tumours from the remaining 14 patients were retrospectively identified as MSI-H using a central laboratory-developed PCR test.

The primary end point used for the FDA review was ORR, as assessed by blinded independent central radiologists (BICRs) using the Response Evaluation Criteria in Solid Tumours (RECIST) guidelines (version 1.1). The ORR was 39.6% (95% CI 31.7% to 47.9%). DoR was considered as a key secondary end point. Although the median DoR was not reached, 78% of responding patients had a DoR of ≥ 6 months. Overall, the safety profile of pembrolizumab was considered acceptable relative to durable responses observed in patients with advanced MSI-H/dMMR cancers.

A total of 16 tumour types were included in the combined data set. Consistent responses were reported between subjects with gastrointestinal (GI) cancer (i.e. CRC, small bowel, gastro-oesophageal junction and pancreas) and subjects with non-GI MSI-H cancer, with ORRs of 36.8% (MSI-H GI, n = 125) and 41.7% (MSI-H non-GI, n = 24). However, the FDA noted that some of the tumours (e.g. breast, prostate, sarcoma and renal cell) were represented by only one or two patients and that there was uncertainty as to whether or not the results apply to all disease types with MSI-H/dMMR status.

The key question considered by the FDA within their review was whether or not the presence of MSI-H/dMMR represents a unique biomarker that predicts a consistent response to pembrolizumab and similar clinical benefit across different primary tumours. In addressing this question, the FDA highlighted specific features associated with MSI-H/dMMR that are common across primary cancers, including increased lymphocytic infiltration and an increased mutational tumour burden with non-synonymous mutations. These features were noted to have been previously identified as correlating with an increased response to checkpoint inhibitors, including pembrolizumab, in tumours that had not been assessed for MSI-H or dMMR. Based on these common histological features, the FDA concluded that there was a strong biologic rationale that MSI-H/dMMR cancer represents a specific subpopulation of patients with cancer who are likely to derive clinical benefit from pembrolizumab.

Pembrolizumab was approved by the FDA for this indication under accelerated approval based on ORR and DoR. Despite a common biology among MSI-H/dMMR tumours, the FDA review also highlighted other differences among patients with different types of cancer that could influence the response to therapy with pembrolizumab (e.g. the degree of immunosuppression related to previous cytotoxic chemotherapy). Given the uncertainties that remain concerning the generalisability of the results to all disease types with MSI-H/dMMR status, a condition of the approval requires the sponsor to submit results of further studies to better characterise the response rate and its duration. These studies are required to include 124 patients with CRC and at least 300 patients with non-CRC, including a sufficient number of patients with prostate cancer, thyroid cancer, small cell lung cancer (SCLC) and ovarian cancer, as well as 25 children.

The FDA review noted that further randomised trials will be challenging to conduct in the histology-independent setting given concerns over equipoise. The FDA also questioned whether or not it would be scientifically appropriate to ‘lump’ all tumour types together into a single randomised trial given the different natural histories. The FDA also noted that, although response may not be entirely predictive of effects on clinical benefit, checkpoint inhibitor therapy, including pembrolizumab, has demonstrated beneficial effects on OS with similar response rates in other tumour types.

In the absence of a companion diagnostic test for the identification of MSI-H or dMMR tumour status, the FDA review noted the uncertainties regarding the use of laboratory-developed tests. These uncertainties concerned the rate of false positives in IHC tests for dMMR and false negatives in PCR tests for MSI-H, and whether or not performance characteristics may differ by the site of the primary tumour. Given these uncertainties, additional post-marketing studies were requested to assess and establish the performance characteristics of MSI-H and dMMR tests.

Food and Drug Administration review of larotrectinib

On 26 November 2018, the FDA granted accelerated approval to larotrectinib for adult and paediatric patients with solid tumours who have a neurotrophic tyrosine receptor kinase (NTRK) gene fusion without a known acquired resistance mutation; who are metastatic or for whom surgical resection is likely to result in severe morbidity; who have no satisfactory alternative treatments; or whose cancer has progressed following treatment. 9

As agreed with the FDA, the submission was supported by pooled safety and efficacy data from the first 55 patients who were enrolled in three multicentre, open-label single-arm studies. These studies enrolled subjects with solid tumours harbouring a NTRK fusion if they met the following criteria:

• documented NTRK fusion, as determined by local testing

• non-central nervous system (CNS) primary tumour with one or more measurable lesions at baseline, as assessed by RECIST 1.1

• received one or more doses of larotrectinib.

The ORR, which was determined by an Independent Review Committee (IRC), was used as the primary end point for efficacy. DoR was a secondary end point, which was defined as the number of months from the start date of partial response (PR) or complete response (CR) to the date of disease progression or death, whichever occurred earlier.

Assuming that the observed ORR was ≥ 50%, a sample size of 55 patients was selected to provide 80% power to achieve a lower boundary of the two-sided 95% exact binomial CI about the estimated ORR exceeding 30%. Ruling out a lower limit of 30% for ORR was considered clinically meaningful. All patients were required either to have progressed following previous systemic therapy for their disease or to have required surgery with significant morbidity for locally advanced disease. The data cut-off time point for the primary analysis was July 2017, approximately 6 months after enrolment of the 55th patient.

The pooled sample included 12 tumour sites, of which the most frequent were salivary gland tumours (22% of patients), soft tissue sarcoma (20%) and infantile fibrosarcoma (IFS) (13%). More common tumours, such as lung or colon cancer, were represented less (n = 4 patients; 7% each) because they tend to rarely express a NTRK fusion. The sample was also heterogeneous in terms of prior cancer therapy, with patients having undergone different types of therapy (i.e. surgery, radiotherapy, systemic therapy) and different numbers of previous lines of therapy (45% having undergone one to two lines, and 35% having undergone three or more lines).

At the time of data cut-off, the estimated ORR was 75% (95% CI 61% to 85%), including 22% of patients with a CR and 53% of patients with a PR. Although the median DoR had not been reached, 30 out of 41 (73%) responders had a DoR of at least 6 months and 16 out of 41 (39%) responders had a DoR of at least 12 months.

The clinical and statistical review included an exploratory subgroup analysis that was performed by study, demographics and tumour type. Based on these analyses, the effectiveness of larotrectinib was reported to be reasonably similar irrespective of age, sex and race; however, no definitive conclusions were made given the limited sample size. A numerical difference in ORR was reported among patients with different tumour types, NTRK gene fusions or status of radiotherapy. Across different tumour types, three tumour types had at least seven patients: salivary gland (n = 12), soft tissue (n = 11) and IFS (n = 7). The ORR in these tumour types was reported to be higher than 75%. Conversely, it was reported that the ORR in colon cancer appeared to be lower (one out of four patients). No response was reported in the two patients with primary CNS lymphoma.

The FDA review concluded that, although the results showed that treatment with larotrectinib results in durable overall responses in patients with a variety of tumour types, there was insufficient clinical experience to conclude that the response rates achieved with larotrectinib were consistent across all NTRK fusion cancers.

A key issue addressed in the review was the potential risk that larotrectinib could be ineffective in some tumour types, even in the presence of a NTRK fusion. The FDA concluded that the risk of ineffectiveness was low owing to the strong rationale presented by the company, which was supported by clinical and non-clinical data. The strength of the evidence was assessed against the following criteria: the ability of the biomarker to identify a population with common features, the similarity of response across tumour types and the ability to reliably identify the biomarker at the screening phase.

The FDA considered the totality of evidence presented by the sponsor to be sufficiently strong to consider pooling the results across trials and patients, supporting a histology-independent indication. The FDA also concluded that, although there was a risk that larotrectinib may be ineffective in some tumours, the level of risk was deemed to be low and was considered acceptable given that the product is approved only for the treatment of patients who have no satisfactory alternative treatment options or whose cancer has progressed following treatment. As a result, the FDA did not consider that patients would be forgoing effective therapies when treated with larotrectinib.

The primary risks of larotrectinib were identified as hepatotoxicity and neurotoxicity. However, these adverse reactions were considered largely manageable and reversible with dose modification or discontinuation. Overall, the toxicity profile of larotrectinib was considered acceptable when considered against the durable effects across different cancer types in patients with limited or no effective treatment options.

The ORR was considered to be a surrogate end point that was reasonably likely to predict benefit, in accordance with the requirements of the accelerated approval process. The clinical effect was deemed to be sufficiently large and the effect was durable, which provided a meaningful advantage over the available therapy for patients with NTRK fusion solid tumours. The population was also considered to have a high unmet medical need given the serious, life-threatening and rare nature of their cancers. However, the FDA specified that the ORR evidence was not sufficiently strong to support a regular approval, given the large number of histological subtypes and the small sample size. This led to a degree of uncertainty regarding the magnitude of the treatment effect of larotrectinib in any single histological subtype.

A key post-marketing requirement is that the company conduct further studies that provide additional data to verify and confirm the clinical benefit of larotrectinib through more precise estimation of ORR and DoR in several specific tumour types [CRC, non-small cell lung cancer (NSCLC), CNS tumours and melanoma]. These tumour types were not well represented in the existing efficacy population. A minimum of 40 patients with cancers other than CRC, NSCLC, CNS tumours, melanoma, soft tissue sarcoma, thyroid cancer, IFS and salivary cancers [e.g. breast cancer, gastrointestinal stromal tumours (GISTs), cholangiocarcinoma and biliary tract cancers] are also required to be studied. ORR and DoR are required as end points and all responding patients are required to be followed for at least 12 months from the onset of response. In addition, a final report is requested from the first 55 patients enrolled with NTRK fusion solid tumours to further characterise the DoR, including follow-up of at least 2 years from the onset of response for responding patients.

Importantly, the FDA concluded that it would not be feasible or appropriate to conduct a randomised trial to demonstrate that larotrectinib improves OS in patients with NTRK fusion. The reasons included the extreme rarity of NTRK fusion cancers, the lack of equipoise in settings without available therapies and the expectations for patient crossover. Consistent with their review of pembrolizumab, the FDA again queried whether or not it would even be scientifically appropriate to ‘lump’ these tumour types together into a single randomised trial, given differences in natural history between different tumour sites.

The identification of positive NTRK gene fusion status was determined in the clinical efficacy analysis set using next-generation sequencing (NGS) for 91% of patients and fluorescence in situ hybridisation (FISH) for the remaining 9% of patients. The company did not submit an application for an in vitro companion diagnostic device. Despite this, the clinical review team was supportive of approval, citing the availability of a reliable non-companion device and the efficacy of larotrectinib. However, the development and validation of a companion diagnostic test by the sponsor was agreed as part of a series of post-marketing commitments.

Food and Drug Administration review of entrectinib

On 15 August 2019, the FDA granted accelerated approval to entrectinib for adults and paediatric patients aged ≥ 12 years with solid tumours who have a NTRK gene fusion without a known acquired resistance mutation, are metastatic or for whom surgical resection is likely to result in severe morbidity, and have progressed following treatment or have no satisfactory standard therapy. 10

This indication was approved by the FDA under accelerated approval based on ORR and DoR. The submission was supported by pooled efficacy and safety results from the first 54 adult patients with unresectable or metastatic solid tumours harbouring a NTRK fusion enrolled across three single-arm studies. All patients were required to have cancer that progressed following effective systemic therapy for their disease, if available, or would have required surgery with significant morbidity for locally advanced disease.

The median age of the patients was 55 years. The most common tumours (≥ 5%) were lung cancer (56%), sarcoma (8%) and colon cancer (5%). In total, 96% of patients had metastatic disease and 4% had locally advanced, unresectable disease. All patients had received prior treatment for their cancer, including surgery, radiotherapy or systemic antineoplastic therapy.

The ORR and DoR, as assessed by BICR using RECIST v1.1, were the primary end points. PFS, as assessed by BICR and OS, was included as a secondary end point. The effectiveness of entrectinib in paediatric patients aged ≥ 12 years was established based on extrapolation of data in adult patients with solid tumours harbouring a NTRK gene fusion and pharmacokinetic data in adolescents enrolled in the STARTRK-NG study. 12

In the first 54 patients, the ORR was 57% (95% CI 43% to 71%). This was clinically meaningful because the results excluded a lower bound of the 95% CI for ORR of 30%. At the data cut-off time point (i.e. 31 May 2018), the median DoR was not reached. Among the 31 responding patients, 55% had a DoR of ≥ 6 months and 39% had a DoR of ≥ 12 months.

Exploratory ORR results for subgroups defined by tumour type and by NTRK gene fusion partner were presented. Although there was no formal discussion of these results, a general disclaimer was provided that noted that the subgroup results should be treated with caution owing to the small sample sizes and the single-arm design.

Only limited details were reported for secondary end points. The estimated median PFS was reported to be 11.2 months (95% CI 8.0 to 14.9 months). Less than 30% of deaths were observed by the clinical cut-off date (31 May 2018), which was considered to be too immature to be considered in the clinical review.

The most serious adverse events reported with entrectinib were congestive heart failure (CHF), CNS adverse reactions, skeletal fractures, hyperuricemia, hepatotoxicity, QT prolongation and vision disorders. Although serious in nature, these events were also reported to be manageable and reversible with dose modification or discontinuation of entrectinib.

The FDA drew similar conclusions for entrectinib to their earlier review of larotrectinib (see Food and Drug Administration review of larotrectinib). Although acknowledging that there was uncertainty regarding the magnitude and durability of the treatment effect of entrectinib in any specific histological subtype of solid tumours, they concluded that the risk of treatment was low, using a similar rationale to that previously described for larotrectinib.

Similar post-marketing requirements were reported for entrectinib to those for larotrectinib. This requires the company to conduct additional single-arm studies to obtain data to verify and further characterise the clinical benefit of entrectinib in an adequate number of patients with common histological tumour types, including colon cancer and melanoma. Additional post-marketing requirements also include the conduct of additional studies to further characterise the risks of CHF and skeletal fractures with entrectinib.

Heterogeneity of effect in basket trials

Heterogeneity of effect across different baskets is a key concern. One way to account for this is to analyse each basket separately as if it was an independent study. For example, a basket study of vemurafenib (Zelboraf, Roche, Basel, Switzerland) in multiple non-melanoma cancers with BRAF V600 mutations used an adaptive Simon two-stage design20 with stopping rules defined independently for each basket, and considered a response rate at week 8 of 15% to be low, a response rate of 45% to be high and a response rate of 35% to be low but still indicative of efficacy. 32 They found that not all tumour types responded homogeneously to treatment, with some tumour types not meeting the prespecified criteria for response. Similarly, the CUSTOM trial31 used Simon’s optimal two-stage design, defining p₀ = 0.3 and p₁ = 0.6 based on previous literature. The trial aimed to identify targets for molecular biomarkers in NSCLC, SCLC and thymic malignancies and to simultaneously evaluate five different targeted therapies in each of the three histologies, which resulted in a total of 15 study arms. A high response rate to erlotinib (Tarceva, Roche, Basel, Switzerland) was identified from only 15 NSCLC patients with an epidermal growth factor receptor (EGFR) mutation, but another therapy, selumetinib (Koselugo, AstraZeneca, Cambridge, UK), failed to achieve a promising response in patients with Kirsten rat sarcoma (KRAS) viral oncogene homologue mutations. 31

However, a separate analysis of each basket does not allow for the possibility that some subgroups may react similarly to the drug, particularly if they share a common biomarker that the novel therapy is targeting. By analysing each basket separately, efficiency may be lost by not allowing information gathered from one basket to inform the next, thus increasing the required sample sizes in each basket. In practice, many standard Phase II designs will ignore potential heterogeneity and pool all patients for analysis, which, in effect, ignores the specific basket-defining tumour characteristics (e.g. histology) and assumes equal efficacy across all baskets. 35 If this approach is taken, trial planning and analysis are similar to a standard Phase II trial and, for example, Simon’s two-stage design can be used. Although allowing analysis with a much smaller number of included patients, pooling all patients ignores the potential for heterogeneity across baskets and effectively assumes that it is zero, which can miss treatments that are active in only some baskets36 and can lead to large biases in overall estimated effects. In addition, if the drug is truly active or inactive in all baskets, this will be an inefficient design. 37,38

Frequentist adaptive designs for basket studies that try to acknowledge this potential for heterogeneity across baskets have been proposed. In the context of Phase II studies with heterogeneous populations, a design that tests global response across the whole population, while allowing a different response for each subgroup, was proposed by London and Chang. 39 Simon’s two-stage design was extended to use a more flexible strategy that both tests each subgroup and tests the combined population, which allows the trial to stop if either a subgroup or the combined population show futility, that is the inability of the study to achieve statistically significant results, at prespecified thresholds (that are not necessarily the same). 40 Negative results in one subgroup would lead to stopping recruitment in that basket alone, unless the combined response for the whole population was below the acceptable threshold. This design leads to smaller sample sizes than separate analyses of each basket when the drug is inactive across all subgroups and to more power when there is activity in all subgroups. It also retains the individual tests for each subgroup, which allows the identification of promising baskets. This design requires prespecification of the expected response rates and prevalence in each subgroup to specify the expected response rate in the overall population. Although the average prevalence in the clinical population may be known, owing to the often small samples recruited, the observed prevalence as the trial enrols patients may be quite different. A design that allows the rejection values to be adjusted depending on the observed prevalence in the trial was proposed by Jung et al. 41

Cunanan et al. 42 later proposed an efficient study design for the specific scenario of the typical basket trial in oncology, which assesses the homogeneity of the baskets’ response rates at an interim analysis, aggregating the baskets in the second stage (i.e. full borrowing of information) if results suggest effectiveness in all or most baskets, or treating each basket separately (i.e. no borrowing) otherwise. Their basic premise is that the design can be made more efficient by aggregating information from separate baskets in which it can be assumed that the drug has similar efficacy, based on an interim analysis. Thus, the second stage of the design could have a much smaller sample size for the same power to demonstrate clinical efficacy. The first stage of the design is based on the parallel, independent two-stage Simon’s design. When each basket has recruited a small number of patients, the heterogeneity in response across baskets is evaluated. If the results support the assumption that the drug’s effects are similar across baskets, either the trial is terminated for futility (if response is low) or a decision is made to continue to the second stage, at which all baskets will be pooled for analysis. If there is evidence of heterogeneity across the baskets, the trial will continue only for those baskets showing a promising level of response and these will be analysed separately at the end of the trial. This type of design answers the overall question of efficacy in the whole population more efficiently when there is evidence of homogeneity at an interim stage, while also shortening trial duration. 43 However, this is at the expense of loss of accuracy at assessing efficacy within each separate basket. 42 A different approach to testing has also been proposed, which replaces the question of whether or not there is response to therapy with the question of whether or not there are differences by tumour type (i.e. across baskets). 44

Although acknowledging the potential for heterogeneity, once a decision has been made on whether or not heterogeneity is present, the analysis proceeds either as separate independent studies for each basket or as a single aggregate study combining all of the baskets. Thus, either complete homogeneity or completely unrelated effects are assumed. A less restrictive assumption is that efficacy is similar (rather than equal or completely different) across baskets, with the different histologies not determining a particular ordering of effectiveness a priori (i.e. the baskets are exchangeable). Bayesian hierarchical models (BHMs)45,46 are particularly suited for this situation because they estimate the heterogeneity and allow information to be borrowed on the effects of the treatment across baskets, increasing precision of estimates compared with analysing all baskets separately, while reducing the chances of obtaining extreme estimates in baskets with few patients. Thall et al. 46 proposed a BHM that produces estimates of efficacy (e.g. probability of response) for each basket that are shrunken towards the mean efficacy (e.g. pooled probability of response) across all baskets. The model is an extension of a Bayesian Phase II design in which the trial is stopped if the posterior probability that the response rate is at least π^* falls below a prespecified cut-off point, and can be applied to both binary and time-to-event (TTE) data. 47 Each basket is assumed to have a different treatment effect (event probability or event rate), θ_j, and these are assumed exchangeable (i.e. similar) and correlated a priori. Specifically, it is assumed that the θ_j follows a BHM, while allowing a separate stopping rule for each basket. Thus, the model will identify subgroups in which results are not promising, which can be dropped at a subsequent stage. Because the effects are assumed to be correlated across baskets, data from each individual basket will provide information on the effects in all of the other baskets, so that, for example, a longer survival time for a patient in a given basket will increase the posterior distributions of all θ_j, on average. In other words, information is borrowed across baskets, which shrinks the observed effects towards the pooled mean effect. Outputs from the resulting analysis include the posterior distributions for the effect (e.g. response or event rate) in each basket, the posterior distributions for the pooled effect across all baskets and the posterior distribution for the heterogeneity across baskets. In addition, a predictive distribution for the effect in a new study sampling baskets from the same overall population can be calculated to reflect the full degree of uncertainty owing to both the sample size and the observed heterogeneity in effects across the observed baskets. A Phase II trial of imatinib in 10 histological subtypes of sarcoma used this design: accrual within a sarcoma subtype would stop if it was unlikely that its response rate was at least 30%. 36,46,48

The BHM was shown to be a better design for a single-arm, non-randomised trial with a tumour response end point when there is a possibility of different effects in different subgroups of patients than Simon’s optimal two-stage design and the Bayesian adaptive design with no borrowing. 49 However, the hierarchical borrowing can make it more difficult to find a single basket in which the treatment is promising, although it is more likely than the other designs to correctly conclude futility or efficacy.

Any borrowing and precision gains from a BHM are advantageous only if the exchangeability assumption is reasonable. An approach for assessing homogeneity at an interim analysis and proceeding with a BHM in the second stage only if efficacy is deemed reasonably homogeneous has been proposed. 50 This approach avoids problems caused by implementing a complete pooling model at the second stage42 or proceeding with a fully exchangeable BHM when there is evidence of outlying baskets.

Hierarchical designs have been criticised when there is insufficient information in the outcome data to determine whether or not borrowing across subgroups is appropriate. 36,51 In addition, unknown between-subgroup heterogeneity, which drives the amount of borrowing, poses a major problem when the number of baskets is small (less than 10, as a rule of thumb)36 because it cannot be well inferred from the data and the results will be sensitive to model specification, in particular to the specification of the prior distribution for the borrowing parameter. 36,52 Alternatives to complete pooling or borrowing across all baskets have been proposed, which extend the BHM to allow borrowing of information across similar baskets while avoiding too optimistic borrowing for extreme baskets. 51,53–57

A model that allows non-exchangeable prior distributions to be specified was proposed for the scenario in which it is not expected a priori that all subgroups will be exchangeable. For example, some tumour types may be associated with a better or worse prognosis and their response to treatment is expected to differ. Different models can be used to implement this assumption: we can accept that a particular tumour characteristic (e.g. prognosis) defines exchangeability so that different categories are formed and exchangeability is allowed only between tumours in the same category (e.g. poor, intermediate and good prognosis), or we can treat the appropriate grouping as a random quantity to be estimated from the data, indexed by a categorical covariate of interest (e.g. prognosis). 53 Thus, the estimation of the treatment effect for a particular subgroup borrows more strength from other subgroups that, according to the prior beliefs, are more likely to be exchangeable, but the models allow the data to correct any prior beliefs that are not supported by the available data. When there is no a priori information on which subgroups might be exchangeable or not, an exchangeable–non-exchangeable model51 allows for selected special exchangeability patterns specified in the model to be determined by the treatment response data. This model extends the BHM to allow θ_j to be either exchangeable with some of the other subgroups or non-exchangeable with any of them, in which case the effect will be estimated independently of all other subgroups. Prior weights for the exchangeable probability of each subgroup are specified to reflect an a priori belief that a subgroup behaves systematically differently to the others. Essentially, the model determines whether some borrowing or no borrowing of information should be carried out across subgroups. Outputs include a global heterogeneity parameter across subgroups and mixture weights that describe the similarity of subgroups in the exchangeable component of the model, while also identifying subgroups that behave differently (i.e. show a low probability of being exchangeable). Although a pooled mean effect for the exchangeable component of the model can be obtained, the focus is on the effects for each individual subgroup, which incorporate different levels of borrowing according to the model. The prior distributions specified for the heterogeneity parameter and for the exchangeability weights can influence the results and need to be specified carefully. The use of this model for trial design requires careful consideration of the specification of the prior distributions and mixture weights, but has been found to perform well in various scenarios. 51 Extensions of these ideas to incorporate more information and, thus, improve performance of the trial design or simplify computation have been proposed. For example, the Bayesian latent subgroup trial design54 defines different latent subgroups within which more borrowing is allowed by jointly modelling biomarker measurements and treatment responses. This allows grouping of different cancers according to biomarker measurements routinely collected during a trial, effectively using internal trial information to inform the adaptive borrowing, which determines the decision to proceed to the next stage. Fujikawa et al. 56 proposed a Bayesian basket design that borrows information across the subgroups that have the most similar posterior distributions based on a prespecified threshold of similarity, which is simple to compute. Decisions can be made at the interim stage to stop or continue with the trial and this design can also determine which subgroups show efficacy in the final analysis, based on predefined criteria. Unlike the fully exchangeable BHM, in these models obtaining and interpreting predictive distributions of effects is not meaningful given that we can no longer reasonably assume that a new tumour type (subgroup) would have been sampled from the same distributions as the observed subgroups (i.e. we cannot assume that all subgroups are exchangeable).

Owing to the increased number of parameters being estimated, the hierarchical approach may increase uncertainty unnecessarily if response to treatment is indeed homogeneous across all subgroups. Therefore, when there is a strong rationale for expecting a uniform level of response it may be preferable to use a simple pooling of information across subgroups. 36 However, a priori assumptions of homogeneity in trial design or analysis need to be carefully justified because, in most cases, basket trials include patients with very clinically heterogeneous tumour types. In addition, the available empirical evidence does not generally support the assumption of homogeneity of activity of drugs across different histologies.

Previous basket trials have shown heterogeneity in the effectiveness of agents across tumour types, which lends support to the a priori assumption that effects may be heterogeneous. A recent trial32 of vemurafenib in 122 patients with BRAF V600–mutated cancers across multiple tumour types (including CRC, NSCLC, Erdheim–Chester disease and Langerhans’-cell histiocytosis, primary brain tumours, cholangiocarcinoma and anaplastic thyroid cancer) found evidence of response in some tumour types, including NSCLC and Erdheim–Chester disease and Langerhans cell histiocytosis, but not in CRC. 32 This heterogeneity in response was also observed in previous separate independent studies, which showed a positive response to vemurafenib in patients with BRAF-positive metastatic melanoma,58 but not in BRAF-positive colon cancer patients. 59 A trial of imatinib,60 a tyrosine kinase inhibitor (TKI), that included 196 patients across 40 different subtypes, found evidence of activity of imatinib in only five malignancies. Another basket trial of imatinib in 10 histological subtypes of advanced sarcoma concluded that, although rare dramatic responses were seen, imatinib was not an active agent in these subtypes, although it had previously shown effectiveness in another subtype of soft tissue sarcoma. 48 Similarly, trastuzumab (Herceptin, Roche, Basel, Switzerland), which is known to be effective in the treatment of women with HER2 (human epidermal growth factor receptor 2)-positive breast cancer,61 was not shown to be effective in HER2-positive recurrent endometrial cancer62 or HER2-positive NSCLC. 63 This evidence suggests that the treatment effects in different cancer types may not be exchangeable. Therefore, the design of basket trials should allow for the possibility of heterogeneity in treatment effects across tumour types, opting only for a design that assumes homogeneity in very special cases or where data from previous stages clearly support it.

Review question

This systematic review sought to address the following research question: ‘What is the strength of the association between response outcomes and PFS, TTP or OS across different types of cancer (primarily advanced or metastatic), based on meta-analyses or meta-regression studies assessing the statistical relationship between these outcomes?’.

Inclusion and exclusion criteria

The inclusion and exclusion criteria for the review are shown in Table 1. Inclusion was restricted to articles that reported meta-analyses and meta-regressions across multiple studies and that reported the strength of association between response outcomes (ORR, CR, PR, VGPR or DoR) and PFS, TTP or OS. The included meta-regressions could themselves include RCTs and/or single-arm studies. However, individual reports analysing single trials or single cohorts were excluded. Included meta-analyses could report absolute associations and/or treatment effect associations. These associations had to be reported as a correlation coefficient (e.g. Pearson r or Spearman’s r_s) and/or a coefficient of determination (R²) between relevant outcomes.

Studies of any cancer and any treatment were included. The review focused mainly on studies of advanced or metastatic cancers (and/or treatment with palliative intent) because these studies were more likely to report PFS and OS. However, studies reporting relevant outcomes were included even where the stage was not specifically restricted to advanced/metastatic disease for all patients or where this was unclear (this applied particularly to haematological cancers). Studies were excluded if they explicitly referred to adjuvant or neo-adjuvant treatment or treatments that are given with curative intent.

Search strategy

Five databases [MEDLINE, EMBASE™ (Elsevier, Amsterdam, the Netherlands), Web of Science™ (Clarivate Analytics, Philadelphia, PA, USA), the Cochrane Database of Systematic Reviews and Cumulative Index to Nursing and Allied Health Literature (CINAHL)] were searched from inception to March 2019. Search terms included cancer terms AND response terms AND terms for PFS, TTP and/or OS AND terms for regression, correlation, prediction, association or relationship AND terms for end point and/or surrogate. Search results were limited to the English language and to studies undertaken in humans. The MEDLINE search strategy is provided in Appendix 3.

In addition, a citation search was undertaken based on two existing meta-reviews77,78 of surrogate relationships; this identified studies that have cited any of the 48 articles included in the review by Fischer et al. 77 and/or any of the 19 articles included in the review by Davis et al. 78 In addition, relevant existing meta-reviews, including Fischer et al. ,77 Davis et al. ,78 Savina et al. 79 and Haslam et al. ,80 and any further reviews identified during searching, were checked for relevant studies.

Study selection process

The titles and abstracts of the articles retrieved by the search were examined by one reviewer and a subset were checked by a second reviewer early in the process, followed by a discussion to ensure that there was consistency in the selection decisions. Full texts were examined by one reviewer and a subset were checked by a second reviewer, with any discrepancies resolved through discussion.

Data extraction

Data were extracted by one reviewer and all data were checked by a second reviewer. The following data were extracted:

• author and date

• cancer type and stage, number of patients, number of included studies and the design of included studies (RCT or single arm and publication dates)

• treatment type, treatment line and other subgroups, as reported

• data type [aggregate-level data or individual patient data (IPD)]

• surrogate and final end points analysed (e.g. ORR to OS)

• response criteria used, if reported (e.g. RECIST)

• measures of outcomes [e.g. hazard ratio (HR), odds ratio (OR), relative risk (RR) or difference between medians]

• statistical methods for correlation and regression, whether weighted, whether adjusted, coefficient reported [e.g. Pearson or Spearman correlation coefficient (r or r_s), regression coefficient of determination (R²)]

• absolute association results (i.e. between absolute values of the surrogate and final end points based on data from individual arms of RCTs or single-arm studies) – correlation coefficient, regression R² and regression equation

• treatment effect association results (i.e. between treatment effects for surrogate and treatment effects for final end points, based on between-group differences from RCTs) – correlation coefficient, regression R² and regression equation

• data, as above, for subgroups

• STE,76 that is the smallest treatment effect on the surrogate that predicts a non-zero treatment effect on the true end point.

Data synthesis

Data were tabulated and described in a narrative synthesis. Plots were constructed to illustrate the reported associations. Some of the included meta-regression studies reported multiple subgroup analyses with differing results. Therefore, for associations between absolute values of end points, the plots show the range of correlation coefficients per study, across all subgroup analyses. Where an included meta-regression study reported on more than one cancer type, these are shown separately on the plots. All types of correlation coefficient were included, for example Pearson’s r and Spearman’s r_s. If no correlation coefficient was reported, Pearson’s r was calculated as the square root of R², if available.

For associations between treatment effects, the plots show the range of regression coefficients of determination (R²) per study, across all subgroup analyses. The plots include both adjusted and unadjusted R² values, as well as values from weighted and unweighted regressions. For studies in which R² was not reported, this was calculated as the square of the Pearson’s (r) correlation coefficient, if available. R² was not calculated from other correlation coefficients, such as Spearman’s r, or where the method of correlation was unclear.

Scoring the strength of association

Two separate sets of criteria have been developed to assess the strength of association between end points. These include the criteria proposed by IQWiG74 (based on the treatment effect association) and the BSES2 criteria75 (based on both absolute and treatment effect associations). In this review, both the IQWiG and the BSES2 criteria were used to assess the strength of association between the surrogate and the final end points.

The IQWiG criteria74 (Table 2) are based on the correlation coefficient (r) for the treatment effect association. Where r was not reported it was calculated as the square root of R², if available. Some slight modifications were made to the IQWiG scoring criteria because the medium score bracket was not clearly defined (see Table 2); these modifications were based on the approach used in the previous review by Savina et al. 79 The IQWiG score was generated based on the magnitude of r, irrespective of its sign (i.e. a negative correlation could generate a high score).

The BSES2 criteria75 (Table 3) require R² values for both the individual and the treatment effect associations. Where R² was not reported, it was calculated as the square of r, if available. BSES2 criteria were used as an adaptation from the original BSES criteria, as described in Savina et al. 79 The original BSES criteria require R² for both individual-level and treatment effect associations and a value for the STE. Given that so few articles report STE, this review used BSES2, which does not require the STE.

Number of included studies

The literature search generated 2829 citations (Figure 1), of which 2630 were excluded during the review of titles and abstracts. In total, 64 references to 63 studies were included in the review. 81–144 The study characteristics for the 63 included studies are shown in Appendix 4. The detailed results of the included studies are shown in Appendices 5 and 6. Studies excluded at the full-text stage, with reasons for exclusion, are listed in Appendix 7.

FIGURE 1.

The PRISMA flow diagram for study inclusion. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Characteristics of the included studies

Full details of the study characteristics for the 63 included studies are shown in Appendix 4 (note that eight references81,109,111,128,130,140,141,144 appear on more than one row because they report on more than one cancer type).

Results of the included studies

Absolute (individual-level) correlation and regression

The range of the absolute (individual-level) correlation coefficients reported in each meta-regression is summarised in Table 6 and illustrated in Figures 2 (for the association between ORR and PFS) and 3 (for the association between ORR and OS). Each horizontal row in the plots illustrates the range of correlation coefficients across all subgroup analyses within a single meta-regression study. Where an included meta-regression reported on more than one cancer type, these are shown separately on the plots. It is worth noting that the meta-regressions varied both in terms of the number of included primary studies (shown as N on the plots) and in terms of the treatment type, line of treatment and precise clinical population; all of these details are provided in Appendix 5, together with correlation coefficients for all individual subgroup analyses.

TABLE 6 - Summary of the absolute (individual-level) correlations per study

AML, acute myeloid leukaemia; CUP, cancer of unknown primary; MM, multiple myeloma; NET, neuroendocrine tumour; NHL, non-Hodgkin’s lymphoma; RCC, renal cell carcinoma.

One study118 of MM reported VGPR/CR to PFS as adjusted R² = 0.64, but this could not be converted to r because it was adjusted.

Notes

Further detail on all studies and outcomes is shown in Appendix 5. The number of studies per outcome may vary from Table 4 given that not all data are in the correct format.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The table includes minor additions and formatting changes to the original table.

Surrogate relationship	Number of studies	Cancer types and references	Range of r or rs across studies and subgroup analyses	Further details
ORR to PFS	12	NSCLC,108,128,141 ovarian,129,135 RCC,126 NHL,117 SCLC,122 MM,118 CRC,115 CUP,125 NET107 and various141	–0.72 to 0.96	See Appendix 5 and Figure 2
ORR to TTP	1	Gastric105	0.41 to 0.56	See Appendix 5
ORR to OS	27	NSCLC,108,112,113,128,131,134,141 CRC,98,115,138 ovarian,129,135 breast,114,127 gastric,105,133 various,123,128,141 pancreatic,100 RCC,81,126 gastro-oesophageal,124 urothelial,81,82 AML,83 SCLC,122 glioblastoma,101 CUP125 and NET106	–0.40 to 1.00	See Appendix 5 and Figure 3
CR to PFS	2	SCLC122 and NHL144	0.22 to 0.83	See Appendix 5
CR to OS	3	NSCLC,112 SCLC122 and gastro-oesophageal124	–0.04 to 0.62	See Appendix 5
PR to PFS	1	SCLC122	0.35 to 0.70	See Appendix 5
PR to OS	1	SCLC122	0.29 to 0.66	See Appendix 5
VGPR/CR to PFS	0	–	a	See Appendix 5
DoR to PFS	0	–	–	–
DoR to OS	0	–	–	–

FIGURE 2.

Correlation (r or r_s) between the absolute (individual-level) value of ORR and the absolute (individual-level) value of PFS. For each study, the plot illustrates the range of correlation coefficients across all subgroup analyses. N represents the number of studies included in each meta-regression. CUP, cancer of unknown primary. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

FIGURE 3.

Correlation (r or r_s) between the absolute (individual-level) value of ORR and the absolute (individual-level) value of OS. For each study, the plot illustrates the range of correlation coefficients across all subgroup analyses. N represents the number of studies included in each meta-regression. AML, acute myeloid leukaemia; CUP, cancer of unknown primary. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Overall response rate and progression-free survival (or time to progression)

The reported correlation coefficients (Pearson’s r or Spearman’s r_s) between absolute ORR and PFS ranged from –0.72 to 0.96, based on multiple analyses within 12 studies across 10 cancer types (see Figure 2 and Table 6; full details in Appendix 5). 107,108,115,117,118,122,125,126,128,129,135,141 Across those studies that report only a single analysis, the correlation coefficient was generally above 0.60; however, some estimates were lower. Confidence intervals around the correlation coefficients were rarely reported (not shown in Figure 2; see Appendix 5). Few separate meta-regressions reported on the same tumour site; therefore, it is difficult to assess whether or not the ORR may be a more reliable surrogate in certain cancer types than others. One study reported on the ORR and TTP (gastric cancer, correlation r_s = 0.41 to 0.56 across subgroup analyses, not shown on the plot). 105

Overall response rate and overall survival

The reported correlation coefficients between absolute ORR and OS ranged from –0.40 to 1.00, based on 27 studies across 15 cancer types (see Figure 3 and Table 6; full details in Appendix 5). 81–83,98,100,101,105,106,108,112–115,122–129,131,133–135,138,141 The CIs around the correlation coefficients, where reported, were generally fairly wide (not shown in Figure 3). The majority of correlation coefficients were above 0.40; however, several estimates were lower. The correlation coefficients reported from multiple analyses within the same study, and those reported across separate studies, did not suggest a clear pattern by cancer type.

Complete response and progression-free survival or overall survival

The correlation coefficients between absolute CR and PFS in two studies of SCLC122 and non-Hodgkin’s lymphoma (NHL)144 ranged from 0.22 to 0.83, while the correlation coefficients between absolute CR and OS ranged from –0.04 to 0.62, based on three studies of NSCLC,112 SCLC122 and gastro-oesophageal cancer124 (see Table 6; full details in Appendix 5).

Partial response and progression-free survival or overall survival

The correlation coefficient between absolute PR and PFS ranged from 0.35 to 0.70 across subgroup analyses within one study of SCLC,122 while the highest correlation coefficient between absolute PR and OS ranged from 0.29 to 0.66 in the same study122 (see Table 6; full details in Appendix 5).

Duration of response and progression-free survival or overall survival

No studies reported on the absolute association between DoR and PFS or OS.

Treatment effect (trial-level) correlation and regression

The range of treatment effect (trial-level) R² values reported in each meta-regression is summarised in Table 7 and illustrated in Figures 4 (for the association between ORR and PFS) and 5 (for the association between ORR and OS). Each horizontal row in the plots illustrates the range of R² values across all subgroup analyses within a single meta-regression study. Where an included meta-regression reported on more than one cancer type, these are shown separately on the plots. It is worth noting that the meta-regressions varied both in terms of the number of included primary studies (shown as n on the plots) and in terms of the treatment type, line of treatment and precise clinical population; all of these details are provided in Appendix 6, together with R² values for all individual subgroup analyses.

TABLE 7 - Summary of the treatment effect (trial-level) R2 per study

BTC, biliary tract cancer; RCC, renal cell carcinoma.

Two studies in CRC92 and pancreatic cancer91 reported Spearman’s correlation coefficients between DoR and OS, ranging from 0.40 to 0.76.

Notes

Further detail on all studies and outcomes is shown in Appendix 6. The number of studies per outcome may vary from Table 4 given that not all data are in the correct format.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The table includes minor additions and formatting changes to the original table.

Surrogate relationship	Number of studies	Cancer types and references	Range of R2 across studies and subgroup analyses	Further details
ORR to PFS	9	NSCLC,84,85,108,140 ovarian,90,135 various130,142 and CRC89	0.18 to 0.94	See Appendix 6 and Figure 4
ORR to TTP	0		–	–
ORR to OS	30	NSCLC,84,85,103,108,109,121,140 CRC,88,89,92,94,136 various,110,120,123,130,142 pancreatic,91,100,116 SCLC,97,104 RCC,95,126 breast,86,99 ovarian,90 prostate,93 BTC119 and soft tissue sarcoma137	–0.08 to 0.84	See Appendix 6 and Figure 5
CR to PFS	1	NHL132	0.45 to 0.93	See Appendix 6
CR to OS	2	Breast99 and SCLC97	0.05 to 0.48	See Appendix 6
PR to PFS	0		–	–
PR to OS	0		–	–
DoR to PFS	0		–	See Appendix 6
DoR to OS	0		a	See Appendix 6

FIGURE 4.

Regression R² between treatment effects (trial level) for ORR and PFS. For each study, the plot illustrates the range of correlation coefficients across all subgroup analyses. N represents the number of studies included in each meta-regression. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

FIGURE 5.

Regression R² between treatment effects (trial level) for ORR and OS. For each study, the plot illustrates the range of correlation coefficients across all subgroup analyses. N represents the number of studies included in each meta-regression. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Overall response rate and progression-free survival

The regression R² values for the treatment effect association between ORR and PFS ranged from 0.18 to 0.94, based on nine studies across four cancer types: NSCLC,84,85,108,140 ovarian cancer,90,135 CRC89 and various solid tumours130,142 (see Figure 4 and Table 7; full details in Appendix 6). The majority of R² values were above 0.40. The R² values that were reported from multiple analyses within the same study and those that were reported across separate studies did not suggest a clear pattern by cancer type. Confidence intervals around the R² values, where reported, were generally fairly wide (not shown in Figure 4; see Appendix 6).

Overall response rate and overall survival

The regression R² values for the treatment effect association between ORR and OS ranged from –0.08 to 0.84, based on 30 studies across 11 cancer types (see Figure 5 and Table 7; full details in Appendix 6). 84–86,88–95,97,99,100,103,104,108–110,116,119–121,123,126,130,136,137,140,142 With the exception of one analysis, all R² values were below 0.60. The R² values that were reported from multiple analyses within the same study and those that were reported across separate studies did not suggest a clear pattern by cancer type. Confidence intervals around the R² values, where reported, were generally wide (not shown in Figure 5).

Complete response and progression-free survival or overall survival

The regression R² for the treatment effect association between CR and PFS ranged from 0.45 to 0.93 in one study of NHL,132 while the regression R² for the treatment effect association between CR and OS within two studies of breast cancer99 and SCLC97 ranged from 0.05 to 0.48 (see Table 7; full details in Appendix 6).

Partial response and progression-free survival or overall survival

No studies reported the treatment effect association between PR and PFS or OS.

Duration of response and progression-free survival or overall survival

No studies reported R² between DoR and OS or PFS. Two studies in CRC92 and pancreatic cancer91 reported Spearman’s correlation coefficients between DoR and OS, ranging from 0.40 to 0.76 (see Table 7; full details in Appendix 6).

Institute for Quality and Efficiency in Health Care and Biomarker-Surrogate Evaluation Schema-2 scores for the strength of association

This section reports the results from the IQWiG and BSES2 scoring for the strength of association between surrogate and final end points. As described in Scoring the strength of association, IQWiG scoring requires a correlation coefficient (r) for the treatment effect association, while BSES2 scoring requires R² values for both the individual-level and the treatment effect associations. IQWiG and BSES2 scores were calculated for all subgroup analyses with sufficient data; therefore, studies reporting more subgroups were more strongly represented in this analysis.

For the IQWiG scores (Figure 6), of 202 analyses (across 63 studies), zero (0%) scored high, 15 (7%) scored medium +, 26 (13%) scored medium, 76 (38%) scored low and 85 (42%) were not evaluable.

FIGURE 6.

Summary of the IQWiG scores across all 202 analyses included in the review. Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Reproduced with permission from Cooper et al. 64 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

For the BSES2 scores (Figure 7), of 202 analyses (across 63 studies), zero (0%) scored excellent, three (1%) scored good, three (1%) scored fair, seven (3%) scored poor and 189 (94%) were not evaluable.

FIGURE 7.

Summary of the BSES2 scores across all 202 analyses included in the review.

Summary of the main findings

1. Use meta-analyses to predict the relationship between the surrogate and the final outcome

As shown in Tables 8 and 9, 14 studies report regression equations for ORR to OS and eight studies report equations for ORR to PFS. These equations could be used together with the observed ORR in the studies of histology-independent therapies to estimate the absolute PFS/OS or the incremental gains in PFS/OS. However, the patient populations included in these studies may not correspond to the populations in the studies of histology-independent therapies in terms of tumour sites or types, and none specifically relate to patients with NTRK fusion-positive cancers (or other relevant biomarkers). From a practical point of view, a number of decisions would be required to apply these analyses within a model: (1) which regression equation to use in instances whereby multiple analyses exist for an individual histology site; (2) the form of regression analysis used to estimate the relationship (i.e. ‘absolute’ regressions that estimate final outcomes for an individual treatment group or ‘trial-level’ equations that predict the treatment effects between groups); and (3) how to model the surrogate relationship where no studies exist for an individual histology site. In addition, concerns regarding the strength of the relationship between ORR and PFS/OS within the tumour sites under consideration should be borne in mind.

It has been suggested that the stringent application of criteria for surrogate validation based on correlations may not be important and that predictions may still be made even where the association is weak, provided that they reflect all uncertainty surrounding the treatment effects. 146 In addition, NICE technical support document (TSD) 20146 notes that the meta-regression approaches included in this review are limited because they ignore the uncertainty associated with the treatment effect on the surrogate end point (which is treated as a fixed covariate in the analysis), the consequence being that predictions based on these regression analyses will fail to fully reflect that uncertainty. Recently developed methods, such as the bivariate random-effects meta-analysis (BRMA) model and its extensions,146,147 provide an approach for both the validation and the prediction of surrogate end points within a Bayesian framework. In principle, this approach could be used to generate predictions of treatment effects on final outcomes in a way that allows for borrowing of information across studies and that fully accounts for all uncertainty surrounding the surrogate relationship. In instances whereby the surrogate association is weak, this would manifest as a wider interval around the prediction and increased uncertainty surrounding modelled outcomes and costs. This approach is intuitively appealing; it would, however, render the published meta-regressions redundant because it would require re-analyses of the input data and the implementation of new meta-analyses for each histology site.

2. Land-marking analysis

This review included only meta-analytic studies and, by design, excluded individual studies that did not include multiple cohorts of patients. Some of the studies that were excluded from the review during the sifting stage adopted a land-marking approach (see Chapter 7, Discussion, for more details of this approach) within individual patient cohorts to explore the impact of response-based outcomes on OS, with differences between responders and non-responders reported in terms of a HR. Given an underlying baseline model of OS for non-responders, it may be possible to estimate the incremental impact on OS by combining the ORRs observed in the histology-independent studies with the HR derived from the land-marking analyses. However, the published land-marking studies generally related to a single tumour type and the study populations do not specifically relate to patients with NTRK fusion-positive tumours (or other relevant biomarkers).

3. Risk prediction models

During sifting, the review authors identified a small number of risk prediction studies. These studies reported multivariable statistical models to estimate the final outcome (OS/PFS) as a function of some response-based variable (e.g. ORR) together with other clinical parameters (e.g. age, sex and clinical characteristics). These studies may also provide a source of HRs for the impact of response on OS/PFS, but, again, these typically relate to a single tumour type and do not specifically relate to patients with NTRK fusion-positive tumours (or other relevant biomarkers).

4. Do not use response as a surrogate for progression-free survival/overall survival

The systematic review suggests that, taken generally, ORR may not be a reliable surrogate for PFS or OS based on current frameworks for surrogate validation. The review did not indicate any particular pattern whereby ORR performs better or worse according to tumour type or site. Even where a means of predicting PFS/OS on the basis of ORR for a given tumour site exists (e.g. using conventional meta-regressions or BRMA), in the absence of a strong relationship between the surrogate and the final end points the resulting estimates may be highly uncertain and difficult to interpret. It should be noted, however, that the alternative may involve extrapolating highly immature PFS and OS data, which are also subject to substantial uncertainty; hence, this may not represent a sufficiently robust solution either.

Exploring heterogeneity in response: case study

To demonstrate the impact of allowing for heterogeneity in response and to explore the potential heterogeneity in effects across tumours, response data were analysed using a BHM framework. 46

For the purpose of this analysis, the response data used were the published efficacy evidence available for the tyrosine kinase (TRK) inhibitor, larotrectinib. 3,153 The results, presented as a post hoc pooling of 55 patients covering 12 tumour types from three non-randomised single-arm Phase I/II basket studies, including the number of patients and responses by tumour type, are shown in Table 11.

We can consider each of the tumour types as a ‘basket’ or group and analyse the response data using a BHM framework to explore the potential heterogeneity in effects across tumours.

Results

For all analyses, 55,000 iterations were run on two parallel chains and the first 5000 iterations were discarded as ‘burn-in’. Model fit statistics are presented in Appendix 10.

The prior distributions used for the base-case analysis are:

⁽⁵⁾ µ ∼ Normal ( − 0.8473 , 10 ) σ ∼ Uniform ( 0.5 )

The BHM estimates substantial between-group heterogeneity (posterior median of 2.86 on the log-odds scale), although there is considerable uncertainty [95% credible interval (CrI) of 0.92 to 4.83] (Figure 8). This suggests that there is considerable variability across tumour types.

FIGURE 8.

Prior and posterior distributions for the between-group heterogeneity standard deviations.

The estimated mean response rate across all tumour types is 0.609 (95% CrI 0.160 to 0.918). This is lower than the mean response rate of 0.745 observed in the efficacy evaluable data set. The response probability predicted for an unrepresented tumour type is 0.569; however, the 95% CrI is wide, meaning that this probability could be as low as 0.2% or as high as 99.9% (Table 12 and Figure 9).

TABLE 12 - Overall posterior and predictive probability of response

Probability	Mean	Median	95% CrI
Posterior	0.609	0.641	0.160 to 0.918
Predictive	0.569	0.649	0.002 to 0.999

FIGURE 9.

Posterior and predictive distributions of response probability.

The estimated probabilities of response for each tumour type are shown in Table 13. The effect of allowing information to be borrowed across the tumour types is to shrink the observed response probabilities towards the pooled mean response probability. Tumour types with a smaller number of patients borrow more information than tumour types with a larger number of patients and, therefore, have values closer to the pooled mean.

TABLE 13 - Probabilities of response for all tumour types

Prior distribution for log-odds of response centred on a probability of 0.3; uniform prior distribution for the between-tumour standard deviation.

Tumour type	Observed response (%)	Estimated mean response based on BHM (%)	95% CrI (%)	Probability of response rate of ≥ 30%	Probability of response rate of ≥ 10%
Sarcoma	91	88	66 to 99	1.000	1.000
Salivary	83	82	58 to 97	1.000	1.000
IFS	100	93	70 to 100	1.000	1.000
Thyroid	100	92	63 to 100	1.000	1.000
Lung	75	73	30 to 98	0.976	0.999
Melanoma	50	52	12 to 89	0.835	0.984
Colon	25	32	3 to 75	0.484	0.854
GIST	100	88	49 to 100	0.996	1.000
Cholangiocarcinoma	0	21	0 to 76	0.281	0.555
Appendix	0	30	0 to 90	0.416	0.650
Breast	0	30	0 to 90	0.415	0.653
Pancreas	0	30	0 to 90	0.413	0.648

Figure 10 shows the posterior distributions of the probabilities of response for each of the 12 tumour types included in the efficacy evaluable data set. Although the observed response suggested that cholangiocarcinoma, appendix, breast and pancreas tumours did not respond to larotrectinib, the posterior distributions of these tumour types are wide and their 95% CrIs suggest that response rates of 76% are plausible.

FIGURE 10.

Posterior distribution for the probabilities of response by tumours type. (a) Sarcoma, salivary, IFS and thyroid; (b) lung, melanoma, colon and GIST; (c) cholangiocarcinoma, appendix, breast and pancreas.

The results were insensitive to the use of the inverse-gamma prior, the half-normal prior and the uniform prior centred on a log-odds of response of 0.5. The results were also insensitive to the use of a more favourable precision of the between-tumour heterogeneity standard deviation of 0.1 and 0.5. For full results of the sensitivity analysis, see Appendix 11.

Implications for the appraisal of histology-independent technologies

Heterogeneity in the treatment effects is likely to be an important issue in the appraisal of histology-independent technologies. As can be seen from the results of the worked example (see Exploring heterogeneity in response: case study), the BHM suggests that there is substantial heterogeneity in response across tumour types. This can be seen in the estimate of the between-group heterogeneity, with the BHM estimating a posterior median standard deviation for the heterogeneity of 2.86 on the log-odds scale, which is considered large. Heterogeneity can also be seen in the predictive distribution of response, appearing in Figure 9 as a bimodal distribution with density concentrated around a probability of response of 0 and 1. This can be explained by the individual tumour response rates shown in Figure 10, which suggest that, even under the assumption of exchangeability of response, there are tumour types in which the data suggest that response is likely and tumour types in which it is not likely.

The results of this analysis challenge the strong assumption of homogeneity in response across such a variety of tumour types when treated with larotrectinib. The assumption of homogeneity may mask important information about empirical evidence of tumour response and the BHM provides a vehicle through which to account for the potential heterogeneity.

Adjustment for confounding bias

A key factor in the reliability of estimates of effectiveness based on observational data is the statistical analysis used; a large number of studies have sought to develop and evaluate methods for adjusting and eliminating bias resulting from confounding. These include methods such as regression analysis, propensity scoring and population-adjusted indirect comparisons [matching-adjusted indirect comparison (MAIC) and Simulated Treatment Comparison (STC)]. 162 These and other methods are frequently used in the literature and have been previously applied and accepted by NICE appraisal committees where no randomised evidence exists. In theory, these methods could be applied in the context of a histology-independent appraisal. Implementing such approaches could, however, be challenging because of the large number of source data sets involved, which means that population characteristics may not be reported across all comparator data sources and would necessarily require strong assumptions about the prognostic value of population characteristics across tumour types. Furthermore, even if a suitable adjusted comparison could be generated, the small sample sizes typically seen in the Phase II trials would be able to account for only a small number of observed characteristics. This limits the potential for these methods to fully account for confounding biases and increases the likelihood of residual confounding bias. Despite these limitations, such methods would generally be considered to be preferable to a naive comparison, which takes no account of differences across groups.

Gaps in the reporting of baseline characteristics, variability in the prognostic value of characteristics across tumour types/histologies and difficulties of matching comparator data to the likely limited available Phase II trial for histology-independent technologies will also create additional challenges of interpretation and validation of comparisons with historical data, as it will be challenging to assess the comparability of patients in the historical control with those in the available single-arm trial data.

Alternative approaches to developing a comparator

Because of these significant concerns of confounding bias and the challenges of generating a truly comparable comparator data set, other approaches to generating a comparator data set should be considered and their limitations explored. For example, two alternative methods outlined in Hatswell et al. 163,164 could be used, in which patients in the single-arm trial are used to generate a control arm.

The first approach proposed by Hatswell et al. 163 uses effectiveness data on non-responders as a proxy for patients not receiving an active treatment. Comparator effectiveness estimates of PFS and OS under this approach would, therefore, be based on observed PFS and OS among non-responders in the integrated efficacy analysis. The advantage of this approach is that all patients in the non-responder subgroup met the same trial inclusion/exclusion criteria and received the same line of treatment. The rationale behind this approach is that patients in whom no response is observed represent those with a lack of treatment effect (because they have no response to treatment) and, therefore, are representative of a counterfactual for whom no effective therapy exists. The patient population is, therefore, likely to be better matched with the intervention arm because they are drawn from the same population.

This approach, however, also requires strong assumptions, namely that there are no differences other than response status between responders and non-responders that explain the survival outcomes and that non-responders derive equivalent benefit to that received on current SoC. The reasonableness of these assumptions is likely to be specific to a particular appraisal; however, as discussed in Chapter 4, the reliability of response as a surrogate is likely to be variable across tumour types. The assumption of no treatment benefit or harm may also not hold because some patients may receive some benefit from treatment, even if they do not have a PR or CR.

When considering the appropriateness of this approach, the relative advantages and disadvantages will need to be considered and it may be that this approach is considered reasonable only where there is substantive evidence of heterogeneity in treatment effects justifying the need to appropriately account for this heterogeneity in the economic analysis.

The second approach164 uses data taken from the trial patients’ previous line of treatment to derive OS and PFS curves. In this approach, the inverse of the ratio between the average TTP on their previous therapy and the mean extrapolated PFS with the active therapy [also called the growth modulation index (GMI) multiplier] is applied to all health outcomes (PFS and OS) for the active therapy. This crude adjustment assumes that the active therapy is more effective in terms of both PFS and OS than the comparator, by the same proportion as the GMI multiplier. Therefore, the resulting GMI-adjusted total mean life-years gained (LYG) and quality-adjusted life-years (QALYs) are assumed to correspond to comparator outcomes and are applied in the calculation of the ICER (based on LYG and QALYs). The main advantage is that effect estimates are drawn from the same population as the intervention arm and, therefore, are better matched; however, there are also disadvantages. First, this can be implemented only for patients who have received a previous line of therapy. Second, it also assumes that the ratio of TTP across lines of therapy is indicative of the treatment effect and it is uncertain to what degree this is likely to hold true. Finally, because this method can estimate PFS only, it requires that assumptions are made about the impact of TTP gains on OS [namely that either OS increases proportionally with TTP or post-progression survival (PPS) is the same across therapies], which, similarly, may not hold true. Further research considering the reasonableness of these assumptions may be helpful. Consideration could also be given to the potential role of expert elicitation to inform these judgements.

Implications for the appraisal of histology-independent technologies

The broad marketing authorisation, heterogeneous populations and uncertainties regarding the position of histology-independent technologies creates a number of significant challenges to creating appropriate historical control data. The confidence in estimates of effect may increase by utilising methods of population adjustment, but the scope of such methods may be more limited in the context of histology-independent appraisals. The assessment of the scope for residual confounding bias is also likely to be made more complicated, further reducing the confidence in comparisons. It is unclear whether or not the use of non-randomised evidence and, in particular, single-arm studies will ever be considered adequate. Alternative methods of developing a comparator may, therefore, be of value to decision-makers and should be considered as alternatives.

Distribution of tumour types

As outlined in Chapter 2 (see Food and Drug Administration review of histology-independent products and European Medicines Agency review of approved histology-independent indications), the evidence likely to be available for decision-making will include a number of histologies or tumour types, with limited data on each. When integrating these clinical data into a cost-effectiveness analysis, one important issue is to consider the distribution of patients across the different tumour types.

One approach would be to utilise the distribution of tumour types present in the clinical evidence to generate an average cost-effectiveness estimate. Underlying this approach, however, is the assumption that the cost-effectiveness of a histology-independent technology does not vary across tumour types or that the proportions of histologies are representative of the proportions eligible to receive the intervention in the full licensed population. The former assumption is very unlikely to hold owing to the potential for differences in effectiveness across tumour types, prognosis, comparators, costs and HRQoL. Furthermore, as shown in the comparison of the distributions in Table 14, there is a mismatch between the distribution of certain histologies in the trial populations. For instance, breast cancer patients represent 11% of the entrectinib trial but only 1.8% of the larotrectinib trial. If we imagine that larotrectinib and entrectinib produce identical clinical results, the resulting average cost-effectiveness estimates of the two distributions will be different given that different tumour types have different testing costs (see Genomic testing for histology-independent drugs), different SoC costs and outcomes, and potentially different prognoses.

The significance of these differences for the overall assessment of cost-effectiveness will depend on the degree of heterogeneity across separate inputs relevant to economic modelling. If the trial distribution is not considered to represent the distribution expected to be seen in the population under a histology-independent license, any decision based on a single ICER estimate for the trial population will be subject to potential bias. The magnitude and direction of this bias will be difficult to determine without a more explicit assessment of heterogeneity in different sources relevant to the economic model.

Where differences between the trial population and the licensed population are considered significant, approaches should be explored that allow for the re-weighting of the clinical population so that the model population better reflects the treated population.

Unrepresented tumour types

A further issue to consider is whether or not the trial evidence encompasses all of the histologies covered by marketing authorisation. If histologies exist that are not represented in the trials but are covered under the marketing authorisation, decision-makers will have evidence on effectiveness from only the subset of the total population that is potentially eligible for the intervention.

For example, within the clinical evidence available for entrectinib (see Table 11), 12 histologies were included. However, it is known that upwards of a further 17 histologies have been shown to harbour NTRK fusions and will be covered by the anticipated marketing authorisation; hence, this total could be even larger. 165 Any decisions made on the evidence alone would, therefore, be implicitly assuming that the 12 included types are representative of the full population.

The impact of the unrepresented population is potentially significant and its importance will depend on the number of unrepresented histologies and the proportion of the eligible population in unrepresented histologies relative to the observed histologies. It is also important to consider unrepresented tumours for which there is significant uncertainty regarding the homogeneity of clinical benefits or significant heterogeneity in costs across tumour types. For example, where there is limited support for the assumption of homogeneous efficacy across histologies, it may be important to characterise the uncertainty in the efficacy within the unrepresented population. Equally, there is significant evidence of variability of testing costs across tumour types and, therefore, ignoring unrepresented tumour types may impact significantly on average testing costs.

Position in the treatment pathway

A further potential limitation regarding the generalisability of the available clinical evidence relates to the position in the pathway at which patients are treated. This is complicated in part because the position in the pathway may vary substantively across tumour types according to the availability of alternative treatments, but also because of the potential for a mismatch between the trial population and the eligible population, as dictated by the marketing authorisation. The latter may be a significant issue because the recruitment of patients to a histology-independent trial is necessarily more complicated and there are potential significant challenges to identifying patients owing to the relative rarity of target genetic mutations. Thus, as observed in the entrectinib and larotrectinib clinical data, patients were recruited across multiple lines of therapy, even within the same tumour type.

This heterogeneity generates a number of issues, not least with respect to the external validity of the trial. Line of therapy may be a significant prognostic factor and failure to adjust for this may impact significantly on estimates of relative effectiveness, particularly if the comparator population is not matched to the position that patients were treated in the treatment arm. This issue may also impact in other ways, including on the final distribution of patients eligible for treatment, because fewer patients will be eligible for treatment in second and subsequent lines of therapy. Furthermore, it may have implications for testing, affecting either the total costs of testing or the population that will be eligible for testing.

Example

The example considers the TRK inhibitor, larotrectinib, which is used for the treatment of solid tumours harbouring a NTRK gene fusion, and the proportion of tumour types presented in the clinical evidence, as outlined in Exploring heterogeneity in response: case study.

First, to quantify the size of the population that will benefit from TRK inhibitors, the total number of patients eligible each year was calculated. This was estimated using the tumour-specific NTRK fusion prevalence, cancer incidence and proportion of patients with advanced or metastatic disease:

The full method for calculating the annual eligible population is described in Appendix 12.

Table 15 presents the calculations of the annual eligible population for TRK inhibitors based on the tumour types represented in the larotrectinib trial. The prevalence of NTRK fusion varies across tumour types, ranging from 92.2% (MASC) to 0.07% (breast cancer). Paediatric patients with infantile fibrosarcoma, a tumour type with a high NTRK fusion prevalence, make up the largest proportion of the eligible population (n = 27). Despite the low prevalence of NTRK fusion, patients with CRC contribute a substantial proportion of the eligible population (n = 23).

The resulting distribution of the eligible population (Table 16) shows that the proportions of tumour types in the eligible population differ substantially to the proportions in the larotrectinib trial (see Table 16). For example, soft tissue sarcoma represents 20% of the population in the larotrectinib trial yet represents only 4% of the population eligible to receive larotrectinib.

Implications for the appraisal of histology-independent technologies

In summary, the trial population may include only a subset of the total population potentially eligible for the intervention. Indeed, it is feasible that the majority of histologies potentially harbouring the biomarker will not be represented in the evidence. Furthermore, matching the line of therapy in the trial population to the eligible population is important given the likely prognostic effect of line of therapy. However, matching can be difficult if there is ambiguity in the treatment position specified within the relevant marketing authorisation.

Overview of molecular testing in the UK

Substantial investment and changes to genomic testing services have been undertaken in the last 5 years after a demand to improve the access to genomic services in the NHS to inform the most effective treatment pathway for a patient with cancer. 158 In 2018, the NHS launched the Genomic Medicine Service and a National Genomic Testing Strategy, which was based in seven genomic laboratory hubs across England. 153 Although this provides positive steps to improve the availability of genomic testing across the UK, the services are still being implemented, leading to limited capacity in some genomic laboratory hubs.

In March 2019, the genomic test directory listed 968 genomic tests available for 64 adult and paediatric tumour types. 166 Although this may seem an exhaustive number of tests for a large proportion of tumour types, it is far from inclusive. Patients with some common tumours, including prostate cancer (a population that contributes 15% of the annual incidence of solid tumours in England), are not eligible for any form of genomic testing because there are currently no effective targeted therapies licensed on the NHS. Although the absence of genomic testing until now may be because of limited evidence of known somatic or hereditary mutation that will be of prognostic or diagnostic value, the provision of a targeted histology-independent therapy would require screening of all cancers, regardless of current availability.

Types of genomic test

Tumourigenesis, the process of cancer growth and development, is driven by genetic alterations that result in sustained cell proliferation or the inhibition of cell division and death. 167 In fact, by the time that a cancer is diagnosed, there are likely to be millions of genetic mutations within a single malignancy. 168

Many of these alterations occur during tumour development but do not contribute to tumour growth, commonly known as ‘passenger mutations’; therefore, these play no functional role in cancer development. These mutations may occur in non-coding sequences of DNA that are removed during the transcription of DNA to RNA as part of gene expression. By contrast, the ‘driver’ mutations are involved in the neoplastic growth of the tumour, which directly result in the prolific growth of the cancer cells. Driver mutations can be differentiated further with respect to whether they solely influence the initial cancer development or whether oncogenic growth and proliferation are dependent on the mutation, regardless of its position in the disease pathway. 169

Therefore, the role of genetic testing is two-fold: first, to detect the presence or absence of a specific mutation and, second, to determine whether the mutation is acting as an oncogenic driver or whether it is merely a passenger mutation in tumourigenesis.

There are a variety of tests that are available to identify the presence of a mutation in individuals. These include DNA- and RNA-based panel tests, whole-genome sequencing (WGS), IHC, FISH and reverse transcription polymerase chain reaction (RT-PCR). Each of these tests determines the presence or absence of a genetic mutation in different ways, from identifying a known driver mutation using targeted tests in DNA and RNA to sequencing the entire genome, or determining the level of expression of a particular protein. The suitability of the alternative types of test will probably depend on the target mutation and the test’s diagnostic accuracy to correctly detect the respective alteration, the prevalence of the genetic mutation within each tumour type and the current testing provision. Table 17 summarises the key characteristics of each test type, noting key advantages and limitations.

Tests may be combined as part of a testing strategy, where confirmatory testing is implemented to verify that a mutation is being expressed. This allows for diagnostic accuracy to be maintained, while reducing the use of more expensive and resource-intensive test types. For example, IHC may be used as a screening tool to detect protein expression, with a further confirmatory test implemented to verify that the protein expression is caused by the mutation of interest. The relevance of strategies based around IHC may, however, become more limited as panel testing using NGS is expanded within the NHS.

Because of the variable provision of testing in the NHS across tumour types, the most appropriate testing strategy will probably depend on the tumour type. For example, all paediatric patients with advanced and metastatic cancer in the NHS will receive WGS at diagnosis by 2020158 and, therefore, any testing strategy is likely to be built on this provision. The appropriateness of each testing strategy will also depend on the prevalence of the genetic alternations across tumour types. Diagnostic accuracy will vary depending on the prevalence of the genetic alteration within each tumour type even when the sensitivity and specificity are held constant (see Appendix 12).

Implications of testing for appraisal of histology-independent technologies

The need for companion diagnostic testing to implement histology-independent technologies has several consequences for cost-effectiveness. These considerations include resource implications associated with implementing testing and the impact of alternative testing strategies on the modelled population, as well as broader implications regarding the feasibility of expanding testing services. These issues are briefly discussed in the following sections, followed by a worked example considering the implementation of testing for NTRK fusions.

Identifying patients eligible for TRK inhibitors based on the presence of a neurotrophic tyrosine receptor kinase fusion: a worked example

This section considers how testing for NTRK fusions might be implemented in the NHS and provides an illustrative example of how both NNS and testing costs might vary across tumour types.

A variety of testing strategies have been proposed for identifying patients with a NTRK fusion, pending the approval of two histology-independent TRK inhibitors. 172,174 The European Society for Medical Oncology (ESMO) proposes that the standard testing pathway should differ depending on the frequency of NTRK fusions in each tumour type and whether or not genomic sequencing is currently provided by the NHS. 166 In the tumour types for which there is a lower frequency of NTRK fusions and for which there is no genomic testing available, it is suggested that IHC is used for initial screening; NTRK gene rearrangements are then confirmed using RNA-based NGS. IHC is high throughput and inexpensive, making it a practical screening tool to use in a large population. Following this with more expensive and highly accurate RNA-based NGS is a plausible testing strategy for identifying tumours. Diagnostic accuracy is further taken into account by stratifying the tumour types into different testing strategies, depending on their NTRK fusion prevalence.

Conversely, it has also been suggested that front-line NGS should be offered to all individuals to detect a NTRK fusion. 174 Although this would require substantial investment, this testing strategy would ‘future-proof’ histology-independent testing because additional mutations could be added to pre-existing panels. This would mean that a single tumour sample could be screened to identify a number of genetic alterations. However, in the short term this testing strategy will require significant resources to implement nationally.

The most exhaustive approach to identify NTRK fusions utilises DNA-based NGS and RNA sequencing. Given that DNA-based NGS is currently available for some tumours, there will be reduced incremental investment in providing RNA sequencing, which would be used to confirm protein expression in the positive cases.

Although there is the potential for NTRK fusions to be observed in any tumour type, current evidence documents the occurrence of NTRK fusions in only around 30 different tumour types. 175–178 However, this list cannot be considered complete. It is plausible that NTRK fusions occur in common tumour types, but with such rarity that they are yet to be detected. There are also likely to be a number of rarer tumour types that express NTRK fusions but are not included in any current database. To align with the available evidence on the prevalence of NTRK fusion, we distinguish between tumour types within a single anatomical site only when there is supporting evidence on the prevalence of NTRK fusions to do so.

Implications for the cost-effectiveness of TRK inhibitors

The costs associated with additional testing for targets are likely to be substantial and will have a significant bearing on the cost-effectiveness of histology-independent technologies. Tumour-specific costs of identifying relevant fusions are also likely to represent a significant source of heterogeneity owing to the variable frequency of targets across tumour types. This heterogeneity in testing costs is likely to mean that for some tumour types for which NTRK fusions are rare, NTRK testing will not be cost-effective. Opportunities to share testing costs across multiple health-care technologies may reduce the cost burden of molecular testing on a specific health-care technology, potentially increasing the financial viability of testing. However, it is currently unclear what mechanism would be used to share testing costs across multiple technologies.

Screening to identify eligible patients

The previous analyses did not include the costs of identifying the population of patients with the biomarker of interest. However, a variety of tests and testing strategies may be required to identify eligible patients. As a result, the cost of patient identification may vary significantly across histologies. Indeed, even if homogeneity in all other model inputs is assumed, the cost-effectiveness estimates will inevitably vary based on differences in the costs of identifying patients with the specific biomarker. Consequently, when evaluating the overall cost-effectiveness of a technology, it is necessary to consider the joint costs and benefits of the testing/treatment strategy. 189,190

Table 27 updates the previous results by including an arbitrary per-patient testing cost of £50. The results clearly demonstrate that even a small per-patient testing cost can result in significant variation in the ICER estimates across the individual histologies.

Tumour-specific costs of identifying biomarker-positive patients are also likely to represent a significant source of heterogeneity owing to the variable frequency of targets across tumour types. This is evident in the individual ICER estimates, which now show much greater variation across different tumour sites than the previous analysis, which excluded per-patient testing costs.

The key variable driving the testing costs and the ICER estimates for the test/treat strategy is the NNS. For now, we assume that the test is perfect: it correctly classifies all individuals as having or not having the mutation (i.e. there are no false positives or false negatives). In this situation, the NNS is 1 divided by the expected frequency of the mutation in each histology. For histologies in which the mutation is very common (‘high frequency histologies’, for example salivary), there is a very small NNS because almost every person (92.9%) screened has the mutation. The opposite is the case for ‘low frequency histologies’ in which the mutation is rare. In pancreatic cancer, 1429 people (i.e. 1/0.07%) need to be screened to identify one individual with the mutation. A testing cost of £50 per test increases the overall costs of Drug X by £71,429, from £37,751 to £109,180 in pancreatic cancer. This increases the ICER from £37,930 to £154,304 in pancreatic cancer. The ICER for each histology has increased, but in the histologies with moderate to high frequency of mutation the increase in the ICER is more modest.

The above analysis assumes that the test (or testing strategy) is perfect; however, if this is not the case, this will result in patients being misclassified. Patients who do not have the mutation will be classified as having the mutation (false positives) and patients who have the mutation will be missed (false negatives). Such misclassifications may have important implications for costs and health. 149,190 The number of false positives and negatives will depend on the testing strategy, test characteristics (sensitively/specificity) and frequency of the mutation in each histology. 165,172,191 The possibility of misclassification presents two tasks: (1) calculating the correct ICER given a specific test or testing strategy, and (2) choosing the optimal test or testing strategy. Both of these tasks require estimates of the costs and QALYs associated with false positives and false negatives, which will probably differ by histology. For costly new treatments and those with significant side effects, false positives may have substantial consequences. The scale of consequences associated with false negatives will depend on the additional benefits of treatment. This means that the consequences of missing a potential patient (false negative) will be larger in those histologies in which the treatment results in larger QALY benefits.

Histology-dependent recommendations and the value of heterogeneity

The assessments presented in Table 29 can also be used to compare the population consequences of making different policy recommendations. Decision-makers, such as NICE, have the option of different approval policies:

1. no stratification – histology-independent approval

2. partial stratification – approval in a clinically defined set of histologies

3. full stratification – approval only in histologies in which cost-effective is demonstrated.

These policies will determine the type of recommendations that are feasible and the relevant health consequences that need to be considered. The following section deals only with approval policies based on expected values, without addressing the impact of uncertainty in decision-making. Uncertainty, the need for further evidence and alternative mechanisms to reduce the risk of decision-making are considered in subsequent sections.

No stratification: histology-independent approval

This represents an ‘all or nothing’ approval policy in which the intervention is approved for all histologies or for none. There is no stratification of decision-making by histology. In this case, the relevant metric is the pooled ICER (or pooled NHB/NMB equivalent) across all histologies. Based on the results shown in Table 29, Drug X would not be approved for use in any histology because the pooled NHB is negative (correspondingly, the pooled ICER would be higher than the £50,000 per QALY threshold). The health system is expected to lose approximately 17 QALYs per year if Drug X was granted a histology-independent approval.

However, a further consideration when making histology-independent decisions is that some histologies, which may harbour the mutation of interest, may not be directly observed in the evidence base at the time of decision-making, despite the inclusion/exclusion criteria of the trial permitting their inclusion. Given that these patients may be treated in clinical practice, consideration should be given to the potential impact of considering histologies that are not represented in the trial data. The larger the incidence of unrepresented NTRK fusion-positive histologies, the greater the influence this can have on decision-making. In this case study, it is estimated that there are 151 NTRK fusion-positive cases in the set of unrepresented histologies each year (see Table 16). This is a larger number than the 125 cases in the observed set of histologies represented in the trial and should be explicitly considered if a histology-independent approval is sought.

If, as in this case study, the economic model is developed around the probability or degree of response in each histology and a BHM has been used to analyse response, the predictive distribution could be used to estimate response in the unrepresented histologies. This assumes that the effects for the unrepresented histologies are exchangeable with the observed histologies. For Drug X, the predictive distribution for response in unrepresented histologies has a mean response probability of 57% and is highly uncertain, with a 95% CrI ranging from 1% to 100%. If estimates for the remaining parameters (e.g. quality of life and testing costs) can be sourced from the literature or generalised from the observed histologies, ICER and NHB estimates can also be estimated for unrepresented histologies.

The results shown in Table 30 include the impact of including unrepresented histologies. To simplify the case study, we collapse all unpresented histologies into one ‘unrepresented’ histology category. The response probability comes from the predictive distribution from the BHM; costs and quality of life are assumed to be the same across all observed and unobserved histologies. The average testing costs for unrepresented histologies were estimated to be £14,322 per patient tested. This estimate was based on observational data on NTRK fusion prevalence (see Appendix 12).

After taking account of the unrepresented histologies, Drug X is now estimated to be cost-effective in the overall population with positive NHBs. In this example, a histology-independent approval, including an assessment of the potential impact of unrepresented tumour sites, would result in an expected overall gain to the health system of approximately 5.79 QALYs (NMB ≈ £290,000) per year. Treating individuals with histologies that are unrepresented in the trial data is expected to result in positive NHB, given the assumptions made here. This is because of the relatively high mean response rate (57%) predicted by the BHM.

Although it may be challenging to identify data to inform benefits in unrepresented histologies, consideration to the magnitude and potential impact of these histologies should be explicitly considered.

Partial stratification: approval in a defined set of histologies

This is similar to the previous approval policy; however, in this case, the intervention is approved only for a clinically defined set of histologies, that is there is partial stratification of decision-making by histology. The relevant metric here is the pooled ICER (or pooled NHB equivalent) for the defined subset of histologies. The basis for selecting a subset of histologies can be based on theoretical and/or empirical grounds. For example, Chapter 2, European Medicines Agency review of approved histology-independent indications, highlighted comments from the SAG to the EMA about larotrectinib that appeared to differentiate the strength of the biological rationale and the available clinical evidence for several specific tumour types (e.g. IFS, salivary gland/MASC, congenital mesoblastic nephroma and GIST). For these specific tumour types, the SAG concluded that efficacy has been established in the absence of available treatments of proven efficacy in terms of convincing clinical efficacy end points and that clinical decisions to use larotrectinib were justified.

To illustrate the implications of a policy decision based on partial stratification, we assume that there is sufficient ground to consider restricting an approval decision for Drug X to only those patients with IFS, salivary gland and GISTs. Evidence for patients with congenital mesoblastic nephroma was not available at the time of the FDA assessment; therefore, these patients are not included in the data used to inform the BHM.

As shown in Table 31, a decision to approve Drug X in only these three individual histologies is expected to result in an overall annual gain to the health system of 26.49 QALYs. Although partial stratification results in fewer patients receiving Drug X than in a full histology-independent approval (i.e. 33 patients annually vs. 276 patients), there would be an overall gain to the health system from a policy decision based on partial stratification. This gain is equivalent to approximately 20.7 QALYs per annum. In other words, a policy to fully approve a histology-independent product could result in an annual loss of 20.7 QALYs to the health system compared with an optimised approval decision based on a partial stratification approach.

The majority of the gains from partial stratification are achieved by avoiding the approval of Drug X in histologies with high testing costs and relatively high incidence (e.g. lung, colon, breast and pancreatic cancer), for which Drug X does not appear to be cost-effective based on current evidence. A further advantage of partial stratification over no stratification is that assumptions about unrepresented histologies can be avoided in decision-making. However, a disadvantage of partial stratification is a potential increase in monitoring costs required to prevent the use of Drug X outside its subset of approved histologies. 151

Full stratification: approval only in histologies in which cost-effectiveness is demonstrated

This is a fully histology-dependent approval policy in which the technology is restricted for use only in those histologies in which it has been shown to be potentially cost-effective based on expected ICER/NHB estimates. Given the ICER/NHB estimates presented in Table 29, Drug X appears to be potentially cost-effective in the following histologies: sarcoma, salivary gland, IFS, thyroid, GIST and appendix. These are the histologies in which NHBs are greater than zero. Equivalently, they each have ICERs below £50,000 per QALY gained. Taking the sum of the NHB across each of these histologies results in an overall annual gain of 35.68 QALYs to the health system from a fully stratified approval decision for Drug X. The expected number of patients treated annually based on full stratification is estimated to be 60.

The additional value of distinguishing between different types of patients represents the VoH. 150–152 In this example, the VoH represents the difference between the NHB of a fully stratified recommendation and a histology-independent recommendation with no stratification. This difference is equivalent to 29.89 QALYs per year.

Exploring the VoH may help to inform NICE committees of the consequences of alternative policy options, in terms of both the expected number of patients who would be eligible to receive a specific new treatment and their overall consequences to the health system. Although a histology-independent approval might be considered appropriate on the basis that this results in an overall positive annual NHB compared with rejecting the technology, it is also important to consider the potential consequences of such an approval policy compared with a more restrictive or optimised recommendation. In this case study, there appear to be significant gains to the health system that could be achieved by an optimised recommendation. Importantly, an approval decision based on partial stratification using only three individual histologies appears to confer approximately 74% of the gains that are potentially achieved based on a full stratification policy.

The consequences of uncertainty

This section introduces value of information (VOI) as a framework to quantify the health effects of uncertainty. VOI analyses can provide decision-makers with metrics to help to understand the drivers of decision uncertainty and assess alternative strategies that could be used to manage this risk. The uncertainty associated with a histology-independent decision (‘no stratification’) is illustrated below. The implications for partial and full stratification will be addressed in subsequent sections.

The NHB results previously reported in Table 30 are illustrated graphically in Figure 11. In addition, uncertainty around the expected (mean) estimates of NHB are represented using a 95% CI. This is computed from the mean and 95% percentiles of the PSA for each histology. Figure 11 also plots the patient-level pooled NHB. This is analogous to the pooled ICER reported in Tables 25 and 26. The pooled NHB is a weighted average of the NHB associated with different histologies. Weights come from the incidence of NTRK-positive histologies reported in Table 30, with a perfect screening strategy assumed.

FIGURE 11.

Net health benefits per person across histologies with uncertainty. Positive numbers favour the intervention (drug) compared with usual care alone.

Figure 11 shows that Drug X is expected to result in additional NHB in sarcoma, salivary, IFS, thyroid and GISTs, with the 95% CI not crossing the line of equivalence with SoC. It is also expected to be cost-effective in the appendix and in those histologies that are unrepresented in the trial, but this is uncertain. This uncertainty can be expressed in terms of the likelihood or probability that Drug X is not cost-effective compared with SoC (i.e. 47% and 39% in appendix and unrepresented cancers, respectively). For lung, melanoma, colon, cholangiocarcinoma, breast and pancreatic cancer, the model estimates that there is approximately 0% chance that Drug X is cost-effective.

As in the previous analysis, the pooled population represents the expected consequences of a histology-independent recommendation. Drug X is expected to result in 0.02 additional QALYs per person treated and there is a 52% chance that it is cost-effective compared with SoC. The pooled estimate relies on the relative incidence of histologies in the target population. Although uncertainty in histology incidence is not addressed quantitatively in this case study, this can be propagated through the PSA in the same manner as other uncertainties.

The health system consequences of decision-making can be better informed with reference to population health. Figure 12 shows the population NHB for each histology and the pooled NHB. This is calculated by multiplying the per-person NHB for each group illustrated in Figure 11 by the incidence for each group (see Table 30).

FIGURE 12.

Population net health effects across histologies with uncertainty. Positive numbers favour the intervention (drug) compared with usual care alone.

Figure 12 shows that, although uncertainty in per-person NHB may be similar across histologies, the consequences of approval and uncertainty vary substantially when the size of populations are taken into account. The figure shows that for many histologies (e.g. sarcoma, salivary, melanoma and breast), the health consequences of approval and/or uncertainty are limited owing to their small population. By contrast, the health consequences associated with the unrepresented histologies are relatively large, as decisions in this group affect 151 individuals each year.

The pooled category represents the health consequences of the ‘no stratification’ approval policy (i.e. a histology-independent approval). A histology-independent approval is expected to result in a gain of 5.79 QALYs per year, on average (consistent with Table 30). However, the 95% CI indicates that this is highly uncertain, with approval potentially resulting in losses of up to 120 QALYs per year (illustrated in Figure 12 by the lower CI, which extends to –120). The following section will describe how VOI methods can be used to quantify the health consequences of this uncertainty to help inform decision-making and approval policies.

Quantifying the health consequences of uncertainty

Histology-independent decision-making (no stratification) is concerned with making approval decisions based on pooled cost-effectiveness estimates. From Figure 12, Drug X is expected to provide an expected benefit of (0.02 × 276 ≈) 5.79 QALYs per year at the pooled population level. However, there is uncertainty about this benefit. VOI methods can be used to quantify the health consequences of uncertainty, that is the risk associated with decision-making with current information. 193,196,197 Uncertainty matters because it means that there is a chance of making the wrong decision. Quantifying the expected health consequences of uncertainty is achieved by multiplying the chance of making a wrong decision by the health consequences of making the wrong decision. This is illustrated in Figure 13.

FIGURE 13.

Estimating the health consequences of uncertainty.

If Drug X is more cost-effective than SoC in the pooled population, there are zero health consequences of uncertainty. The tall left-hand bar in Figure 13 shows that there is estimated to be a 52% chance that Drug X is cost-effective in the pooled population. This corresponds to a 52% chance of zero consequences of uncertainty.

Making an incorrect decision (e.g. approving Drug X when it is not cost-effective) will have health consequences. For the pooled population, there is a 48% chance that the decision to approve Drug X is incorrect. As shown in Figure 13, these health consequences are not uniform. There is a greater chance of more limited consequences than a smaller chance of greater consequences. Figure 13 shows that there is a 21% chance of Drug X resulting in a loss of 25 QALYs per year (second bar from the left). There is a 19% chance of a loss of 75 QALYs per year (third bar from the left), and so on. The weighted average over this range of outcomes provides an estimate of the health consequences of uncertainty. This is estimated to be 29.52 QALYs per year, equivalent to approximately 0.108 QALYs per person.

This quantitative approach to the risks associated with uncertainty can be used to assess policy options that address this risk. In the following sections, we will illustrate three approaches to managing this risk: further data collection, pricing agreements and stratified decision-making.

Managing risk through further data collection

Further data collection is one approach to reduce risk associated with uncertainty. The imprecision in parameter estimates owing to limited sample size (e.g. OS) can be reduced by collecting data on these parameters. 198

Decisions about further data collection are important because under current policy arrangements when NICE is unable to approve a technology for routine use owing to parameter uncertainties it may recommend it for inclusion into the CDF if it is eligible. 199 Topics that are eligible for the CDF are reimbursed for a time-limited duration following the development of a managed access agreement (MAA). The MAA consists of (1) a data collection agreement (DCA), which specifies the data that must be collected that could sufficiently resolve the parameter uncertainties identified by the appraisal committee, and (2) a commercial access agreement, which ensures that the technology is reimbursed at a cost-effective price during the period of the MAA. A technology remains in the CDF until the data collection agreed in the DCA is complete; it then proceeds to reappraisal and exits the fund.

The MAA covers the entire eligible population determined by the NICE guidance, which means that entry to the CDF is equivalent to an ‘approval with research’ decision, which is reassessed after the data collection period (usually 2 years). 199,200 The assessments required to inform the suitability of an ‘approval with research’ decision over ‘only in research’, approve and reject are covered in detail elsewhere. 197 Explicit consideration of these assessments could aid the transparency of CDF entry requirements. However, it is beyond the scope of this report to suggest reforms to CDF processes or to determine the appropriate size of the CDF budget. These issues have been commented on elsewhere and require further research. 200–202

The aim here is to provide a framework to understand how the CDF, in its current form, can help to address the risk associated with histology-independent technologies. The intention is to demonstrate how a unified decision framework could enable CDF data collection arrangements to be considered alongside other risk reduction strategies (e.g. pricing schemes and stratified decision-making).

Managing risk through pricing schemes

The NICE process allows for consideration of a variety of pricing schemes, including patient access schemes, commercial access agreements and flexible pricing. 206 These schemes can facilitate pricing arrangements, such as simple discounts or more complex ‘pay-for-performance’ arrangements. In this section, we illustrate the effect of a simple discount and a pay-for-performance scheme on uncertainty and the expected value of a technology.

Simple discounts have been identified in previous research as an effective approach for payers to reduce the risk of approving technologies that are not cost-effective. 194,195,200 Reducing the price of a technology that is expected to be cost-effective has two implications: (1) the value of implementing the technology will increase owing to the resources saved and (2) the risk of the technology not being cost-effective will decrease. Under the current price (£1250 per month), a histology-independent recommendation for Drug X is expected to result in 5.79 additional QALYs per year. However, owing to uncertainty in parameters, there is also a 48% chance that Drug X is not cost-effective. As shown previously, the expected health consequences of this uncertainty have been estimated to be 29.52 QALYs per year.

With a 10% simple discount (£1125 per month), the expected value of Drug X is estimated to increase to 21.4 QALYs per year. Furthermore, the risk of Drug X not being cost-effective is reduced to 42%. The potentially negative health consequences associated with the uncertainty are reduced to 23.91 QALYs per year (a reduction in risk of 5.61 QALYs). A 20% simple discount (£1000 per month) increases the expected value of Drug X to 30.04 QALYs per year and reduces the consequences of uncertainty to 21.3 QALYs per year (a reduction in risk of 8.22 QALYs).

To illustrate the use of more complex pricing schemes, we also implemented a pay-for-performance scheme. In this scenario, the undiscounted cost of Drug X (£1250 per month) is incurred only if a patient responds to treatment. This has two impacts on risk and cost-effectiveness. First, this acts in a similar manner to a price discount because the average response is expected to be approximately 60% according to the BHM (i.e. reducing the effective price by 40%). Second, the risk associated with Drug X not resulting in the expected outcomes is now shifted from the payer to the company. 194 These two impacts reduce the health consequences of uncertainty to the health system. With this pricing scheme, the expected value of Drug X increases to 51.95 QALYs per year and reduces the consequences of uncertainty (i.e. the expected risk) to 9.35 QALYs per year (a reduction in risk of 20.17 QALYs).

The examples here illustrate how alternative pricing approaches can be used to increase the expected value of a technology, as well as impacting the risk and consequences associated with uncertainty.

Managing risk through stratified decision-making

The previous discussion described how uncertainty can be addressed when making histology-independent decisions in which the technology is expected to be used across all histologies (no stratification). In this section, we discuss how to apply these same principles under partial and full stratification.

Partial stratification

The previous sections described how uncertainty and associated risk could be managed through further data collection and pricing schemes. Both assumed that the health technology would be approved for use in all histologies or none (i.e. histology-independent approval decisions). In this section, we discuss stratification as an additional approach to reducing risk in approval decisions for products with a histology-independent marketing authorisation.

In the case of partial stratification, the intervention is approved only for a clinically defined set of histologies. The relevant metric for decision-making is the pooled NHB for the subset of histologies of interest. Therefore, it is the uncertainty in this pooled NHB that is of relevance to decision-making. Figure 14 illustrates the uncertainty in pooled NHB for the approval of Drug X in IFS, salivary gland and GISTs only.

FIGURE 14.

Population net health effects with uncertainty for partially stratified decision-making. Positive numbers favour the intervention (drug) compared with usual care alone.

Figure 14 shows that Drug X is expected to provide positive NHB in each of IFS, salivary gland and GISTs individually. Implementing Drug X in this subset is expected to result in approximately 26.5 additional QALYs per year over SoC (this corresponds to Table 31). Figure 14 graphically represents the uncertainty in this estimate. The 95% CI for the pooled effect is far from the line of equivalence between Drug X and SoC, indicating that there is not much uncertainty in cost-effectiveness in this subset of histologies. Given the model assumptions, it is estimated that there is now a 0% risk that Drug X is not cost-effective. This means that the risk in approving Drug X has been eliminated, without the need to carry out additional research or wait for research to report. For this reason, a routine commissioning decision may be considered appropriate for this specific subset of tumour sites.

Full stratification

Under full stratification, there is the option to make different decisions for different histologies. This is a fully histology-dependent approval policy in which the technology is restricted for use only in those histologies that it has been shown to be cost-effective. Figure 15 illustrates the population-level uncertainties in making fully stratified recommendations.

FIGURE 15.

Population net health effects with uncertainty for fully stratified decision-making. Positive numbers favour the intervention (drug) compared with usual care alone.

Approval in sarcoma, salivary gland, IFS, thyroid, GIST and appendix cancers is expected to provide a positive NHB for all. Approval of Drug X in the remaining histologies is expected to result in a loss of population health. Figure 15 also shows that uncertainty about health benefits (or losses) differs across histologies. The 95% uncertainty bounds cross the line of equivalence for melanoma and appendix cancers only.

Because separate approval decisions can be made for each histology, the risk associated with decision-making should be estimated for each histology separately. The risk associated with uncertainty in each histology is reported in Table 32, along with the expected annual value of approving the treatment for each histology.

TABLE 32 - Benefits of approval and further research for fully stratified decision-making

Subgroup	Population-level £50,000 per QALY threshold
	Incidence	Health impact of uncertainty per year, QALYs (NMB, £)	Health impact of approval per year, QALYs (NMB, £)
Sarcoma	5	0.02 (1216)	2.88 (144,046)
Salivary gland	2	0 (113)	1.56 (78,201)
IFS	27	0.01 (577)	22.35 (1,117,570)
Thyroid	6	0.01 (665)	4.32 (216,219)
Lung	17	0.03 (1398)	–8.52 (–426,230)
Melanoma	3	0.23 (9950)	–0.23 (–11,276)
Colon	23	0.02 (1045)	–15.19 (–759,374)
GIST	4	0.01 (407)	2.57 (128,728)
Cholangiocarcinoma	0	0 (2)	–0.31 (–15,727)
Appendix	16	1.1 (55,078)	1.99 (99,383)
Breast	5	0 (0)	–6.19 (–309,616)
Pancreas	17	0 (0)	–21.78 (1,089,034)
Total	125	1.41 (70,452)

Table 32 shows that, of the histologies included, the largest risks are associated with decisions about appendix and melanoma histologies. This is because these two histologies have uncertainty bounds in Figure 15 that cross the line of equivalence. Given that further research appears most valuable in these histologies, this may help to prioritise further data collection. EVPPI assessments can be applied to these histologies to understand the parameters that are driving uncertainty. For other histologies, it is clear that Drug X is cost-effective (e.g. IFS) or not cost-effective (e.g. pancreas) based on current evidence; therefore, there appears to be limited risk of making the wrong decision and little value in further research.

Table 32 also illustrates that the total risk associated with decision-making has reduced for stratified decision-making compared with no stratification. The expected health consequence of uncertainty was 29.52 QALYs per year for no stratification. This was zero for partial stratification and approximately 1.41 QALYs per year for full stratification (implying a reduction of 28.11 QALYs).

The change in uncertainty (as measured by VOI) from less stratification to more stratification has been called the ‘dynamic value of heterogeneity’ in the literature. 151 It should be noted that increasing stratification may increase or decrease the uncertainty in decision-making. When the characteristic that the treatment is being stratified by is important in explaining heterogeneity (such as histology in the case study), stratification will increase the value of implementing the treatment while reducing the risk of making an incorrect decision. This is because variability in outcomes is translated into heterogeneity. However, if a stratification characteristic contains little information to distinguish outcomes, uncertainty may increase with stratification owing to sample splitting. 151

Comparing approaches to risk management in histology-independent technologies

When making a histology-independent approval decision, with current evidence and without any discount, Drug X appeared cost-effective based on expected values but the health consequences of uncertainty were estimated to be 29.52 QALYs per year. Three approaches to risk management were explored: further data collection, pricing schemes and stratified decision-making.

The upper bound for the value of further data collection on OS was expected to be 12.16 QALYs; this is compared with a reduction in risk of 5.61 QALYs from a 10% price discount, 8.22 QALYs from a 20% discount, 20.17 QALYs from a pay-for-performance scheme and a reduction of 28.11 QALYs from stratification.

The magnitude of uncertainty resolved through data collection will depend on which parameters can be informed by feasible research. Owing to institutional or ethics constraints, data collection may not be possible for some parameters and this places limits on this approach to risk management.

As discussed previously, the degree of uncertainty resolved through stratifying by histology will depend on the importance of histologies in explaining heterogeneity in cost-effectiveness. In cases in which cost-effectiveness does not vary significantly across histologies, the risk reduction from stratification will be lower. There may also be additional costs associated with (partially or fully) stratified recommendations, for example the costs of monitoring clinician behaviour to ensure that treatments are not being used in histologies for which they do not have approval. 150,151 In principle, this cost can be incorporated into the analysis of alternative policy options; however, reliable data to predict these costs may be difficult to find. 207

When considering the impact of pricing schemes, the magnitude of risk reduction will depend on the pricing arrangement. 194 For simple discounts, the risk reduction will increase with the scale of the price reduction. Neither a 10% nor a 20% simple discount reduced the risk of approval as much as either further data collection or stratified decision-making in the case study. A pay-for-performance scheme reduced the risk more than further data collection, but not as much as stratified decision-making. It should be noted that, when comparing price reductions (or stratification) with data collection, it is important to take account of the fact that data collection takes time to report whereas the other risk management policies can theoretically begin immediately.

Pricing schemes and stratified decision-making can also increase the value of approving technologies in addition to addressing risk. A 10% and 20% discount increased the value of Drug X from 5.79 to 21.4 and 30.04 additional QALYs per year, respectively. Partial and full stratification increased this to 26.49 and 35.68 QALYs per year, respectively. The gain in value from stratification is a result of making more optimised decisions. This has been called the ‘static value of heterogeneity’ in the literature. 151 The pay-for-performance scheme increased the potential value of approval by the greatest extent. It resulted in an additional 51.95 QALYs per year from the approval of Drug X. This gain is mostly because of the substantial discount implied by the pay-for-performance scheme. Because these approaches to risk management (further data collection, pricing schemes and stratified decision-making) are not mutually exclusive, each one can be used in combination to address the risk of approving a technology that is not cost-effective. 195

Limitations of the analysis and directions for future research

A limitation of the analysis in this section is that ‘unrepresented histologies’ are included as a homogeneous group. In reality, there may be significant heterogeneity between different unrepresented histologies. The sections on stratified decision-making assumed that Drug X could be approved only in represented treatments. Theoretically, this need not be the case. If unrepresented histologies were not treated as a homogeneous group but were considered individually, it is likely that for some histologies Drug X would be expected to be cost-effective and for others it would not. The uncertainty surrounding each would also differ. If approval for individual unrepresented histologies was feasible, the decision uncertainty remaining after full stratification would be larger than reported in Managing risk through stratified decision-making. This is because the uncertainty reported for fully stratified decision-making (1.41 QALYs per year) considers uncertainty only in the represented histologies. Including uncertainty in unrepresented histologies would necessarily increase this.

For the sake of clearly illustrating the core principles of decision-making under uncertainty, other simplifying assumptions were made. Namely, one-off infrastructure costs, population prevalence and test uncertainty were not explicitly modelled; these assumptions should be relaxed in future research. 197

One-off infrastructure costs are relevant to calculate per-person testing costs. In the case study, we have assumed a one-off testing cost of £50 per individual tested. However, testing approaches based on NGS may require large up-front investments in infrastructure. A recommended approach to incorporate capital costs, such as testing infrastructure, is to divide the one-off expenditure by the total population of patients who are expected to use the infrastructure. 197,208 For histology-independent technologies, this includes individuals across a range of histologies and over the expected lifetime for the infrastructure. This has several potential important implications for decision-making.

The first is that any stratification of approval by histology will necessarily mean that testing costs will be spread over a smaller number of patients. This will have the effect of increasing per-person testing costs when treatments are approved for subsets of histologies, reducing the expected health gains associated with stratification. Second, if reimbursement decisions are changed before the end of the assumed lifetime of the one-off infrastructure investment and some proportion of these costs are not recoverable, this has important implications for decision-making under uncertainty. The presence of significant irrecoverable costs increases the costs associated with initially implementing then subsequently removing a technology from general use. Taking account of these costs will tend to favour more conservative approaches to decision-making, which demand less uncertainty before a treatment is approved for widespread use. 197 This has implications for the CDF because MAAs stipulate approval of technologies for the entire eligible population as determined by the NICE guidance alongside research. 199 Explicit consideration of significant irrecoverable costs in this context will make the costs of inclusion into the CDF more transparent.

A third implication of investment costs is that testing infrastructure, such as NGS, may provide a basis for the use of other health technologies that use the same infrastructure. This means that the population of patients who are expected to use the infrastructure extends across all treatments and indications expected to use the infrastructure.

A further simplification of the case study was that it was assumed that the test that was used to identify eligible patients was perfect, that is it results in zero false positives and zero false negatives. The reality of testing will differ in two ways: (1) the accuracy of a test may not be perfect and will, therefore, misclassify a certain proportion of patients and (2) the false-positive and false-negative rate will be estimated with uncertainty, meaning that the rate of misclassification may not be known with certainty. For point (1), the consequences of misclassification and the analytical approaches to deal with this have been discussed in Screening to identify eligible patients. For point (2), if uncertainty in the false-positive and false-negative rate can be parameterised, the health consequences of this uncertainty can be managed using the same EVPPI methods as illustrated in the case study.

The case study was also built on a simplified surrogate relationship between response and survival. Survival was assumed to be determined by response, and, conditional on response or non-response, it was assumed to be homogeneous across histologies. The aim of this model was to link heterogeneity in response to heterogeneity in costs and health outcomes. However, as discussed in Chapter 4, the relationship between response and survival is highly uncertain, variable and may be very weak. Further research is required to better inform how surrogate outcomes, such as response, can be linked to costs and health outcomes.

Comparing approaches to risk management in histology-independent technologies compared data collection, pricing schemes and stratified decision-making as alternative approaches to manage risk and increase the health impact of decision-making. Considering the full range of options has important implications for price negotiations in histology-independent technologies. The health impacts of stratified decision-making could be used as a benchmark in negotiating discounts required for histology-independent approval. For example, to obtain a histology-independent approval, the reimbursement decision-maker could require a pricing scheme sufficient to reduce risk to the level that would exist under stratified decision-making. Any approval policy will create a specific set of incentives for research and pricing strategy. 209,210 Further research is required to understand the incentives provided by current arrangements and the potential benefits of changes to policy.

Finally, the case study focused on histology as the main source of heterogeneity. However, heterogeneity could be explored using a range of alternative characteristics and subgroups. To move from histology as the main source of heterogeneity to considering a wider range of characteristics requires an understanding of how different characteristics can be utilised and combined in different ways in decision-making. 151 How best to decide on which characteristics to utilise in decision-making and how they should interact is a complex question that requires further research. It is also important to note that the case study has focused on observable sources of heterogeneity. Inevitably, there will be unobservable sources of heterogeneity (e.g. unobserved differences between patients and/or studies) that cannot be explicitly addressed but will need to be taken into account by decision-makers when interpreting these findings.

Treatment effectiveness

Complex innovative study designs are increasingly used to improve the efficiency of the drug development process and to speed up regulatory approval and the access of drugs with new mechanisms of action. Adaptive basket trials are particularly suited to assess the efficacy of histology-independent drugs, although their reliance on surrogate outcomes, small sample sizes and mostly uncontrolled designs pose challenges for HTA. Adequately designed and analysed basket studies that assess the homogeneity of outcomes and allow borrowing of information across baskets, where appropriate, are recommended. In particular, the use of comparative and randomised designs and primary outcomes that can adequately predict the clinical outcomes of interest is recommended where feasible.

The potential for heterogeneity in treatment effects, either across tumour types or across other characteristics, is likely to be an important issue in the appraisal of histology-independent technologies. Careful consideration should be given to the appropriateness of the assumptions of homogeneity of treatment effects and NICE committees should expect to see an exploration of this assumption in company submissions. Bayesian hierarchical methods, which are frequently used in the analysis of basket trials, may provide a useful vehicle with which to explore any heterogeneity. Where there is evidence of heterogeneity in treatment effects and estimates of cost-effectiveness, consideration should be given to optimised recommendations.

Counterfactual

Generating a counterfactual is likely to be challenging in the context of histology-independent technologies and, in the absence of randomised evidence, it is likely that no single approach will be able to provide robust estimates of relative effectiveness. Companies developing histology-independent technologies, therefore, should be encouraged to consider several alternatives. Consideration should be given to the relative strengths and weaknesses of these alternatives when evaluating the most appropriate comparison. Evidence on the prognostic and predicative performance of the biomarkers should also be considered where possible, although it is recognised that such data may be limited at the time of submission.

Generalisability

The trial evidence available to support the approval of histology-independent technologies may differ substantially from the patients eligible for treatment in practice. Significant differences may, for example, be seen in the distribution of tumour types, positioning of the technology and subsequent treatments received. The potential for heterogeneity in treatment effects means that differences between the trial population and the eligible population may have an important impact on estimates of cost-effectiveness. Where possible, it is important that such differences are properly accounted for. Consideration of the differences between the trial population and the eligible population should also be borne in mind when considering an appropriate counterfactual data set.

Trial evidence supporting histology-independent technologies may not offer complete coverage of the eligible population. For this reason, there may be no effectiveness evidence supporting a proportion of the eligible population. Appropriate consideration should be given to these unrepresented tumour types in the appraisal of histology-independent technologies. BHM may be able to provide an estimate of the distribution of treatment effects in this population. Data collection plans, where considered appropriate, should consider the potential for collecting evidence in unpresented tumours to better inform estimates of effect. Consideration should also be given to the fact that unrepresented tumours are not a single tumour type and may be heterogeneous. For this reason, blanket approval or collection of data in unrepresented tumours may not be appropriate.

Genomic testing

Genomic testing is likely to be integral to identify patients eligible for histology-independent therapy. Genomic testing costs may vary substantially across tumour types and, therefore, represent an important potential source of heterogeneity that should be appropriately considered. It is possible that some tumour types will not be cost-effective on the basis of genomic testing costs alone. Current NICE guidance provides that testing should be included where necessary to support a new health-care technology. Investment in universal provision of genomic testing, however, generates challenges to this model because some testing strategies may be used to identify multiple potential targets. In principle, it may, therefore, be appropriate to apportion testing costs over several technologies. It is currently unclear how this should be undertaken or who should make such judgments.

Model structure

Alternative sources of heterogeneity that may impact cost-effectiveness estimates (e.g. baseline risk, treatment effect, costs and HRQoL) should be explicitly acknowledged and appropriately reflected in any economic model. Where an economic analysis is developed using a partitioned survival analysis approach based on the direct extrapolation of TTE end points, appropriate exploration of the validity of pooling PFS and OS across prespecified subgroups and histologies (where data permit) should be undertaken (e.g. separate presentation of Kaplan–Meier curves and landmark PFS and OS rates). The process of internal and external validation should be clearly described.

The BHM approaches may be useful to support the validity of pooling PFS and OS data. Where there is substantive evidence of heterogeneity in treatment effects, consideration should be given to alternative model structures that are better able to reflect this heterogeneity, including the use of landmark response approaches. If such a model is used, evidence supporting the proposed surrogate final relationship should be presented and uncertainty surrounding the surrogate relationships included in the model should be fully characterised. Although concerns remain regarding the validity of response as a surrogate for PFS and OS, a surrogate-based modelling approach informed by predictions from meta-analyses that capture all relevant uncertainty may be preferable to the extrapolation of heavily censored and potentially confounded PFS and OS data. The BRMA approach outlined in DSU TSD 20146 is recommended to ensure that all uncertainty around the surrogate relationship is reflected in the predictions used in the model.

Consideration of uncertainty

Uncertainty is inherent to all decisions made by NICE and other reimbursement agencies, but may be particularly acute when considering histology-independent technologies owing to the limitations of the underlying evidence base. When considering the implications of uncertainty, due consideration should be given to the scale and consequence of decisions because often populations may be small with limited consequences to the overall health system. Quantification of the consequences of uncertainty using NHB may provide an important framework to add to the assessments already routinely specified with the existing TA methods guide4 to quantify decision uncertainty. Routine presentation of such metrics should be considered.