Cancer diagnostic tools to aid decision-making in primary care: mixed-methods systematic reviews and cost-effectiveness analysis

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 16/12/04. The contractual start date was in April 2017. The draft report began editorial review in March 2019 and was accepted for publication in March 2020. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© Queen’s Printer and Controller of HMSO 2020. This work was produced by Medina-Lara et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2020 Queen’s Printer and Controller of HMSO

Scientific background and rationale

The rate of cancer survival in the UK is lower than the European average for most cancers: for example, the 5-year survival rate for stomach cancer is 17.2% in the UK, compared with the European average of 25.1%; for colon cancer, it is 51.8% in the UK, compared with the European average of 57%. 1 Efforts to reduce the time to making a cancer diagnosis have the potential to improve prognosis,2 because earlier diagnosis is associated with earlier cancer stage at diagnosis,3 and earlier treatment is associated with improved survival. 4 There is also the potential to reduce presentation via emergency admissions, and to prevent the poorer survival associated with that route of diagnosis. 5 National cancer screening programmes in the NHS (for breast, bowel and cervical cancer) and the National Awareness and Early Diagnosis Initiative (NAEDI) (to increase public awareness of the signs and symptoms of cancer4) are intended to improve early diagnosis. As many individuals go through primary care as a route for diagnosis,5 efforts there could improve cancer survival.

Cancer diagnosis in primary care is not straightforward. Symptoms of cancer are commonly seen, but mostly have non-cancer origins. 6 Of those individuals referred from primary care via the 2-week-wait (2WW) referrals for suspected head and neck cancer, approximately 9% were ultimately diagnosed with cancer. 7 The type and presence of symptoms can vary greatly,8 and it is not surprising that patients can have multiple general practitioner (GP) consultations before being referred, especially for those cancers that have less well-known signs and symptoms. 9 Thus, tools to help improve cancer diagnosis in primary care have great potential to affect diagnoses and subsequent treatment options, leading to better outcomes for patients.

Diagnostic prediction models combine multiple predictors, such as symptoms and patient characteristics, to obtain the risk of the presence or absence of a disease in an individual patient. 10,11 These prediction models can then be used to develop diagnostic tools (such as a website risk calculator or a mouse mat detailing estimates of risk depending on features) to assist doctors in estimating probabilities, and can potentially influence doctors’ decision-making. 11 To evaluate diagnostic prediction models, there are three important stages, or types, of studies: prediction model development, prediction model validation and assessment of the impact of prediction models in practice (generally implemented as diagnostic tools). The first two are often conducted as part of the same study, and are generally evaluated using a single cohort design. These types of studies are commonly found in the diagnostic prediction literature, with some studies also reporting results of an external validation. 12 To assess the impact of the prediction model (the third stage), comparative studies are required to evaluate the ability of the tool to guide patient management. In the literature on prediction models in general, very few diagnostic prediction models that are developed go on to be evaluated for their clinical impact. 12

Tools currently available to GPs to help cancer diagnosis, beyond the National Institute for Health and Care Excellence (NICE) guidelines for suspected cancer referral,6 are based on the following diagnostic prediction models:

1. the risk assessment tool (RAT) developed by Hamilton et al. ,13 which provides estimates of cancer risk for 17 cancers, based on symptoms alone

2. the QCancer^® (ClinRisk Ltd, Leeds, UK) tool, which estimates the risk of 10 cancers, based on symptoms and patient characteristics, such as age, smoking status and body mass index.

There are clear differences in the derivation of the RAT and QCancer. The RAT used a case–control design to predict likely cancer diagnosis, whereas QCancer used a cohort design. Many of the QCancer prediction models have subsequently been externally validated and reported to have good diagnostic performance. 14,15 There has, however, been no comparison of the clinical effectiveness of these diagnostic tools in clinical practice, or research on whether or not GPs currently have access to and are using these tools.

In 2013, Hamilton et al. 13 reported an increase in cancer referrals and investigations associated with the introduction of RATs as mouse mats and desktop flip charts for lung cancer and colorectal cancer (CRC), and an increase in the awareness of GPs of cancer symptoms, especially those symptoms that are less known in those cancers. 16 A 2015 evaluation of an electronic version of RATs for lung cancer and CRC highlighted the potential issue of prompt overload from the system, cautioned on potential variation in data used by the tool, and the extent to which the aid might increase pressure on secondary care owing to increased referral (a finding that could be generalised to all such diagnostic tools). 17

An Australian study using simulated GP consultations explored the implementation of an aid based on QCancer. 18 The study found that GPs agreed that the diagnostic aid was potentially useful in practice, but noted that different GPs interpreted the same set of symptoms differently, leading to inconsistent estimates of risk from the QCancer aid. In collaboration with the NAEDI, Macmillan Cancer Support developed, with BMJ Informatica, and evaluated the introduction of an electronic clinical decision support (eCDS) containing the RAT and QCancer for colorectal, lung, oesophagogastric, pancreatic and ovarian cancers. It was found that the impact of eCDSs varied across practice, from no impact on referrals to increased referrals and investigations in other practices,19 with use of eCDSs leading to further investigation or referral of the patient that would not have occurred otherwise in 19% of cases.

However, there is very little evidence as to whether or not these tools have led to increased or quicker cancer diagnoses, and, ultimately, to impacts on patient quality of life or survival. A study protocol for a randomised controlled trial (RCT) to evaluate eCDSs for assessing symptoms indicative of stomach cancer was published in 2016. 20 Other diagnostic prediction models have been developed in the UK, such as that reported by Iyen-Omofoman et al. 21 for lung cancer and the Bristol–Birmingham (BB) equation for CRC,22 plus those developed outside the UK, such as Benign, Lonely, Irregular, Nervous, Change, Known (BLINCK) clues in Australia for skin cancer23 and that developed in the USA for ovarian cancer,24 which may have the potential to be useful in the NHS context. However, little is known about whether or not, and how, these diagnostic prediction models and tools affect patient outcomes, and would affect NHS resources.

Although we are unclear about the evidence on the clinical effectiveness of these diagnostic tools to affect patient quality of life and survival, a systematic review conducted by Neal et al. 25 found a large number of studies looking at the impact on patient outcomes of reducing diagnostic and/or treatment intervals for cancer. Only a small number of studies were found to be of high quality, and there was substantial variation in the type of intervals evaluated and the findings within and between cancer types. Compared with other cancer types, studies of colorectal, breast, head and neck, and testicular cancers and melanoma suggested that shorter time intervals were associated with improved patient outcomes. However, for each of these cancer types, there were also studies reporting no association between time interval and patient outcome.

The possible trade-offs between the costs and the harms, and the inherent uncertainty, of using these diagnostic tools in primary care are also unclear, as well as the extent to which reducing times to diagnosis and/or treatment could affect patient outcomes.

Aims and objectives

The aim of this project was to evaluate the evidence on the development, validation, clinical effectiveness and cost-effectiveness of cancer diagnostic tools in primary care, and to understand the extent to which existing tools are currently used in the primary care setting in the NHS.

The objectives were to:

1. identify evidence evaluating the clinical effectiveness of symptom-based diagnostic tools that that could be used to inform cancer diagnosis decision-making in primary care (see Chapters 2 and 3)

2. identify and summarise studies reporting the development, validation or accuracy of any diagnostic prediction model that could be used as a tool to aid cancer diagnosis in primary care (see Chapters 2 and 4)

3. update a previous systematic review25 assessing the association of the durations of different intervals in the diagnostic process to clinical outcomes, and conduct a more in-depth evaluation of CRC studies, focusing on methods, to identify studies that are likely to provide the best estimate of the impact of diagnostic intervals on patient outcomes for informing the decision-analytic mode (see Chapter 5)

4. use a decision-analytic model to explore uncertainties in the cost-effectiveness of using symptom-based diagnostic tools, including the impacts on health service resource use, costs and patient outcomes, using CRC as an example (see Chapters 6 and 7)

5. understand the extent to which GPs currently have access to cancer diagnostic tools and are using them in primary care to inform their decision-making (see Chapter 8).

Introduction

Two systematic reviews were conducted: systematic review (SR) 1 and SR2.

The research question (SR1) was ‘what evidence is there for the clinical effectiveness and cost-effectiveness of symptom-based diagnostic tools that could be used to inform cancer diagnosis decision-making in primary care?’. It was anticipated that limited evidence would be identified; therefore, a second review was planned (SR2) to identify studies reporting the development and validation of any diagnostic prediction model that could be used as a tool to help cancer decision-making in primary care, that is to provide a list of cancer models that might have the potential to be developed into diagnostic tools. This chapter clarifies the definitions of diagnostic tools and prediction models used and describes the search methods employed for SR1 and SR2. The results are presented separately in Chapters 3 (SR1) and 4 (SR2).

Diagnostic prediction models can be categorised according to the different stages of development and evaluation of the model (Table 1). However, a number of predictive research studies have differed in what they consider to fall under the ‘predictive model research’ header. This variation primarily related to whether or not they incorporated predictor finding studies as an initial stage and, at the other end of the spectrum, whether or not they incorporated impact or implementation studies. For the purpose of our reviews, we have differentiated between prediction models and prediction tools. Diagnostic prediction models are defined as multivariate statistical models that predict the probability or risk that a patient currently has cancer based on a combination of known features of that patient, such as symptoms, signs, test results and patient characteristics. 26 Symptoms could be self-reported by the patient, or prompted by a physician’s questioning. Signs and test results are identified in primary care via routine testing (e.g. full blood count, urine dipstick testing, clinical signs), and patient characteristics are also determined in primary care (e.g. sociodemographic variables, personal and family history). The prediction tools implement the models to provide a numerical risk of having cancer. As examples, prediction tools may be mouse mats or desktop flip charts, or may be integrated into information technology (IT) systems.

As a 2009 study by Moons et al. 28 points out, studies assessing the impact of prediction tools need designs and outcome measures that are different from those used in studies that develop and evaluate prediction models. Studies of predictive research26,27,29,30 also differ slightly in the way they categorise studies, based on whether they considered the evaluation of internal validity to be part of the model development stage or a separate stage.

Table 1 summarises the spectrum of potential study types and clarifies the inclusion and exclusion criteria for SR1 and SR2.

Methods

The systematic reviews were conducted in accordance with good practice guidelines. 31 As the inclusion and exclusion criteria were very similar for SR1 and SR2, the same search strategy was used for both reviews; however, two separate protocols were developed for reviewing the evidence. Further details are presented in this chapter.

Results

Studies identified

Search phrases were finalised and searches were run in May 2017 (see Appendix 1). A total of 9352 records were obtained through database searching. Additional reference and citation searches on tool names resulted in another 4171 records. After de-duplication, 9780 records were obtained. The database searches were updated in January 2018, resulting in 631 additional new records (after de-duplication), and again in November 2018, when 702 hits were identifeid. Discussions with collaborators led to the identification of relevant grey literature, but no such studies were deemed eligible for inclusion.

As hits were screened simultaneously for inclusion in SR1 or SR2, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)33 flow diagram shows the results for both reviews: 260 full-text articles were screened: five studies met the inclusion criteria for SR1 and 41 studies met the inclusion criteria for SR2 (Figure 1).

FIGURE 1.

The PRISMA flow diagram of the included studies for SR1 and SR2.

Details on the methods for data extraction, on assessing the risk of bias of the included studies and on the findings of the SRs are presented in Chapters 3 (SR1) and 4 (SR2).

Objective

The objective was to identify evidence evaluating the clinical effectiveness of symptom-based diagnostic tools that that could be used to inform cancer diagnosis decision-making in primary care.

Methods

Data extraction

To extract relevant data from each included study, standardised data extraction forms were used, which evolved following piloting and discussion among reviewers. One reviewer (BG) extracted the data, which were checked by a second reviewer (RL). Extracted data included cancer type(s); study design; country; sample size; patient recruitment (with inclusion and exclusion criteria); characteristics of the tool (including whether based on symptoms alone or other features in addition to symptoms); definition of outcomes (including the number of cancer diagnoses, time to cancer diagnosis, stage of cancer at diagnosis, resection rates, patient health-related quality of life, other patient-reported outcome measures); main results, including confidence intervals (CIs); and subgroup analyses, when available.

Results

Discussion

This review attempted to summarise existing evaluations of feature-based cancer diagnostic tools used in primary care. Our strategy was able to identify a limited number of heterogeneous studies that did not provide strong evidence of the impact of feature-based diagnostic tools on patient-related outcomes or referral patterns. The small number of studies (n = 5) and the heterogeneity in reported outcomes did not allow for a meta-analysis of the results. The included studies provided limited evidence of the clinical effectiveness of using diagnostic tools.

A few other reviews have looked at feature-based cancer diagnostic tools in primary care. Williams et al. 40 conducted a systematic review of studies that described, validated or assessed the impact of CRC diagnostic tools. However, they did not identify any studies that tested whether or not patients who were diagnosed with the aid of the tool fared better than those who were diagnosed without it. Schmidt-Hansen et al. 41 conducted a similar review of lung cancer tools and found limited evidence to support the recommendation of any of the identified risk prediction tools, owing to lack of external validation or cost impact assessment. Similarly, Usher-Smith et al. 42 concluded that, even though some of the prediction models had the potential for clinical application, there remains considerable uncertainty about their clinical utility.

Other reviews have looked at cancer RATs in primary care. 43,44 However, they differ from this review in that they reported on tools that estimate the risk of developing cancer in the future, rather than the risk of having an undiagnosed cancer based on current signs and symptoms.

Although the intention of our review was to explore tools based only on prediction models, it was not clear whether or not, in practice, the impact of such tools could be isolated from other decision-making tools available to practitioners, such as diagnostic algorithms45 or guidelines. 6 With limited evidence available on the impact of implemented diagnostic models, we decided to report on identified studies on the algorithm-based tools as well; however, the evidence was still sparse.

Among the limitations of the included studies were lack of randomisation, lack of patient-related outcomes and use of models developed on different populations. The outcome measures used by some of the studies make it difficult to interpret reports of an increase in referral rate without including reasonable assessment of the appropriateness of the referral or subsequent impact on cancer versus non-cancer diagnosis.

Furthermore, concerns on the quality of the studies make it unclear whether the lack of effect was due to poor implementation of the tools in practice, insufficient uptake by the GPs or limited marginal contribution of the tools in assessing the risk of cancer. The best-quality study35 also failed to show a significant effect; however, the composite intervention used, combining older versions of several instruments (developed on populations from a different country), could have limited the clinical effectiveness of the diagnostic tools. These findings could be further obfuscated by publication bias, whose magnitude on this topic remains unknown.

Conclusion

Current evaluations provide limited evidence of the impact on patient outcomes of using feature-based cancer diagnostic tools in primary care. Better research is needed to provide these data, possibly through better study design and choice of outcomes. However, identifying the ideal approach may not be straightforward. Practical reasons may highlight the potential need for a cluster and pragmatic trial design. Arguably, by comparing average times to diagnosis, patients not prioritised for quick referrals are less at risk of being missed. The debate, however, is ongoing on the most appropriate outcomes for evaluating interventions to improve cancer diagnosis and referral.

Objective

Systematic review 2 was conducted as a complementary study to SR1, described in Chapter 3. The objective was to identify and summarise studies reporting the development, validation or accuracy of any diagnostic prediction model that could be used as a tool to aid cancer diagnosis in primary care.

Methods

Results

Discussion

Looking at symptom-based cancer diagnostic prediction models currently under development, we were able to identify 43 studies in total. The majority of these reported on various aspects of just two such modelling versions, QCancer and RAT, both developed in the UK. Most of the reported work also seemed to be located in Europe, with two of the models developed in the Netherlands.

The majority of the models included in this review were developed only with the sample used to derive the model. With the exception of the RAT (colorectal) model, there were no reports to suggest that models were being updated based on new available data. Some models were validated using split-sample techniques. A number of models, in particular the ones developed under the QCancer name, were externally validated in independent studies.

The two main models highlight important knowledge gaps; the development of the QCancer models was based on higher-quality data (cohort data) than that of the RATs, and were mostly externally validated, but lack impact assessment. By contrast, the RAT series has more evidence of impact on practice, but was developed from case–control studies and has limited external validation.

Our systematic review was limited to the inclusion of diagnostic prediction models; however, the search strategy also highlighted a number of studies concerned with the development of symptom-based decision tools that were not based on prediction models, such as scoring systems, as well as the development of diagnostic prediction models for secondary care. These are listed in Appendix 3 (see Tables 43 and 44) as a resource for further research.

Evidence suggested that there has been a great deal of external validation work for QCancer, whereas we found only one attempt at external validation of RATs. Ideally, this is an area for further development of the RATs and the four other models that have not yet been externally validated.

Conclusion

To our knowledge, this is the first systematic review of diagnostic prediction models for use in primary care to aid cancer diagnosis. We have identified models that have the potential to be developed into tools and be used by GPs. However, there are gaps in the literature that we have identified.

Currently, most research on developing symptom-based cancer risk prediction models is concentrated in Europe and, in particular, the UK. QCancer and RATs are the dominant prediction models. Although there has been a great deal of external validation work done for QCancer, only one study was identified that reports the external validation of RATs.

Introduction

An update of a previous systematic review by Neal et al. 25 (published in 2015) was undertaken to examine the association between different durations of time from first symptom to diagnosis or treatment and clinical outcomes across all major cancers for symptomatic presentations, to investigate whether or not a more timely cancer diagnosis is associated with more favourable outcomes. It had been anticipated that there would be very little evidence on whether or not diagnostic tools have led to increased or quicker cancer diagnoses and, ultimately, impacts on patient survival. This updated review would therefore provide important information to inform the proposed decision model of a cancer diagnostic pathway for assessing the impact of using diagnostic tools.

The cancer diagnostic pathway is complex, incorporating multiple key time points and intervals from the patient first experiencing symptoms to receiving a definitive diagnosis or treatment (Figure 2). An initiative to improve the design and reporting of early-cancer diagnosis research included a review of existing instruments used to measure time points and intervals was undertaken. The project also incorporated a consensus conference approach combined with nominal group techniques to develop a series of recommendations for definitions and methodological approaches, which culminated in the Aarhus consensus statement, published in 2012. 92 The Aarhus checklist is a resource for early-cancer diagnosis research that aims to promote greater precision and transparency in both definitions and methods used. 92

FIGURE 2.

Illustration of the different interval types. T1, time from symptom onset to first seen in primary care (‘patient interval’); T2, time from symptom onset to referral to specialist care; T3, time from symptom onset to first seen in specialist care; T4, time from symptom onset to diagnosis; T5, time from symptom onset to treatment; T6, time from first seen in primary care to referral to specialist care (‘referral interval’); T7, time from first seen in primary care to first seen in specialist care; T8, time from first seen in primary care to diagnosis (‘diagnostic interval’); T9, time from first seen in primary care to treatment; T10, time from referral to specialist care to first seen in specialist care; T11, time from referral to specialist care to diagnosis; T12, time from referral to specialist care to treatment; T13, time from first seen in specialist care to diagnosis; T14, time from first seen in specialist care to treatment; T15, time from diagnosis to treatment (treatment interval).

The broader evaluation of how reductions in time to diagnosis or treatment can affect patient outcomes with any cancer not only allows the consideration of the probable impact of using diagnostic tools in these cancers, but also provides a greater evidence base on the probable impact that could be expected from reducing time to diagnosis and/or treatment in general. A more in-depth evaluation of CRC studies was undertaken in this updated review, as this was the focus of our decision-analytic model. This in-depth evaluation focused on the methodology used in those studies. The decision-analytic model focused on the use of diagnostic tools in an adult population; therefore, unlike the previous review of diagnostic intervals by Neal et al. ,25 we did not consider childhood cancers as part of the updated review. The previous review and this updated review considered any type of interval along this pathway, which were grouped according to accepted definitions, relating to the patient, primary care, secondary care or combinations of any of those categories. 92

Aim

The aim of this review was to update a previous systematic review25 assessing the evidence linking the durations of different intervals in the diagnostic process to clinical outcomes. A more in-depth evaluation of CRC studies, focusing on methods, was conducted to identify studies that are likely to provide the most valid estimate of the impact of diagnostic intervals on patient outcomes for informing the decision-analytic model.

Methods

Results

Studies identified

Our searches identified 1810 references after de-duplication, of which 104 were considered relevant and retrieved as full-text studies. Thirty-five studies were identified that met the inclusion criteria (Figure 3). This includes one study for which the interlibrary loan was still outstanding at the time of the report; therefore, the data extraction is based on an abstract. This study122 included patients with well-differentiated neuroendocrine tumours.

FIGURE 3.

Updated review: the PRISMA flow diagram.

A summary of the included studies is provided in Appendix 4 (see Table 51). Two studies considered multiple cancer sites, whereas the remaining studies evaluated a sole cancer: bladder (n = 1), breast (n = 4), colorectal (n = 8), lung (n = 5), lymphoma (n = 1), myeloma (n = 1), neuroendocrine (n = 1), oral (n = 2), ovarian (n = 2), pancreatic (n = 2), penile (n = 1), prostate (n = 1), sarcoma (n = 3) and testicular (n = 1). The studies were from a wide range of countries. Eight studies were undertaken in the UK, five in Canada and four in Spain; two studies were undertaken in each of the following countries: the USA, the Netherlands, Mexico and Japan; and one study was carried out in each of the following countries: China, France, Islamic Republic of Iran, Israel, Italy, Montenegro, Poland, Rwanda, Sri Lanka and Uganda.

Discussion

Conclusion

The overall findings of the updated review highlight the uncertainty in the evidence base for the extent to which reducing times to diagnosis and/or treatment leads to improved patient outcomes. There is still a lack of firm evidence to demonstrate an association between timely diagnosis and favourable outcomes in those with CRC. The evidence indicates a U-shaped relationship, but also reflects considerable heterogeneity in terms of the methods and intervals evaluated, and highlights some important methodological challenges in evaluating whether or not timely diagnosis leads to better outcomes. The findings support the need to expedite the diagnosis of symptomatic patients and to work to prevent the preventable delays that some patients continue to experience.

Objective

The economic decision-analytic model explored the uncertainties regarding the clinical effectiveness and cost-effectiveness of the tools to aid cancer diagnosis decision-making in primary care in the NHS. Given the scope of the current study, it was not feasible to model the impact of diagnostic tools for all common cancer types for which diagnostic tools exist. Instead, at the outset, we chose to model one common cancer, CRC, to illustrate a case for which we believe there are good a priori reasons why diagnostic tools in primary care could affect patient outcomes and may be effective and cost-effective. CRC is among the cancers with the shorter diagnostic intervals (31–60 days); cancers with longer diagnostic intervals, for example bladder cancer (61–91 days) and lung cancer (91–120 days), would have greater scope for diagnosis. If clinical effectiveness and cost-effectiveness results are favourable for CRC, the modelling approach could be extended to other cancers, but this is beyond the scope of the current study.

We will use the decision-analytic model to link empirically demonstrated impacts on short- and medium-term outcomes to the effects of treatment, to estimate the impact on longer-term outcomes. In particular, we will explore the potential impacts on health service resource use, costs and patient outcomes in relation to CRC. This chapter focuses on the sources of information from previous chapters, a methodological review of the decision-economic models on CRC that have been published and the data for populating the decision-analytic model.

Summary of findings for the decision-analytic model

Systematic review of existing economic decision-analytic models for colorectal cancer

Results

Studies identified

A first electronic search retrieved a total of 2730 hits; after de-duplication, 2129 were screened, of which 2109 hits were excluded. In addition, we were aware of two key cost-effectiveness studies that were missed in the search and decided to run a new search with a more refined set of search terms (for the revised search, see Appendix 5) to identify more cost-effectiveness models. We assessed the full text of 20 studies from the first search, seven additional studies from the revised search and one new study from the updated search. After a more in-depth screening of these 28 studies, 13 papers were excluded: for nine papers, the disease model description was not based on the Dukes’ or AJCC staging; three screening models did not include a full disease model; and the last excluded study was a costing study of setting up a screening programme. Reference-searching led us to three additional studies; 18 studies, in total, met the inclusion criteria for the review. Further details are presented in Figure 4.

FIGURE 4.

Economic decision-analytic models: the PRISMA flow diagram. a, Revised strategy; b, updated search using the revised strategy; c, a total of 546 hits were screened with the updated search; and d, further exclusion details are presented in Appendix 5.

Study characteristics

Table 13 provides the overview of the included models. Over 50% (10/18) of the studies were conducted in Europe (seven of these were in the UK), six in North America (four in the USA and two in Canada) and two in Asia.

Over 80% (15/18) of the studies were two-part models, mainly screening and full disease progression models (12/18); only three studies were diagnosis and natural history disease models. The other three models included in this review were disease progression models only. Two exceptions were made of studies that did not meet the inclusion criteria. The study by Cantor et al. 157 evaluated a patient decision aid for CRC screening; it was reviewed as it was deemed to be important for developing the diagnosis part of the model. Similarly, the model by Zauber et al. ,147 which excluded resource use and costs, was included despite not meeting the inclusion criteria.

In Table 14, the study characteristics are presented. Westwood et al. 161 is the only study of a diagnostic strategy in the population of interest to our study, namely a comparison of faecal immunochemical tests (FITs) at various thresholds against faecal occult blood tests (FOBTs) in a low-risk symptomatic population in a primary care setting. Of the two remaining studies of diagnostic strategies, only one144 compared two or more diagnostic strategies, and, although it is conducted in a primary care setting, it is concerned with a high-risk symptom only (rectal bleeding in a person aged 55 years) in US primary care. The remaining study, by Tappenden et al. ,155 is a whole-disease model, which captures the complete pathway of CRC patients from a healthy state to clinical disease down to the detail of treatment sequences used by age and disease stage. This study’s aim is not sufficiently detailed to account for the diagnostic pathway of low-risk symptoms, although it accounts for the role of a RAT in the transition from asymptomatic to clinical disease,39 and its main interest to our purposes lies in the methods used to build the disease history model.

Further, the disease history model in the study by Tappenden et al. 145 is used by most of the included screening studies145,149,154,156 summarised in Table 14 with few modifications. In common with most models, this model used a lifetime time horizon and considered cases aged 50–60 years.

None of the identified studies evaluated a diagnostic tool. Therefore, attention will be devoted in the rest of this chapter to reviewing the suitability of identified models to accommodate any evaluation of such tools, and the quality of the available evidence synthesised by existing models to inform the assessment of tools using any suitable existing or new model.

Critical appraisal: key model features

The critical appraisal data extracted for the 18 included model-based economic evaluation studies included are presented in Appendix 5, and the findings are summarised in this section.

Over 60% (11/18) of the studies considered the NHS perspective and only four studies144,147,150,158 included a societal perspective. The perspective in three studies146,157,159 was not explicitly stated, but, given the costs included, it appears to be their respective health systems’ perspectives.

There were marked differences between the models’ assumptions for the long-term rate of CRC disease progression. In 40% (8/18) of the studies,145,147,148,151,152,155,156,161 the most common assumption was to apply an annual disease progression rate, the choice of which was clearly defined by the history of the disease. The selection of key parameters was justified in all the studies, although the quality of the chosen parameters was not fully discussed in one study. 150

Fifty per cent of the studies (9/18) used the QALY as an outcome measure; life-years were used in six studies,146,147,150,153,157,159 and three studies149,154,161 included both QALYs and life-years. The parameterisation into the models was described with sufficient detail and with all the relevant sources of data in most of the studies (14/18). However, the relevance of the assumptions was difficult to understand in two studies,144,160 and how the data were incorporated into the model was unclear in two studies. 159,160

Over 80% (15/18) of the studies addressed properly the different types of uncertainty; in particular, point estimates and values for sensitivity analyses were stated and justified in most of the studies, except for three studies150,159,160 in which assumptions were less clearly described. All studies except for Pil et al. 158 justified the values used for the sensitivity analysis.

Probabilistic sensitivity analysis was undertaken by 55% (10/18) of studies, and one study153 justified the absence of probabilistic sensitivity analysis based on the lack of evidence on sampling variation of parameter values. In contrast, this information was vaguely stated in, or missing from, seven studies. 144,146,147,150,157,159,160 The choice of parameter distributions used was clearly reported in 50% (9/18) of the studies; however, five studies144,154,157,159,160 omitted the information with respect to second-order uncertainty.

The internal consistency was evaluated in 10 studies only. 144,145,147–149,152,153,155,156,161 In the absence of data on key parameters, the model was calibrated against independent data in eight studies,145,147,148,151–153,155,156 such as the rate of disease progression in asymptomatic CRC states.

Modelling approaches

Five different types of decision models were identified in the review: (1) 11 Markov state-transition models,144–146,148,149,151,152,154,156,159,161 (2) four microsimulation models,147,150,153,160 (3) one Markov model combined with a decision tree,158 (4) one decision tree157 and (5) one discrete event simulation model. 155 The colorectal health cancer states were modelled using the AJCC staging (9/18) or the Dukes’ staging systems (8/18). The model diagnosis by Cantor et al. 157 did not consider any staging.

Markov models were preferred because of their flexibility and simplicity in populating and calibration processes. Specifically, the transition within health states can be easily represented by a cohort in a Markov model. Given the limited number of data on key disease history model parameters, a Markov model may be more suitable to synthesise the clinical and economic evidence for evaluating diagnostic, screening and surveillance strategies in CRC than other types of models such as microsimulation and discrete event simulation models. The only study that used a discrete event simulation model used such an approach to accommodate the distribution of time to event survival and disease progression outcomes of CRC treatment, which were found to have limited importance in the results of the only diagnostic study of a low risk of suspected cancer in primary care (Westwood et al. ;161 see Appendix 5). Likewise, the three studies150,153,160 that used microsimulation models used such an approach for its convenience to account for the interactions between individual characteristics and risk factors of the screening tests and combining cohorts with different characteristics to produce results averaged across subgroups of the patient population.

Detailed review of individual models

UK models

Tappenden et al. 145 constructed a model for England incorporating three elements: (1) a screening intervention model that included subsequent colonoscopy surveillance; (2) a state-transition model that simulated the natural history of colorectal neoplasia from normal epithelium, to adenoma and carcinoma sequence; and (3) a mortality model that considered age-specific other causes mortality, CRC mortality and mortality from perforation due to endoscopic procedures. Transitions between model health states were calculated using an annual cycle length until the entire model cohort was absorbed into the dead state. The model assumed that the incidence of adenomas was 1.60% and that the incidence of cancer from adenomas was 3.26%. Five screening options versus no screening were evaluated: biennial FOBTs, for people aged 50–69 and 60–69 years; two once-only flexible sigmoidoscopies, for people aged 55 and 60 years; and the fifth option was once-only flexible sigmoidoscopy for individuals aged 60 years, followed by biennial FOBTs for individuals aged 61–70 years.

The UK model developed by Lee et al. 149 was a two-part model: (1) the evaluation of screening strategies for CRC and (2) a state-transition Markov model representing the natural history of colorectal neoplasia (from normal colorectal epithelium to adenoma–carcinoma sequence, to death). The four screening strategies evaluated were (1) FOBT every 2 years, (2) flexible sigmoidoscopy every 10 years, (3) optical colonoscopy every 10 years and (4) computerised tomography colonography (CTC) every 10 years. The authors used data from the study by Tappenden et al. 163 and recalibrated the data to adjust for estimates from recent observational data on CRC incidence, mortality and staging of cancer at time of diagnosis. The progression from normal epithelium to low-risk adenoma, low-risk adenoma to high-risk adenoma, and high-risk adenoma to Dukes’ stage A CRC was 1.2%, 2.4% and 3.4%, respectively. The mortality for other causes was modelled as an age-dependent probability based on UK life tables,164 and the probability of dying as a result of endoscopic perforation from either the screening procedure or the polyp removal (polypectomy) was included in the model as a potential complication of the interventions.

The two interlinked models developed by Whyte et al. 152 had (1) a model that described the screening interventions and surveillance of CRC (from invitation to screening, screening test, follow-up, diagnostic tests and surveillance) and (2) a natural history of colorectal neoplasia model (from normal colorectal epithelium to adenoma–carcinoma sequence to death). The screening intervention used the following diagnostic strategies: (1) no screening; 2) biennial guacal faecal occult blood test (gFOBT) at 60–69 years; (3) biennial gFOBT at 60–74 years; (4) immunochemical faecal occult blood test (iFOBT) at 60, 65 and 70 years; (5) biennial iFOBT at 60–69 years; (6) biennial iFOBT at 60–74 years; (7) flexible sigmoidoscopy at age 55 years; (8) flexible sigmoidoscopy at age 55 and 65 years; (9) flexible sigmoidoscopy at age 55 years and biennial gFOBT at ages 66–74 years; (10) flexible sigmoidoscopy at age 55 years and biennial iFOBT at ages 66–74 years; (11) flexible sigmoidoscopy at age 55 and iFOBT at ages 60, 65 and 70 years; (12) flexible sigmoidoscopy at age 55 years and biennial iFOBT at ages 60–74 years; and (13) flexible sigmoidoscopy at age 55 years and biennial iFOBT at ages 56–74 years. The authors assumed that probabilities associated with the transition to low-risk adenomas, the transition from low- to high-risk adenomas and the transition from high-risk adenomas to Dukes’ stage A CRC were age dependent.

An interconnected model developed by Gomes et al. 154 comprised (1) diagnostic test and the subsequent adenoma surveillance model for symptomatic patients in secondary care, and (2) a Markov model, based on CRC natural history in the UK simulating the development of CRC in the population (from normal colorectal epithelium to adenoma–carcinoma sequence to death). The model assumes a variable annual rate of polyp development of 1.9% to 3.3% and an annual incidence of CRC from adenomas of 3.4%. The model evaluated the impact of three-dimensional CTC versus optical colonoscopy to improve the detection of adenomas and CRC.

Tappenden et al. 155 developed a patient-level model that simulates the disease progression and treatment pathways from pre-clinical disease through detection, diagnosis, adjuvant/neoadjuvant treatments, follow-up, curative/palliative treatments for metastases, supportive care and eventual death. The authors assumed a detailed age-specific incidence of CRC and adenoma prevalence based on Bayesian Markov chain Monte Carlo methods. The whole-disease model considered all the processes that are linked to CRC from screening, surveillance and treatment.

The model developed by Whyte et al. 156 comprised (1) the evaluation of the effect of a campaign to increase the awareness of the signs and symptoms of CRC and encourage self-presentation to a GP, and (2) a model to represent the development of CRC in the population (from normal colorectal epithelium to adenoma–carcinoma sequence to death). The authors assumed an increased presentation rate of 10% in each CRC Dukes’ stage and an age-dependent incidence of adenomas (low and high risk) and CRC.

Westwood et al. 161 constructed a model for England incorporating three elements: (1) a decision model reflecting the diagnosis of CRC, (2) a Markov state-transition model to estimate the long-term costs and effects related to treatment and progression of CRC and (3) a Markov state-transition model to estimate effects for patients without CRC. The transitions between the model health states of the two Markov models were calculated using an annual cycle length until the entire model cohort was absorbed into the dead state. The model assumed the incidence of adenomas and the incidence of cancer (from adenomas reported in Tappenden et al. 145) to be 1.60% and 3.26%, respectively. The model evaluated three options of symptomatic individuals that presents to primary care: (1) FITs, (2) gFOBTs or (3) no triage tests at all (referral straight to colonoscopy).

European models

The Irish study by Sharp et al. 151 was a Markov state-transition model with three interlinked components: (1) the impact of screening and subsequent adenoma surveillance, (2) a natural history of colorectal neoplasia model (from normal colorectal epithelium to adenoma–carcinoma sequence to death) and (3) the impact on mortality. The authors assumed that 14% of individuals progressed from normal epithelium to stage I cancer. The second model component used gFOBT, FIT and flexible sigmoidoscopy for screening, and the diagnosis used the sensitivity and specificity of colonography and CTC. Finally, mortality in the model was assumed to be from CRC, endoscopic bowel perforation or other causes; other causes were obtained from Irish life tables and were modelled to be age dependent within each Markov cycle. The risk of dying from endoscopic perforation was considered only when using flexible sigmoidoscopy, diagnostic investigations and adenoma surveillance. The model assumed that individuals had a higher probability of dying if they had a more advanced cancer stage.

The work by Goede et al. 153 was based on the MIcrosimulation SCreening ANalysis (MISCAN) microsimulation model, which had three integrated components: (1) a model evaluating the screening strategies of one-sample and two-sample FIT screening with variable intervals, age ranges and cut-off levels (age at which to start screening: 45, 50, 55 and 60 years; age at which to stop screening: 70, 75 and 80 years; and screening interval at 1, 1.5, 2 and 3 years); (2) the natural history component part that simulates the development of CRC in the population (from normal colorectal epithelium to adenoma–carcinoma sequence to death); and (3) a demography component that simulates individual life histories without CRC to form a population. Authors assumed an age-dependent adenoma incidence and adenoma progression, whereas the CRC incidence rate was based on the observed incidence rate in the Netherlands.

The model developed by Pil et al. 158 comprised two interconnected submodels: (1) a decision tree to simulate the screening process based on FITs and (2) a state-transitional Markov model simulating the natural progression of the disease (from normal colorectal epithelium to adenoma–carcinoma sequence, diagnosis and treatment, follow-up to death). The authors assumed a sex- and age-dependent adenoma prevalence varying from 1.6% to 6.01%, and a CRC prevalence ranging from 0.11% to 0.84%.

North American models

Allen et al. 144 used a Markov decision model composed of three elements: (1) a model to evaluate four diagnostic strategies of a patient with rectal bleeding, (2) a model to simulate the natural history of colorectal neoplasia for patients with rectal bleeding (from no serious pathology to adenoma–carcinoma sequence and surveillance to death) and (3) a mortality model to estimate mortality among patients with CRC. The authors assumed that the progression of smaller adenomatous polyps to large polyps would take an average of 10 years and that the progression of large adenomatous polyps to invasive cancer would also take an average of 10 years. Once invasive cancer was detected, the average transition of 2 years, 1 year, and 1 year for Dukes’ stage A to B, B to C, and C to D staging, respectively, was assumed. It was also assumed that 7% of individuals progressed from adenomas to Dukes’ stage A and that the prevalence of adenomas was 16%. The diagnostic model evaluated (1) watchful waiting, (2) flexible sigmoidoscopy, (3) flexible sigmoidoscopy with consequent air-contrast barium enema and (4) colonoscopy. Finally, the mortality estimates used within each stage of cancer were obtained from the Surveillance, Epidemiology, and End Results (SEER) cancer registry.

Zauber et al. 147 used two previous developed microsimulation models [Simulation Model of Colorectal Cancer (SimCRC) and MISCAN] to reproduce the natural history of colorectal neoplasia (from no lesion to adenoma–carcinoma sequence to death). The semi-Markov microsimulation model simulated the effect of screening and other interventions on the incidence and mortality of CRC. It is a hybrid model, made of a Markov model and a discrete event simulation that described the progression of the underlying colorectal disease (e.g. from adenoma to carcinoma sequence) in an unscreened population. Authors assumed an age-dependent variability of adenoma prevalence varying from 10.2% to 36.7% and a CRC incidence varying from 5.3% to 7.3%. The model evaluated the following screening strategies: (1) no screening, (2) colonoscopy, (3) FOBT, (4) flexible sigmoidoscopy with biopsy and (5) flexible sigmoidoscopy combined with FOBT. For each strategy, individuals were evaluated at the following ages: 40, 50, 60, 75 and 85 years. FOBT strategies considered screening intervals of 1, 2 and 3 years; for the sigmoidoscopy and colonoscopy strategies, the intervals considered were 5, 10 and 20 years. The prevalence of pre-clinical cancer and adenomas was assumed to be zero.

The two-part model in Heitman et al. 148 included (1) the impact evaluation of screening strategies on average risk in a North American population and (2) a Markov model reproducing the natural history of the colorectal neoplasia (from normal colon to non-advanced/advanced adenomas, to CRC, to death). The authors assumed a prevalence of 17% of non-advanced adenomas, 3.8% for advanced adenomas and 0.1% for CRC. The screening strategies evaluated were (1) gFOBT or FIT annually, (2) faecal deoxyribonucleic acid (DNA) test every 3 years, (3) flexible sigmoidoscopy or CTC every 5 years and (4) colonoscopy every 10 years. The mortality rate was derived from an age-dependent population from Canada and the CRC mortality rates observed for patients with CRC according to their stage at diagnosis.

Knudsen et al. 150 developed a microsimulation model based on a previous study147 (SimCRC) to represent the natural history of colorectal neoplasia (from normal colon to non-advanced/advanced adenomas, to CRC, to death) and the relative effects of screening strategies. The prevalence, size, location and multiplicity of adenomas and the incidence of CRC was assumed from the SEER programme. The model evaluated five rescreening strategies for individuals with a negative colonoscopy result at 50 years of age: (1) no further screening, (2) continuing colonoscopy every 10 years, (3) rescreening with annual highly sensitive guaiac-based FOBT, 4) annual FIT or (5) CTC every 5 years. Rescreening was assumed to begin at 60 years (10 years after the negative colonoscopy result) for all strategies.

Cantor et al. 157 developed a decision-analytic model comprising a decision tree to evaluate the impact of a decision aid for CRC screening. The study does not present a disease model and there is no indication of patient characteristics. The model evaluated two strategies: use of decision aid and no use of the decision aid. The decision aid supported the selection of the patients who would undergo screening. Each patient undergoing screening can have one of three diagnostic tests: (1) flexible sigmoidoscopy, (2) FOBT or (3) colonoscopy, or not undertake screening. The study assumed a change in the preferences of the screening, with a consequent increase in the number of tests, number of patients undergoing the screening and costs.

The model by Coldman et al. 160 (OncoSim-CRC) comprised (1) a model to predict the outcomes of biennial screening in a cohort for eight FIT threshold values of between 50 and 225 ng/ml and (2) a model simulating the natural history for the development of CRC (from normal colorectal epithelium to adenoma–carcinoma sequence, to death). The authors assumed that the prevalence of adenomas was age, sex and size dependent. The age- and sex-specific CRC incidence and mortality rates were derived from the Canadian population.

Asian models

Tsoi et al. 146 developed a Markov model comprising three submodels that evaluated which primary screening procedure should be adopted to reduce CRC incidence. The diagnostic tests considered were (1) FOBT, (2) flexible sigmoidoscopy and (3) colonoscopy. The annual mortality rates of CRC diagnosed at different stages were based on the Hong Kong Cancer Registry,165 and the mortality due to perforation of endoscopic tests was considered to be 10%. The authors assumed a prevalence of 10% for the first stage of CRC and, that 14% of individuals progressed from normal epithelium to stage I cancer.

Wong et al. 159 used an approach similar to that of Tsoi et al. ,146 but focused on developing a Markov process model to evaluate five screening strategies: (1) flexible sigmoidoscopy as a primary screening test every 5 years at the ages of 50, 55, 60, 65 and 70 years; (2) colonoscopy as a primary screening test every 10 years at the ages of 50, 60 and 70 years; (3) flexible sigmoidoscopy for each female subject every 5 years at the ages of 50 and 55 years; (4) flexible sigmoidoscopy for each female subject every 5 years at the ages of 50, 55, 60 and 65 years; and (5) flexible sigmoidoscopy for each female subject every 5 years at the ages of 50, 55, 60, 65 and 70 years. The age-stratified incidence of CRC of the Hong Kong population was based on reports from the Hong Kong Cancer Registry.

Model population and time horizon

UK models

In the models developed by Tappenden et al. ,145 Lee et al. 149 and Whyte et al. ,156 individuals were aged 30 years. In the model developed by Westwood et al. ,161 individuals entered the model at age 40 years, and in the models developed by Whyte et al. 152 and Gomes et al. ,154 individuals were aged 50 years. The only study that included individuals from birth is Tappenden et al. 155 All of the UK models followed individuals until death.

European models

In line with the UK models, in two of the European models, by Sharp et al. 151 and Pil et al. ,158 individuals entered the model at the ages of 30 and 50 years, respectively. However, only the model developed by Sharp et al. 151 followed up individuals until death, whereas, in the model developed by Pil et al. ,158 individuals were followed up until the age of 70 years. In comparison, the study by Goede et al. 153 modelled the age distribution of the Dutch population in 2005 and all the cohort individuals were followed up until death.

North American models

In two of the US models,144,147 the cohorts entered at age 40 years and were followed up to the age of 100 years or death. However, Heitman et al. 148 and Knudsen et al. 150 considered older cohorts (aged 50 years); both studies followed up individuals to death. Although the cohort of the Canadian model148 entered at the age of 45 years and was followed until death, Cantor et al. 157 did not specify any age for their cohort.

Asian models

In both Asian models, the cohort entered at the age of 50 years, but the model developed by Wong et al. 159 followed up individuals to 70 years of age and the model by Tsoi et al. 146 followed up individuals to 80 years of age.

Critique

For the purposes of the current research questions, the review has helped to identify the Westwood et al. 161 model as a useful precursor for the diagnostic phase. This model is important from two perspectives: first, it is the only evaluation of primary diagnostic tests for CRC identified by our review, and, second, the population included are the low-risk symptomatic population of interest. The focus of the critique will therefore be its methodology, as well as the source of the evidence used to populate its key parameters to inform the approach taken in the novo model. In particular, we will review the mechanisms used to link the diagnostic pathway to the Markov model of disease treatment used by Westwood et al. 161 in terms of their suitability for incorporating a diagnostic phase that has enough granularity to allow an evaluation of costs and health benefits of tool-based diagnostic strategies.

The analysis by Westwood et al. 161 informed the NICE Diagnostic Guideline Number 30140 and assessed the relevant low-risk population for using the diagnostic tools of interest to our study. A strength of the model is that it considers an intervention, FIT, that has now become the recommended standard practice in the UK. The model has policy appeal as it considers the option of referring all symptomatic patients directly to secondary care, a natural option for a country where policy-makers are concerned about expediting access to diagnosis and treatment as a way to improve cancer survival outcomes.

An attractive feature of the Westwood et al. 161 model is that it shows that detecting more CRC cases with diagnostic tests comes at the cost of imposing more unnecessary use of health-care resources and health risks from invasive investigations to the great majority of patients who do not have CRC. When the costs and life-years are calculated over the denominator of all patients presenting with symptoms of suspected CRC, the importance of the trade-off between sensitivity and specificity becomes apparent. The costs of colonoscopies and their associated risk are modelled in the Westwood et al. 161 model to explore the trade-offs between gains in sensitivity of diagnostics in primary care at the expense of exposing more false positives to invasive tests. Our de novo model analysis will follow a similar approach, which will serve to highlight that gains in sensitivity of diagnostic strategies are achieved at the expense of their specificity, especially among younger patients who are at lower risk of CRC. When modelling the impact on colonoscopies, Westwood et al. 161 make the simplifying assumption that the costs of CRC treatment are not linked to the clinical outcomes, but this simplification does not affect the results in a significant way because of the low proportion of cancer patients.

A limitation of the Westwood et al. 161 model is the quality of the evidence on the diagnostic accuracy of FITs, which was obtained from high-risk patient populations included in studies that, for the most part, were not conducted in a primary care setting. This weakness is problematic because the values of FIT of 10 µg haemoglobin (Hb)/g faeces, the recommended new standard, has sensitivity values close to 100%, which effectively limits the ability of any diagnostic tool to add to the primary care diagnostic yield (see Chapter 7). Furthermore, although referring all patients to secondary care may be the least risky option, the low prevalence of CRC means such a strategy is unaffordable and is not used in practice. 140

In terms of methods, the analysis by Westwood et al. 161 adopts a linked-data approach, whereby survival outcomes of FOBTs are driven by the ability of the test to correctly identify patients with CRC before the disease has progressed to advanced disease stages and become less responsive to treatment. The model uses disease progression data from the disease history model developed by Tappenden et al. 145 and Whyte et al. 156 to determine the extent to which delays in referral from lack of diagnostic accuracy would affect long-term life expectancy. The limitation of this approach is the high degree of uncertainty in the rates of disease progression underlying any such analysis, which were derived from calibrations of the disease history model to observed data on CRC incidence and distribution of stage at diagnosis. We adopt a similar approach in our model, given the lack of more reliable data on the relationship of delayed diagnosis and stage at diagnosis, but raise this weakness as an area that needs further research.

A further limitation of the diagnostic phase of the Westwood et al. 161 model is the way the time to referral is analysed. In particular, the delays incurred by false-negative cases are not explicitly modelled in terms of diagnostic accuracy of the tests being compared, but simply assumed to last for an arbitrary amount of time, which, in their base-case analysis, was 6 months. This adds a high degree of uncertainty in the absence of any data used by the authors to justify the extent of the delay. We sought to address this limitation by building a de novo model of the diagnostic pathway that was fully consistent between the diagnostic accuracy and the number of visits and the referral interval.

In terms of the granularity of the model used to link the diagnostic pathway to the Markov model of disease treatment, the Westwood et al. 161 model uses an annual cycle length to model health transitions. A limitation of an annual cycle is that it may not have sufficient granularity to measure the predicted outcomes, especially given small differences involved in accuracy and referral intervals between strategies. We addressed this limitation in the de novo model by using a 28-day cycle to link the diagnostic pathway to the Markov disease model.

The Westwood et al. 161 model did not account for the short-term survival benefits of reducing the time to referral and treatment, as symptomatic patients with CRC who are undiagnosed, and thus untreated, are subject to increased death risks, relative to those who have been diagnosed and started treatment, especially in disease stages II/III (Dukes’ stage B/C). We addressed this limitation by building a de novo model of the diagnostic pathway that accounts for the short-term survival benefits of reducing the time to referral and treatment.

The effect of delays in diagnosis on the mental health of a patient with CRC is also omitted from the Westwood et al. 161 model. No evidence on the magnitude of such an impact was found in the literature reviewed for this study, and the studies reviewed failed to explore the importance of that effect for the cost-effectiveness of diagnostic strategies in primary care. Our de novo model, therefore, is unable to incorporate these effects. Measures of health-related quality of life among undiagnosed and untreated CRC patients are lacking, and are difficult to measure because of the very nature of studying undiagnosed patients.

Conclusions

The review highlights differences between existing economic models of cancer diagnosis and screening. A common approach to the problem of lack of evidence on clinical outcomes of diagnostic and screening strategies has been to model long-term outcomes of CRC using a model of CRC disease progression in asymptomatic disease obtained by calibrating a disease history model to data on CRC incidence and disease stage at diagnosis. 145,147,148,151,152,155,156,161 We applied the same approach in our de novo model, given that the updated review provided no direct evidence on clinical outcomes of diagnosis, but we recommend that further research on this issue is undertaken.

Our review found no evidence on the cost-effectiveness of diagnostic tools for managing patients in primary care with suspected CRC, but identified one study of FITs161 in the low-risk population of interest that modelled the diagnostic phase. The model explored the trade-offs between gains in sensitivity of diagnostics in primary care at the expense of exposing more false positives to invasive tests, and is a useful precursor for the diagnostic phase of our de novo model. In addition, our critique of the Westwood et al. 161 model identified three areas in which we could make improvements to the diagnostic phase:

1. Time to referral – time to referral will be explicitly modelled to be consistent with diagnostic accuracy of the respective test and the distribution of the number of visits before referrals in electronic health records or retrospective reports from surveys of patients’ experiences. This will include modelling delays incurred by false negatives. By coherently modelling the sensitivity of primary care diagnostic strategies and the delay in referrals and treatment, the uncertainty in relative clinical effectiveness between diagnostic strategies is reduced relative to arbitrary estimates of diagnostic delay used by the existing model in this area.

2. Cycle length to model health transitions – our model used a 28-day, rather than annual, cycle to get accurate measure of predicted outcomes, especially because of the small differences involved in accuracy and referral intervals between strategies.

3. Cancer-related mortality – the short-term survival benefits of reducing the time to referral and treatment will be explicitly modelled.

The review of existing models also highlighted that the health-related quality-of-life outcomes in the diagnostic phase have received very limited attention. Our de novo model is similarly unable to explore the health-related quality-of-life outcomes in the diagnostic phase, as much of the policy focus has been devoted to improving cancer survival.

Aim and objectives

The earlier chapters highlight the uncertainty in the evidence base for decision tools, which make it difficult to assess the cost-effectiveness of the use of risk tools. In this chapter, we use a decision-analytic model to demonstrate the uncertainty inherent in the current evidence base, and show the probable impact that use of the tools in clinical practice may have on patient outcomes and NHS resources. The primary role of the decision modelling is not the estimation of the single most likely point estimates of costs per QALY associated with each diagnostic tool. Instead, the decision-analytic model and the evidence that is available are used in this chapter to explore the probable range of costs per QALY, and to ask questions about the likely impact of the diagnostic tools, given the current evidence base.

The objectives of the decision-analytic model were to examine the following questions:

• What are the possible impacts on patient quality of life or survival if the diagnostic tools reduce time to diagnosis?

• Will the benefit to cancer patients who are identified earlier by diagnostic tools outweigh any disutility in extra patients who do not have cancer being referred for further investigation?

• How big an improvement in quality of life would be needed to warrant the use of these tools if there are no survival impacts associated with the diagnostic tools? Would this quality-of-life improvement be justifiable given the evidence we have?

• Could a cancer diagnostic tool be considered cost-effective if it reduces the period of extreme anxiety for patients by expediting investigation and management?

• Where are the gaps in the evidence base and where is more research needed?

Furthermore, we developed a model that anticipates evidence development and can be used alongside any studies measuring impact on patient outcome directly in the future to explore implications for cost-effectiveness. We will also be able to use the model to identify the parameters that contribute most to the overall decision uncertainty about the cost-effectiveness of decision tools and where additional research might be targeted using expected value of partial perfect information. 166–168 If considered effective and cost-effective, now or in the future, the model could also be developed to assess the budget impact of introducing cancer diagnostic tools in different populations.

It was not feasible to model the impact of diagnostic tools for all common cancer types for which diagnostic tools exist. Instead, we chose to model one common cancer: CRC.

Colorectal cancer was chosen, based on the following criteria, to provide a best-case example of how the diagnostic tools could affect patient outcomes:

• Tools exist – to compare diagnostic tools with each other and with no tool, we need diagnostic tools to be available for the specific cancer type. There are existing tools for CRC, as identified in SR2. In particular, there are RATs75,81 and QCancer80 for CRC, which are now within GP systems and, therefore, potentially available for GPs to use.

• Common cancer – CRC is the third most common cancer in the UK, accounting for 11% of cancers in women and 13% in men. 169

• Whole-disease models exist – it is important that whole-disease models exist, and, although not made available by the original authors, could be replicated by published methods and adapted to incorporate the diagnostic pathways of interest.

• Wide agreement on patient management –it will be important that there is wide agreement on the treatment and management of individuals to minimise uncertainties elsewhere in the clinical pathway. CRC is a cancer for which there is wide agreement on the treatment and management of individuals.

• Evidence that use of diagnostic tools changes practice – to explore the impact of the tools, it will be important to identify a cancer type for which there is existing evidence to show where the diagnostic tools will impact, and what that impact will be. Only a few studies assess the impact of RATs in SR1 that were related to CRC. These studies reported that increased cancer referrals and investigations for CRC were associated with use of tools. 13,35

• Evidence that change in practice affects patient outcomes – it is not enough that increased referrals and investigations are associated with use of the tools; there also needs to be existing evidence of the impact of earlier diagnosis on patient outcomes. This might include earlier stage at diagnosis, higher resection rates or improved survival, as well as improved quality of life of patients. There is some evidence to show that long (and very short) diagnostic intervals are associated with increased mortality in CRC. 99,135,137 However, as indicated in Chapter 5 and summarised in Chapter 6, Updated review, there is great uncertainty associated with estimates of the effect of expeditious CRC diagnosis on cancer stage or patient survival.

Although the implications of prolonged time to tests, diagnosis and treatment vary by cancer type, among the low-risk symptoms there are many similarities in the diagnostic options within primary care (i.e. refer all, watch and wait, refer for further tests). Therefore, the model developed can be viewed as a template for modelling the effect of diagnostic tools in cancers other than CRC, and can be revised to include prevalence and test sensitivity/specificity and costs specific to other cancers.

Methods

Conceptualisation of the model

Patient and public involvement

Involving members of the public in health research projects is becoming standard practice, but there are fewer reported examples of patient and public involvement (PPI) in health economic or modelling studies. 170,171

At the outset of putting together the diagnostic pathway of the model, we organised a workshop to meet with patients who have experienced bowel cancer diagnosis to seek their advice on our understanding of the current diagnosis pathway and to listen to their experiences of being diagnosed. This was facilitated by Dr Emma Cockcroft from the PPI team from the Peninsula Collaboration for Leadership in Applied Health Research and Care, which is based at the University of Exeter Medical School. This part of the study did not require separate ethics approval because it was underwritten by the current ethics approval granted to the PPI group at the University of Exeter.

Dr Cockcroft wrote an invitation to the Bowel Cancer West cancer treatment centre in Plymouth to participate in the PPI, which was published on their Facebook page (Facebook, Inc., Menlo Park, CA, USA). After several weeks, two patients contacted us and we opted to have a session in which we listened to their experiences and journeys through the diagnosis pathway and asked them to tell us their opinion about our diagrammatic representation and understanding of the diagnostic pathway. We then asked their opinions about the diagnostic pathway within primary care and our diagrammatic representation of this.

Two individuals diagnosed with bowel cancer, who were brother and sister, attended the meeting. Their bowel cancers were diagnosed through a test to assess whether or not they had inherited the gene after their father passed away from bowel cancer. Both diagnoses occurred at secondary care, so neither of them had any experience of diagnosis practice in primary care. Although at this point we knew that we would not obtain information on the diagnostic pathway in primary care, we decided to take this opportunity to discuss our understanding and modelled representation of the diagnostic pathway. After listening to their descriptions of the pathway and their very difficult experiences with multiple operations, we asked them if they could identify any potential misrepresentation of reality in an animated presentation covering the progression through the diagnostic pathway in stages, from the patient attending general practice owing to experienced symptoms to the moment when the GP decides to refer the patient to secondary care for a colonoscopy. We received very encouraging, although limited, feedback on our diagnosis pathway.

Other letters of invitation to Bowel Cancer West were published via Facebook, but, after weeks of not receiving any messages, we decided to focus on the other key stakeholder, the GP.

We opted to conduct this process in two steps. First, we opted to consult with a recently retired GP from Somerset to assess whether or not our explanations of key economic concepts and our diagnostic pathway were clear and suitable for presentation to a wider number of health professionals. As a warming-up task, a short interview was carried out in which we focused on the number of years of practice by the respondent and her experience with cancer diagnosis, that is number of patients diagnosed per year, some issues with cases diagnosed at late stage and similar topics, which allowed us to build a clear picture on the process and interactions taking place at the general practice involving symptomatic patients with suspected cancer.

What we learnt from this interview corroborated the evidence available:

• too many guidelines, so it is difficult to keep all of them in mind

• very few patients per year picked up at early staging in primary care

• symptoms easily wrongly attributed to other diseases, mainly in those aged > 65 years with comorbidities.

We then moved to have a more in-depth conversation of the diagnostic pathway with the same retired GP. However, we changed our prior strategy and let her tell us the different steps that a GP would take if they suspect that a patient may have bowel cancer. When the reasoning behind the choices was unclear, we asked for further clarification. Notes were taken that were compared with our preconceived diagnostic pathway, thereby permitting us to have a better understanding of the current pathway for bowel cancer at general practices.

A third workshop was organised at a local general practice in Exeter. We prepared a modified diagnosis pathway, constructed using the current NICE guidelines172 and complemented with the strategies learnt from the previous interactions with the patients and the retired GP. To this diagnosis pathway we added the strategy of the ‘diagnostic tool’. The diagnostic pathway was printed on colour A7 pages that we brought with us for discussion with the GPs.

All the practice’s GPs (n = 7) attended the workshop, as well as two nurses, and the group discussion was transcribed. The GPs had very different levels of experience in terms of number of years of practice. Some of their opinions were as follows:

Tools would be useful if you could use as a justification for referral – but currently still have to meet NICE criteria to get onto 2WW. The criteria are ‘rigid’ and some referrals might get bounced back if the results from a blood test are borderline.

We use a ‘tool’ which is a calculated risk in our head.

Consider litigation – normally more likely to get sued for not doing something.

Borderline’ cases often bounced back by secondary care – they would need to accept the ‘clout’ of the diagnostic tool/aid.

We also asked them if there was any reason why they would not consider using the tool. The answers were as follows:

Stuck in ways.

Doing similar anyway – just in heads.

Don’t have the tools ‘to hand’.

We asked in what circumstances they would consider using the diagnostic tool:

To verify the ‘gut feeling’ was true, not only for referral, but for reassurance, when deciding against urgent referral or more tests, etc.

A follow-up question was if the diagnostic tool would help them to decide whether or not a patient needed referral:

Seemed this would only be the case if a tool had NICE backing.

After these questions, all the different strategies in our diagnosis pathway were discussed, which resulted in Figure 5. An additional one-to-one meeting was held with one of the GPs to ask for further clarifications.

FIGURE 5.

Diagnosis pathway model.

The diagnostic pathway

A decision-analytic model was used to investigate the key areas of uncertainty in the economic evaluation of primary care tools for informing decisions on referrals of suspected cancer patients from primary to secondary care in England.

The analysis compares costs and clinical outcomes between a diagnostic strategy whereby primary care doctors use the diagnostic algorithm in conjunction with information from investigations in primary care to decide whether or not a patient presenting with low-risk symptoms needs to be referred to secondary care for further investigations, and a strategy of primary care investigations alone. We compare the intervention strategy of RATs (CAPER)75 or QCancer80 as a triage to primary care investigations (which, in the model, is represented by FITs) or direct referral to secondary care (intervention) with the following comparator strategies:

• ordinary referral to secondary care without prior primary care investigations for all patients (‘refer all’ strategy)

• primary care investigations alone

• send home/wait (‘send home and wait’ strategy).

In Figure 5, we illustrate in the diagnosis pathway for patients with low-risk symptoms who do not fulfil the 2WW criteria.

In the absence of clinical studies providing evidence on the clinical outcomes of these strategies, the effect of the diagnostic algorithm on patient management is modelled using a linked-data approach, which translates evidence of diagnostic accuracy into clinical outcomes in terms of life expectancy, health-related quality of life and health-care costs. In the model, the gain in sensitivity with the algorithm may cause an increase in the number of patients being referred to secondary care, which, in turn, results in the earlier identification of CRC cases; this is the main driver of outcomes in the model. An increase in sensitivity may also change the composition of diagnosis by disease stage, but this is harder to model without further data. The clinical benefits of the diagnostic tools depend on detecting cases at earlier disease stages than those that would occur under the status quo, without use of the diagnostic tools.

Interventions

The intervention involves use of the diagnostic tool and referral of those individuals above the threshold score of 35, which corresponds, approximately, to 2% in RAT and 0.5% in QCancer, to secondary care for investigation, which, in the model, is assumed to be colonoscopy. Although some patients undergo alternative tests such as flexible sigmoidoscopy, CTC and barium enema because of the presence of certain symptoms, older age or patient choice, only colonoscopy was modelled, as it is the most common test used. 155 This assumption was intended to avoid unnecessary complexity, given the exploratory aims of the model. For those individuals with a score below the threshold, the diagnostic tool is followed by primary care investigations, which, in the model, are assumed to consist of FITs at the 20-µg threshold (hereafter referred to as FIT). A subgroup of patients may also be sent home without a FIT if the presenting symptoms result in a score below a certain minimum level. We do not have adequate information on diagnostic accuracy for this group of patients deemed to be at too low a risk to warrant investigation; therefore, it is assumed that all patients below the threshold undergo a FIT. As a result, our analysis assumes that using the tools will always result in some further action. Figure 6 shows the decision model for patients with CRC and Figure 7 shows the decision model for patients without CRC.

FIGURE 6.

Diagnostic decision model for patients with CRC. p1, number of cases with a positive test result using the diagnostic tool/total number of CRC cases; p2, number of cases with a positive test result using the FIT test/total number of CRC cases.

FIGURE 7.

Diagnostic decision model for patients without CRC. p3, number of patients with a negative test result using the diagnostic tool/total number of patients without CRC; p4, number of patients with a negative test result using the FIT test/total number of patients without CRC.

A key difference between the diagnostic model for CRC patients and the model for non-CRC patients in Figures 6 and 7 is that the proportion of patients referred to secondary care is based on the sensitivity (p1 and p2) of the tests for CRC patients and specificity (p3 and p4) of the tests for patients without CRC. In addition, patients without CRC are assumed not to return to general practice after a negative test result.

Disease history model linking test results and clinical outcomes

To model the implications of avoidable delays in referral and diagnosis, we employed the updated version of the disease model developed by Tappenden et al. ,145 as described in Whyte et al. 156 As we could not obtain the model from Whyte et al. ,156 we replicated it from information published on its methods. 145,156,174,175 As this disease history model was not developed for evaluating diagnostic strategies for identifying cancer patients in primary care, we adapted it to focus entirely on the symptomatic stage population of present interest. An illustration of the adapted Markov model of health-state transitions experienced by CRC patients visiting their GP with symptomatic disease is presented in Figure 8. Briefly, an undiagnosed patient presents to primary care with symptoms at one of the four possible disease stages and is managed according to one of the diagnostic strategies investigated in this study. The patient’s disease is detected and he/she is referred to secondary care for investigation with a certain probability (p), which we assume is equal to the sensitivity of the strategy, or is missed with a probability of 1 – p. Once in secondary care, the patient undergoes colonoscopy and, because this investigation has sensitivity approaching 100%, the transition probability to a cancer diagnosis from being undiagnosed equals the overall sensitivity of the strategy, p, times the probability of surviving until the end of the cycle, 1 – d.

FIGURE 8.

Markov model of health-state transitions for CRC cases.

Colorectal cancer patients missed by the diagnostic strategy remain untreated and are at risk of progressing to the next stage of disease severity. The probability of disease progression from one cycle to the next is equal to the probability of a false-negative result with the diagnostic strategy times the probability of remaining alive by the end of the cycle times the probability of disease progression (q), q × (1 – d) × (1 – p). If, after repeated cycles of visits to the GP, the patient remains undiagnosed, the disease stage D would be reached, for which the only possible transition, other than to starting treatment, is to death from CRC or other causes.

In this model, a patient with CRC who, under the intervention, is identified earlier than under the control, say Dukes’ stage A or B instead of C or D, derives both short-term and long-term health benefits. The short-term benefits occur as soon as treatment starts in the form of death risk reduction, whereas the longer-term benefits arise from avoiding disease progression to Dukes’ stages C or D, for which treatment options are less effective. In terms of Figure 8, the excess death risk of not receiving treatment is captured by the relative magnitude of death probabilities on and off treatment for each of the Dukes’ disease stages A, B, C and D: d_TA < d_A, d_TB < d_B, d_TC < d_C and d_TD < d_D, respectively. The long-term survival loss from delays in diagnosis is measured by the positive relationship between the probability of CRC-related death and disease severity, that is d_TA < d_TB < d_TC < d_TD. The reduction in death risk is assumed to continue for the rest of the patient’s life, apart from the maximum survival ceiling imposed by the background mortality risks included in the model, as measured in life tables176 for the general English population of the same age and sex. In the scenario analysis, we explore the effects of adopting the assumption in Whyte et al. ,152 whereby patients who remain alive at the end of the 5-year period after diagnosis have, from then on, the same cycle probability of death as that of the general population of the same age and sex, that is negligible CRC death risk.

For patients with no CRC, a two-state Markov model was used that followed the initial decision tree of Figure 5. After the initial decision of whether or not to refer those patients, those referred undergo colonoscopy and are exposed to its associated small risk of adverse events, whereas those not referred are spared such exposure. Patients who are alive after colonoscopy or not referred to colonoscopy are in the alive health state and may transition to death from non-CRC causes, as determined by the death risks in English life tables, or remain alive at the start of the next cycle.

Mechanism of effect

In the absence of direct clinical effectiveness data, we assume a relationship between the sensitivity of both the algorithms and routine referral practice and the time to diagnosis. We break the time from symptom presentation to diagnosis into (1) the time from presentation to primary care to referral for investigation (the ‘referral interval’ in Chapter 5) and (2) the time from referral to specialist care to diagnosis and treatment, and assume that the diagnostic algorithms affect only the referral interval, while the time from referral to diagnosis and treatment remains fixed. Thus, in our analysis, a reduction in the diagnostic interval associated with any improvement in primary care diagnostic accuracy is equal to the reduction in the time from presentation to referral that is mediated through the number of primary care visits before referral.

Given our chosen population, we model the clinical effectiveness as a function of the diagnostic outcome of the first visit, assuming that, under current clinical practice, true-positive cases are referred after two visits, to allow for primary care investigations to be conducted and discussed between the patient and the doctor. False-negative cases would see their symptoms persist and would return for a third, or possibly more, visits. Because we have no obvious way to infer how a change in primary care diagnostic accuracy would affect the distribution of second and subsequent visits, we assume that the sensitivity and specificity are independent across multiple visits to primary care. This may be incorrect because individuals who are missed on an initial visit are more likely to be missed on a second visit, and so on. Indeed, we infer the probable extent of this in primary care referral practice in Table 15 and investigate the implications in sensitivity analyses (see Scenario analyses). Thus, a probability of referral after two, three, four and five or more visits is calculated, respectively, as:

⁽¹⁾ P ( n = 2 ) = sensitivity .

⁽²⁾ P ( n = 3 ) = ( 1 − sensitivity ) × sensitivity .

⁽³⁾ P ( n = 4 ) = ( 1 − sensitivity ) 2 × sensitivity .

⁽⁴⁾ P ( n ≥ 5 ) = ( 1 − sensitivity ) 3 .

TABLE 15 - Implicit average median time to referral under current standard practice

N/A, not applicable.

Lyratzopoulos et al. 173

Weighted average calculated by the authors from data reported by Lyratzopoulos et al. 173

Assuming independence of sensitivity values across multiple visits.

Weighted average calculated by the authors from data reported by Lyratzopoulos et al. 173 and predicted frequency distribution of number of pre-referral visits.

Referral after	Actual frequencya	Median days to referrala	Implicit sensitivity by visitb	Predicted frequencyc	Median days to referrald	Source
2 visits	0.76	18	0.76	0.76d	18	Analysis of data from the English National Audit of Cancer Diagnosis in Primary Care 2009–10, on 1170 general practices (≈ 14% of all English practices)173
3 visits	0.12	39	0.50	0.18	39
4 visits	0.05	56	0.42	0.045	56
≥ 5 visits	0.06	122	1.0	0.01d	122
Weighted average	1	29	N/A	1.0	25

Under the diagnostic sensitivity of 0.76 implicit in the results reported by Lyratzopoulos et al. ,173 the predicted frequency for three, four and five or more visits is 18%, 4% and 1%, respectively, which is in contrast with the actual frequencies of 12%, 5% and 6%, respectively. Consequently, the predicted median referral interval of 25 days is lower than the actual figure of 29 days (see Table 15).

The length of time to referral (i.e. diagnosis in our model) is derived from the number of primary care visits from previous research. 173 Time to referral and the number of primary care visits before referral are strongly positively associated, with a threshold of 16 or 17 days discriminating between cancer patients referred after three or more and two or fewer visits. 173

In our Markov model of 28-day cycles, the length of the referral interval is equal to the number of initial repeat GP visits times 28 days (Table 16). The resulting times to referral implicit in the sensitivity values of the tests in each strategy adopted for the model are presented in Table 16. These base-case values correspond to sensitivity and specificity values of 69% and 77%,141 61% and 91%80 and 53% and 99%175 for the RAT, QCancer and FIT of 20 µg Hb/g faeces, respectively, after adjusting for the 80% compliance with diagnostic results, so that 20% of those testing positive would not be referred but sent home. These values are an approximation to the model results, which also account for the attrition caused by death from CRC or other causes before referral, so that the values in Table 16 are greater than the model outputs by 2–2.5 days. When the sensitivity values implicit in the number of pre-referral visits reported in a retrospective patient experience survey (Lyratzopoulos et al. 173) are used for FITs, the reduction in the time to referral relative to the comparator is only 7 days for QCancer and 8 days for the RAT (see Appendix 5).

TABLE 16 - Mean time to referral of CRC patients by strategy in the model: base-case analysis

N/A, not applicable.

Derived from the sensitivity values for the component tests times the 80% rate of compliance of referral practice with the test results; see Table 19.

The modelled comparator strategy assumes that every time the patient returns two GP visits are required for a referral decision to be made, one for the return visit and second, follow-up visit, for the review and discussion of the results of primary care investigations ordered in return visit.

Positive test results with the tool result in referral after the first visit, whereas positive test results with a FIT after negative result with the tool takes two visits.

Positive test result with FIT alone results in referral after two visits, an initial and a follow-up visit to discuss results.

Referral after	QCancera		Comparatorb		Referral after	RATa		Comparatorb
	Predicted frequencyc	Mean time to referral (days)	Predicted frequencyd	Mean time to referral (days)		Predicted frequencyc	Mean time to referral (days)	Predicted frequencyd	Mean time to referral (days)
1 visit	0.49	28	N/A	28	1 visit	0.55	28	N/A	28
2 visits	0.27	28	0.42	28	2 visits	0.21	28	0.42	28
3 visits	0.12	56	N/A	N/A	3 visits	0.13	56	N/A	N/A
4 visits	0.07	56	0.24	56	4 visits	0.05	56	0.24	56
≥ 5 visits	0.06	93	0.34	123	≥ 5 visits	0.06	93	0.34	123
Weighted average	1	37	1	67	Weighted average	1	37	1	67

A note of warning is due as to the lack of comparability between the accuracies of the RAT and QCancer. Their specificity and sensitivity values were derived from studies using different designs (a case–control study for the RAT and cohort data from electronic records for QCancer) and populations. This would render any comparison of the tools biased. We note that similar arguments could also be raised in relation to the comparison of each tool with the comparator, as this is also based on an indirect comparison of heterogeneous single-arm accuracy studies; this fact reflects the low quality of data available for analysis.

To complete the link from diagnostic accuracy outcomes and number of cycles before referral (which determines the time to treatment because the time from referral to diagnosis and treatment is assumed to be unaffected) with survival outcomes, we use the relationship between time to diagnosis and disease stage at diagnosis. The only available source for this unobservable parameter is provided by a calibration exercise of a decision model based on Tappenden et al. 155 It reports rates of progressive transition between (non-clinical) Dukes’ stages A to B, B to C and C to D, and from each of these to death, which, for stages earlier than D, was exclusively unrelated to cancer. We assume that the disease progression rates from this source, which were intended to depict the experience of patients in asymptomatic states, apply to our symptomatic population, but combine these with CRC mortality rates reported by disease stage in untreated patients (Liu et al. 177) Table 17 presents the mean time to progression from Dukes’ stages A to B, B to C and C to D implied by the disease progression rates adopted.

TABLE 17 - Dukes’ stage transitions

Tappenden et al. 145

Dukes’ stage transition	Annual transition probability (%)	Mean time to progression (months)	Source
A to B	58.29	12.3	Authors’ calculations from reported model-calibrated dataa
B to C	65.55	9.0
C to D	86.48	4.3

The reductions in time to diagnosis and treatment made possible by the smaller number of initial repeat visits to the GP have the ultimate impact of increasing the likelihood of starting treatment at a stage of disease for which treatment is more effective. We employ data on relative survival (with respect to the general population) by disease stage at diagnosis, depicted in Figure 9, to model this longer-term health benefit of earlier access to treatment with more accurate GP diagnosis and referral. 178 We present the expected survival over time of an individual who is diagnosed while at Dukes’ stage B and the counterfactual, improved survival, scenario that would have been observed had this simulated case been diagnosed earlier while still in Dukes’ stage A (Figure 10).

FIGURE 9.

Relative survival of CRC patients diagnosed in 1996–2002 in England. Survival curves by disease stage. Source: National Cancer Registration and Analysis Service. 178

FIGURE 10.

Actual and counterfactual survival curves for a patient who is diagnosed at Dukes’ stage B.

In Table 18, the link between diagnostic accuracy and disease stage at diagnosis, which is mediated by the number of pre-referral visits (cycles) and underpins the clinical effectiveness in the model, is made explicit for each strategy. The more accurate intervention strategies shift the distribution of disease stage at diagnosis towards the earlier disease stages of Dukes’ stages A and B, and C (not shown in the table). The overall percentage of stages A/B presented in the table do not account for background mortality effects; therefore, they differ by about 1 percentage point from the final model estimates, which do account for such mortality. When the sensitivity values implicit in the number of pre-referral visits reported in a retrospective patient experience survey (i.e. Lyratzopoulos et al. 173) are used for FITs, there is a 1-percentage-point gain in the proportion of patients with Dukes’ stages A/B at diagnosis with both QCancer and the RAT, relative to the comparator (see Appendix 5). Therefore, an improvement in the sensitivity of primary care diagnostic testing would be expected to reduce the time to referral and, provided this is translated into a one-to-one reduction in the time to diagnosis and treatment, increase the relative frequency of earlier disease stages at diagnosis. Stage at diagnosis would then determine the long-term clinical outcomes and costs in the model.

TABLE 18 - Disease stage at diagnosis of CRC patients by strategy in the model: base-case analysis

Referral after	QCancer		Comparator		Referral after	RAT		Comparator
	Predicted frequency	Dukes’ stage A/B (%)	Predicted frequency	Dukes’ stage A/B (%)		Predicted frequency	Dukes’ stage A/B (%)	Predicted frequency	Dukes’ stage A/B (%)
1/2 visits	0.65	54	0.42	54	1/2 visits	0.68	54	0.42	54
3/4 visits	0.23	51	0.24	51	3/4 visits	0.22	51	0.24	51
5/6 visits	0.08	48	0.14	48	5/6 visits	0.07	48	0.14	48
7/8 visits	0.03	45	0.08	45	7/8 visits	0.02	45	0.08	45
9/10 visits	0.01	42	0.05	42	9/10 visits	0.01	42	0.05	42
≥ 10 visits	0.01	≤ 40	0.07	≤ 40	≥ 10 visits	0.00	≤ 40	0.07	≤ 40
Weighted average	1	52	1	50	Weighted average	1	53	1	50

The values of key model parameters are presented in Table 19. These values were identified from reviews of previous decision models in CRC, described in Chapter 6. This choice of data sources was partly determined by the fact that, in contrast to our research plans, the decision model of Whyte et al. 156 was not made available to our study. As a result, we had to reallocate the research resources that had been originally assigned for conducting a systematic review of model parameter values to building the model from scratch by replicating its published methods. Further details of the methods and data used to populate the disease model parameters are given in Appendix 5.

TABLE 19 - Model parameter values used in the base-case analysis

HR, hazard ratio; PSSRU, Personal Social Services Research Unit.

Parameter	Value	Comment	Source
Clinical
Starting age, 70 years
Prevalence of CRC	0.015	Symptomatic low-risk population, as in analysis informing NICE Diagnostic Guideline Number 30140	Hippsley-Cox 201280
Disease stage at presentation		Age-specific distribution by Dukes’ stage	Simulated by authors from replicated disease history model (Tappenden 2007145)
Sensitivity of QCancer	0.610	Based on the top 10% risk score in their tool, which gives a risk threshold of 0.5% (PPV 1.5%). We recalculated their sensitivity and specificity of this threshold to exclude observations in the top 1% risk score range (5.2% risk threshold) and thus get closer to the low-risk population (0.1–3.0% prevalence); unfortunately, the source does not give the required breakdown of results to exclude people above the 3% threshold	Hippsley-Cox 201280
Specificity of QCancer	0.910
Sensitivity of the RAT	0.69	Based on the 2% risk threshold	Hamilton 2005141
Specificity of the RAT	0.77
Sensitivity of the FIT	0.526	FIT 20 µg Hb/g faeces	Murphy 2017175
Specificity of the FIT: 50–69 years old	0.988
Specificity FIT: ≥ 70 years old	0.963
Compliance with colonoscopy referral	0.80		Tappenden 2007145
Colonoscopy sensitivity for CRC	1	Murphy 2017175 reports a value of 0.966 from a systematic review (i.e. van Rijn 2006179)	Assumption
Colonoscopy specificity for CRC	1	Assumption	Murphy 2017175
Probability of perforation with colonoscopy	0.0017		Atkin 2002180
Probability of death with perforation	0.0582		Gatto 2003181
Probability of bleeding	0.0044		Atkin 2002180
Transition probabilities		4 weekly	Whyte 2014156
Dukes’ stage A to B	0.0651	Reported model calibration to CRC incidence and stage at diagnosis	Tappenden 2007145
Dukes’ stage B to C	0.0787
Dukes’ stage C to D	0.1427
CRC death risk Dukes’ stage A	0.00	Post Dukes’ stage A at diagnosis	Exponential hazard function fitted to digitised Kaplan–Meier curves up to 5-year relative survival for CRC patients (diagnosed 1996–2002) by stage at diagnosis in England178
CRC death risk Dukes’ stage B	0.00	Post Dukes’ stage B at diagnosis
CRC death risk Dukes’ stage C	0.01	Post Dukes’ stage C diagnosis
CRC death risk Dukes’ stage D	0.07	Post Dukes’ stage D at diagnosis
HR untreated CRC in Dukes’ stage A	1	Survival of patients refusing treatment with newly diagnosed CRC in Taiwan (Province of China) during 2004–8. Treatment refusal was defined as ‘not undergoing any cancer treatment within 4 months of confirmed cancer diagnosis177’	Liu 2014177
HR untreated CRC in Dukes’ stage A	1.22
HR untreated CRC in Dukes’ stage A	1.22
HR untreated CRC in Dukes’ stage A	1.22
Costs
Cost of FIT	£3	Threshold 20 µg Hb/g faeces. Published data, excludes costs of screening campaign in the original source	Murphy 2017175
Cost of GP visit	£31	PSSRU’s unit cost per surgery consultation lasting 9.22 minutes, without qualification costs and including direct care staff costs	Curtis 2018182
Cost of colonoscopy	£615	Costs of colonoscopy with or without polypectomy	Whyte 2014,156 NHS reference costs, screening centre estimates152
Cost of treating bowel perforation	£5559	Major surgery	Whyte 2014,156 NHS reference costs
Cost of bleeding	£304	Cost of admittance for bleeding (overnight stay on medical ward), NHS reference costs	Whyte 2014156
Health-care cost, Dukes’ stage A	£3178	Lifetime estimate, age specific, figure presented is for people aged 70–79 years Published data from whole-disease model simulations	Whyte 2012174
Health-care cost, Dukes’ stage B	£3455
Health-care cost, Dukes’ stage C	£4485
Health-care cost, Dukes’ stage D	£4365
Utilities
Cancer free	0.91	Whyte 2014.156 General population utility decline with age by sex was accounted for using the linear model proposed by Ara 2010183	Ness 1999184
Dukes’ stage A	0.74
Dukes’ stage B	0.70
Dukes’ stage C	0.50
Dukes’ stage D	0.25

Calculations of costs and benefits in the model

Our Markov model consists of a repeated iteration of 28-day cycles, whereby patients accrue costs and benefits depending on the health state occupied in each iteration. We chose this cycle length as the most appropriate for capturing the frequency with which patients with unresolved symptoms return to their GPs for care. By aggregating the proportion of patients occupying the various health states in each 28-day cycle across the modelled time horizon of 30 years, we calculated the total time alive and total time spent in each state. We applied costs and health-state utility values corresponding to the different states and aggregated across health states and time periods to calculate total health-care costs and total QALYs. The exception to this method is the estimation of health-care costs of cancer treatment, which are implemented as a single lifetime cost payoff incurred at diagnosis, as in the original model by Whyte et al. 156 The modelled accumulated total life-years, QALYs and lifetime health-care costs for the intervention and the status quo strategies, over a period of 30 years after an initial presentation with symptoms to the GP, are presented for a hypothetical cohort aged 70 years at first presentation. We initially focus on the predicted clinical outcomes for the subgroup of CRC cases and then present the results for the overall cohort presenting to their GP with symptoms suggestive of CRC.

Table 20 compares the main features of our model with existing models relevant to our study question. Common to ours and the previous models is the use of a disease history model developed by Tappenden et al. 145 Whyte et al. ’s156 model is a model of screening and therefore does not address the question of interest here. The main differences between our and Westwood et al. ’s161 work is that we explicitly model the time to referral consistent with the diagnostic accuracy of the respective test strategies and the distribution of the number of visits before referrals; we chose a shorter cycle length (28 days vs. annual) to allow greater granularity for predicting outcomes; and we model the short-term survival benefits of reducing the time to referral and treatment. Like Westwood et al. ’s161 and Whyte et al. ’s156 models, we have implemented a probabilistic sensitivity analysis, but with the caveat that this type of sensitivity analysis was of limited relevance because most of the uncertainty derives from structural assumptions, such as how the length of the diagnostic and treatment interval is determined, defining the status quo diagnostic practice and measuring its diagnostic accuracy. Consequently, we did not conduct any value-of-information analysis.

TABLE 20 - Key model characteristics of the de novo model and previous models

GBP, Great British pounds; N/A, not applicable.

Study	Perspective	Population	Diagnostic strategies	Outcome measure(s)	Model structure	Model duration	Uncertainty analysis	Discount rate (%)	Base year and currency
Our current model	UK NHS	Symptomatic patients aged ≥ 40 years presenting to primary care who are at low risk of CRC	QCancer + FIT RAT + FIT FIT alone	Cancer mortality, disease stage at diagnosis, life-years, QALYs	Markov model with monthly cycles	Lifetime	Threshold analysis (diagnostic accuracy) Scenario analysis	3.5	2018; GBP
Whyte 2014156	UK NHS	General population	N/A	Cancer mortality, QALYs	State-transition model with annual cycles	Lifetime	Two-way sensitivity and probabilistic sensitivity analyses	3.5	2012; GBP
Westwood 2017161	UK NHS	Symptomatic patients aged ≥ 40 years presenting to primary care who are at low risk of CRC	FIT FOBT No triage, refer all to colonoscopy	QALYs, life-years	Markov state-transition model with annual cycles	Lifetime	Scenario analysis and probabilistic sensitivity analysis	3.5	2015; GBP

Deterministic sensitivity analysis

To investigate areas of uncertainty in our model, we arbitrarily varied clinical parameters by 20% above and 20% below their base-case analysis values. In addition, we undertook scenario analyses for the following parameters:

• alternative sensitivity and specificity values for the comparator (inferred from the number of GP visits before referral173 and from receiver operating characteristic data for version 3 of NICE guidelines22)

• assuming 100% compliance with the test result (i.e. all positive cases are referred and all negative cases are not referred to secondary care)

• no mortality benefit of treatment in stages A/B, by setting the hazard ratio to 1

• restricting excess CRC-related death risk to first 5 years156

• limit time horizon of analysis to 5 years

• alternative utility values adopting a common value for all CRC disease stages145

• alternative ‘send home/wait’ comparator strategy, whereby, at the initial visit, the GP sends the patient home without conducting a FIT; from the second visit onwards, they manage the patient as in the ‘FIT given to all’ strategy

• modified intervention strategy whereby patients who are above the risk threshold of the tool are referred directly to secondary care, as in the original intervention strategy (see Figure 2), whereas those below the threshold are sent home, rather than being subjected to primary care investigations, as assumed in our base-case analysis

• assuming no survival gains and maximum quality-of-life (utility) gains (i.e. zero utility values for false-negative cases until diagnosis) from more expeditious referral and diagnosis in patients with CRC.

We present our results in 2017/18 prices. We provide tentative estimates of incremental costs per life-year gained. We also present results in terms of incremental cost per QALY gained, discounting costs and QALYs at an annual rate of 3.5%.

Results

Tornado analysis

A tornado analysis is used to provide a graphical representation of the degree to which the incremental cost per QALY gained from tools is sensitive to independent variation by 20% of clinical parameters (Figure 11). The tornado analysis shows that the source of greatest uncertainty in the incremental cost per QALY originates from the sensitivity of the standard test strategy. A reduction of 20% in the false-negative rate, that is an increase in the sensitivity of the FIT from its base-case value of 53% to 62%, increases the discounted cost per QALY gained with the QCancer tool from £28,704 to somewhere above £48,633 (see Figure 11). A reduction of 20% in the same parameter reduces that figure to £17,842 per QALY gained. The algorithm’s specificity is the second most important economic parameter, as it determines how much of the increased detection of CRC by the primary care diagnostic workup is obtained at the expense of patients without CRC who are referred, but have no need, for colonoscopy. Naturally, the cost of colonoscopy is the single most influential cost parameter in the results; reducing its cost from £615 to £492 reduces the cost per QALY gained with QCancer to £22,883.

FIGURE 11.

Sensitivity of results to arbitrary increases and decreases of 20% in model parameter values.

Probabilistic analysis

A probabilistic sensitivity analysis was conducted for the base-case analysis by accounting for sampling uncertainty in the key model parameters (see the tornado plot in Figure 11 and see Appendix 5, Table 66, for information on the distributions used for those parameters). The first and third quartiles of the incremental discounted costs’ distribution were £40 and £44 per patient, respectively, whereas those for incremental discounted QALYs were 0.0012 and 0.0017, respectively (Figure 12). The probabilistic estimate of the incremental cost per QALY gained with the tool using the case of QCancer is £28,522, which is very similar to the deterministic estimate of £28,703.

FIGURE 12.

Cost-effectiveness plane: diagnostic tool (QCancer) vs. no tool.

Figure 13 presents the probability of the tool being cost-effective as a function of the willingness to pay for one QALY gained. This figure implies a 95% CI for the incremental cost per QALY gained with the tool of £17,000–50,000. A note of caution is warranted as this figure does not capture the uncertainty inherent in the assumption that the health benefits with the tool are due to its increased sensitivity, as it translates to fewer visits and, therefore, shorter times to referral, and that these reduced times in turn translate into diagnoses at earlier disease stages. Owing to this underestimation of the uncertainty in this probabilistic sensitivity analysis, we did not attempt to analyse the expected value of additional research information.

FIGURE 13.

Cost-effectiveness acceptability curve: tool (QCancer) vs. no tool.

Threshold analysis

The tornado analysis has highlighted the sensitivity of the cost per QALY gained to the sensitivity of the FITs and the specificity of the diagnostic tools. Given the key role of these parameters in the results, Figure 14 depicts the levels of specificity of the triage tool required for the intervention to meet the £30,000 threshold of cost-effectiveness, at varying levels of sensitivity of the FIT. At the levels of sensitivity of ≥ 50% that have been reported for a FIT of 20 µg Hb/g faeces,175 the minimum level of specificity needed by the tool to serve as a cost-effective triage intervention in primary care is 0.89.

FIGURE 14.

Threshold analysis.

Discussion

In this chapter, we present exploratory analyses based on a simple model of the diagnostic pathway that has as its starting point symptomatic patients presenting to primary care and undergoing an initial clinical assessment, which is the same starting point considered by NICE Guideline Number 126 and the Diagnostic Guideline Number 30. 140 We then combine the model with an adapted disease history model for CRC from published studies (see Chapter 6), using a shorter cycle length and accounting for the excess risk of mortality due to undiagnosed CRC. By developing these features in our model, we were able to produce predictions in terms of meaningful clinical outcomes. The model was then used to identify the parameters that contribute most to the overall decision uncertainty about the cost-effectiveness of decision tools, which are good candidates for assessment as research priorities using expected value of partial perfect information. 166–168 Our model is expected to undergo further refinement as new evidence emerges, thus becoming an effective live research resource for evaluation and decision-making analysis.

Our analysis of key areas of uncertainty in the economic evaluation of the diagnostic tools led us to the following findings. First, a key determinant for the relative clinical effectiveness and cost-effectiveness of a diagnostic tool-based pathway is the diagnostic accuracy of current standard practice, which is currently unknown. We can only infer what this accuracy might be from the limited data available in a single report of findings from a retrospective survey. 173 Alternatively, as we have also done in this chapter, one may assume that patients are being managed by the results of the FIT, using the diagnostic accuracy estimates for this test available in the literature. 175 This may be a more consistent approach, but limits the scope allowed for accounting for clinician’s own judgement in the referral decision-making process.

Second, there is a lack of evidence for evaluating the impact of diagnostic tools in clinical practice. It is not known how these tools would be used in actual practice and how any impact they might have on referral intervals would affect clinical outcomes. We have opted for modelling the expected accuracy of using the diagnostic tools to triage patients for primary care investigation with a FIT, under the strong assumption that the diagnostic accuracy of FITs does not depend on the outcome of the diagnostic tool used for triage.

Third, there is little evidence on the prevalence of cancer in the low-risk population in which the diagnostic tools are intended for use. The value of this parameter is crucial for the cost-effectiveness of using the tool to detect more cancer patients early, which depends on limiting the cost imposed on the overwhelming majority of patients who present with symptoms of suspected cancer but turn out later not to have the disease. Previous analyses, including those informing the NICE Guideline Number 126 and Diagnostic Guideline Number 30,140 have assumed that the CRC prevalence in the low-risk population of interest here is 1.5% based on expert opinion, which, naturally, is highly uncertain. We have thus adopted this value in our base-case analyses. In any case, our exploratory analysis suggested that the specificity of the tools will also determine whether or not these may be used efficiently, without incurring excessive costs of colonoscopies performed in people without cancer, and are sustainable for a universal public health system such as the English NHS.

Fourth, the absence of data on the rate of CRC disease progression prior to diagnosis is also of importance. In our analysis, the slower the rate of progression, the lower the relative clinical effectiveness and cost-effectiveness of diagnostic tool triage to primary and secondary care investigations, as the tool triage would detect fewer extra cases before they had reached advanced disease stages, for which treatment is less effective, relative to those currently being detected in routine practice. However, the tornado analysis revealed that the sensitivity of current primary care practice, the specificity of the tool and the way clinicians would use the tools are more important sources of uncertainty than the disease progression in the evaluation of these tools.

Fifth, uncertainty regarding the decision-making process currently used in primary care for referring patients to secondary care owing to suspected CRC prevents drawing conclusions about the optimal intervention strategy. When we compared the intervention with a strategy of sending symptomatic patients home on the first visit and using the FIT-alone strategy for those patients who return with unresolved symptoms for a second visit, the proportion of non-CRC patients who return for the second visit determines whether or not a diagnostic tool approaches levels of cost-effectiveness. Thus, the economic value of the tools will depend on the clinical effectiveness of delaying access to colonoscopy as the alternative strategy for identifying patients who need referral, which may avoid unnecessary use of health-care resources and health risks of invasive investigations, at the cost of putting some patients with CRC at risk by delaying their identification and treatment.

Another limitation in our analysis is the lack of evidence on diagnostic accuracy from any direct comparative study of the modelled strategies. Thus, our sensitivity and specificity values are derived from single studies that are likely to be different in terms of population, as reflected, for example, in the underlying prevalence rate, or study design; for example, the evidence from one of the tools is derived from a case–control study, whereas the evidence for the other tool was obtained from a cohort study. Therefore, we cannot directly compare QCancer and RATs, as this is likely to lead to biased results.

In addition, we have not modelled the heterogeneity in current standard practice; therefore, we have not accounted for the variation in the costs and benefits of using the tools depending on local situations. It is conceivable that practices with robust referral systems may have little to benefit from using the tools, whereas areas without a sound referral system may find them beneficial. The model assumes 100% compliance of GPs with the strategies (both FITs and decision tools). However, it is clear that, currently, not all GPs use decision tools. More research is therefore needed to understand how compliance with strategies varies across localities.

The model does not account for the resource constraint effects imposed by limited capacity to do colonoscopies. Thus, although our model accounts for the benefits of faster access to colonoscopy, it does not include the opportunity costs of those patients whose access to colonoscopy will be delayed as a consequence of further demands placed on such service with limited capacity. This is an area of further research.

Conclusion

The decision-analytic model was developed to analyse the uncertainty inherent in the current evidence base, and to ask questions about the probable impact of the diagnostic tools, given the current lack of evidence. The model showed the importance of accounting for the excess mortality effects of CRC in symptomatic patients who are undiagnosed, and extends previous models in CRC by explicitly modelling a simple diagnostic referral pathway in primary care. Despite the high degree of uncertainty on evaluating the effect of the diagnostic tool on primary care referral of suspected CRC and health outcomes, our model has helped us to identify the parameters for which uncertainties contribute most to the overall decision uncertainty about the cost-effectiveness of decision tools, namely the sensitivity of current standard practice, the specificity of the diagnostic tool, the prevalence of CRC in the relevant primary care patient population and the cost of colonoscopy.

In the concluding section, we return to the original questions about the probable impact of the diagnostic tools on the costs and effects for patients presenting with symptoms of suspected CRC. For each question we summarise the initial findings from the model and highlight where the uncertainties still remain.

• What are the possible impacts on patient quality of life or survival if use of diagnostic tools reduces time to diagnosis?

  The model-based analysis suggests that the diagnostic tools would reduce the number of CRC patients who die without a diagnosis from 12% to 8%. The increased accuracy of the tool relative to the comparator also reduces the referral interval, which equals a reduction in the time to diagnosis and treatment by assumption. Owing to the challenges of measuring the effect on health-related quality of life post diagnosis, the amount of this type of benefit is highly uncertain. Overall, our modelled analysis suggests that, depending on the sensitivity of current practice, the diagnostic tools may extend life expectancy by somewhere between 4 days and 2 months among symptomatic patients aged 70 years presenting to their GP with undiagnosed CRC, relative to a hypothetical current practice of management according to results of a FIT alone. This range of variation in benefit highlights the importance of establishing current practice and its primary care diagnostic accuracy.

• Will the benefit to cancer patients identified earlier by diagnostic tools outweigh any disutility in extra patients referred for further investigation who do not have cancer?

  In the absence of evidence, our analysis suggests that the tools may help to improve diagnostic accuracy in primary care, and result in earlier referrals to secondary care. If the reduction in the referral interval results in a shortening of the diagnostic interval of the same magnitude, our model predicts that the amount of benefit to patients with CRC identified earlier outweighs the loss in quantity and quality of life in those patients referred for further investigation who do not have CRC. However, although the benefit to cancer patients would more than compensate the risks to life from exposing the overwhelming majority of patients without cancer to the risks of colonoscopy, it is unlikely to justify the additional health-care costs associated with such exposure.

  A sensitivity analysis was used to explore the uncertainty around the estimated cost-effectiveness results and to highlight the parameter values that were driving the results. The sensitivity analysis revealed that the cost-effectiveness results were particularly sensitive to uncertainty around the sensitivity of current standard practice and the specificity of the tool. In a further threshold analysis, we explored the levels of sensitivity of current standard practice and the specificity of the tool in which the decision tool would meet the £30,000 cost-effectiveness threshold. At the levels of sensitivity of ≥ 50% that have been reported for a FIT of 20 µg Hb/g faeces,175 the minimum level of specificity needed by the tool to serve as a cost-effective triage intervention in primary care is 0.89.

  The limited available data on current practice in the UK suggests that the sensitivity of primary care referrals is already too high for the diagnostic tools to make a significant impact on CRC patient survival. Although our base-case analysis produced an estimate of 2 months of life extension with the tools when compared with a FIT of 20 µg Hb/g faeces, this gain is reduced to 2 weeks when the sensitivity value derived from the number of visits before CRC diagnosis in routine practice is used, and to 4 days against a FIT of 10 µg Hb/g faeces. Given the very low prevalence of CRC in the relevant primary care population, this amount of benefit may be insufficient to outweigh the risks to life from exposing a vast number of patients without CRC to colonoscopy. Furthermore, whether or not the tools are cost-effective will depend on the prevalence and the sensitivity of current referral practice and the specificity of the tool.

• How big an improvement in quality of life would be needed to warrant the use of these tools if there are no survival impacts associated with the diagnostic tools?

  We investigated this question by assuming that the increased accuracy of the tool relative to the comparator does not result in any survival benefit but would avoid any loss in quality of life arising from false-negative results, whereas the quality of life of a false-positive result remains unaffected. The benefits gained from the tools are then largest when the losses from false-negative results are the biggest, as would occur when quality of life of these false-negative patients falls to zero. Our analysis, however, shows that even preventing these extreme losses in quality of life for false-negative patients is insufficient to make the diagnostic tools cost-effective (the ICER for QCancer is then £68,102 and for the RAT is £190,294). Thus, without survival benefits, the quality-of-life benefits of the tools alone would be unlikely to be sufficient to justify their costs.

• Could a cancer diagnostic tool be considered cost-effective if it reduces the period of extreme anxiety by expediting investigation and management in patients, even if it made no impact on patient outcome?

  The model results do not account for any negative impact of referrals on a patient’s quality of life due to anxiety, or the impact of colonoscopy on patient quality of life, for which we found no evidence in the literature. Therefore, we cannot provide an answer to our study question of whether or not a cancer diagnostic tool reduces the period of extreme anxiety associated with an uncertain CRC diagnosis by expediting investigations and management. Answering this question would require prospective studies measuring the outcomes of patients in primary care, but it is questionable whether or not such a study is feasible given the difficulty to define the patient population and the low number of cases expected in a single practice each year. Nevertheless, our exploratory analyses suggest that, even if plausible reductions in the diagnostic interval brought about by the use of the tools were to equate to reductions in the amount of time patients experience extreme anxiety, the tool would not be cost-effective without extending patient survival.

• Where are the gaps in the evidence base and where is more research needed?

  The updated review showed that the statistical relationship between diagnostic delay and cancer patient outcomes is not suitable for deriving valid estimates of the effect of expeditious CRC diagnosis on cancer stage or patient survival. In particular, the documented non-monotonic (e.g. U-shaped) relationship between diagnostic interval and mortality does not imply an effect mediated through the impact of diagnostic interval on stage of diagnosis. Given the lack of valid surrogate or clinical outcome data for building an economic model of diagnostic tools or decision-aid models in primary care, our cost-effectiveness analysis was limited to an exploration of the expected effects of time delays to diagnosis associated with the relative accuracy of the diagnostic tools. This limited aim will naturally be subject to more sources of uncertainty than an analysis based on clinical surrogate outcomes, but reflects a pragmatic approach given the quality of the data available.

  In the absence of direct clinical effectiveness data, we assumed a relationship between the sensitivity of both the diagnostic tool and current routine referral practice and the time to diagnosis. In the model, the time from symptom presentation to diagnosis consists of (1) the time from presentation in primary care to referral for investigation (the referral interval; see Chapter 5) and (2) the time from referral to specialist care to diagnosis. We then assumed that the diagnostic algorithm affects only the referral interval, while the time from referral to diagnosis and treatment remains fixed. Thus, in our analysis, we assume that the improvement of primary care diagnostic accuracy with the tool results in fewer visits to the GP before referral, and that the reduction in the accompanying time to referral is equal to the reduction in the time to diagnosis and treatment. We have explicitly modelled the average time to referral and diagnosis and treatment, based on the number of repeat appointments to make the mechanisms of effect transparent and open to scrutiny. The relationship between the number of repeat appointments and the time to referral is supported by evidence from a retrospective patient experience survey. 173 More research is needed on the impact of the different strategies on the number of GP appointments before referral and diagnosis.

  The review highlighted that there was a lack of direct comparative accuracy evidence between the strategies/tests involved in the model. Our model is therefore based on indirect comparisons of single-arm accuracy studies and assumptions about how the tools would be used to triage patients for investigations in secondary care. There is limited evidence on the diagnostic accuracy of decision aid tools in the UK low-risk population of interest, and the methods used to evaluate the tools that have any relevant evidence, QCancer and the RAT, differ in terms of how the relevant low-risk population is defined, that is ‘top 1 to 9% risk’ versus ‘low risk but not no-risk’,75 and the study design used to develop or evaluate them, that is observational cohort versus case–control, respectively. As QCancer was developed from a large cohort from electronic medical records, data were not prospectively collected and may thus lack relevant predictors to primary care practitioners. The RAT, in contrast, warrants validation in a large representative UK primary care population, because evidence on its diagnostic accuracy is based on a retrospective patient selection design. In addition, the added value of these tools is highly uncertain as the status quo has not been defined or rigoroursly evaluated, as reflected by the poor quality of the evidence informing the latest NICE guidance (Diagnostic Guideline Number 303), including the FIT, which has been recommended by such guidance but is not yet universal at the time of writing.

  More research is also needed to verify the relationship between diagnostic delay and cancer patient outcomes. This is a difficult area to study because of unobserved confounding factors, but one for which access to electronic medical health records with information to construct detailed symptoms libraries has opened new opportunities for fruitful research.

  The sensitivity analysis revealed that the cost-effectiveness results were particularly sensitive to uncertainty around the diagnostic accuracy of current standard practice and the specificity of the tool. We assume that all patients in the population of interest, who we defined as those with ‘low risk but not no-risk symptoms’,75 are akin to the CAPER study population. 75 This patient population includes the majority of symptomatic patients who do not present with high-risk symptoms such as rectal bleeding or severe anaemia, and therefore are not diagnosed through the 2WW referral pathway. The best available evidence on accuracy of the tool is for the QCancer tool, as it is based on a large cohort analysis. However, even these sensitivity and specificity parameters were not for the same population of interest here, as QCancer values include patients with red-flag symptoms, who may be deemed appropriate for 2WW referrals. Therefore, in our model, we calculated the sensitivity and specificity of QCancer for the top 10% risk threshold after excluding the top 1% risk observations from the published data77 to be 61% and 91%, respectively. In a further threshold analysis, we explored the levels of accuracy of current standard practice and the specificity of the tool in which the decision tool would meet the £30,000 cost-effectiveness threshold. At the levels of sensitivity of ≥ 50% that have been reported for a FIT of 20 µg Hb/g faeces,175 the minimum level of specificity needed by the tool to serve as a cost-effective triage intervention in primary care is 0.89. More research is clearly needed on the specificity of these tools among the population of interest.

  In addition, we have not modelled the heterogeneity in current standard practice; therefore, we have not accounted for the variation in the costs and benefits of using the tools depending on local situations. It is conceivable that practices with robust referral systems may have little to gain from using the tools, whereas areas without a sound referral system may find them beneficial. The model assumes 100% compliance of GPs with the strategies (both FITs and decision tools). However, it is clear that, currently, not all GPs use decision tools. More research is therefore needed to understand how compliance with strategies varies across localities.

  The model does not account for the resource constraint effects imposed by limited capacity for colonoscopies. Thus, although our model accounts for the benefits of faster access to colonoscopy, it does not include the opportunity costs of those patients whose access to colonoscopy will be delayed as a consequence of further demands placed on such service with limited capacity. This is an area of further research.

  Finally, there were insufficient data to evaluate one of the a priori aims of the model, which was to explore whether or not the tools will become cost-effective if they reduce the period of extreme anxiety, even if they made no further impact on patient outcome. Answering this question would require prospective studies measuring the outcomes of patients in primary care, but is questionable whether or not such a study is feasible given the difficulty to define the patient population and the low number of cases expected in a single practice each year.

Introduction

Diagnosing cancer quickly once patients become symptomatic remains an important priority in the UK. 185–187 A key part of the UK’s strategy for early cancer diagnosis was to lower the threshold for fast-track referrals of symptomatic patients for investigation from an estimated 5% to an explicit 3% risk of undiagnosed cancer. This was implemented via the 2015 revision of NICE guidelines on recognition and referral for suspected cancer. 6 A potential source of diagnostic delay remains the UK’s ‘gatekeeper’ system, whereby a GP decides whether or not a symptomatic patient meets the criteria for investigation for suspected cancer. 188 Indeed, fast-track referrals for cancer are less likely when patients present with ‘low-risk but not no-risk’75 symptoms of cancer than when they present with ‘alarm’ symptoms. 189

A number of models and algorithms have been developed to quantify the risk of an undiagnosed malignancy in symptomatic patients (see Chapters 3 and 4). 42,44 Some of these models have been translated into cancer decision support tools for clinical use, helping GPs to identify patients who warrant investigation for suspected cancer.

Systematic review 1 (see Chapter 3) identified two tools that are available to support doctors assess the risk of undiagnosed skin cancer. 36–38 These tools are adapted to Australian guidelines and have limited applicability in the UK. For the internal cancers, there are two main types of cancer decision support tools available in the UK: RATs and QCancer. RATs are available for a number of specific cancer sites, and use symptoms and test results to estimate the risk that a patient has an underlying cancer. 51,66,69–71,73,81,83,90 They were identified in SR1 (see Chapter 3) as having been evaluated in their mouse mat and desktop forms. QCancer tools estimate the risk of undiagnosed cancer based on symptoms, test results and patient risk factors. They are available for specific cancer sites,32,52,80,82,84,85 and also in sex-specific, person-centred forms, which estimate an individual’s risk that they have a number of different cancers. 53,54 QCancer tools were identified as validated cancer prediction models in SR2 (see Chapter 4).

In 2013, Macmillan Cancer Support, BMJ Informatica, the Department of Health and Social Care, and Cancer Research UK collaborated to incorporate RATs and QCancer into GP IT systems, renaming them collectively as ‘electronic cancer decision support tools’. RATs are fully integrated with the GP IT software INPS Vision (In Practice Systems Ltd, London, UK) and will soon be available via SystmOne (The Phoenix Partnership, Leeds, UK). QCancer is available via EMIS Web (EMIS Health, Leeds, UK), and QCancer tools are also freely available on the internet (www.qcancer.org). NHS Digital data indicate that, together, EMIS Web and Vision had 62% of the market share of GP IT systems in 2015. 190 In addition, RATs are available in mouse mat and flip chart forms; in January 2012, they were distributed to all 10,000 general practices in England as part of the National Cancer Action Team’s Supporting Primary Care Project. 185 Information about QCancer and RATs is available via the Cancer Research UK and Macmillan Cancer Support websites. 191,192

The electronic cancer decision support tools have three main functions:

1. alert/prompt – cancer risk scores appear automatically on opening a patient’s record

2. cancer risk assessment – GPs can request a patient’s cancer risk score, using the symptom checker

3. searches/report – GPs can routinely search records and produce summaries indicating patients who may need follow-up (‘safety-netting’).

Methods

A cross-sectional postal survey was designed and conducted in line with conduct and reporting guidelines for surveys. 194

Results

Sample characteristics

Responses were received from 473 GPs and from three GP registrars based in 227 practices. Responses from the GP registrars were not analysed separately, because of the small number of GP registrars providing responses. The response rate at the practice level was 23.3%; at the practitioner level, it was 10.3%. The responding practices had a median of 6 [interquartile range (IQR) 4–8] GPs, of whom a median of 2 (IQR 1–3) responded to the survey. The mean within-practice response rate was 43.7% (95% CI 39.3% to 48.1%). Unprompted comments written on incomplete surveys returned by practice managers, or messages received by e-mail or telephone (see Data entry and coding) indicated that lack of time (n = 12) and lack of awareness of the tools (n = 6) were the most common reasons for non-response.

Of the 476 respondents, 294 (61.8%) had been practising as a GP for ≥ 11 years, and 299 (62.8%) worked between five and eight sessions per week (Table 26).

TABLE 26 - Numbers of years in practice and number of consultation sessions worked per week

Demographic information	GPs
	n	%
Years in practice
< 1	20	4.2
1–5	57	12.0
6–10	57	12.0
11–20	145	30.5
21–30	111	23.3
> 30	38	8.0
Missing	48	10.1
Total	476	100.00
Number of consultations sessions worked per week
1–2	10	2.1
3–4	70	14.7
5–6	140	29.4
7–8	159	33.4
9–10	40	8.4
> 10	5	1.1
Missing	52	11.0
Total	476	100.0

EMIS Web was the most frequently used IT software (96/227, 42.3%), followed by TPP SystmOne (74/227, 32.6%) and then INPS Vision (32/227, 14.1%). These relative frequencies largely reflect the national market share (Table 27).

TABLE 27 - General practice software

Microtest Health, Bodmin, UK.

General practice IT software (eCDS tool)	Responding general practice		National market share (%)
	n	%
EMIS Web (QCancer)	96	42.3	52.4
TPP SystmOne (none)	74	32.6	33.0
INPS Vision (RAT)	32	14.1	9.9
Egton Medical Information Systems PCS (none) (EMIS Health)	11	4.9	0.01
Egton Medical Information Systems LV (none) (EMIS Health)	5	2.2	0.02
Other (none)	6	2.2	3.3
Microtest (none)a	3	1.3	1.4
Total	227	100	100

The IMD profile of the sample is generally representative of that of the national population (Figure 15).

FIGURE 15.

Index of Multiple Deprivation profiles.

Discussion

To the best of our knowledge, this is the first UK-wide survey of the availability of cancer decision support tools. Based on the responses received, cancer decision support tools, in either paper or electronic format, are available to GPs in approximately one-third (36.6%, 95% CI 30.3% to 43.1%) of UK practices. The proportion of general practices where at least one GP had access to the tools and was likely or very likely to use them was 16.7% (95% CI 12.1% to 22.2%). There are no current plans to re-release the paper-based tools, so the expectation is that electronic clinical decision support tools will become the norm. For this reason, it is useful to report separately the availability of electronic clinical decision support tools in UK general practices, which was 19.0% (95% CI 14.0% to 24.6%). Currently, the tools are available only via EMIS Web and INPS Vision, and approximately one-third of the practices using these software systems had opted to download or activate them. The software will shortly be integrated into SystmOne, which is estimated to have 33% of the market share in the UK. Between them, EMIS Web, SystmOne and INPS Vision represent > 95% of the general practice software systems available;190 therefore, in the near future, it is reasonable to assume that nearly 100% of GPs will have access to the electronic clinical decision support tools, should they choose to download/activate them.

There was no evidence that access to the tools was associated with a change in either the rate of 2WW referrals or the proportion of those who were referred transpiring to have cancer.

Conclusion

Our survey indicates that cancer clinical decision support tools are currently not widely used in the UK. This may contribute to our findings in SR1 (see Chapter 3) and SR2 (see Chapter 4) that there is limited evidence that these tools have an impact on patient outcomes. The upcoming integration of the tools into SystmONE will further increase the potential use of these tools.

As the levels of uptake are relatively low, it remains possible to carry out a RCT to assess whether or not these tools are genuinely helpful in improving the selection of patients for investigation. The attendant benefits of improved selection would include better targeting of investigation resources, potentially earlier diagnosis and reduced treatment costs. 13–15,19,88,89,91,186 Such a trial should include a study of barriers to tool use, and ways to overcome them, as well as a health economics arm.

What evidence exists on the clinical effectiveness and cost-effectiveness of cancer diagnostic tools in primary care? (Systematic review 1)

Systematic review 1 identified a limited number of heterogeneous studies that provided weak evidence of the impact of these diagnostic tools on patient-related outcomes or referral patterns. This finding is in line with previous reviews across different types of cancers,40,44 confirming the uncertainty surrounding the clinical utility of diagnostic tools in primary care. Only two RCTs were identified, and the only existing trial of RATs found no statistically significant difference in time to diagnosis between the intervention and control groups. No studies evaluating the cost-effectiveness of these tools was identified, even though the definition of diagnostic tool used in SR1 was extended to include the implementation of any diagnostic model or algorithm.

The results of SR1 are limited by heterogeneity between the studies in terms of the tools evaluated, the populations and the outcomes used. Furthermore, the findings are constrained by the design of the included studies, for example non-randomised designs and infrequent use of patient-reported outcomes.

What diagnostic prediction models exist in the literature that have the potential to be used as tools in primary care? (Systematic review 2)

Eleven site-specific cancer diagnostic prediction models were identified, and covered a range of common and less common cancer sites. These included different versions of the QCancer and RAT models, both developed in the UK. All of the models identified had been developed in Europe.

The risk of bias across studies reporting the development or validation of models was mixed. Notably, the QCancer models were developed using cohort data, and most have been externally validated, but lack impact assessment. In comparison, the RAT models have gathered more evidence of impact on practice, but have been developed from case–control studies, and with limited external validation. With the exception of the CRC RAT model, there were no reports to suggest that models were being updated based on new available data.

What evidence exists on the association between different diagnostic interval durations and patient outcomes? (Updated review)

The findings of a previous systematic review25 were updated (see Chapter 5) to further examine the association between different durations of time from first symptom to diagnosis or treatment and clinical outcomes across all major cancers. A more in-depth evaluation was conducted of CRC to inform the model.

The literature suggests a U-shaped relationship between diagnostic interval and unfavourable outcomes, which suggests the clinical need to expedite the diagnosis of symptomatic patients to avoid the preventable delays that some patients continue to experience. The review also identified important biases and other factors that may affect the findings of studies in this field.

The majority of the CRC studies found ‘no association’ between various intervals and patient outcomes. A small number of studies (n = 4, although three used the same or overlapping population) reported a positive association between shorter intervals and patient outcomes, but, paradoxically, a small number of studies (n = 3) also found a negative association.

What is the likely impact that use of the tools in clinical practice may have on patient outcomes and NHS resources? (Economic model)

There are clear gaps and uncertainties in the evidence base that need to be addressed to fully assess the clinical effectiveness and cost-effectiveness of the tools. There is no available evidence that using the tools has an impact on clinical practice.

To explore trade-offs in the benefits, harms and associated NHS resources related to the use of diagnostic tools in primary care, we developed a simple model of the primary care diagnostic pathway for patients with symptoms suspected to be caused by CRC. Exploration of the model indicates uncertainty as to whether or not the amount of benefit to patients who are identified earlier by diagnostic tools and confirmed as having cancer outweighs the loss in quantity and quality of life in those patients referred for further investigation who do not have cancer. In the absence of good-quality evidence, our model predicts that use of a diagnostic algorithm would extend life expectancy by between 4 days and approximately 2 months among symptomatic patients aged 70 years presenting to their GP with undiagnosed CRC, relative to a hypothetical current practice of management according to results of a FIT alone. Although this amount of benefit may be larger than the loss of life from exposing the overwhelming majority of patients without cancer to the risks of colonoscopy, it is unlikely to justify the additional £25 incurred in health-care costs with the tool per low-risk patient presenting to primary care suspected to have CRC.

The ability to fully evaluate the cost-effectiveness of using diagnostic tools in primary care is restricted because of a lack of evidence on their impact on clinical practice and diagnostic intervals, as well as on current standard practice. In the absence of such evidence, our analysis had to rely on two key assumptions: first, about the relationship between the sensitivity of the test and number of GP visits before referral, which has previously been shown to correlate with time to referral, and, second, about the relationship between the time to referral and time to diagnosis and treatment, so that a reduction in the referral interval was assumed to correspond to an equal reduction in the time to diagnosis and treatment.

Because of the uncertainty behind these assumptions, cost-effectiveness analysis was not the objective of our study. However, we were able to identify that, in addition to research to verify the validity of these assumptions, other key sources of uncertainty in the evidence should be the focus of research, including an evaluation of the tools in comparative studies in low-risk populations and relative to current primary care diagnostic practice. It is noted that the diagnostic accuracy studies should be conducted across the low-risk subpopulation, defined by age and prevalence, and that the sensitivity of standard practice and the specificity of triage tools are two of the most influential parameters for cost-effectiveness, which is also driven by the cost of colonoscopy.

To what extent do general practitioners have access to and use existing cancer decision support tools? (General practitioner survey)

We conducted a UK-wide survey of GPs on the availability of cancer decision support tools. Cancer decision support tools, in either paper or electronic format, were available to GPs in 36.6% (95% CI 30.3% to 43.1%) of UK practices. The proportion of general practices where at least one GP has access to the tools and was likely or very likely to use them was 16.7% (95% CI 12.1% to 22.2%). The proportion of general practices in the UK with access to an electronic clinical decision support tool was 19.0% (95% CI 14.0% to 24.6%).

The survey indicates that cancer clinical decision support tools are currently not widely used in the UK. There was no evidence that access to the tools was associated with a change in the rate of 2WW referrals, or the proportion of those referrals transpiring to have cancer. The subanalyses also did not provide any evidence of an association between access to the tools and practice-level diagnostic activity. This is, perhaps, not surprising, given that the electronic tools, in particular, have been available for only a short time and our survey suggests that they are not yet embedded in clinical practice.

Strengths

Systematic reviews 1 and 2, and the updated review, followed prespecified systematic review protocols, and the team conducting the reviews are independent and experienced in systematic review methodology. The decision model was specifically developed to reflect the primary care pathway for patients with symptoms suspected to be caused by CRC. We were able to represent key relationships required to produce predictions in terms of meaningful clinical outcomes and identify the main parameters contributing to uncertainty in the model. The model is expected to undergo further refinement as new evidence emerges, and to be used as a template for modelling the effect of diagnostic tools in other cancers.

The GP survey is the first UK-wide survey of the availability and use of cancer decision support tools. It is also the first national study to attempt to link use of the tools with changes in the rate of 2WW referrals or cancer diagnoses.

Limitations

The reviews and decision model were limited by a lack of evidence, and poor study quality. The best-quality study in SR135 was conducted in Australia, and used a composite intervention including RATs (with older versions of several instruments that were developed using populations from a different country). Not only does this limit the generalisability of results, but it also hinders examination of why no impact of patient outcomes was found. For instance, could it be due to low uptake, poor implementation or limited marginal contribution of the tools in assessing the risk of cancer?

Systematic review 2 identified a number of potential models for use in primary care, yet considerable gaps in the literature exist. There is a disparity in the available evidence on the impact of the tools and the extent to which models have been validated (e.g. whether or not they have been validated at all, and if so, if it has been in the appropriate populations). Comparison of the identified models is limited by heterogeneity between different prediction models for different cancers.

Because there was heterogeneity between studies in the updated review of diagnostic intervals, a ‘vote-counting’ approach was taken, which does not account for the magnitude of any effect. The potential for selective outcome reporting also limits the interpretation of the evidence on an association between duration of pre-diagnostic intervals and patient outcomes. It is therefore very difficult to say whether or not there is an association between different interval durations and patient outcomes, although the more recent evidence of a U-shaped relationship requires further examination.

The decision model has several limitations as it relies on a linked-data approach in a context of little available evidence. In particular, in the absence of direct clinical effectiveness data, we assume a relationship between the sensitivity of both the algorithm and routine referral practice and the number of primary care visits before referral, which has been shown to be associated with the referral interval, and that any difference between diagnostic strategies in the implied referral interval equals their difference in time to diagnosis and treatment. The lack of relevant data on the diagnostic accuracy of the current referral practice in primary care, and on the prevalence of cancer in populations at low risk are identified, not surprisingly, to be crucial in calculating the magnitude of the trade-offs in the decision model. Crucially, we do not know how the tools are being used and our main analysis of a diagnostic tool that is used to select patients for direct referral to secondary care and manage the rest by standard investigation may not necessarily be relevant to many localities using the tool. Furthermore, there were insufficient data to evaluate the effect of the tools on quality of life that is mediated by a reduction in the diagnostic interval and associated experience of severe anxiety by patients and their relatives, although our optimistic exploratory analysis revealed that this effect is unlikely to render the tools cost-effective, without further impact on patient survival.

The survey was limited by a lower than expected response rate, with a small likelihood of response bias (those aware and using the tools may have been more likely to respond). This may limit the ability to conclude that the tools do not lead to increased diagnostic activity or, indeed, the ability to characterise referral systems that may be able to benefit more from the tools.

Uncertainties

There are clear gaps and uncertainties in the evidence base that need to be addressed to fully assess the clinical effectiveness and cost-effectiveness of the tools. There is uncertainty about the relative accuracy of the tools, as there is no single study that has compared them, or evaluated any one of them relative to standard practice or against any comparator. The tools that we focused on had evidence from single-arm studies of different designs and population prevalence. Our model-based evaluation was undertaken in the context of the recent publication of a policy recommendation for FITs to replace FOBTs as routine tests. In primary care, the evidence on FITs is limited as the tests have not been evaluated in the population relevant to our study. This means that even the evidence from indirect comparisons of the tools with FITs is limited to diagnostic accuracy data that may not be valid for our study purposes.

Moreover, there is uncertainty about the interaction between GP, tool and the patient. The intervention is not just the tool, but how the GP interacts with the tool. A better understanding of this may help to disentangle uncertainties on whether or not a lack of evidence of the clinical effectiveness of the tools is due to poor implementation, insufficient uptake or the assumptions on which the tools are based.

Given the uncertainities in the evidence base, it is also possible that rapid referral can do more harm than good, or at least more harm than people might expect. As well as the false positives, there is the likelihood that expediting investigation in one group will lead to fewer resources being available for the group that are not expedited. The group not expedited will contain cancer cases, as sensitivity is around 50%, and slower diagnosis could lead to worse outcomes for these cases. These uncertainties and the associated potential for an inappropriate allocation of resources suggests the need for future research to address these issues, as outlined in Chapter 10, Recommendations for research.

Given the difficulty of conducting robust clinical studies of diagnostic tools for suspected cancer referrals in primary care, evaluations may pragmatically focus on intermediate outcomes, such as referral intervals, time to diagnosis and treatment initiation. In addressing the structural uncertainty arising from translating an effect on referral intervals into an effect on the time interval until diagnosis, observational studies may fruitfully exploit routine electronic health-care records in which, for example, changes in referral policy guidance could opportunistically be used as a natural experiment to assess the impact of earlier referrals on the diagnostic interval.

Recommendations for practice

The uncertainties about the clinical effectiveness of cancer diagnostic tools in primary care, compounded by continued lack of clarity about the relationship between delay to diagnosis or treatment and outcome, suggest that further recommendations for increased use of these tools may be premature. It may be that the limited uptake of the tools observed in our survey is appropriate, given the limited and mixed evidence base we have found.

Recommendations for research

It is undeniable that the rationale for cancer diagnostic tools in primary care is strong. Therefore, the priority should be for further research to confirm or refute the potential of cancer diagnostic tools to improve cancer outcomes. QCancer and RATs are well-established tools. We therefore tentatively suggest that further research should concentrate on QCancer and RATs.

The research on these tools should include:

1. Continued model validation – RATs should try to emulate the validation methods employed by QCancer. Adhering to good conduct and reporting guidance on studies of prediction tools is essential to maximise the value of the studies and to improve the ability to review them.

2. QCancer versus RATs – the performance of the two tools should be compared. It would be particularly helpful to see how similar the risks predicted by the two tools are for identical patients.

3. Assessment of clinical effectiveness – although some assessment of the clinical effectiveness of RATs has been performed, this has been inconclusive, so there is a need for this to be performed for both tools. Studies should collect information on time to diagnosis and treatment and stage at diagnosis, as well as health outcomes (if feasible). In terms of design, cluster randomised trials have been shown to be feasible, and are preferable to quasi-experimental designs already attempted. Given the likely small effect on health outcomes in particular, it is essential that the sample size of any study is appropriately large. A Phase II cluster RCT evaluating the impact of RATs for gastro-oesophageal cancer is ongoing. 20

4. Assessment of the tool in GP patient consultations – as the tool is not used in isolation, research to understand the complex nature of the intervention is warranted to investigate the interaction between GP, tool and patient. This may be part of an interventional study.

5. Assessment of quality-of-life implications of delayed and incorrect diagnosis. We do not know what the effect of delayed diagnosis is on the health-related quality of life of cancer patients and their families during the diagnostic delay and after delayed diagnosis. The anxiety caused by persistent unresolved symptoms would require retrospective observational studies recruiting subjects in large secondary care centres.

6. Assessment of cost-effectiveness – although the main current limitation is the absence of clinical effectiveness evidence, researchers should anticipate how cost-effectiveness will be addressed when such evidence becomes available. Incorporating assessment of cost-effectiveness into trials may be one approach, but experience suggests that, to generalise and extrapolate from the trial results, continued enhancement of the economic model developed in this project should be pursued and used to synthesise the latest evidence from points 1 to 3 above. Thus, our model may provide an iterative framework for informing the design of studies described in point 3, and be populated by the results of those studies.

7. Assessment of impact on other referral pathways – clinical effectiveness studies should consider not just the referral/diagnosis time in expedited pathways, but also the impact on the other pathways, as cancer cases might be slowed by a shift in resources to the expedited pathway.

8. Finally, the challenges of interpreting observational data exploring the relationship between delay and outcome continues, as demonstrated in the updated review. For further studies answering this question to be useful, they need to address issues of unmeasured confounding such as grade and stage, as well as problems of survivor time or selection bias implicit in the period of follow-up over which the studies are conducted. These are problems limiting the quality of existing evidence, some of which may be improved by developing agreed standards for providing complete and accurate reporting of all methods used and outcomes measured.

Contributions of authors

Professor Antonieta Medina-Lara (https://orcid.org/0000-0001-7325-8246), Associate Professor in Health Economics, acted as the project lead, worked on the systematic review of the economic models evidence and on the development of the de novo economic model, and supported the GP survey.

Dr Bogdan Grigore (https://orcid.org/0000-0003-4241-7595), Postdoctoral Research Associate, led SR1 and SR2.

Ms Ruth Lewis (https://orcid.org/0000-0003-0745-995X), Senior Lecturer, led the updated review and worked on SR1 and SR2.

Dr Jaime Peters (https://orcid.org/0000-0003-1778-3518), Senior Research Fellow, supported SR1 and SR2 and the writing-up of the economic model.

Dr Sarah Price (https://orcid.org/0000-0002-2228-2374), Research Fellow, undertook the design, data collection and analysis, and the writing-up of the GP survey.

Dr Paolo Landa (https://orcid.org/0000-0001-6532-6747), Research Fellow in Health Economics, worked on the systematic review of the economic models and on the de novo model.

Ms Sophie Robinson (https://orcid.org/0000-0003-0463-875X), Information Specialist, carried out the electronic searches for SR1, SR2, the review of economic models and the review of parameter values for the de novo economic model.

Professor Richard Neal (https://orcid.org/0000-0002-3544-2744), Professor of Primary Care Oncology, and Professor William Hamilton (https://orcid.org/0000-0003-1611-1373), Professor of Primary Care Diagnostics, provided support to the project on the clinical aspects of the project.

Professor Anne E Spencer (https://orcid.org/0000-0002-8163-3103), Associate Professor in Health Economics, provided senior advice for the review of the economic models, the de novo model and the GP survey.

Data-sharing statement

Requests for survey data or a copy of the model should be submitted to the corresponding author for consideration. Access to anonymised data or a dummy-data populated model may be granted following review.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care.