Dynamic contrast-enhanced CT compared with positron emission tomography CT to characterise solitary pulmonary nodules: the SPUtNIk diagnostic accuracy study and economic modelling

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 09/22/117. The contractual start date was in August 2012. The draft report began editorial review in December 2019 and was accepted for publication in September 2020. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© Queen’s Printer and Controller of HMSO 2022. This work was produced by Gilbert et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2022 Queen’s Printer and Controller of HMSO

Background and rationale

Although the incidence of lung cancer is slowly reducing in the UK, the number of new cases diagnosed each year is > 47,000. 2 A proportion of patients with lung cancer present with a solitary pulmonary nodule (SPN) on diagnostic imaging tests; these patients form an important subgroup, as early-stage disease has excellent survival rates following surgical resection. 3 However, not all SPNs are due to lung cancer and the accurate characterisation of SPNs is an ongoing diagnostic challenge with significant associated health costs. A 2010 observational study4 found that the average US Medicare expenditure for clinical management of a patient with a SPN was US$50,233 (£30,363) when the SPN was malignant and $22,461 (£13,577) when it was benign. With the advent of NHS England’s Targeted Lung Health Checks programme,5 piloting low-dose computerised tomography (CT) lung cancer screening in England, the number of patients with a SPN requiring further investigation will increase substantially. A previous Health Technology Assessment (HTA) review6 noted that CT screening is associated with a relatively high false-positive rate and that subsequent investigations constitute a significant cost. Furthermore, SPNs are a common finding in whole-body screening CT examinations offered to asymptomatic individuals by independent sector providers. Typically, the costs of follow-up investigations from these examinations are incurred in the public sector (UK National Screening Committee, appendix 17). Novel cost-effective approaches to the assessment of SPNs would be of value to the NHS.

Trial design

The trial was designed in accordance with the guidance for the methods of technology appraisal issued by NICE40 and adopted by NICE in formulating its guidance for the use of PET in the staging of lung cancer.

The trial was designed as a prospective observational study to assess the diagnostic performance and incremental value of DCE-CT by the addition of this modality to PET/CT in a cohort of 375 patients with a SPN, with the trial protocol previously published. 39 The trial flow diagram is presented in Figure 1.

FIGURE 1.

Flow diagram of the SPUtNIk trial. IPCARD-SPN, Identifying symptoms that Predict Chest And Respiratory Disease – Solitary Pulmonary Nodule; SPUTniK, Single PUlmonary Nodule Investigation.

Participants

Setting and recruitment pathway

Patients were identified at local MDT meetings, at the time of referral for investigation of a SPN, or at referral to the PET centres for PET/CT on the basis of having a single dominant pulmonary nodule on a CT scan with uncertain aetiology.

An invitation letter and patient information sheet were sent to potential patients, along with their PET/CT appointment letter, inviting them to participate in the study.

Local research and NHS staff approached potential patients either in clinic or by telephone to:

• explain the study and/or provide the patient information sheet

• note the age, sex and smoking history of a patient

• confirm eligibility for the study.

Patients were given an appointment for PET/CT, and were booked for DCE-CT on either the same day or within 14 days of the PET/CT appointment. (Note that, if there were scheduling issues, appointments could be up to a maximum of 21 days apart.) Some sites chose to make the DCE-CT appointment at the time of the PET/CT appointment, following consent, if there were constraints on scanner time.

The Single PUlmonary Nodule Investigation (SPUtNIk) study patient information sheets and Identifying symptoms that Predict Chest And Respiratory Disease – Solitary Pulmonary Nodule (IPCARD-SPN) questionnaire were given to patients either in clinic or by post.

Outcomes

Adverse events

All adverse events ocurring within the 30 days following the study DCE-CT were reported to the SPUtNIk database. For full details, see the SPUtNIk study protocol. 39

Registration

Sites registered patients to the SPUtNIk study by sending a signed registration form via fax or e-mail attachment to the Southampton Clinical Trials Unit (SCTU). Sites were given a block of SPUtNIk codes. Patients were allocated trial identifier codes consecutively, following consent. On registration, SCTU staff checked eligibility and confirmed the patient trial identifier code by e-mail. Registration and DCE-CT could not take place before informed consent was signed; however, registration with SCTU could take place before or after DCE-CT, because there was a possibility that some clinics would schedule the DCE-CT outside office hours.

Blinding

The study was a non-randomised diagnostic accuracy and comparative health economic effectiveness trial; therefore, blinding was not necessary to meet the trial objectives. However, the results of the DCE-CT were not reported to the participants’ clinicians, so as not to bias the assessment of the diagnostic accuracy of DCE-CT.

Data collection

Sample size

For the primary outcome measures in this study, the diagnostic characteristics for DCE-CT and PET/CT were used for the sample size calculations. Use of the other outcome measures that are related to the economic analyses was prevented by the prior need for detailed characterisation of the decision trees. We consider the sample size needed to detect particular accuracy for each test, separately, and then when the tests are used in conjunction.

Statistical methods and data analysis

We considered the diagnostic accuracy of positive PET/CT and DCE-CT scan results, both separately and in conjunction, in relation to a diagnosis of lung cancer by 2 years. In these analyses, we will be able to include only those patients in whom the presence of lung cancer by 2 years is confirmed.

Initially, the separate diagnostic performance of PET/CT and DCE-CT were examined using the predefined diagnostic criteria. For PET/CT, this was based on the combined PET grade and CT grade (see Table 3 for grade breakdown). This was classified as positive for malignancy if one of the following criteria was met: grade 4 on PET or CT, or grade 3 on both PET and CT, or grade 2 on PET and grade 3 or 4 on CT. For DCE-CT, 15-Hounsfield unit (HU) enhancement is prevalent in the literature when scanning patients at an X-ray energy of 120 kVp. We proposed scanning patients at 100 kVp to obtain a higher signal-to-noise ratio. The attenuation of iodine increases with decreasing photon energy. Phantom measurements verified that an iodine concentration measuring 15 HU at 120 kVp measured approximately 20 HU at 100 kVp; as a result, 20 HU was chosen as the prespecified threshold for malignancy.

In addition, we examined the full range of possible threshold values to see if performance could be improved. This involved using the grading of individual CT and PET images, as well as the maximum and mean SUV from the PET image, peak enhancement, peak and mean HU from the DCE-CT and the reporting radiologist’s classification of the SPN from the PET/CT and DCE-CT results. The classification produced by the site radiologists was their expert opinion and was based on all available information from either the PET/CT or DCE-CT (correspondingly), as well as any prior information from the initial staging CT. An indeterminate option was available here. At each threshold, sensitivity and specificity were estimated (with 95% CIs), and the receiver operating characteristic (ROC) curve calculated. The optimal cut-off point from the range of values is reported based on keeping the sensitivity above 90% and maximising specificity within this limitation. An alternative cut-off point that provides the best trade-off in sensitivity and specificity is also reported. The sensitivity, specificity, PPV, NPV and ODA at these cut-off points, as well as the pre-defined values, are presented with 95% CIs in this report. When translating sensitivity and specificity to PPVs and NPVs, we assumed a particular prevalence, using the value seen in our study. We report PPVs and NPVs for a range of other prevalences reported in the literature, identified from our systematic review. The ODA is the percentage of cases that are correctly classified, regardless of whether they were cancers or non-cancers.

The diagnostic performance of PET/CT and DCE-CT combined was examined using the same techniques as previously described, with patients classed as ‘positive’ if they had both PET/CT and DCE-CT positive scan results, and all other patients classed as ‘negative’.

A logistic regression model was undertaken, including key diagnostic elements from both PET/CT and DCE-CT. This included (but was not limited to) the individual imaging components that were described previously, size measures from each of the scans and predefined measures such as the standardised perfusion value (SPV). The SPV is the maximum enhancement multiplied by the subject’s body weight and divided by the dose of iodine received. These variables were used on their original scale as covariates in a logistic regression to produce a risk score and predicted probability of lung cancer for each individual, based on their specific test values. This predicted probability came from a transformation of the linear predictor from the logistic regression model: the predicted ‘risk’ for each individual. A cut-off value was used to decide a high-risk score (which predicts an adverse outcome) and a low-risk score (predicting a good outcome). The calibration of the model was assessed by grouping patients into deciles ordered by predicted risk and considering the agreement between the mean predicted risk and the observed number of true lung cancer cases in each decile (sometimes referred to as the expected vs. observed statistic). The derived diagnostic rule was cross-validated by comparing the classification of each patient with their outcome of confirmed lung cancer, allowing an estimate of the sensitivity and specificity of the prediction model. By varying the chosen cut-off level, a ROC curve was produced, summarising the sensitivity and specificity of the predictive rule across the full range of cut-off points. The overall discriminatory ability was summarised as the area under the receiver operating characteristic (AUROC) curve with 95% CI. The most suitable cut-off level was selected using the same rules as before. The internal validity of the final model was assessed by the bootstrap resampling technique to adjust for overoptimism in the estimation of model performance due to validation in the same data set that was used to develop the model itself. These models will be exploratory only and, before they can used or considered useful, they would require external validation in a separate data set.

If these methods showed poor accuracy performance (in terms of calibration and/or discrimination), then the logistic regression model was extended to include additional patient-level covariates (such as time from ¹⁸F-FDG injection to PET/CT scan) in addition to test results. Demographic information was considered, as well as clinical and imaging features considered indicative of a higher likelihood of SPN malignancy. The performance of the model was also evaluated at the site level, when possible, to ascertain whether or not model performance is consistent in each site or, if it is not, the variability in performance across sites, and whether or not this can be improved by tailoring the prevalence in each site.

Interim analysis and data storage

No interim analysis was planned for this study, although there were a series of Data Monitoring and Ethics Committee meetings that occurred throughout the duration of the data collection process.

Data and all appropriate documentation will be stored for a minimum of 15 years after the completion of the trial, including the follow-up period.

Accreditation of positron emission tomography centres

Accreditation of dynamic contrast-enhanced computerised tomography centres

Centres performing DCE-CT for the SPUtNIk study underwent an accreditation process developed specifically for this trial and conducted by the Radiation Protection Department of the Mount Vernon Cancer Centre, East and North Hertfordshire NHS Trust.

Two phantom types were designed for the trial. The SPUtNIk chest-equivalent phantom (Figure 2) was constructed by the Clinical Physics Department at St Bartholomew’s Hospital, and the SPUtNIk water-filled Radiographer QA Phantom (Figure 3) was constructed by the Bioengineering Department at the Mount Vernon Cancer Centre.

FIGURE 2.

The SPUtNIk chest-equivalent phantom. Outer annulus, diameter 290 mm, composed of water-equivalent material WT1 (Clinical Physics Department, St Bartholomew’s Hospital); middle annulus, diameter 240 mm, composed of lung-equivalent material LN10; central circle, diameter 100 mm, composed of water-equivalent material WT1. Six holes, each 30 mm in diameter, designed to hold a vial containing iodine solution at the following concentrations: 0.0, 0.3, 0.6, 1.6, 2.4 and 4.7 mg/ml. Phantom thickness: 100 mm.

FIGURE 3.

The SPUtNIk water-filled radiographer quality assurance phantom. Perspex cylinder: 250 mm outer diameter, 244 mm internal diameter. Water filled with release plug to fill/empty phantom. Three internal chambers, each of 30 mm internal diameter, with plugs to fill/empty with iodine solution at the following concentrations: 0.6, 1.6 and 2.4 mg/ml.

A clinical scientist from the DCE-CT accreditation team visited each site and, with the lead radiographer, entered the weight-dependent trial protocols into the scanner and saved them with clearly identifiable names. The chest-equivalent phantom was scanned under the trial protocol (see the SPUtNIk study protocol39). One SPUtNIk water-filled radiographer QA phantom was issued to the centre and scanned, as per the QA protocol (see the SPUtNIk study protocol39). Digital Imaging and Communications in Medicine- (DICOM-)standard images of both phantoms were saved to a compact disc for later analysis. The peak energy (in kVp) and half-value layer of the X-ray beam at 100 kV were measured using the RaySafe™ Xi R/F Detector (RaySafe, Billdal, Sweden), positioned in the scan plane and using a scout projection, taking care to avoid attenuation by the table/couch.

Information and instruction were given to the lead radiographer for QA for the SPUtNIk trial and for data anonymisation and transfer to the PET Core Lab for storage and further analysis of patient images. Images exported from each scanner were checked to ensure that the correct scan protocol had been set up. The mean and standard deviation in CT number in the iodine inserts of both the chest-equivalent phantom and the Radiographer QA phantom were extracted using IQWorks [http://iqworks.org/ (accessed 12 October 2018)]. The line of best fit between CT number and iodine concentration in each phantom was assessed using Microsoft Excel^® 2010 (Microsoft Corporation, Redmond, WA, USA), and the gradient was taken as the iodine calibration factor at 100 kV. Sites were issued a certificate of accreditation and informed of the iodine calibration factor in the SPUtNIk chest phantom for their scanner.

All scanners were required to undergo regular planned preventative maintenance and CT number calibration, at frequencies recommended by the manufacturer.

Fourteen sites recruited patients as part of the study. Participants underwent DCE-CT on one of 16 scanners, as summarised in Appendix 2, Table 28.

The mean iodine calibration factor in the lung-equivalent phantom was 32.2 HU/(mg/ml), with a coefficient of variation across the scanners of 6.6%. The mean iodine calibration factor in the water-filled radiographer phantom was 30.0 HU/(mg/ml), with a coefficient of variation of 4.5%.

Dynamic contrast-enhanced computerised tomography acquisition design

The DCE-CT acquisition settings were chosen to reasonably standardise dose and image quality across all manufacturers and models of scanner. The full procedure is listed in the SPUtNIk study protocol. 39

Computerised tomography scans taken at 120 kVp are prevalent in the literature; however, by scanning at a lower photon energy, the increased attenuation of iodine could be exploited. For a given iodine concentration, the increased attenuation at low energy results in a higher CT number. Scans of phantoms simulating patients of different weight explored the use of 100 kVp and 80 kVp. At 80 kVp, the X-ray tube current required to maintain a desired signal-to-noise ratio was sufficiently high to exclude most CT scanners. However, at 100 kVp, a good signal-to-noise ratio could be achieved for all phantom sizes. Therefore, 100 kVp was chosen.

Tube current modulation alters the X-ray current, and, consequently, the photon intensity, with variation in patient attenuation during CT. The variation in manufacturer approach to tube current modulation would have resulted in differences in dose and image quality, depending on patient weight and scanner type. Standardisation of tube current modulation between manufacturers was beyond the scope of this trial; therefore, fixed tube currents were used across three weight categories: < 60 kg = 200 mA, 60–90 kg = 350 mA and > 90 kg = 500 mA. These tube currents were chosen to maintain a signal-to-noise ratio independent of patient weight.

At the time of trial inception, 64 × 0.6 mm collimation was representative of modern scanners. Slice thickness and interval were chosen to provide adequate noise statistics for the measurement of CT number enhancement, with enough sensitivity to choose the slice position and with maximum enhancement for nodules of > 8 mm in diameter. A reconstruction kernel with similar appearance and noise level across each manufacturer was chosen.

Although some scanners included in the trial had an option of iterative reconstruction, this was not used, so as to standardise the dose and image quality of all CT systems, regardless of scanner model and manufacturer.

Accreditation of positron emission tomography centres and technical quality control of positron emission tomography scans

Imaging protocols with guidance for performing the PET/CT (see SPUtNIk protocol39) were provided to sites, with local clinical protocol followed if the PET/CT was carried out as standard of care prior to study entry. Of the 380 patients recruited to the study, 373 were eligible for the study and underwent PET/CT imaging from January 2013 to December 2016. The scans were sent to the UK PET Core Lab, St Thomas’ Hospital, for technical checks (Table 2).

Any deviations from the expected patient preparation, image acquisition or scan processing were flagged for discussion with a PET expert. A total of 370 (99.2%) PET scans were submitted for technical review. The results of technical review were as follows:

• Sixteen scans used a new reconstruction not accredited for the study called point-spread function (PSF) modelling.

• Patient blood glucose was not measured/not provided for seven patients; one patient had a blood glucose level of > 11 mmol/l.

• Fasting status was not provided for 27 scans, nine patients fasted for < 6 hours, but all fasted for at least 4 hours and their blood glucose was within the expected range.

• All scans covered at least the angle of jaw to mid-thigh.

The mean injected activity was 352.6 MBq (range 148.1–464.0 MBq). The national diagnostic reference level (DRL) for ¹⁸F-FDG is 400 MBq and it is recommended to use a weight-based protocol of 4.5 MBq/kg, with a minimum injected activity that is dependent on the scanner model and acquisition parameters used. The mean injected activity used for the SPUtNIk patients was 4.9 MBq/kg (range 2.5–12.2 MBq/kg) and patient weight ranged from 30 to 150 kg (mean 76.8 kg). A plot of the injected activity as a function of patient weight is shown in Figure 4.

FIGURE 4.

Injected activity as a function of patient’s weight for all patients.

The mean uptake time was 68.6 minutes (range 49.0 to 117 minutes). The distribution of uptake times is shown in Figure 5; for 72% (n = 267) of PET scans, the uptake time was within 60 ± 10 minutes. For 80% (n = 295) of scans, the uptake time was within 50–75 minutes. Only one scan had an uptake time of < 50 minutes, and 74 had an uptake time of > 75 minutes. The standard-of-care protocol for one centre was a 90-minute uptake time, rather than the 60-minute uptake time in the other sites.

FIGURE 5.

Uptake time (time between injection and PET scan start) for all patients.

Radiation dose

In general, a dose of 7.6 mSv is quoted for the radiotracer injection based on 400 MBq and 6.5 mSv for the CT part, based on national DRLs for CT as part of a PET-CT. This gives a total of 14.1 mSv for the PET-CT. However, local practice varies and the expectation is that the PET injected dose is lower for SPUtNIk, as many sites used weight-based injected activities.

Radiographer-led quality control of dynamic contrast-enhanced computerised tomography scanners

Each site was issued with a water-filled radiographer QA phantom and shown how to carry out tests on its scanner. Baseline measurements were performed by the clinical scientist, who then created a radiographer QA spreadsheet with tolerances and pass/fail criteria for the site. All results were logged on the QA spreadsheet, a copy of which was sent to Mount Vernon for analysis at regular intervals throughout the trial. To ensure that QA was not excessively burdensome at participating sites, the frequency of DCE-CT radiographer QA phantom scans was set to either weekly (for sites with a high trial patient volume) or before each patient scan (for less frequent trial scans). QA was not carried out within 7 days of trial DCE-CT on 24 occasions.

Across the 14 sites, QA was carried out a total of 753 times. This ranged from 148 instances at one site to just four at another. Some sites carried out radiographer QA more often than the trial required, whereas other sites, with particularly low recruitment, carried out QA a few times only. In total, there were just six (0.8%) instances when the measured iodine calibration factor fell outside ± 5% of the baseline calibration factor measured at the accreditation visit.

There was no marked variation in iodine calibration factor over time for any of the scanners. The maximum difference between baseline calibration factor (measured at accreditation) and mean calibration factor (over the length of the trial) was –2.1%, with the mean variation being 0.2%. There were no systematic trends.

Sites were asked to record any new tubes that were installed in the scanners. Only one new tube was reported during the trial. The baselines and tolerances for QA measurements were reset following installation of the new tube. Although the measured CT numbers in the phantom did change slightly, the enhancement relative to water was not significantly different and the iodine calibration factor did not change. The cause of the tube change was not ascertained; however, the changes in tube output or beam quality that might be expected prior to the failure of an X-ray tube had no effect on the radiographer QA results or iodine calibration factor. There was one other step-change in CT numbers during the trial. No explanation for this could be found. The enhancement relative to water and iodine calibration factor did not change, so the scanner remained in the trial.

Technical quality control of dynamic contrast-enhanced computerised tomography scans

Reconstructed field of view

A significantly larger reconstructed field of view will affect how accurately the region of interest (ROI) is drawn within the nodule and the number of voxels included in the CT number measurement. Four scans had a field of view within a few mm of the 150 mm required in the protocol. This had little effect on the ROI or CT number measurement. These patients were included. Nine scans were reconstructed with a significantly larger field of view (303–426 mm) than specified in the protocol. The nodule diameter for these scans was reviewed against the reconstructed field of view to determine the number of voxels contained within the ROI. If the value was greater than would be measured for an 8-mm nodule with the protocolled 150-mm field of view, the scan data were included for analysis. Five scans were excluded on this basis, as relatively small nodules were scanned on a large field of view. This combination of small nodule and large field of view resulted in fewer voxels sampled for measurement of average CT number, thereby reducing the confidence and accuracy of the CT number measured.

Reconstructed slice thickness and slice interval

One scan was reconstructed with a slice interval of 2.5 mm, one scan with a slice thickness of 2 mm and 18 scans with an interval of 3 mm (protocol 2.5- or 3-mm slices with a 2-mm interval). This could have a small effect on the choice of central slice of the nodule for CT number measurement. The deviations are not significant, and all scans were included for final analysis.

Contrast-enhanced imaging times

Contrast-enhanced CT scans should have been acquired at 60, 120, 180 and 240 seconds post iodine injection. Three patients were scanned at incorrect time points. One patient was scanned with an additional 23-second delay to the 2-, 3- and 4-minute time points. Iodine uptake usually takes place within the first minute; the short delay to the final three acquisitions was thought to have little effect on the measured CT number, so this scan was included. A second patient had the 60-second post-contrast scan carried out correctly, but the remaining three acquisitions were carried out at 174, 209 and 270 seconds. The patient effectively had no 2-minute scan, and the peak CT number may have been missed. This scan was excluded from analysis. A third patient had the 60-second post-contrast scan in the wrong location, not including the nodule. The remaining three acquisitions were carried out at the correct times and location. The peak enhancement value was 29.9 HU, which is greater than the predicted 20 HU threshold for malignancy. This scan was included in the analysis.

X-ray tube current and patient mass

The DCE-CT protocol has three weight-based mA settings: < 60 kg = 200 mA, 60–90 kg = 350 mA and > 90 kg = 500 mA. Seven patients were scanned with the wrong mA value. When a patient received a higher current than intended, the image quality was improved and the scans were included in the trial. When a patient’s weight was slightly over or under a weight boundary, the data were included for analysis. One patient was scanned with automated tube current modulation. The mA value delivered for the central slice of the nodule was compared with the protocol mA value. The patient was included, as the enhancement value was significantly lower than the expected threshold for malignancy; therefore, a lower-dose (higher-noise) scan would not have altered the status of this patient’s data.

Radiation dose

A high-dose DCE-CT strategy was used to keep the image noise low and get more accurate enhancement values. Imaging was undertaken at multiple post-contrast time points. The total effective dose for an average patient for the whole DCE-CT examination (whole-chest scan to locate nodule, then one pre-contrast and four post-contrast nodule scans) is ≈ 30 mSv. This was calculated using the ImPACT CT dose calculator [www.impactscan.org/ (accessed 1 October 2018)].

Reconstruction kernel

Royal Papworth Hospital installed a new Siemens Force CT scanner (Siemens Healthineers AG, Erlangen, Germany) during the trial. The Force CT scanner does not have the same reconstruction kernels as other Siemens scanners used in the trial. During the protocol set-up visit, a variety of reconstruction kernels were reviewed. The kernel with noise statistics most similar to the B30 kernel in the trial protocol was selected.

Dynamic contrast-enhanced computerised tomography injection rate

Four patients had DCE-CT examinations using a contrast injection rate of 3 ml/second, rather than the 2 ml/second prescribed in the protocol. A linear-systems approach (see Report Supplementary Material 1) was used to determine patient-specific correction factors to compensate for this difference in injection rate. The model showed that the faster contrast injection rate could have significantly affected the enhancement values for images acquired at 60 seconds, but the impact on later time points would be minimal. As three of the four cases showed significant nodular enhancement at later time points, it was considered unlikely that their DCE classification as benign or malignant would have been affected by the incorrect contrast injection rate. The remaining cases were excluded from the trial as a result of analysis failure for other reasons.

Window centre and width

In addition to the deviations above, 19 scans were viewed with a window width of 400 HU (protocol is 350 HU). This has no significant impact on how the nodules are visualised; therefore, all scans were included in the final analysis.

Measurement software for contrast enhancement

Site accreditation involved measurement of iodine CT number on CT scanner workstations to determine the iodine calibration of each scanner, and also on reporting workstations or picture archiving and communication systems. The measurement of iodine CT number was tested on both CT workstation and reporting workstation to ensure that no additional processing was applied to the images that might affect the trial.

Phantom scans were compared at eight different sites with three different manufacturers of scanner (seven different scanner models) and four different reporting workstation manufacturers. The radiographer QC phantom contains three inserts of varying iodine contrast concentration, equivalent to approximately 20, 50 and 70 HU (at 100 kV) embedded in a larger volume of water. The CT number, standard deviation and enhancement relative to water were compared for measurements taken on the CT scanner and reporting workstation. Measurements for three slices in the phantom were taken, with care taken to avoid air bubbles.

The maximum deviation in CT number for a given slice was 1.4 HU for the highest iodine concentration. At the clinically relevant threshold of 20 HU, the maximum deviation seen was 1.0 HU. There was no overall positive or negative trend in the values, and the differences seen were within the standard deviation (typically 6–10 HU) of the sample of CT numbers within each ROI.

There was no evidence of a difference between the measured CT number and the enhancement value for measurements taken on the CT scanner or reporting workstations.

Positron emission tomography and dynamic contrast-enhanced computerised tomography interobserver variability: site versus core read

Positron emission tomography

Nuclear medicine specialists or dual-trained radiologists read the PET/CT examinations. For the SUV, the ROI was placed using the threshold technique, as in standard practice.

Originally, 10% (n = 41 to account for potential losses due to missing data) of the total patients undergoing PET were selected for a second read by a core laboratory. Owing to high variability between the recruiting sites, documented results and the core laboratory read results, a full core read of all the PET data sets was performed. The original site reads were performed by the reporting physician at each of the study centres. Image analysis was performed using the onsite software used in routine clinical practice. All core laboratory reads were performed by a single radiologist who was not involved in the primary reads. The core laboratory read was blinded to the original diagnostic CT, the DCE-CT and the original site’s read. A ROI was defined based on the nodule size and location, with alterations made when there was a discordance in the location of the PET uptake relative to the nodule on the CT that could be attributed to motion/respiration. Quantitative assessment of the nodule uptake was performed using the maximum standardised uptake value (SUVmax) and the mean standardised uptake value (SUVmean). The nodule CT and PET characteristics were semiquantitatively graded based on visual inspection, as described in Table 3.

TABLE 3 - Semiquantitative grading scheme for both the PET and CT component of the PET/CT examination combining the anatomical features from the CT and the metabolic features from the PET exam

M, metastasis.

Grade	Significance	PET/CT	Attenuation correction CT
0	No evidence of malignancy	No visible uptake	Round, well-defined lesion with laminated or popcorn calcification
1	Low probability of malignancy	Uptake less than mediastinal blood pool	Inflammatory features, for example air bronchograms, enfolded lung
2	Indeterminate	Uptake comparable to mediastinal blood pool	Smooth well-defined margins, uniform density
3	High probability of malignancy	Uptake greater than mediastinal blood pool	Lobulated, spiculated or irregular margins
4	Very high probability of malignancy	Evidence of distant metastases (i.e. M1 disease)	Evidence of distant metastases (i.e. M1 disease)

A total of 354 cases were included in the reproducibility analysis for the SUVmax, with six excluded because of lack of an identifiable nodule, multiple nodules or missing data. The SUVmax demonstrated a small, but significant, bias between the site read and core laboratory read, and high variability between the two reads with wide limits of agreement (LOAs) (mean difference 0.37, LOA –3.36 to 4.11; paired t-test p < 0.001). Despite this, when considering a threshold for the SUVmax of > 2.5 as being malignant, there was excellent agreement between the site read and core laboratory read [κ (unweighted): 0.93; p < 0.001].

As the SUVmax LOAs were wider than expected, we further reviewed the steps in the scanning and analysis. Several sites were identified to be using PSF reconstruction algorithms, which can result in an increase in SUVmax of up to 25%. Data submitted to the core laboratory were all reconstructed using the standard algorithms, but the reporting sites would have had access to both; therefore, it is possible that the values from the PSF reconstruction would have been recorded. When the 108 cases from those sites were excluded, leaving 247 cases in the analysis, the mean difference was –0.06 (LOAs –2.29 to 2.42; paired t-test p = 0.41), producing significantly improved LOAs and the loss of bias.

Figure 6 contains the Bland–Altman plots for the site and core laboratory reads for the SUVmax, with and without PSF sites included.

FIGURE 6.

Bland–Altman plot comparing the SUVmax obtained at the recruiting site with that obtained at the core laboratory. (a) Across all sites; and (b) excluding those with PSF.

A total of 339 cases were included in the reproducibility analysis for the SUVmean, with 15 cases excluded because of lack of documentation of site read of the SUVmean. There was a small, but significant, bias in the SUVmean between the site read and the core laboratory read, with wider LOAs than those for the SUVmax (SUVmean: mean difference –0.38, LOAs –4.41 to 3.67; paired t-test p < 0.001).

Intraobserver variability was also examined, using two different software packages [Xeleris™ (GE Healthcare, Chicago, IL, USA) and ADW 4.4 (GE Healthcare)], with a minimum of 4 months between reads to minimise the chances of retention between scans. This showed high levels of agreement for the SUVmax (mean difference –0.04, LOAs –0.28 to –0.21; paired t-test p = 0.07), and reasonable agreement for the SUVmean, with higher variability (mean difference 0.34, LOA –1.81 to –2.49; paired t-test p = 0.06). The two outliers in both cases were nodules that lay close to the heart or to the diaphragm, making accurate measurement challenging because of the proximity of the nodule to adjacent structures, and, therefore, prone to variability.

There was excellent agreement between the sites on the visual semiquantitative grading of the PET uptake [κ (with squared weighting): 0.87; p < 0.001]. The greatest variation in scoring occurred around grade 2, which is ‘uptake equivalent to the mediastinum’. Only eight (2.3%) cases were called as equivalent to the mediastinum at the sites, whereas 41 (11.6%) cases were called as equivalent to the mediastinum at core laboratory read. Given that the SUVmax agreement was tight in and around a SUVmax of 2–2.5, which is typically equivalent to the mediastinal blood pool, this suggests that it is very subjective as to when the visual uptake becomes greater, or less, than the mediastinum.

There was fair agreement in the CT visual grading of the nodules [κ (with squared weighting): 0.33; p < 0.001], with disagreement between site read and core read most pronounced for grade 2 lesions. As the original CT was blinded to the core laboratory, but not to the site readers, it may be that subtle spiculation, lobulation or heterogeneity of attenuation not evident on the low-dose attenuation-corrected CT scan may, in fact, have been determined by the standard CT scan, causing this marked upgrade in scoring. Alternatively, it may be that determining whether there is true spiculation or simply background parenchymal lung disease causing contour irregularity is a poorly reproducible finding on CT.

The combined ¹⁸F-FDG-PET/CT assessment was classified as positive for malignancy if one of the following criteria was met:

• grade 4 on ¹⁸F-FDG-PET/CT

• at least grade 3 on both PET and CT appearances

• grade 2 on PET and grade 3 or 4 on CT.

Based on this grading, there was good overall agreement on the presence of malignancy [κ (unweighted): 0.70, p < 0.001).

Based on the above, the SUVmax appears to be the most robust measurement across sites. It has both a lower mean difference and narrower LOAs than the SUVmean. Using a SUVmax cut-off point of 2.5 yielded significantly higher agreement between sites and the core read for malignancy than visual grading of the lesions. CT grading was particularly prone to high observer variability, which affected the diagnosis of malignancy in a large number of cases.

Dynamic contrast-enhanced computerised tomography

Thoracic radiologists or dual-trained radiologists read the DCE-CT. Training was given using a manual and a short video. The ROI was placed using mediastinal settings (window width: 400 HU, window level: 40 HU) in the axial plane with the largest diameter of the nodule. The two-dimensional size was measured in millimetres by taking the longest diameter of the lesion and the perpendicular diameter in the axial plane.

Twenty per cent (n = 66 to account for potential losses due to missing data) of the total DCE-CT scans were selected for a second read. A larger number of DCE-CT scans were chosen to reflect the fact that DCE-CT was not generally used in routine practice, and thus was potentially more prone to variability in the analysis. A training video was prepared to ensure that site readers were analysing the DCE-CT scans using a similar technique, with the technique used in the study as previously published. 45 The selection criteria were weighted to ensure that all centres had at least one scan second read and that the number of scans selected from each centre was representative of the proportion of the total scans performed at that centre. The individual scans from each centre were randomly selected. The site reads were performed using the on-site software used in routine clinical practice. All central reads were performed on a single software platform (syngo.via, Siemens Healthineers AG, Erlangen, Germany) by a single reader. The second reader was blinded to the original diagnostic CT, the PET/CT and the original site read. Quantitative assessment of the nodule enhancement was performed using maximum enhancement.

Maximum enhancement was comparable in 63 of the 66 cases (three cases had missing data). There was no significant difference between the site read and the core laboratory read (mean difference 2.57 HU, LOAs –34.5 to 39.6; paired t-test p = 0.29). The Bland–Altman plot for maximum enhancement is shown in Figure 7.

FIGURE 7.

Bland–Altman plot comparing maximum enhancement obtained at the recruiting site with that obtained at the core laboratory.

Considering a threshold of maximum enhancement of ≥ 20 HU as being malignant, there was good agreement between the site read and the core laboratory read [κ (unweighted): 0.75; p < 0.001].

There was no systematic bias in the values obtained by the DCE-CT central read and the sites in either the maximum enhancement of the nodules or the wash-out. There were, however, wide LOAs between the two measures, suggesting substantial variability in the technique. Further work is required to determine if this is due to differences in analysis software/scanners, or due to variations in the size and precise location of each of the ROIs at each of the time points.

Methods

The study was prospectively enrolled in PROSPERO (CRD42018112215). The study has been reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy Studies (PRISMA-DTA) statement. 46

The condition to be studied was the diagnostic test accuracy of SPNs. The inclusion criterion was studies examining SPNs being worked up for malignancy; studies that included participants aged < 18 years and studies of pure ground glass nodules were excluded. The intervention of interest was DCE-CT. CT scans were included as long as there was a minimum of both a pre-contrast-enhanced and post-contrast-enhanced CT data set for the quantification of the degree of enhancement. The gold standard against which the test was examined was required to be histological diagnosis of malignancy obtained from either needle biopsy or surgical resection, with benign status confirmed either histologically, or with follow-up imaging showing no growth at 2 years or resolution. We considered both prospective and retrospective diagnostic accuracy studies that contained sufficient data to construct contingency tables in order to assess true-positive, false-positive, true-negative and false-negative results.

To identify articles of interest for review, MEDLINE and EMBASE were searched from their inception until October 2018 for published studies on the diagnostic accuracy of DCE-CT in the characterisation of pulmonary nodules. The full search strategy is documented in Appendix 5. Titles and abstracts of studies retrieved using the search strategy, and those from additional sources, were all independently screened by two reviewers (Jonathan R Weir-McCall and Stella Joyce) to identify studies that potentially met the inclusion criterion. The full texts of these potentially eligible studies were retrieved and independently reviewed by the two reviewers to assess eligibility. When there was a disagreement between the reviewers, a consensus was reached through discussion. The references of the retrieved full-text articles were screened for further articles of interest; if any articles were found, these were retrieved if they had not been identified by the original search strategy.

A single reviewer (Jonathan R Weir-McCall) used a standardised, pre-piloted form to extract data from the included studies for assessment of study quality and evidence synthesis. Extracted information included study population, participant demographics and baseline characteristics; details of the CT scanning hardware, scanning technique and diagnostic threshold used; study methodology; nodule size range and eventual diagnosis; diagnostic accuracy metrics; and radiation dose. Two reviewers authors (Jonathan R Weir-McCall and Stella Joyce) independently assessed the risk of bias in the included studies through the use of the second version of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) questionnaire. 47 Discordance in the scoring of bias between the two reviewers was resolved by a third reviewer (Lena-Marie Dendl).

Two deviations occurred from the original pre-registered protocol. A size threshold was not prespecified in the original protocol, yet, during the literature review, it became readily apparent that the upper size limit included in studies varied markedly. Although the Fleischner9 and BTS8 guidelines state that the upper limit of a SPN size is 30 mm, we allowed up to 40 mm for the purpose of this analysis because of the high quality of many of the studies using this threshold, and the granularity it would provide the review. An analysis was performed to compare studies with nodules of > 30 mm with studies with nodules of ≤ 30 mm, as described in Statistical analysis, to determine the effect this might have on the results. The original protocol called for the analysis of SPNs; several studies recruited based on the detection of a SPN, but, if an additional nodule was detected at the time of the index test, they included, analysed and followed up both lesions. Despite not being SPN studies, these were included in the analysis as they reflect routine clinical practice whereby new lesions can frequently be detected on interval studies, or picked up when CT is performed following detection of a nodule on chest radiographs.

Statistical analysis

Numbers of true positives, false positives, true negatives and false negatives were extracted from the studies and used to form 2 × 2 contingency tables, which were used to derive sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR) and diagnostic odds ratio (DOR). Results were pooled using the lme4 package in R (Rstudio, version 1.1.463, RStudio, Inc. Boston, MA, USA; R, The R Foundation for Statistical Computing, Vienna, Austria) to perform a bivariate binomial random-effects meta-analysis. 48 This uses a binary (logit) generalised linear mixed-model fit by maximum likelihood (using a Laplace approximation). Bivariate summary receiver operating characteristic (SROC) curves were constructed using the bivariate random-effects model outputs to populate the SROC plot in Review Manager version 5.3. (RevMan, The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark). To identify potential sources of heterogeneity, we stratified a secondary analysis into subgroups according to characteristics such as sample size, lesion size, risk of bias (low vs. high/indeterminate), diagnostic thresholds and whether or not the diagnostic threshold was prospectively set. These were included as covariates, in turn, in a meta-regression analysis, with analysis of statistical significance between models performed using a likelihood ratio test of nested models. For sample size, the threshold at which to split the data was arbitrarily set at 100, to represent larger samples that were less likely to be prone to bias due to outliers. For mean nodule size, the sample was split at 20 mm, to provide a reasonable split of the data. For maximum nodule size, the data were split based on whether or not the study included nodules of > 30 mm, as the 30-mm diameter is considered by most guidelines as the upper threshold for a lesion to be called a nodule, after which it is considered to be a mass. The effect of publication date was examined by splitting on the median (2008), with studies published in the previous decade considered to be more representative of modern CT technology. In studies reporting the diagnostic accuracy of multiple thresholds, the optimal threshold was used in the primary analysis.

In the secondary analyses examining different thresholds, studies were included in each subgroup analysis if they had reported the threshold of interest. Thresholds with one or two studies reporting the same threshold were not considered for meta-analysis. To test for study publication bias and heterogeneity, a Galbraith plot was created to examine the interaction between the efficient score and variance, with the Harbord test used to test for funnel plot asymmetry. 49 All statistical analyses were performed using RStudio version 1.1.463. Forest plots and SROC curves were generated using RevMan version 5.

Results

Of 3028 potential papers identified by the literature review, 22 met the inclusion criterion. An additional study was located from the references of the included papers, resulting in 23 studies in the final analysis. Figure 8 details the study flow diagram of the studies identified and screened for eligibility, and the reasons for study exclusion.

FIGURE 8.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of the papers identified by the literature search, screened for eligibility and included in the final study. MRI, magnetic resonance imaging; SPECT, single-photon emission computerised tomography; XR, X-ray.

The 23 included studies incorporated data from 2397 patients with 2514 nodules. Out of 2514 nodules, 1389 (55.3%) were malignant. The studies were predominantly retrospective single-centre studies. Appendix 5, Tables 31 and 32, details the study characteristics of each of these studies and the scanning technique, injection protocol and reconstruction algorithm used in each.

The results of the QUADAS-2 bias and applicability assessment are summarised in Figure 9. Appendix 5, Table 33, documents the individual bias scores for the seven domains for all included studies. Bias in patient selection was unclear in a large number of studies [14/23 (61%)] because of a lack of reporting of the sampling of patients for the diagnostic test accuracy evaluation, with many retrospective studies not clearly documenting whether or not consecutive cases were included. Risk of bias in the index test was high in a large number of studies [12/23 (52%)] because of a lack of prespecification of the intended threshold to be used, and, in several studies, multiple techniques of enhancement of quantification were used simultaneously (including, but not limited to, absolute contrast enhancement, relative contrast enhancement, wash-in, wash-out, wash-in and wash-out, and area under the enhancement curve). Bias regarding the reference standard was unclear in the majority of studies [18/23 (78%)], with the blinding of the reference standard to the index test infrequently reported. Flow and timing had a similarly high-rate frequency of uncertain bias [15/23 (65%)], with the delay between the index test and reference standard infrequently reported. Concerns regarding the applicability of the included studies to the review question were low for the majority of the studies (see Figure 9).

FIGURE 9.

The QUADAS-2 scoring summary of the included studies.

The results of the individual studies’ sensitivities and specificities are collated in a forest plot in Appendix 5, Figure 24, with all studies reporting a per-nodule diagnostic accuracy. The pooled analysis of the 23 studies is reported in Appendix 5, Table 34. The pooled sensitivity and specificity were 94.8% (95% CI 91.5% to 96.9%) and 75.5% (95% CI 69.4% to 80.6%), respectively (the SROC plot is presented in Figure 10), with a PLR of 3.86 (95% CI 2.99 to 4.74), a NLR of 0.07 (95% CI 0.03 to 0.10) and a DOR of 56.6 (95% CI 24.2 to 88.9). Only two distinct enhancement thresholds were reported by more than two studies, with the pooled analysis for each of these reported in Appendix 5, Table 34. Of these, a threshold of < 20 HU enhancement for the differentiation of a malignant from a benign nodule had the highest DOR of 142.5 (95% CI –36.4 to 321.3), maintaining a high sensitivity of 98.3% (95% CI 95.1% to 99.4%) and moderate specificity of 71.0% (95% CI 63.1% to 77.8%). However, it should be noted that the CIs were wide for this threshold, as there were too few data points, with far narrower CIs for a 15-HU threshold (see Appendix 5, Table 34).

FIGURE 10.

Bivariate SROC curve of the included studies.

The aqua circles indicate individual studies and the orange circle indicates the summary point. The purple dashed line is the 95% confidence region for the summary operating point, whereas the light-blue dashed line is the 95% prediction region (which is the confidence region for a forecast of the true sensitivity and specificity in any future study).

The subgroup analyses for heterogeneity studies with a low risk of reference standard bias demonstrated a borderline higher sensitivity and lower specificity than studies with an intermediate/high risk of reference standard bias (p = 0.044). No difference was present between subgroups when studies were split based on CT technique; sample size; mean or maximum nodule size; threshold prospectively or retrospectively set; or the presence of patient selection bias, index test bias or flow and timing bias (p > 0.1 for all, full results are described in Appendix 5, Table 35). In particular, there were no significant differences in the pooled sensitivity or specificity between studies that only included nodules of ≤ 30 mm (and therefore met the current definition of SPN) and studies that included larger nodules of up to 40 mm in size (p = 0.07 for between-group differences in sensitivity and specificity).

The Galbraith plot (see Appendix 5, Figure 25) demonstrated multiple studies falling outwith the 95% CIs, consistent with a significant interstudy heterogeneity in findings, but there was not any significant asymmetry in the plot (p = 0.87) to suggest publication bias.

Discussion

This meta-analysis demonstrates a high sensitivity and moderate specificity for DCE-CT for the diagnosis of SPNs, with a pooled sensitivity and specificity of 94.8% and 75.5%, respectively. However, the study quality was indeterminate in a significant number of the studies, with only one multicentre study and a large number of small studies. Although the analysis shows promising results for the technique, the low quality of the included studies must be taken into account; further carefully designed high-quality multicentre studies are required.

The current Fleischner guidelines9 for further investigation and management of indeterminate SPNs call for either PET/CT or biopsy if the nodule is > 8 mm in size; DCE-CT is not mentioned in the diagnostic pathway, despite inclusion of the technique in the 2005 version of the guidelines. 10 The BTS guidelines8 state that DCE-CT should not be used when PET is available, although it is acknowledged that there is little evidence to support this beyond the historical prerogative of PET/CT. A 2018 meta-analysis12 of PET/CT including 20 studies with 1557 participants reported a sensitivity and specificity of 89% and 70%, respectively, and a DOR of 22. These results are similar to the DCE-CT results obtained in this meta-analysis: 23 studies including 2397 participants demonstrated a pooled sensitivity, specificity and DOR of 95%, 76% and 57, respectively. This suggests that DCE-CT could replace PET/CT as an equivalent diagnostic technique. A formal comparison of the two techniques is recommended to confirm this, as there are a limited number of studies directly comparing DCE-CT with PET/CT. The limited number of studies currently precludes the ability to perform a meta-analytic comparison. Ohno et al. 33 compared DCE-CT with both PET/CT and dynamic contrast-enhanced magnetic resonance imaging (MRI) in a single-centre study of 198 patients, and found that DCE-CT outperformed both MRI and PET/CT in specificity and accuracy. This contradicted the results of Yi et al. ,17 who found, in a single-centre study of 119 participants, that PET/CT was more sensitive than DCE-CT, with specificity equal to that of DCE-CT. Thus, further work is required to directly compare these two modalities. Another technique that has a growing body of evidence is that of diffusion-weighted MRI. Whereas PET/CT examines metabolism and DCE-CT measures perfusion, diffusion-weighted MRI quantifies the movement of water within the lesion. A 2019 meta-analysis50 of diffusion-weighted MRI for the diagnosis of indeterminate SPNs has suggested superiority of this technique over PET/CT, with a pooled sensitivity, specificity and DOR of 83%, 91% and 50, respectively, for diffusion-weighted MRI, compared with 78%, 81% and 15, respectively, for PET/CT. 50 Given the differing nature of the three parameters in question, further research is needed, both to compare them and to determine whether the information from perfusion, diffusion and metabolism are complementary or duplicative in improving diagnostic accuracy.

The equivalent sensitivity, specificity and accuracy in this meta-analysis of DCE-CT, compared with previous meta-analyses of PET/CT, provide supportive evidence for consideration of the incorporation of DCE-CT into the diagnostic pathway of pulmonary nodules. CT machines are both more commonly found and more readily accessible in hospital settings than PET/CT equipment. A dynamic contrast examination is very similar to a standard contrast CT procedure, which is commonly undertaken at all hospitals and requires no additional equipment. A PET/CT examination requires the injection of a radioactive substrate, which needs to be delivered reliably to centres undertaking PET examinations. The requirement of such a supply chain can have significant impact on service flexibility and can result in imaging cancellations when there is disruption or delay in the delivery of the radioactive agent. 51 Future studies examining whether or not certain subgroups of pulmonary nodules (such as those of small size), or nodules found in patients with different risk profiles and likelihood of malignancy, may have more to gain from a DCE-CT examination than from a PET/CT procedure are also required. Similarly, a tiered approach using DCE-CT as the first diagnostic test and using the result as a gatekeeper, with PET/CT as the follow-on examination, may allow for a more considered, nuanced approach targeting a more appropriate workup, utilising the strengths of both techniques. Indeed, such an approach has been shown to be a cost-effective approach for the diagnosis of SPNs. 26 Robust direct comparative accuracy studies of DCE-CT and PET/CT in the same population, and cost-effectiveness studies are warranted to test the various diagnostic pathways.

There are several limitations with the current meta-analysis. The quality of the included studies was frequently indeterminable because of a lack of reporting of key metrics. The studies were almost exclusively single centre, and frequently retrospective, both of which are likely to amplify the apparent diagnostic accuracy of the technique. In addition, the dynamic contrast acquisition technique and the metrics for the quantification of the enhancement were heterogeneous throughout the studies. Although these factors did not appear to have an impact on the accuracy of meta-regression, a standardised acquisition and analysis technique should be agreed upon to improve reproducibility, to facilitate comparison between trials and to allow more widespread adoption.

In conclusion, we have found a high diagnostic accuracy of DCE-CT for the diagnosis of pulmonary nodules, although study quality was poor or indeterminate in a large number of cases. These findings support the current investigation into how DCE-CT may complement or augment the current diagnostic pathway of pulmonary nodules.

Methods

The methods for the systematic review were outlined in a research protocol prior to undertaking the review; the review was registered on the PROSPERO database (CRD42019124299). The systematic review aimed to identify economic evaluations of diagnostic imaging techniques for characterising SPNs. The review was not limited to studies that included PET, with or without integrated CT, and DCE-CT in their diagnostic strategies, as the review had broader aims:

• to identify and summarise evidence on the cost-effectiveness of alternative imaging modalities and combinations of imaging modalities for the characterisation of SPNs, with a particular focus on the relevance of such evidence for decision-making in the NHS

• to identify methods used in the cost-effectiveness studies (studies using model-based synthesis, data from observational studies or a combination) and the range of outcomes reported.

Results of systematic review of economic evaluations

The original search identified 382 candidate publications, of which 362 were excluded by screening titles and abstracts. Of the 20 papers retrieved for full screening, 12 did not meet the inclusion criteria (Figure 11) (a list of relevant excluded studies can be found in Appendix 8). Eight studies met all inclusion criteria: Gambhir et al. ,22 Dietlein et al. ,24 Keith et al. ,21 Comber et al. ,26 Gould et al. ,20 Gugiatti et al. ,25 Tsushima and Endo27 and Lejeune et al. 28 The first updated search identified 98 candidate publications, of which 90 were excluded by screening the titles and abstracts. Of the eight papers retrieved for full screening, seven did not meet the inclusion criteria (see Figure 11, and see Appendix 8 for a list of relevant excluded studies). One study met all inclusion criteria: Deppen et al. 55 The second updated search identified 184 candidate publications, with all 184 excluded after screening titles and abstracts. As a result, nine papers were included in the review.

FIGURE 11.

Study selection: original and updated searches.

Centres

A total of 16 centres participated in the SPUtNIk trial: three in Scotland and 13 in England. These 16 sites recruited a total of 380 participants (median per site 21.5 participants, range 2–74 participants). See Appendix 10 for site details.

Screened patients

A total of 2541 patients were screened (Figure 12). The most common reasons for screen failure were the presence of more than one nodule [n = 413 (19%)], patient declining [n = 296 (14%)], nodule outside the size range [n = 306 (14%)] and malignancy within the previous 2 years [n = 264 (12%)].

FIGURE 12.

The SPUtNIk study standards for the reporting of diagnostic accuracy studies (STARD) flow diagram.

Recruited patients

Of the 2541 patients screened, 380 were recruited to the final study.

Participant withdrawal

Fifteen participants withdrew consent between recruitment and the performance of one or both imaging techniques (see Figure 12). Eight of these participants declined both imaging techniques, and seven withdrew consent prior to DCE-CT. Four participants withdrew consent between the imaging techniques and completion of the 2-year follow-up, prior to receiving a confirmed diagnosis. One participant died prior to receiving a diagnosis; their cause of death was not related to the nodule.

Participant follow-up

A total of 312 participants completed follow-up with usable information from both imaging modalities. Of these, 205 underwent histological/pathological sampling and 191 out of 312 (61.2%) of the SPNs were lung cancer (see Table 14).

Numbers analysed

A total of 380 participants were recruited to the study; however, six were found to be ineligible because of either multiple nodules (n = 4) or having a nodule that was actually outside the size range (n = 2). Of the remaining 374 participants, 362 underwent PET/CT and 329 underwent DCE-CT. Of the 12 participants who did not undergo PET/CT, eight declined, two could not undergo PET/CT for technical reasons, one could not undergo PET/CT for a medical reason and one had a nodule that could no longer be detected. Of the 45 participants who did not undergo DCE-CT, 15 declined, 13 had a nodule that could no longer be detected, nine could not undergo DCE-CT for technical reasons and eight could not undergo DCE-CT for medical reasons (see Figure 12).

Of the completed scans, two of the PET/CT images were later found to be unusable because of a resolution recovery issue and a higher-than-allowed blood glucose level. Eleven of the completed DCE-CT scans were later found to be unusable, with the field of view being too large in five of the scans, the nodule being too small to report in five of the scans and a single case of an incorrect contrast injection rate. This resulted in 360 participants having usable PET/CT scan data and 318 having usable DCE-CT scan data. A total of 317 participants had usable scan data for both modalities; 312 of these had a complete nodule status at 2 years.

Baseline data and demographics

Baseline characteristics for all recruited individuals, eligible participants consented to the study and the subset of participants with usable data from both scans and a 2-year outcome status are summarised in Table 7. This includes the approximate location of the identified SPN and a targeted medical history.

The sex balance of the recruited sample was relatively even, with slightly more male participants (53%). The median age was 69 years, with a range of 35–89 years. Of the recruited participants, 299 out of 380 reported a history of smoking, although only 92 (25%) reported themselves as current smokers (77/312 in the main analysis set). Sixty-six out of 380 (57/312 in the main analysis set) participants were never smokers. The location of the identified SPN was relatively evenly spread across the sites, with the right middle lobe occurring less frequently (7%), and the right upper lobe most frequently (29%). The comorbidities reported were thought to be representative for this age group, suggesting that a representative sample was recruited. Fifty-two out of 380 (38/312 in the main analysis set) participants reported having had a previous malignancy > 2 years before entering the study. The 264 individuals screened for the study with a malignancy within the previous 2 years (see Figure 12) were not recruited to the study.

Baseline computerised tomography, fluorodeoxyglucose positron emission tomography–computerised tomography and dynamic contrast-enhanced computerised tomography results

The timings between the imaging procedures and the imaging results are presented in Table 8. Of the 312 participants who successfully completed the 2-year follow-up with usable information for both PET/CT and DCE-CT, 282 (90%) underwent DCE-CT within 2 weeks of PET/CT (median delay 1 day, range 0–32 days). A total of 307 (98%) participants underwent both PET/CT and DCE-CT within 3 weeks of each other. In 154 instances (49%), both the PET/CT and DCE-CT occurred on the same day.

From the baseline CT, most [212 (74%)] of the nodules were classified as ‘grade 3: lobulated, spiculated or irregular margins’, with evidence of consolidation or inflammation in 31 (10%) of the nodules. Lymph node involvement was present in 24 (8%) of the cases and none of the images showed evidence of distant metastases. Presence of metastases on this baseline CT was one of the study exclusion criteria.

Despite being a low-dose examination, the CT grading from the PET/CT was very similar to that of the baseline CT, with the most common grade remaining grade 3 [208 (71%)]. On grading the PET portion of the examination, 161 (52%) had uptake greater than the mediastinal blood pool (grade 3), 17% had no uptake and 10% had a similar uptake to the mediastinal blood pool (grade 2).

The mean of the SUVmax was 4.75 (with a range of 0–35.3) and the mean of the SUVmean was 2.51 (with a range of 0–20.8). There were slightly more missing data for the SUVmean, as not all sites report this as standard. In only 121 cases (39%) was the radiologist able to make a definitive diagnosis from the PET/CT images, with the remainder (61%) classified as indeterminate. When following the study protocol for determining the status of the PET/CT result, a diagnosis of cancer was reached for 161 (52%) cases. It was not possible to reach an indeterminate result using the rules in our protocol. In 40 (13%) cases, there was lymph node involvement, and in four (1%) there was evidence of metastases.

From the DCE-CT images, the mean of the peak enhancement across all slices was 48.6 HU (range 0–179 HU). The most common grade remained grade 3: lobulated, spiculated or irregular margins [223 (74%)]. For 82 cases (27%), the radiologist involved was able to make a definitive diagnosis from the DCE-CT images; an indeterminate diagnosis was given for the remaining 227 cases (73%). When using a peak enhancement threshold of ≥ 20 HU, a diagnosis of cancer was reached for 267 (86%) cases.

The sizes of the SPNs, as measured from the baseline CT, PET/CT and DCE-CT, are presented in Table 9. Furthermore, Table 9 is subdivided by the 2-year outcome status of the SPN (cancer or non-cancer). The mean of the maximum diameter from the baseline CT scan was 15.9 mm, with a range of 8–30 mm. Generally, the sizes from the PET/CT scan are similar to those from the baseline CT scan, with the DCE-CT measurements tending to be smaller. When splitting the results by outcome status, malignant nodules are larger, on average, than the non-cancerous nodules, although there is considerable overlap between the groups.

For a diagnosis of malignancy, histological confirmation was obtained or, if biopsy/resection was not possible, an increase in nodule size with MDT certainty of malignancy.

Prevalence of cancer

At the end of 2 years’ follow-up, 191 (61%) participants had confirmation of lung cancer. Table 10 shows that there is a variation in the prevalence across the recruiting sites, with values ranging from 11 to 80%, although only three sites have a prevalence of < 42%. There was also a difference in the total number of participants recruited across these sites, meaning that some of these estimated percentages will be imprecise. Cancers were confirmed by histological diagnosis, except for participants undergoing SABR, or when biopsy was not considered possible.

Table 11 shows the type of cancers and non-malignant nodules by smoking status. The most common type of cancer was non-small-cell (76%), with adenocarcinoma making up 74% of that subgroup. The squamous cell cancers (30/191) were all found in the current or ex-smokers groups. Benign disease was confirmed by biopsy in 27 cases and by up to 2 years’ follow-up with CT for 94 patients. The single SPN in the ‘other’ category was part of a diagnosis of diffuse neuroendocrine cell hyperplasia.

A total of 178 participants had findings on their PET/CT other than their pulmonary nodule. Of these findings, the majority were pulmonary findings that would have been detected on the initial recruitment CT, such as emphysema, fibrosis or pleural plaques, and were not considered to be significant incidental findings. Seventy-three findings were extrapulmonary; the locations of these are summarised in Table 12. In the head and neck group, 11 participants had abnormal FDG uptake in the thyroid, five had abnormal uptake in a parotid, six had abnormal uptake in the nasopharynx/tonsils and the remaining three had abnormal uptake in the sinuses or soft tissues. Seven indeterminate breast lesions and seven hiatus hernias were detected, and nine with gastro-oesophageal uptake suggestive of gastro-oesophageal reflux disease were detected. The 25 colonic findings were all indeterminate focal hotspots within the colon and rectum, with the sigmoid colon most commonly affected. In two participants, the increased uptake was multifocal, involving two sites. Three indeterminate kidney lesions were identified, with a single obstructive ureteric calculus. Six thoracic/abdominal aortic aneurysms were detected, requiring follow-up. Further incidental findings included two participants with diffuse large- and medium-vessel uptake suggestive of a vasculitis (one with polyarticular uptake consistent with an inflammatory arthritis and the other with skin plaque uptake consistent with psoriasis). It was not possible to follow up all of these abnormalities to determine if these were confirmed pathology or incidental findings, so no further analysis was performed.

Diagnostic accuracy of positron emission tomography–computerised tomography and dynamic contrast-enhanced computerised tomography

The various gradings from both the PET/CT and DCE-CT are cross-tabulated against the outcome status of the nodule at 2 years in Table 13. The outcome status for the non-cancers has been further subdivided into those who received a negative biopsy and those who did not have a biopsy. This shows the performance of the various diagnosis gradings. Table 14 displays the diagnostic accuracy characteristics of the key diagnosis gradings at the pre-defined cut-off points. It displays sensitivity, specificity, NPV, PPV and ODA, with 95% CIs.

When comparisons were made according to nodule sizes of 8–15 mm, just over 15 mm to 20 mm, and just over 20 mm to 30 mm, the sensitivity of PET/CT increased with increasing nodule size, with a corresponding drop in specificity, and an increase in diagnostic accuracy from 75.5% to 83.1% (see Appendix 11) (Figures 13 and 14). However, when doing the same analysis with DCE-CT, there was relatively little change in the sensitivity or specificity with increasing nodule size, although there was a slight improvement in diagnostic accuracy.

FIGURE 13.

Sensitivity, specificity and diagnostic accuracy by nodule size: PET/CT and DCE-CT. Acc, accuracy; Sens, sensitivity; Spec, specificity.

FIGURE 14.

Sensitivity, specificity and diagnostic accuracy by nodule size: PET/CT and SUVmax > 2.5. Acc, accuracy; Sens, sensitivity; Spec, specificity.

The following study objectives are addressed in Table 14:

• Primary objective 1: to determine, with high precision, the diagnostic performances of DCE-CT and PET/CT in the NHS for the characterisation of SPNs.

• Secondary objective 1: to assess, in an NHS setting, the incremental value of incorporating the CT appearances of a SPN into the interpretation of integrated PET/CT examinations.

• Secondary objective 2 (part 1): to assess whether or not combining DCE-CT with PET/CT is more accurate in the characterisation of SPNs than either test used alone or in series.

The imaging technique with the highest sensitivity is DCE-CT (see Table 14), which achieves a value of 96.9% when using an enhancement cut-off point of ≥ 15 HU, and 95.3% when using an enhancement of ≥ 20 HU, whereas PET/CT achieved a sensitivity of only 72.8%. However, this increased sensitivity is associated with low specificity, whereby DCE-CT achieves only 20.7% and 29.8% for enhancement cut-off points of ≥ 15 HU and ≥ 20 HU, respectively, compared with 81.8% for PET/CT.

When assessing the incremental benefit of adding the CT results on top of the PET/CT results (see Table 14), both the PET/CT values based on the PET grading alone and the SUVmax of ≥ 2.5 perform better than the combined PET/CT results in all areas. This suggests that, in this sample, there is no evidence that adding the CT component is worthwhile.

When looking at the combination of DCE-CT with PET/CT (see Table 14), the performance is similar to that of PET/CT, but with a slightly lower sensitivity of 70.2% and a higher specificity of 83.5%. This results in the NPV being slightly lower, at 63.9%, and the PPV being marginally higher, at 87.0%. Considering that this combination is equivalent to using one of the imaging techniques and then using the other only if the first was positive (as to be test positive requires a positive result on both modalities), this potentially opens up a cost-saving approach, as DCE-CT is less costly (see Chapter 7) than PET/CT. The ODA is lower, at 75.3%, for a combined approach than for PET/CT alone (76.3%), but this small drop in performance may be deemed acceptable.

Combination of dynamic contrast-enhanced computerised tomography with positron emission tomography–computerised tomography

Figure 15 shows ROC curves for the prespecified rules and the best-performing combinations following exploratory modelling with logistic regression using the key elements from the imaging scans (including nodule sizes from the scans) to see if a better combination of PET/CT and DCE-CT results is possible. Eight different models are displayed in the ROC curve alongside their AUROCs. Higher AUROCs indicate a better diagnostic performance across the full range of possible cut-off points.

FIGURE 15.

The ROC curves for the diagnostic performance of the prespecified and best-performing exploratory models (N = 300).

Table 15 shows a tabulation of the diagnostic performance for the list of models presented in Figure 15. It also includes a model comparison (compared with PET/CT) and 95% CIs for both the AUROCs and differences in the AUROCs.

Figure 15 and Table 15 show that the best-performing models are the SUVmax for a single variable model and the combination of SUVmax with DCE-CT peak enhancement for a combination of multiple variables. Both of these two diagnostic models produce statistically significantly better AUROCs than the PET/CT protocol that we described earlier (based on the combination of PET/CT and CT elements). For absolute optimum performance, the combination of both SUVmax and DCE-CT peak enhancement performs the best, but the additional performance will be offset by any additional cost and complexity of needing to perform both imaging techniques. However, it does suggest that these scanning technologies can be used in combination to benefit performance.

For ease of comparison, the best cut-off options for all of the models are presented together in Table 16 (including both a 90% minimum sensitivity cut-off point and a balanced sensitivity and specificity cut-off point). From Table 16, it can be seen that changing the SUVmax cut-off point to using a threshold of ≥ 1.8 produces an increased performance of 91.0% for sensitivity, 63.0% for specificity, 81.5% for NPV, 79.7% for PPV and 80.3% for ODA, compared with using the 2.5 cut-off point (see Table 14). The best-performing single cut-off point involves combining the SUVmax with DCE-CT peak enhancement and using a threshold probability of ≥ 0.43. This produces sensitivity of 90.5%, specificity of 68.9%, a NPV of 82.0%, a PPV of 82.2% and ODA of 82.1%. There is very little to choose between the 90% threshold and optimum cut-off point for this model, with both being very close and performing similarly across the spectrum of values; however, the 90% sensitivity cut-off point correctly classifies one additional participant, compared with the optimum cut-off point (82.1% ODA vs. 81.8% ODA). However, for these models to be used in practice, they would require external validation to show that this was not just an overly optimised performance in the data on which the model was generated.

Of the predefined combination models, the SUVmax/SPV outperforms the SUVmax/DCE-CT peak enhancement quite substantially across the full spectrum. The SUVmax/SPV model performs well and reaches close to the ‘best’ model, with a sensitivity of 90.0%, a specificity of 61.0%, a NPV of 79.1%, a PPV of 78.8% and ODA of 78.9% at its optimum threshold. All of the values for this model are, however, worse than for the SUVmax and DCE-CT peak enhancement combination that was produced via a logistic regression model.

Assessing barriers to recruitment and actions taken to increase enrolment

Introduction

The main objectives of this chapter were to use decision-analytic modelling to assess and compare the likely costs and outcomes resulting from the use of DCE-CT, PET/CT, or a combination of DCE-CT and PET/CT as part of the management strategy for patients presenting with SPNs.

This chapter also provides a detailed description of the modelling approach, the methods and the main assumptions underpinning our chosen analysis, namely a cost–consequences analysis (CCA). The chapter continues with a section outlining the results from the main analysis and the sensitivity analyses, which identify the factors deemed to have the greatest impact on the costs and consequences of diagnostic strategies compared. The concluding section summarises the findings of the economic evaluation.

Deciding to allocate resources on the provision of diagnosing malignancy in SPNs means that these resources will be forgone and will not be available for alternative use (either to provide any other method of diagnosis of malignancy in SPNs or in the provision of different health-care interventions). The (economic) cost of this decision is the opportunity to obtain the health benefits had the resources been allocated in a different way, which, in economics, is called the opportunity cost. Economic evaluation provides decision-makers with important information about the opportunity cost of a decision. It involves the analysis of the costs and effects of one course of action, compared with alternative courses of action. 52

In the current context, economic evaluation involves the assessment of costs and different types of benefits associated with alternative tests for identifying malignancy in SPNs. The form of economic evaluation adopted is a CCA. A CCA assesses alternative courses of action in terms of their impact on costs and a number of outcomes (or consequences) measured in natural or clinical units. Its aim is not to aggregate these consequences into a single measure, such as QALYs, but to highlight the trade-offs between costs and different outcomes that are implicit in a choice between the courses of action being compared. With this approach, non-health impacts can be incorporated in the analysis, something that cannot be readily achieved with QALYs and a cost–utility analysis. 64

Methods

Model overview

As seen in the systematic review of economic evaluations (see Chapter 5), there are no existing economic evaluations directly comparing the use of DCE-CT and PET/CT in the diagnosis of malignancy in SPNs. Therefore, a decision-analytic model has been developed to synthesise evidence from the literature and the SPUtNIk trial, and to evaluate the costs and effects of these alternative regimens. The model was designed using the software TreeAge Pro 2018, release 2 (TreeAge Software, Williamstown, MA, USA).

The model is based on a hypothetical cohort of patients aged 69 years, of which 47% are females, presenting with a SPN with a diameter of 8–30 mm. These characteristics were based on the characteristics of the sample recruited to the clinical study (see Table 7). Cost data were expressed in Great British pounds, reflecting prices for the year 2018. Costs were estimated from an NHS and Personal Social Services perspective. The costs and consequences of each strategy were estimated for a 2-year time horizon for the major outcomes, and projections beyond this time horizon were attempted for secondary outcomes (i.e. life expectancy and QALYs). This horizon also reflects the trial’s length of follow-up.

In addition to the primary CCA, two complementary cost-effectiveness analyses were also conducted. The first of these estimated the incremental cost per malignant case treated and the second estimated the incremental cost per correctly managed case. The first analysis puts weight on correctly detecting and managing disease when it is present (i.e. it focuses on true positives). The second analysis considers that there is value in correctly managing those without a malignancy, as well as those with a malignancy (i.e. it focuses on true positives and true negatives).

The model has been built as a probabilistic model. This means that uncertainty around input parameters can be characterised by assigning probability distributions to these parameters to reflect uncertainty around the mean values. Monte Carlo simulation was used to explicitly characterise how uncertainty in the input parameters is translated into uncertainty around the results, which, in turn, were presented with their associated uncertainty (lower and upper estimates).

Model type and rationale for model structure

The decision tree model simulates the management pathway and estimates the expected costs and consequences of a cohort of patients presenting with a SPN (of size 8–30 mm) until they are treated or until a 2-year surveillance period has been completed. The likelihood of particular courses of actions within the model is determined by a set of rules (i.e. probabilities). Different courses of action reflect alternative management pathways, which are explicitly illustrated in the decision tree. Each management pathway is assigned costs and health outcomes that are associated with events experienced by patients as they journey through the pathway. The costs, health outcomes and the likelihood of following each management pathway can be expressed as both point estimates and probability distributions.

In the primary analysis, patients entering the decision tree were assumed to be assessed for SPNs with initial diagnostic imaging using PET/CT, DCE-CT or a combination of DCE-CT and PET/CT. For the third option, individuals undergo PET/CT only if the initial DCE-CT results are indeterminate. Following the results of the imaging test, patients could be managed by immediate resection, SABR, TNB or watchful waiting (by follow-up CT). The choice of management was dependent on the outcome of the tests taken.

The structure of the decision tree was informed by models reviewed in the systematic review of economic evaluations (see Chapter 5) and developed in collaboration with experts in the SPutNIk study. The model structure also reflected the diagnostic and management pathways outlined in the current BTS guidelines for the investigation and management of pulmonary nodules. 8

The decision tree can be considered as having three main components:

1. The risk stratification component. This is the ‘diagnostic component’. It stratifies patients according to the risk of having a malignant tumour, that is into groups of patients with SPNs identified as positive, negative or indeterminate for malignancy (hereafter described as ‘indeterminate’). An indeterminate result could be considered a result for which clinical management decisions are taken in the face of higher uncertainty.

2. The patient management component. The second component involves patient management; it includes the procedures undertaken following an initial test result. The available management options considered after an initial test result are immediate surgical resection, SABR, confirmation by immediate TNB, watch and wait, or combinations of these options.

3. Estimating costs and outcomes. The final component of the decision tree includes the costs and outcomes for each management pathway. Short-term outcomes captured in the model include the proportion of patients managed correctly (i.e. treatment for malignant cases, and no treatment for benign cases), the proportion of malignant nodules not receiving immediate treatment (i.e. ending up in ‘watch-and-wait’ management) and the cost of treatment within the 2-year period. Life expectancy and quality of life are also captured to describe the long-term impact of each screening test strategy.

These three components are described in more detail in the following three subsections. They are followed by a fourth subsection, outlining the key model assumptions.

Risk stratification component

The probability of characterising a SPN as being positive or negative for malignancy or of having an indeterminate result is conditional on the prevalence of malignancy and summary measures of the diagnostic performance of the tests estimated in the SPutNIk study. Calculations for diagnostic test outcomes constitute the first part of the model, and they take the following general forms:

⁽¹⁾ p Positive = [ prevalence × sensitivity ] + [ ( 1 − prevalence ) × ( 1 − specificity ) ] .

⁽²⁾ p Negative = prevalence × ( 1 − sensitivity ) + ( 1 − prevalence ) × specificity .

While Figure 16 shows that an imaging test result is produced in a single node with three possible outcomes (positive, negative and indeterminate), the decision model calculates this in two stages. It does this, first, by calculating the probability of a diagnostic (i.e. determinate) result, then, second, given a determinate result, by calculating the probability of a positive and negative test result, with the prevalence of malignancy updated in each step of these calculations. The estimation of the probability of a diagnostic test result uses a prevalence estimate based on a formula that is derived using the classification of the test results, as set out in Table 17.

TABLE 17 - Classification of results based on test outcome and final nodule status

Test outcome	Final nodule status
	Malignant	Benign
Cancer	a	b
Non-cancer	c	d
Indeterminate	e	f
Total cases	g	h

The prevalence of malignancy is determined by the total number of malignant and benign cases:

⁽³⁾ Prevalence = Malignant Malignant + Benign = g g + h .

The sensitivity and specificity for a determinate test result are based on the diagnostic tests yield in patients with malignant and benign tumours:

⁽⁴⁾ Sensitivity ( diagnostic ) = a + c g .

⁽⁵⁾ Specificity ( diagnostic ) = f h .

The posterior prevalence of malignancy in the population with diagnostic test results is determined by applying the PLR to the pre-test prevalence, expressed as an odds ratio (OR):

⁽⁶⁾ PLR = sensitivity 1 − specificity .

Worked examples of these calculations, using diagnostic outcome data from the SPutNIk study, are reported in Appendix 12.

The probabilities of a positive or a negative test result in the population with diagnostic test results is determined in the same way, except that sensitivity and specificity are estimated using the more typical form based on the number of true-positive (TP), false-negative (FN), true-negative (TN) and false-positive (FP) test results:

⁽⁷⁾ Sensitivity = TP TP + FN = a a + c

⁽⁸⁾ Specificity = TN TN + FP = d b + d .

The same procedures are applied to determine the probabilities of positive, negative or indeterminate results for each diagnostic test (PET/CT, DCE-CT and TNB).

Patient management

Patient management options following a test result were based on the BTS guidelines for the investigation and management of pulmonary nodules. 8 The guidelines recommend management options including surgical excision, SABR, image-guided biopsy or CT surveillance (described as a ‘watch-and-wait’ strategy) depending on the risk of malignancy. These management options have been incorporated into the decision tree. The likelihood that they are offered depends on the initial result of diagnostic imaging using PET/CT or DCE-CT.

The following section provides a narrative description of the decision tree and the schematic representation in Figure 16. For illustrative simplicity, the tree depicted in Figure 16 uses ‘clones’, which simply indicate that identical copies of parts of the management pathway are repeated in a number of places.

FIGURE 16.

Simplified scheme of decision tree, including clones of subtrees.

Positive for malignancy

Patients classified as positive for malignancy, by PET/CT or DCE-CT, are offered immediate surgical excision, ablative radiotherapy or a confirmatory biopsy (those cases classified as positive for malignancy at biopsy are then offered surgical excision or ablative radiotherapy). Surgical excision was also considered to have a diagnostic component based on the histopathology of excised tissue. Patients with a negative finding on histopathology were assumed to have a wedge resection without requiring a lobectomy, in comparison with those with a positive finding on histopathology, who would undergo a full lobectomy.

Patients with a negative biopsy, following a positive diagnostic imaging result, are followed-up with a watch-and-wait strategy. The watch-and-wait strategy involves repeat CT (at 3, 12 and 24 months) to assess tumour growth. Patients in whom tumour growth is indicative of malignancy are offered immediate surgical excision, ablative radiotherapy, no treatment or a confirmatory biopsy. It is assumed in the model that confirmatory biopsy during the watch-and-wait period always yields definitive diagnostic results.

Patients in whom biopsy yielded an indeterminate result, following a positive imaging test result, were offered immediate surgical excision, ablative radiotherapy, watch and wait or a second confirmatory biopsy. As above, people followed up using a watch-and-wait strategy are offered surgical excision, radiotherapy or a diagnostic confirmatory biopsy if tumour growth indicative of malignancy is observed. If a second confirmatory biopsy is offered, patients with positive and negative results are managed as outlined previously. However, if the second biopsy also yields indeterminate results, patients are offered surgical excision, radiotherapy or the watch-and-wait strategy.

Indeterminate

Patients whose tumour is classified as indeterminate are offered surgical excision, confirmatory biopsy or the watch-and-wait strategy. Patients offered a confirmatory biopsy or watch-and-wait strategy are offered the same options as outlined previously for those with a positive test result.

In the imaging test strategy whereby DCE-CT and PET/CT are used in combination, if DCE-CT classifies a tumour as indeterminate, PET/CT is then used as a confirmatory imaging test. In this case, the patient management decisions are based on the PET/CT results.

Negative for malignancy

Patients classified as negative for malignancy are likely to be predominantly entered in a watch-and-wait period. However, the decision tree has been structured to allow for patients with a negative diagnostic imaging result to be offered surgical excision or a confirmatory biopsy also. However, this is expected to occur in the minority of patients only.

Complications and mortality associated with invasive procedures

Mortality and non-fatal complications that may occur during surgical resection, ablative therapy or biopsy are also included in the model. To avoid excessive branching in the decision tree, non-fatal complications are implicitly included by weighting procedure-related costs and health-related utility decrements (expressed in QALYs) by the probability of non-fatal complication. Operative mortality is explicitly included in the model structure by adding a branch for procedure-related mortality after each invasive procedure.

Model outcomes

The current economic evaluation takes the form of a CCA, meaning that multiple outcomes are estimated in the model. In the current model, the focus is on short-term outcomes occurring during the 2-year period following the initial imaging test(s). In addition to these, secondary long-term outcomes, such as life expectancy, are also estimated and presented. A brief summary of these outcomes is presented in Box 1.

BOX 1 - Summary of outcomes included in the decision tree

• Costs associated with the diagnostic imaging and subsequent management pathway.

• Accuracy (malignant tumours correctly treated and benign tumours left untreated).

• Malignant tumours detected and treated.

• Malignant tumours missed and left untreated.

• Malignant tumours entering the watch-and-wait strategy (receiving delayed or no treatment).

• Benign nodules inappropriately/unnecessarily treated.

• Operative deaths for patients with malignant and benign nodules.

• Life expectancy.

• QALYs.

Model outcomes were evaluated at each decision-tree terminal node. Outcomes such as costs and utilities were summed along the path leading to each terminal node, and were dependent on the diagnostic and treatment options available in each pathway. Other outcomes, such as accuracy, correctly treated malignant tumours, and life expectancy, were calculated for each decision node and were dependent on the distribution of malignant and benign cases at these terminal nodes. The prevalence of malignancy at each terminal node following a series of diagnostic tests (PET/CT, DCE-CT, TNB) was determined by the prevalence of malignancy prior to each diagnostic test and by the sensitivity and specificity of each diagnostic test. For any diagnostic test or series of diagnostic tests, the post-test prevalence of malignancy was estimated by applying the appropriate likelihood ratios to the pre-test prevalence, as described in Appendix 12. Therefore, when multiple diagnostic tests were carried out along a particular management pathway (e.g. a TNB following PET/CT), the post-test prevalence of the earlier diagnostic test (e.g. PET/CT) became the pre-test prevalence of the later diagnostic test (e.g. TNB).

Model assumptions

To estimate the cost–consequence of the alternative imaging strategies for patients presenting with SPNs, a number of pragmatic structural assumptions had to be made:

• The process following a positive, negative or indeterminate result is identical for all screening strategies.

• Following a positive screening test result, patients are not allowed to immediately enter the watch-and-wait strategy.

• In the case of patients having two consecutive indeterminate biopsy results, a third biopsy is not an option.

• In the case of those SPNs that are under CT surveillance, biopsy always produces diagnostic results. This was carried out to confine the model’s time horizon to 2 years, by preventing patients from re-entering a new CT surveillance period after an indeterminate biopsy result.

• Solitary pulmonary nodules with a negative or indeterminate imaging test result are not allowed to be treated by ablative radiotherapy without a prior confirmatory biopsy.

• Other than patients treated with ablative radiotherapy following a positive imaging test result, patients are treated with ablative radiotherapy only if they are unfit or unwilling to undergo surgery.

Results

Base case

Table 19 presents the results of the CCA. On average, DCE-CT alone was the least costly strategy, and the difference in costs between PET/CT alone and a combination of DCE-CT and PET/CT was small, on average, and may lack economic significance. However, a combination of DCE-CT and PET/CT was the best alternative strategy when comparing patient outcomes. This strategy correctly identified the highest proportion of patients with malignant disease (46.7%), had the lowest proportion of malignant cases left without treatment (13.71%) and achieved the lowest proportion of inappropriate treatment in patients with a benign nodule (9.0%). These results led to the appropriate management of 84.4% of patients, according to their malignancy status.

TABLE 19 - Costs and consequences results for base-case analysis

SD, standard deviation.

Single outcomes	PET/CT (SD)	DCE-CT (SD)	DCE-CT and PET/CT (SD)
Cost (£)	4013 (206)	3305 (199)	4058 (210)
Accurately managed cases (%)	82 (1.6)	77.8 (2)	84.4 (1.4)
Malignancies treated (%)	44.2 (2.5)	40.1 (2.5)	46.7 (2.4)
QALYs	7.64 (0.25)	7.43 (0.26)	7.76 (0.24)
Life expectancy (years)	10.5 (0.32)	10.22 (0.34)	10.65 (0.31)
Delayed or no treatment (%)	20.31 (1.84)	25.63 (2.15)	17.18 (1.59)
Malignancies missed (%)	16.2 (1.68)	20.49 (2.04)	13.71 (1.43)
Benign cases treated (%)	9.91 (1.25)	9.8 (1.25)	9 (1.23)
Operative deaths (%)	1 (0.05)	0.92 (0.05)	1.05 (0.05)
Operative deaths for benign cases (%)	0.17 (0.02)	0.16 (0.02)	0.15 (0.02)
Operative deaths for malignant cases (%)	0.96 (0.05)	0.87 (0.05)	1.01 (0.05)

In addition, PET/CT, on average, performed better than DCE-CT in reducing the malignant cases receiving late or no treatment. However, when the two tests were used in combination, the proportion of malignant cases having delayed or no treatment was reduced by a further 3.13% (20.31% vs. 17.18%). DCE-CT was associated with the lowest life expectancy, on average (10.22 years), and fewest QALYs (7.43), of all the strategies, as a result of the larger number of patients with malignant nodules left untreated, which reduces a patient’s life expectancy and quality of life. A combination of DCE-CT and PET/CT was associated with the highest life expectancy (10.65) and QALYs gained (7.76) per patient.

Finally, the combination of DCE-CT and PET/CT was associated with the lowest proportion of operative deaths for patients with benign nodules (0.15%), because of the higher overall test specificity. However, the same strategy was associated with an increased proportion of operative deaths in all patients with SPNs, compared with DCE-CT (1.05% vs. 0.92%, respectively), because of the larger number of patients who are offered procedures with associated fatal complications.

Results for the incremental cost per malignant case treated and incremental cost per correctly managed case are presented in Table 20. PET/CT is excluded from Table 20, as it was an extendedly dominated strategy (i.e. a combination of DCE-CT and DCE-CT plus PET/CT strategies would be a more efficient strategy than PET/CT alone). The cost-effectiveness acceptability curves for the incremental cost per malignant case treated and incremental cost per correctly managed case are presented in Figures 17 and 18, respectively.

TABLE 20 - Cost-effectiveness results for base case analysis

Strategy	Cost (£)	Incremental cost (£)	Effectiveness (%)	Incremental effectiveness (%)	ICER
Per malignant case treated
DCE-CT	3305		40.1
DCE-CT and PET/CT	4058	753	46.7	6.61	11,395
Per correctly managed case
DCE-CT	3305		77.8
DCE-CT and PET/CT	4058	753	84.4	6.65	11,323

FIGURE 17.

The cost-effectiveness acceptability curve for cost per correctly treated malignancy.

FIGURE 18.

The cost-effectiveness acceptability curve for cost per correctly managed case.

Figure 17 shows the probability of each strategy being cost-effective for a wide range of maximum WTP-per-malignant-case-treated thresholds. When the WTP ceiling ratio per correctly treated malignancy was < £9000, DCE-CT was always the most preferable strategy. However, when the WTP ceiling ratio for one more correctly treated malignancy increased to > £15,500, the strategy that combines DCE-CT and PET/CT became the most cost-effective, with a probability of 1.

Another analysis (see Figure 18) on the incremental cost per correctly managed case took into account also those benign cases who were correctly provided no treatment. In this analysis, DCE-CT was more likely to be the preferred strategy when the WTP ceiling ratio per correctly managed case was < £11,395. However, when the WTP ceiling ratio per additional correctly managed case was increased beyond £16,000, DCE-CT combined with PET/CT became the strategy most likely to be considered cost-effective.

Sensitivity analyses

Exploratory univariate sensitivity analyses

A univariate deterministic sensitivity analysis was conducted to identify the parameters with the highest impact on costs and on the proportion of correctly managed cases. Since the base-case results showed that PET/CT was an extendedly dominated strategy, the impact of the parameters on the incremental costs and incremental correctly managed cases were presented for DCE-CT and DCE-CT combined with PET/CT.

The results from the univariate sensitivity analysis showed that the main drivers for the cost difference between the DCE-CT and DCE-CT plus PET/CT strategies are the initial prevalence of malignancy, the accuracy of PET/CT and the costs associated with PET/CT and surgical excision (Figure 19). The results also showed that, when the sensitivity of biopsy drops (depicted with a blue bar), the cost difference between the two strategies decrease, substantially. The cost of procedures, such as biopsy, radiotherapy and CT, had a moderate impact on the cost difference between the DCE-CT and DCE-CT plus PET/CT strategies. The cost of complications had no impact on the results. Overall, varying the model parameters over a range of 50% did not make DCE-CT a more costly strategy than DCE-CT plus PET/CT.

FIGURE 19.

Tornado diagram showing impact on incremental costs of changes in key parameters.

The impact of changing the model parameter values by 50% was also estimated on the effectiveness of the diagnostic strategies when defined by the proportion of correctly managed patients (Figure 20). The univariate sensitivity analysis showed that the accuracy of PET/CT and biopsy, the diagnostic yield of DCE-CT in benign cases and the initial prevalence of malignancy had the greatest impact on the incremental effectiveness between DCE-CT and DCE-CT plus PET/CT. Other parameters, such as the proportion of patients with a growing nodule being treated, the proportion of malignant nodules showing growth over the surveillance period and the likelihood of undergoing surgery immediately after the imaging test results, all had a moderate impact on effectiveness. Overall, this sensitivity analysis showed that the sensitivity of PET/CT was the only parameter that, if reduced by 50%, would make DCE-CT the more effective strategy.

FIGURE 20.

Tornado diagram showing the impact on the incremental cost per correctly managed case.

In total, the diagnostic accuracy of PET/CT, the initial prevalence of malignancy, and the diagnostic yield of DCE-CT in benign cases were the main drivers of the results when comparing DCE-CT with DCE-CT plus PET/CT. This was also the case when comparing the effectiveness of these strategies in terms of correctly treated malignancies and QALYs. In addition, when the two strategies were compared in terms of total QALYs, the results were also sensitive to the life expectancy following a lobectomy and the quality of life of benign and treated malignant cases.

Scenario in which indeterminate results are not allowed in the model

For this analysis, costs and consequences of four imaging test alternatives, which are assumed to always yield diagnostic test results, were estimated. In this scenario, PET/CT and DCE-CT are based on the diagnostic accuracy of those tests where indeterminate cases are treated as positive. PET/CT (alt) and DCE-CT (alt) are based on the diagnostic accuracy as derived from a pre-determined set of thresholds.

As Table 21 shows, PET/CT (alt) was, on average, the least costly strategy (£3957), whereas PET/CT was the most costly strategy (£4945). In contrast, PET/CT (alt) was the imaging strategy associated with the least favourable outcomes when considering the number of correctly managed patients, patients with malignant tumours receiving delayed or no treatment, and missed malignancies. When accounting for operative death for benign cases and the proportion of benign cases inappropriately treated, PET/CT (alt) was associated with the most favourable outcomes, because of the high specificity compared with the other diagnostic test strategies.

TABLE 21 - Costs and consequences results of scenario excluding indeterminate results

SD, standard deviation.

Single outcomes	PET/CT (SD)	DCE-CT (SD)	PET/CT (alt) (SD)	DCE-CT (alt) (SD)
Cost (£)	4945 (287)	4469 (287)	3957 (275)	4358 (279)
Accurately managed case (%)	92.8 (0.8)	92.8 (0.8)	85 (1.9)	92 (0.9)
Malignancies treated (%)	55.8 (2.7)	55.8 (2.7)	47.4 (3.1)	54.9 (2.7)
QALYs	8.26 (0.22)	8.26 (0.22)	7.75 (0.26)	8.2 (0.23)
Life expectancy	11.3 (0.29)	11.3 (0.29)	10.63 (0.33)	11.22 (0.3)
Delayed or no treatment (%)	4.65 (0.74)	4.65 (0.72)	15.98 (2.16)	5.95 (0.94)
Malignancies missed (%)	3.95 (0.69)	3.95 (0.68)	13.35 (2)	5 (0.89)
Benign cases inappropriately treated (%)	15.06 (2.16)	15.08 (2.16)	6.36 (1.67)	14.08 (1.97)
Operative deaths (%)	1.29 (0.06)	1.29 (0.06)	1.05 (0.07)	1.26 (0.06)
Operative deaths, benign cases (%)	0.25 (0.03)	0.25 (0.03)	0.11 (0.02)	0.23 (0.02)
Operative deaths, malignant cases (%)	1.2 (0.06)	1.2 (0.06)	1.02 (0.07)	1.18 (0.06)

Overall, PET/CT and DCE-CT yield more favourable results with regards to patient outcomes than the other two strategies. PET/CT and DCE-CT achieve almost identical patient outcomes, with DCE-CT costing, on average, £476 less than PET/CT.

Discussion

The results of the CCA suggest that using DCE-CT alone is likely to reduce the overall costs related to the imaging strategy and management pathway of people presenting with a SPN (of 8–30 mm in diameter). However, current practice (i.e. PET/CT) was associated with better outcomes, including, but not limited to, the accuracy of treatment according to malignancy status, as well as the life expectancy and quality of life of patients with SPNs. Furthermore, the analysis suggests that the combined use of the two imaging techniques is likely to further improve outcomes for a minimal additional cost, compared with current practice. Multiple univariate one-way sensitivity analyses showed that the prevalence of malignancy and the sensitivity of PET/CT are the model parameters with the highest impact on incremental costs and incremental correctly managed cases.

The results of this economic evaluation should be interpreted with caution, as assumptions had to be made about the management of patients. In particular, owing to the limited evidence concerning the management of patients following a DCE-CT or DCE-CT plus PET/CT test result, it was assumed that these patients were managed identically to patients with the equivalent PET/CT test result. That is, the model assumes that a positive test result provides the same information to guide patient management regardless of how it was made and that patients who receive a true-positive result do not differ regardless of how a diagnosis is made. In addition, extrathoracic findings, which could potentially increase the clinical value (as well as the costs) associated with ¹⁸F-FDG-PET/CT, and the costs associated with the treatment of these findings, were not included in the analysis.

The choice of model structure and the limited evidence on utility weights following a patient’s diagnosis and management mean that the estimated QALYs associated with each imaging test strategy should not be considered robust. It is partly for this reason that an incremental cost per QALY has not been estimated (a further reason is that we did not estimate costs beyond 2 years). Hence, further research is required to obtain more evidence for model inputs, particularly for those related to the management following DCE-CT or DCE-CT plus PET/CT, and long-term outcomes such as life expectancy and quality of life.

The structure of the decision-analytic model is a simplified representation of the management of patients with SPNs following the results of an imaging test. The model was used to synthesise evidence from multiple sources of evidence, with the SPUtNIk trial being the primary source of evidence. There is uncertainty surrounding both the specific model inputs and the underlying structure of the model. Hence, important costs and benefits may not be accurately estimated. Nevertheless, considerable efforts were made to ensure that the best available evidence was used and that the model reflects real-life decisions (i.e. inclusion of indeterminate imaging test results) and the management pathway to the greatest extent.

Background

The SPUtNIk study participants were invited to complete a self-administered symptoms questionnaire [Identifying symptoms that Predict Chest And Respiratory Disease (IPCARD-SPN)]82 at the point of recruitment to the SPUtNIk study (see the SPUtNIk study protocol39). The questionnaire (see the SPUtNIk study protocol39) was designed to record a broad range of chest, respiratory and systemic symptoms that might predict lung cancer, and it had demonstrated good content validity, test–retest reliability and completion rates in validation studies. 82 The IPCARD-SPN questionnaire was developed in a population with early-stage lung cancer and includes items designed to elicit symptoms often normalised by lung cancer patients prior to diagnosis. 83

Aims

• To identify symptoms that predict malignant SPNs in a population with indeterminate SPNs.

• To determine whether or not the inclusion of symptoms found to distinguish between malignant and non-malignant nodules increases the diagnostic value of DCE-CT and PET/CT.

Methods

Results

Model accuracy

The area under the curve for model 1 (Figure 21) was 0.708 (95% CI 0.641 to 0.775). The optimum threshold for distinguishing between those with and those without lung cancer for model 1 that provided the best balance between sensitivity and specificity was c = 0.5 [with a specificity of 0.61 (95% CI 0.51 to 0.71), a sensitivity of 0.70 (95% CI 0.61 to 0.78) and a Youden’s index of 0.31 (95% CI 0.12 to 0.49)]. Not restricting the sample to observations for which there were complete data during model development led to models with greater areas under the curve. However, variable selection might then be determined by missing data, rather than strength of association with lung cancer. See Appendix 17, Table 64, for the predicted probabilities for the diagnosis of lung cancer within 12 months of surveillance (model 1).

FIGURE 21.

The ROC curve for model 1.

The symptoms and risk factor variables discarded from model 1 (previous malignancy, nodule size at baseline scan, ‘a niggle, ache or pain that feels like wind or indigestion but not associated with eating’ and ‘more colds or flu in the previous 12 months’) were added, in turn, to model 1 in the larger sample with complete data for the model 1 variables (n ranged from 204 to 224). Independent associations with lung cancer approached, but did not reach, statistical significance (p ≤ 0.05) for three of these four variables (Table 24).

TABLE 24 - Independent associations with lung cancer for variables re-entered into model 1a

Variables were re-entered using a sample with complete data for the final model 1 variables.

Risk factor/symptom	n	OR (95% CI)	p-value
Previous malignancy	224	2.7 (1.13 to 6.45)	0.025
Nodule size at baseline scan	222	1.06 (0.99 to 1.12)	0.055
More colds and flu in the previous 12 months	210	2.25 (0.93 to 5.43)	0.071
A niggle, ache or pain that feels like wind or indigestion but not associated with eating	204	2.08 (0.97 to 4.47)	0.06

Model accuracy

The area under the curve for the model predicting lung cancer at year 2 (Figure 22) was 0.72 (95% CI 0.653 to 0.782). The optimum threshold for distinguishing between those with and those without lung cancer for model 2 was c = 0.5, with a sensitivity of 0.79 (95% CI 0.71 to 0.85), a specificity of 0.51 (95% CI 0.40 to 0.61) and a Youden’s index of 0.29 (95% CI 0.11 to 0.46).

FIGURE 22.

The ROC curve for model 2.

Discussion

The completion of the IPCARD-SPN questionnaire by SPUtNIk study participants provided the opportunity to identify symptoms that might predict early lung cancer in a population under surveillance for SPNs; as far as we are aware, the potential for symptomatic diagnosis has not yet been investigated in this population. The IPCARD-SPN questionnaire was designed to include a broad range of symptoms and lay symptom descriptors that might distinguish between lung cancer and non-malignant diagnoses in populations with differing spectrums of disease and comorbidities. 82 As symptoms that predict lung cancer might be expected to differ between populations with different spectrums of disease,87 the IPCARD-SPN questionnaire does not provide a score, but elicits a range of potential predictors that might then be used to generate population-specific risk scores.

Exploratory analyses identified four symptoms associated with lung cancer diagnosis within the first year of surveillance. When forward stepwise regression analyses were used to select symptoms for a multivariable model, two of these four symptoms, ‘sweats in the previous 3 months, not caused by the menopause’ and ‘unexpected tiredness first experienced in the previous 3 months’, remained in the model [adjusted for age, sex and years smoked (model 1)]. The risk factors ‘nodule size at baseline scan’ and ‘previous malignancy’, although associated with lung cancer in univariate analyses, did not improve model fit, and therefore exited the model. Symptoms associated with lung cancer diagnosis within 2 years of surveillance in single-symptom analyses (adjusted for age and sex) were not independently associated with lung cancer and did not improve model fit once ‘years smoked’ was added to the model. The addition of ‘previous malignancy’ and ‘nodule size at baseline scan’ improved the performance of the model predicting lung cancer diagnosis at year 2 (model 2).

Conclusion

The development cohort provided by the SPUtNIk study has enabled the identification of candidate symptom predictors for the diagnosis of lung cancer within the first 12 months of the surveillance of indeterminate pulmonary nodules; these symptom predictors can now be further investigated in fully powered studies that systematically record respiratory and cardiovascular comorbidities.

Main findings

Strengths and limitations

To our knowledge, this is the largest multicentre prospective trial to date comparing DCE-CT with PET/CT in the diagnosis of indeterminate SPNs in the same patients. We have shown that DCE-CT has higher sensitivity and that ¹⁸F-FDG-PET/CT has higher specificity and better accuracy. Sensitivity for ¹⁸F-FDG-PET/CT is lower for nodules of < 20 mm, whereas DCE-CT is unaffected by nodule size. A major strength of this study is that it was pragmatic, reflecting real-life practice, and the comparison was conducted in the same cohort of patients. Patients were recruited from accredited centres from NHS hospitals across the UK in a prospective manner before diagnosis in a standardised setting, and are representative of the type of patients with an undiagnosed SPN. The average age, smoking status, presence of COPD and historical exposure to asbestos, coal or silica were similar to those seen in patients referred for investigation through outpatient clinics and lung cancer MDT meetings.

From a technical perspective, a broad range of acquisition parameters for the imaging tests were accepted, reflecting current NHS practice, and were similar to those that would be used if this technique was widely adopted. Our study used nodule diameter as the entry criteria, but clinical practice has moved towards volumetric measurement to determine which diagnostic pathway to follow. However, the size criteria we used are still relevant. In this study the radiation dose for DCE-CT was higher, at 30 mSv, than a standard PET/CT dose of 14.1 mSv. The DCE-CT dose was high to reduce noise and improve the sensitivity of the technique. However, the high radiation dose of DCE-CT would be a concern and would need to be reduced before adoption in routine clinical practice.

From the health economics perspective, a major strength is that the modelling was based on an updated systematic review of the current evidence linked to standard NHS costs and applied to results from this study. The exploratory analysis benefits from a large well-curated data set with robust outcome data, from which it has been possible to derive new meaningful cut-off values for different sizes of nodules. However, it is recognised that these findings need to be validated in a separate unrelated data set.

One limitation of the study was that the specificity of 29% obtained by DCE-CT was significantly lower than the 75% that the meta-analysis of the literature would suggest. The frequently poor/indeterminate quality of the included studies in the meta-analysis raises the possibility that significant biases were present, inflating the actual accuracy of the technique. The other possibility is that the large number of centres in the current study included many with relatively limited experience of the technique, with reporting often being undertaken by non-thoracic radiologists, which could, in turn, lead to a more conservative assessment, with overcall, rather than undercall, of the likelihood of malignancy. This is only the second multicentre analysis of DCE-CT, to our knowledge, with the other analysis being performed in centres with almost a decade of experience developing and subsequently reporting the technique. 15,90 Future exploration of techniques to reduce this in-built reader variability and impact of experience are warranted. Standardisation of the technique with automated/semiautomated contouring of the nodules or volumetric analysis of the nodules to account for the effects of breathing changing the relative position of the ROI are both avenues worthy of further research. Another shortcoming was the low recruitment rates and the length of time it took to recruit patients, mainly as a result of governance and regulatory changes in the R&D departments delaying set-up of the additional sites. However, the study remains relevant and the examinations use current techniques. Finally, the reported 61% malignancy rate of nodules in the current study is relatively high. This is representative of solitary nodules of this size detected in clinical practice, and is in keeping with previous meta-analyses of MRI and PET in SPNs;12,50 it is substantially higher than the malignancy rate of screening-detected SPNs, such as in the National Lung Screening Trial (15.0% malignancy in nodules of 10–30 mm)91 and the Dutch–Belgian Lung Cancer Screening (NELSON) trial (15.2% malignancy in nodules of > 10 mm). 92 Previous work has shown the sensitivity of a technique to be relatively robust to disease prevalence and for the specificity to increase with falling prevalence. 93 It can be postulated that the diagnostic accuracy of DCE-CT would be similar, or even further improved, for SPNs found in a screening population. However, this requires further evaluation.

Regardless of these limitations, the study remains relevant and the examinations use current techniques that can be easily applied in everyday clinical practice.

The clinical predictors of unexpected tiredness first experienced in the previous 3 months, and more colds or flu in the previous 12 months were positively associated with lung cancer, which is highly novel and shows some interesting results, but it is emphasised that these are exploratory findings and of little value until demonstrated in a prospective study on a different data set.

Lessons for the future

It is very resource intensive to undertake follow-up over a 2-year period following recruitment. This is particularly hard for the research nurses who are often charged with this task. With the advent of NHSX, all digital NHS data will be available from one legal entity. As information on outcomes and resource use is now collected from electronic patient records, it should be possible to create an electronic CRF that can pull data into the study database. Obviously, this can be carried out for routinely collected data only, but this could go a long way towards reducing trial costs, and the challenges of identifying personnel at each site to extract data from patient records and enter it into another database. Attempting to pilot this approach in this ethics-approved study, in which patients had consented to have their records examined, was not possible because of governance concerns raised from several quarters. Streamlined flow of patient data into clinical trial databases from electronic patient records and central NHS databases would greatly facilitate many clinical trials.

Future research

In recent years, several major lung cancer screening studies have shown that screening high-risk populations with low-dose CT confers an overall mortality benefit. 94,95 The randomised US National Lung Screening Trial of three annual low-dose CT examinations in high-risk smokers showed a 20% reduction in mortality, compared with chest radiography,94 which has been sustained on extended follow-up. 96 The NELSON97 population trial of regular CT screening in people aged 50–74 years who are at high risk of cancer from smoking showed a reduction in mortality over a 10-year period of 26% in men and 39% women, compared with a control arm. In this trial, there was a 0.9% cancer detection rate, with 50% found at stage I. 97 As a result, CT screening programmes are being developed in a number of countries. In the UK, following a successful pilot study in Manchester that specifically targeted screening at current smokers and those of low socioeconomic status, NHS England has recently set up targeted lung health checks at 14 pilot sites in 10 schemes across England to evaluate CT screening in high-risk populations. 5,98 It is hoped that these pilots will, in due course, lead to the roll-out of a national screening programme in England.

With the advent of screening, the number of patients with a SPN requiring further investigation will increase substantially. A previous HTA review6 noted that CT screening is associated with a relatively high false-positive rate and subsequent investigations constitute a significant cost. Furthermore, SPNs are a common finding in whole-body screening CT examinations offered to asymptomatic individuals by independent-sector providers. Typically, the costs of follow-up investigations from these examinations are incurred in the public sector (UK National Screening Committee, appendix 17).

The findings from the SPUtNIk study raise the issue as to how DCE-CT could be used in the context of a screening programme. Given that the prevalence of lung cancer in a high-risk screened population (0.9–3%) is much lower than in the SPUtNIk study, it is recommended that PPV and NPV calculations for both DCE-CT and PET/CT are undertaken with this lower prevalence, especially with smaller lesions. As DCE-CT costs less, it could be tested as part of a two-step screening procedure whereby a SPN is found and confirmed by an algorithm and a DCE-CT examination is undertaken at the same visit as the screening CT. Validation of the new SUV and enhancement thresholds for SPNs according to size would be worthwhile, and combining this with radiomic information from the CT and PET examinations might improve accuracy. Further work is required to reduce the radiation dose of DCE-CT without compromising the sensitivity.

Further work on the model is warranted. The internal validity of the final model was assessed by the bootstrap resampling technique to adjust for overoptimism in the estimation of model performance due to validation in the same data set that was used to develop the model itself. It is important to test the validity of the model in an external, unseen data set of SPNs.

Exploration of an integrated examination with DCE-CT performed with the PET examination would be a very efficient way to combine information from both the techniques at a reduced examination cost. This could be piloted to estimate clinical effectiveness and cost-effectiveness.

In summary, the research recommendations are as follows:

• explore the integration of the DCE-CT component into the PET/CT examination for the characterisation of SPNs

• explore the feasibility of two-stage CT lung screening with DCE-CT at the same visit if a suspicious nodule is found

• undertake analysis of PET/CT and DCE-CT by tumour type, grade and size using different SUVs and enhancement thresholds to improve accuracy

• develop a new protocol for DCE-CT with a lower radiation dose suitable for the newer CT machines.

Ethics committee

The SPUtNIk trial was approved by the South West Research Ethics Committee. It received ethics approval on 30 July 2012.

Sponsor

University Hospital Southampton NHS Foundation Trust sponsored this trial and approved the original protocol and subsequent amendments to the study.

Trial Steering Committee members

• Dr Toby Maher (Consultant Respiratory Physician, Chairperson).

• Dr Simon Padley (Consultant Radiologist).

• Professor Jannet Dunn (Statistician).

• Mrs Lisa Lamond (PPI member).

• Professor Stephen Duffy (Statistician).

Data Monitoring Committee members

• Mr Seth Seegobin (Chairperson).

• Professor Willie Hamilton.

• Professor Vicky Goh.

Collaborators

We would like to thank Jeremy Jones for his contribution to designing the study, developing the health economic model and writing report chapters.

Contributions of authors

Fiona J Gilbert (https://orcid.org/0000-0002-0124-9962) (Professor, Honorary Consultant Radiologist and Co-Chief Investigator) was involved in the design of the study, delivery of the study, interpretation of the results and writing of the report.

Scott Harris (https://orcid.org/0000-0001-5774-1537) (Associate Professor of Medical Statistics) was involved in the design of the study, delivery of the study, statistical analysis, interpretation of the results and writing of the report.

Kenneth A Miles (https://orcid.org/0000-0002-5920-381X) (Honorary Professor, Radiology and Nuclear Medicine) was involved in the design of the study, delivery of the study and interpretation of the results.

Jonathan R Weir-McCall (https://orcid.org/0000-0001-5842-842X) (University Lecturer, Radiology) was involved in the delivery of the study, interpretation of the results and writing of the report.

Nagmi R Qureshi (https://orcid.org/0000-0003-2674-449X) (Consultant Cardiothoracic Radiologist) was involved in the design of the study, delivery of the study and interpretation of the results.

Robert C Rintoul (https://orcid.org/0000-0003-3875-3780) (Reader in Thoracic Oncology) was involved in the design of the study, recruitment of participants, delivery of the study and interpretation of the results.

Sabina Dizdarevic (https://orcid.org/0000-0002-6715-3744) (Principal Lead Consultant in Imaging and Nuclear Medicine) was involved in the design of the study, delivery of the study and interpretation of the results.

Lucy Pike (https://orcid.org/0000-0002-4381-3349) (Clinical Scientist) was involved in the design of the study, and took responsibility for the PET accreditation and quality assurance, and chapter writing.

Donald Sinclair (Medical Physicist) was involved in PET accreditation and quality assurance aspects of the SPUtNIk study, and chapter writing.

Andrew Shah (https://orcid.org/0000-0001-8843-3406) (Head of Radiation Protection) was involved in the design of the study and took responsibility for the DCE-CT accreditation and quality assurance, and chapter writing.

Rosemary Eaton (https://orcid.org/0000-0001-6091-0477) (Clinical Scientist) was involved in the design of the study and took responsibility for the DCE-CT accreditation and quality assurance, and chapter writing.

Andrew Clegg (https://orcid.org/0000-0001-8938-7819) (Professor of Health Services Research), Valerio Benedetto (https://orcid.org/0000-0002-4683-0777) (Research Associate, Health Economics) and James E Hill (Senior Lecturer, Health Economics) were involved in the systematic review of cost-effectiveness, delivery of the study and chapter writing.

Andrew Cook (https://orcid.org/0000-0002-6680-439X) (Associate Director, SCTU) oversaw the study management and was involved in the interpretation of the results.

Dimitrios Tzelis (https://orcid.org/0000-0002-6524-9048) (Health Economist) and Luke Vale (https://orcid.org/0000-0001-8574-8429) (Professor of Health Economics) were involved in the health economics model development and analysis of model results, interpretation of the results and chapter writing.

Lucy Brindle (https://orcid.org/0000-0002-8933-3754) (Associate Professor in Early Diagnosis Research) developed and conducted the analysis for the IPCARD-SPN questionnaire substudy and was responsible for chapter writing.

Jackie Madden (https://orcid.org/0000-0002-4604-731X) (Trials Manager) and Kelly Cozens (https://orcid.org/0000-0001-9592-9100) (Senior Trials Manager) were responsible for study management.

Louisa A Little (https://orcid.org/0000-0002-0083-7279) (Senior Trials Manager) oversaw the study management, and was involved in the interpretation of the results and chapter writing.

Kathrin Eichhorst (https://orcid.org/0000-0002-3666-4189) (Data Co-ordinator) was responsible for data management.

Patricia Moate (Patient Representative) and Chris McClement (Patient Representative) were patient representatives on the Trial Management Group (TMG).

Charles Peebles (https://orcid.org/0000-0003-3369-2173) (Consultant Radiologist), Anindo Banerjee (Consultant in Thoracic Oncology and Tuberculosis), Sai Han (https://orcid.org/0000-0001-5482-9648) (Consultant in Nuclear Medicine), Fat Wui Poon (https://orcid.org/0000-0003-1793-7580) (Consultant Radiologist), Ashley M Groves (https://orcid.org/0000-0003-0358-0795) (Director, Institute of Nuclear Medicine), Lutfi Kurban (Consultant Radiologist), Anthony J Frew (Consultant Radiologist), Matthew E Callister (https://orcid.org/0000-0001-8157-0803) (Consultant in Respiratory Medicine), Philip Crosbie (https://orcid.org/0000-0001-8941-4813) (Clinical Senior Lecturer and Honorary Consultant in Respiratory Medicine), Fergus V Gleeson (https://orcid.org/0000-0002-5121-3917) (Professor of Radiology), Kavitasagary Karunasaagarar (https://orcid.org/0000-0003-3151-4760) (Radiology Consultant) and Osei Kankam (https://orcid.org/0000-0003-3637-6133) (Consultant Respiratory Physician) were involved in the design of the study and were responsible for recruiting participants.

Steve George (Consultant Clinical Epidemiologist and Co-Chief Investigator) was involved in the design of the study, delivery of the study and interpretation of the results.

All authors reviewed the final report.

Publications

Qureshi NR, Rintoul RC, Miles KA, George S, Harris S, Madden J, et al. Accuracy and cost-effectiveness of dynamic contrast-enhanced CT in the characterisation of solitary pulmonary nodules – the SPUtNIk study. BMJ Open Respir Res 2016;3:e000156.

Qureshi NR, Shah A, Eaton RJ, Miles K, Gilbert FJ, Sputnik Investigators. Dynamic contrast enhanced CT in nodule characterization: how we review and report. Cancer Imaging 2016;16:16.

Weir-McCall JR, Joyce S, Clegg A, MacKay J, Baxter G, Dendl LM, et al. Dynamic contrast-enhanced computed tomography for the diagnosis of solitary pulmonary nodules: a systematic review and meta-analysis. Eur Radiol 2020;30:3310–23.

Gilbert FJ, Harris S, Miles KA, Weir-McCall JR, Qureshi NR, Campbell Rintoul R, et al. Comparative accuracy and cost-effectiveness of dynamic contrast-enhanced CT and positron emission tomography in the characterisation of solitary pulmonary nodules [published online ahead of print December 9 2021]. Thorax 2021.

Data-sharing statement

Individual participant data will be made available, including data dictionaries, for approved data-sharing requests. Individual participant data will be shared that underlie the results reported in this article, after de-identification and normalisation of information (text, tables, figures and appendices). The study protocol and statistical analysis plan will also be available. Anonymous data will be available for request, from 3 months after publication of the article, to researchers who provide a completed data-sharing request form that describes a methodologically sound proposal, for the purpose of the approved proposal and, if appropriate, sign a data-sharing agreement. Data will be shared once all parties have signed relevant data-sharing documentation, covering SCTU conditions for sharing, and, if required, an additional data-sharing agreement from the sponsor. All data requests should be submitted to the corresponding author for consideration.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care.

Trial Management Group

Chris McClement was the original PPI member of the trial development team and was fully engaged and involved in the development of the final study protocol. Sadly, before the trial opened to recruitment, Chris McClement passed away. Patricia Moate took over as PPI representative on the TMG and assisted the trial team thereafter, and was an integral part of the TMG.

Patricia Moate was a dedicated PPI member and was involved in most of the TMG meetings prior to becoming ill in 2018. She was fully committed at all times and added valuable comments and points for discussion at all meetings that she attended. Although disabled through her condition, Patricia still managed to attend meetings in person, join conference calls or submit, via e-mail, comments on documents for consideration by the team.

Patricia made the following contributions:

• attended TMG face-to-face meeting when they took place and she was able to do so

• attended TMG conference calls and reviewed all necessary documents that were tabled for discussion at each conference

• reviewed the protocol when applicable and contributed to amendments

• reviewed all patient-facing documentation, providing valuable advice and points for discussion

• commented on patient pathways and gave advice regarding burden on the patient to assist with troubleshooting patient recruitment issues in the earlier, and also the later, phases of the trial.

Some specific contributions:

• Recruitment logs – Patricia was involved in all discussions regarding recruitment issues among sites and provided some valuable input. For example, she raised some concern about a site that was recording ‘panic attack’ as the main reason consented patients had not undergone DCE-CT, whereas other sites had not. This was investigated by the trial team at the time, and it was discovered that differences in language/terminology used between sites was the problem. This led the trial team to ask sites to be more specific in terms of language on recruitment logs to enable us to provide a more accurate CONSORT flow diagram.

• Recruitment difficulties and delays – certain legalities concerning PET and DCE-CT being performed at an alternative trust for one particular site led to a delay of > 1 year in opening to recruitment. The site eventually opened, but, because of these legalities, was soon closed again. Patricia was very concerned that patients were being prevented from taking part in valuable research because R&D departments between collaborating NHS departments could not solve this problem. She advised that we write to the head of the trust to highlight this as an issue. On further discussion within the TMG, it was decided that a letter be sent to the head of the trust and the local CRN.

• Substudy – it was decided that the IPCARD-SPN questionnaires be sent a second time to SPUtNIk patients at 18 months’ study participation. This was widely discussed within the TMG, as it raised questions about who the questionnaires should and should not be sent to and how we would ensure that it was not sent to deceased or ill patients. Patricia contributed to these discussions and helped the team rationalise their decision to re-send the questionnaire and helped in the development of the associated letters of information and patient information documentation.

Trial Steering Committee

Lisa Lamond was recruited to the TSC at its inception. Lisa made the following contributions:

• attended TSC face-to-face meetings when they took place

• attended TSC conference calls and reviewed all necessary documents that were tabled for discussion at each conference

• reviewed the TSC reports and actively contributed to discussions regarding trial progress and any issues tabled for discussion

• attended the final TSC results meeting and reviewed the lay summary for the final report, and will support the team in disseminating the results.

Search strategy

Database: EMBASE.

Date range searched: 1974 to 28 February 2019.

Database: Ovid MEDLINE(R).

Date range searched: 1946 to 28 February 2019.

Date searched: 28 February 2019.

1. Solitary Pulmonary Nodule.mp. [mp=ti, ab, hw, tn, ot, dm, mf, dv, kw, fx, dq, nm, kf, px, rx, an, ui, sy] (5712)

2. SPN.tw. (4399)

3. (pulmonary or lung or chest or pleura).tw. (2,368,476)

4. 2 and 3 (1041)

5. ((solitary or coin or single or discrete or indeterminate) adj3 (lung or pulmonary or “chest wall” or pleura) adj3 (lesion* or lump* or nodule* or lobe or lobes)).tw. (5990)

6. 1 or 4 or 5 (8912)

7. “Computed Tomography”/ (677,626)

8. Tomography, X-Ray Computed/ (390,442)

9. Tomography, Spiral Computed/ (18,211)

10. Contrast Media/ (133,577)

11. (Dynamic adj3 enhance*).tw. (20,256)

12. (contrast adj3 enhance*).tw. (118,030)

13. “DCE-CT”.tw. (423)

14. (DCE and CT).tw. (1154)

15. “perfusion”.tw. (351,014)

16. “computed tomography”.tw. (485,681)

17. 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16 (1,750,697)

18. 6 and 17 (4539)

19. exp accuracy/ (160,921)

20. (diagnos* or sensitiv* or specific* or accura*).tw. (13,447,155)

21. differenti*.tw. (2,568,154)

22. 19 or 20 or 21 (14,882,434)

23. (letter or editorial or comment or historical article).pt. (3,666,677)

24. 22 not 23 (14,742,125)

25. 18 and 24 (3148)

26. diagnosis, computer-assisted/ or image interpretation, computer-assisted/ (100,696)

27. 6 and 24 and 26 (221)

28. 25 or 27 (3211)

29. limit 28 to humans (3028).