MAVARIC - a comparison of automation-assisted and manual cervical screening: a randomised controlled trial

Evidence of the effectiveness of cervical screening

The NHS Cervical Screening Programme (NHSCSP) began a managed programme of call and recall in 1988 and is estimated to save as many as 5000 lives per year in the UK. 1 It has become recognised as one of the world’s leading cervical cancer prevention programmes.

Harnessing new technology to improve service efficiency is a key strategy of the NHSCSP. Desirable advances in cytology include improving sensitivity and specificity, and reducing human workload. The number of tests processed by the screening programme has dropped significantly in recent years owing to service improvement. The roll-out of liquid-based cytology (LBC), completed in 2008, has seen the number of inadequate samples (and the associated repeat testing) drop from 9% in 2004–5 to 2.9% in 2007–8. 2 The implementation of six sentinel sites for human papillomavirus (HPV) triage and test of cure around England has reduced the number of repeat tests taken by triaging women on the basis of their HPV results. Women attending for routine tests who are found to have a low-grade abnormality and a positive HPV result are referred directly to colposcopy without repeat cytology testing, and those who are HPV negative are returned to routine recall without cytological follow-up. 3 National roll-out of HPV triage and ‘test of cure’ would further reduce the amount of cytology, and allow women either to be diagnosed and, if necessary, treated more quickly, or to be returned to routine recall.

Current manual screening practice

Current programme guidelines recommend that all cytology is primary screened; slides reported as negative or inadequate receive a rapid review, and slides that are suspected to be abnormal are reviewed and reported by senior laboratory staff. 4 It is recommended that cytoscreeners do a maximum of 5 hours of microscopy work in a 24-hour period, with a complete break from the microscope at least every 2 hours. 5 With the introduction of LBC, rapid screening is carried out by screening staff performing a rapid review of the whole slide in 90 seconds. Current screening techniques are time-consuming and require a large and committed laboratory workforce. Despite the effectiveness of the screening programme, cytoscreeners have often felt under pressure, particularly when failures receive media attention.

Screening schedule and coverage

Currently, women aged 25–49 years are invited every 3 years, and women aged 50–64 years are invited every 5 years. 2 Of the 3.6 million women aged 25–64 years who were screened in 2008–9, around 6.7% received an abnormal result. 6 In the same period there were 134,000 referrals to colposcopy prompted by an abnormal screening result, 28.9% of which were for results of moderate dyskaryosis or worse,6 the remainder resulting from low-grade cytological abnormalities.

Despite the efficiency of the call–recall system, coverage for the year 2007–8 fell below 80% for the first time, at 78.6%. 7 There has been particular concern in recent years over the fall in attendance in the under-30s, although this trend was bucked during 2009 following the occurrence of cervical cancer in a media celebrity. A total of 3.6 million women aged 25–64 years were screened in 2008–9 compared with 3.2 million in 2007–8 – an increase of 11.9% with an increase in coverage to 78.9% [with a range of 65.8%–85.8% between primary care trusts (PCTs)]. 6 The durability of this increase will not be confirmed until the publication of screening statistics for tests taken in 2009–10.

Future programme considerations

Alongside the question of whether or not to implement automated screening, there are several organisational challenges that face the NHSCSP. In 2007 the Department of Health published the Cancer Reform Strategy. 8 This document recommended that in order to achieve the Government’s target of a 14-day turnaround time (from cervical sample being taken to the result being received by the woman), laboratories and screening offices should be reconfigured to make them larger and more efficient. Some laboratories currently operate as ‘hub and spoke’ with larger central laboratories processing the LBC samples and returning them to the smaller laboratories for screening. Amalgamation of smaller laboratories will see further changes to this service configuration. NHS pathology services as a whole across England are also under review by the Department of Health as part of the NHS Pathology Improvement Programme,9 which may have further implications for the NHSCSP’s laboratory infrastructure.

The HPV vaccination programme will also have an impact on screening once vaccinated girls enter the screening programme. (Girls are vaccinated at ages 12–13 years. This began in September 2008 when a 3-year catch-up campaign began to vaccinate older girls aged 14–17 years.) Screening intervals and follow-up protocols will need to be reviewed once the evidence base regarding screening in a vaccinated population becomes clearer. The importance of following up the screening outcomes of recently vaccinated girls was stressed by the Advisory Committee on Cervical Screening (ACCS) during the review of current screening policy in women aged 20–24 years. 10 Following recommendations from the ACCS, the Department of Health decided against making any changes to current policy regarding screening in women aged 20–24 years. Instead, further education of general practice staff will ensure that symptomatic women aged < 25 years are assessed appropriately. 11

Development of technologies

Two US Food and Drug Administration (FDA)-approved automated machines were developed in the 1990s, the AutoPap^® 300 QC (NeoPath, Redmond, WA, USA) and the PapNet^® (Neuromedical Systems Inc., Suffern, NY, USA), both systems being designed to work with conventional cytology slides. AutoCyte had also developed a machine known as the AutoCyte-Screen which was able to read AutoCyte-Prep slides (now BD SurePath LBC). Despite the initial promise of the technology none of these machines is now available. AutoCyte and NeoPath merged to form TriPath Imaging Inc. (Burlington, NC, USA) and discontinued both the AutoCyte and the AutoPap 300 QC, replacing the systems with the AutoPap Primary Screening System, which is now known as the BD FocalPoint™ GS Imaging System (BD Diagnostics, Franklin Lakes, NJ, USA).

There are currently two commercially available FDA-approved automated screening systems – the BD FocalPoint GS Imaging System and the ThinPrep Imaging System (Hologic™, Bedford, MA, USA). The BD FocalPoint Slide Profiler scans the slides and assigns each one a rank according to the likelihood of there being abnormal cells present. The slides are assigned to quintiles, with quintile 1 containing the highest ranking slides. The machine also categorises slides into one of four of categories: review (comprising quintiles 1–5), no further review (NFR; up to 25% of slides), process review (indicating a technical problem) and quality control review (requiring a full screen). NFR designates the 25% of slides least likely to contain an abnormality which could be reported as negative and archived without human reading. Slides that are flagged for review by the system are examined by screening staff using the BD FocalPoint Guided Screener Workstation (previously known as TriPath Slide Wizard^®). This comprises a standard screening microscope fitted with an electronic stage linked to a desktop computer. The Workstation directs screening staff towards 10 electronically marked fields of view (FOVs) on the slide. If abnormal cells are seen in any of the FOVs the entire slide is screened and appropriate action taken in line with laboratory protocols. The BD FocalPoint Guided Screener (GS) Imaging System has received FDA approval to scan both conventional and BD SurePath LBC slides.

In contrast, the ThinPrep Imaging System is designed to work with ThinPrep LBC slides (stained with the Hologic Imager stain) alone. The ThinPrep Imaging System scans all of the slides and selects 22 FOVs which are presented to screening staff on the review scope. The review scope comprises a Hologic automated screening microscope with a motorised stage to guide screeners to each of the 22 FOVs. If an abnormality is suspected in any of the 22 FOVs then a full screen of the slide is undertaken. Unlike the BD FocalPoint GS Imaging System, the ThinPrep Imaging System does not assign scores to slides and is therefore unable to rank and select slides for archiving without further intervention, or to select slides for quality control (QC) reviewing.

Capability of automated cytology

Two systematic reviews have been published on the potential of automated screening technologies. 14,15 A review commissioned by the Health Technology Assessment (HTA) programme and published in 2005 concluded that there was a need for rigorous, unbiased public sector research into the effectiveness of automated screening technologies. 14 One drawback of this review was that the majority of the papers included relate to the now obsolete PapNet and AutoPap 300 QC systems. An earlier review by the New Zealand HTA programme reached a similar conclusion and recommended large-scale prospective trials to be conducted under normal laboratory conditions with reliable gold standards for diagnostic verification. 15 This review also focused on technologies that are no longer commercially available. As yet there have been no systematic reviews that focus on the two currently available technologies which are under appraisal in the MAVARIC study.

Table 1 summarises previous ‘controlled’ studies, in which there was a general pattern of increased rates of abnormality detection in the automated arm. The studies are, however, characterised by methodological weaknesses including the use of outdated systems, using split samples, the use of manually read conventional (as opposed to liquid-based) cytology, using the same slide set for retrospective comparative readings and not reporting histological outcomes. There has not been a single rigorous prospective randomised comparison of manual and automated reading which has been specifically powered to show superiority or non-inferiority, in terms of detection of any lesion of cervical intraepithelial neoplasia grade II (CIN2) or worse (CIN2+).

Productivity and cost-effectiveness

Automation has productivity implications for staff time reviewing slides in the laboratory with potential for cost savings in staff time. There are also additional costs associated with the automated equipment. The HTA programme’s systematic review concluded that there were productivity gains associated with automation when compared with manual reading with conventional cytology. 14 Studies published since, which have evaluated the cost and productivity implications associated with using the ThinPrep Imaging System and BD FocalPoint GS Imaging system, have suggested that automation results in both increased productivity and increased costs. In all studies the authors found that automation resulted in at least a 50%25,26,29 increase in productivity, with the biggest increase reported being 56%. 32 A study based in Italy which estimated the costs associated with automated screening concluded that similar costs to manual screening could be achieved only if 60,000 samples per year were processed by the AutoPap Primary Screening System (now BD FocalPoint GS Imaging System) with a 30% NFR rate. 33

There is also a lack of rigorously evaluated data relating to the incremental cost-effectiveness of automated screening compared with manual reading. The HTA programme’s systematic review concluded that there were insufficient data to draw any conclusions regarding the cost-effectiveness of automated screening and acknowledged that the papers included in the review did not consider the effect of combining LBC with the technologies. 14

Becton Dickinson FocalPoint Guided Screener Imaging System

Currently, there are two ongoing evaluations in the UK involving the BD FocalPoint GS Imaging System. Cervical Screening Wales began an evaluation in 2006 to assess the utility of the BD FocalPoint GS Imaging System for QC by comparing the 10 FOVs with the current manual QC method. The technology has been used as an additional QC tool. All slides were then manually primary screened. This evaluation has since been extended to include four laboratories across Wales, and was due to be completed by March 2010. A similar evaluation was also undertaken at Derby City Hospital. This study was completed in early November 2009, over 40,000 slides were included. In both studies the slides were sent to Source Bioscience’s (formerly Medical Solutions) laboratory in Nottingham for scanning, with the images being read remotely at the trial sites (Wilma Anderson, Source Bioscience Plc., 2010, personal communication).

ThinPrep Imaging System

The Scottish Government Health Department has commissioned a feasibility study of the ThinPrep Imaging System which began in 2008 and aims to compare 40,000 manually read ThinPrep LBC slides with 40,000 ThinPrep Imaging System read slides. The trial has been running in six laboratories – two laboratories processing and reviewing ThinPrep Imaging System slides plus four remote reviewing laboratories. 36 The analysis of the first two phases of the study showed that the ThinPrep Imaging System performed as well as manual screening. 37 The results of phase 3 of the study involving the Review Scope Plus are described in Chapter 4. There are three further feasibility studies taking place in England: one based in Ashford and a second based in Taunton; a QC evaluation study is also taking place in Northampton General Hospital (Glenn Weatherley, Hologic, 2009, personal communication).

The characteristics of further studies involving the ThinPrep Imaging System that are ongoing worldwide are summarised in Table 2.

Epidemiology of human papillomavirus

It is now universally accepted that HPV infection by so-called ‘high risk’ types is essential for the process of cervical carcinogenesis. 38 There are > 100 different HPV types based on differences in genetic sequences. Of these, > 20 oncogenic types are associated with cervical cancer and, of these, type 16 alone is thought to be responsible for up to two-thirds of all cases. 39 Types 16, 18, 31, 33 and 45 are probably responsible for almost 90% of cervical cancers. 40 HPV including all high-risk types is considered to be responsible for virtually 100% of cervical cancer. 38 There are two crucial implications from this. The first is that prevention of high-risk HPV infection will prevent the chain of events that leads to cervical cancer, which has resulted in the production of prophylactic vaccines based on virus-like particles. 41,42 Beginning in 2008, a prophylactic vaccination programme directed against types 16 and 18 was established across the UK, directed at girls aged 12–13 years with a one-off catch-up programme over 3 years to vaccinate girls aged 14–18 years. The second has been the development of HPV tests which can be used diagnostically. The rationale of these is that women who test HPV-negative are not at risk of cervical neoplasia, and so HPV testing can be used to distinguish HPV-positive women who are at risk from the HPV-negative women who are not.

Current technologies

The first HPV deoxyribonucleic acid (DNA) test to receive FDA approval was the so-called Digene high-risk HPV Hybrid Capture^® 2 (HC2) (Qiagen, Crawley, UK) test in which a cocktail of 13 high-risk types are tested, which can be used with the liquid cytology medium and which does not require the step of polymerase chain reaction (PCR) to amplify the viral DNA. This test has become the current standard by which emerging tests need to be compared with, in terms of sensitivity and specificity. It is the test currently used in the NHSCSP sentinel sites protocol both for triage and for test of cure for cervical intraepithelial neoplasia (CIN)-treated women, and was adopted for the MAVARIC study (see Triage and test of cure). New tests have been developed and others are under development. These tests rely on PCR and most test for DNA, but two test for ribonucleic acid, believed by the manufacturers to achieve greater clinical specificity. Another feature of several new tests is the ability to genotype HPV, with the intention of adding specificity to clinical testing by identifying types such as type 16 which are most strongly associated with high-grade CIN. Some testing kits will combine generic testing for a mixture of high-risk types, with restricted genotyping. Others will rely solely on genotyping. The full potential of HPV testing for cervical screening has yet to be realised.

Triage and test of cure

Triage was employed to achieve maximal detection of underlying CIN2+ in the MAVARIC trial. The use of HPV testing to triage women with low-grade cervical cytology has already been referred to above. Various studies have demonstrated the value of HPV triage in terms of avoiding the need for colposcopy for HPV-negative women as well as increasing the relative sensitivity for detecting CIN2+ compared with repeated cytology. 43–45 These benefits of HPV triage were demonstrated in the NHSCSP pilot study, although it did result in an increase in rates of colposcopy referral. 46 These benefits included immediate colposcopy referral, avoiding failure by women to comply with repeat cytology, and increased rates of CIN2+ suggesting that either CIN was being diagnosed more rapidly or triage was more sensitive than repeat cytology, or indeed an element of both.

Test of cure is a term coined for HPV testing following treatment of CIN. A process of long-term cytological surveillance has evolved which has resulted in 10-year annual cytological follow-up in England for treated women found to have CIN2+. Test of cure using HPV testing exploits its high negative predictive value (NPV), to identify the large majority of women who are HPV negative following treatment (who are therefore at very low risk) and allowing them to be returned to routine recall. An assessment of HPV testing as test of cure in the NHS system was undertaken in a recently published study of 900 treated women. 47 The incidence of cytological abnormality over 2 years among women who were cytology negative and HPV negative at 6 months was sufficiently low to recommend return to routine recall. This would save many thousands of women multiple annual follow-up cytology and this approach has been incorporated into the current sentinel sites protocol. Some samples in MAVARIC underwent HPV test of cure as part of this protocol.

Primary screening

Primary screening using HPV testing is not relevant to the MAVARIC study, which is based on primary screening by cytology. Nonetheless, there is a strong rationale for considering a move to HPV testing in the future based on three considerations:

1. greater sensitivity than cytology

2. the potential for increased screening intervals

3. greater throughput efficiency than cytology.

It should be recognised that the NHSCSP is extremely effective, based as it currently is on cytology. In the future, however, the majority of screened women will have been vaccinated, strengthening the rationale for HPV as the initial test. Published randomised trials indicate that HPV and cytology combined do not increase the overall detection of CIN2+ and CIN grade III (CIN3) or worse (CIN3+) over two successive rounds of screening,48–50 but HPV as a single initial test could be a cost-effective means of screening if suitable strategies can be developed to manage HPV-positive women. Such strategies could combine reflex cytology, HPV genotyping and biomarkers.

Randomisation of technologies

Initial cluster randomisation between technologies was performed at the general practice level (Figure 1) because it was not feasible for both cytology systems to be used within a single practice. The overall aim of this randomisation was to ensure, as far as possible, that sources allocated to the two systems should have similar population numbers and include women with similar underlying risk. Randomisation was stratified by PCT to take account primarily of variation in Townsend Deprivation Score, but also in ethnic minority composition and screening interval. Sources within each PCT were assumed to have closer levels of these risk indicators. Community clinics were included as a separate stratum. There were a total of nine PCT strata; seven consisted of one PCT where the PCT was expected to contribute large numbers and two strata consisted of more than one PCT (grouped by high or low deprivation) for PCTs where only a small number of women were expected to contribute [i.e. contributing fewer general practitioner (GP) practices]. Owing to the variation in population size, the sources were ordered by decreasing size (number of women) within each PCT and block randomisation of four sources at a time ensured that similar numbers from each PCT were allocated to each of the two techniques.

FIGURE 1.

Randomisation flow chart.

The numbers 1 –6 in Table 3 show the various possible allocations of two As and two Bs within a block of four where A coded for ThinPrep LBC and B for BD SurePath LBC.

For each PCT stratum the sources were therefore ordered by decreasing size and a series of random digits were generated, with each digit giving the randomisation for a block of four of the sources. For example, in a series of random digits such as 21234, the first number, 2, allocated the first four sources in the PCT to ABAB, the number 1 the next four to AABB and so on until all the sources in the PCT had been allocated to A (ThinPrep LBC) or B (BD SurePath LBC).

Randomisation of slides

The Cancer Screening Evaluation Unit (CSEU) provided two spreadsheets, one for each system, with unique numbers allocated to either the manual-only arm or the paired comparison arm. In the laboratory a query was set up to run on the CliniSys information technology system to pick up all samples eligible for the study and populate the appropriate randomisation spreadsheet. Laboratory randomisation lists were prepared by the laboratory trial co-ordinator and placed alongside the appropriate slides ready for screening.

Incorporation of trial into the NHS Cervical Screening Programme sentinel sites protocol

In September 2006 the Manchester Cytology Centre agreed to become one of the NHSCSP’s sentinel sites for HPV triage, making reflex HPV testing (triage) of low-grade cytological abnormalities routine for all NHS cervical screening samples received at the laboratory. This removed the need for the option to opt out of HPV testing and the LREC agreed that women need no longer be given the opportunity to opt out of the trial. Randomised practices began working to the sentinel site protocol from mid-2007 after consultation with the local Cervical Screening Steering Groups.

Monitoring

The trial was monitored by the HTA programme in July 2007 and by Central Manchester University Hospitals NHS Foundation Trust R&D Office in March 2009, receiving a satisfactory report on both occasions.

Processing of samples for cytology testing

The cytology samples were received in the Manchester Cytology Centre in either BD SurePath or ThinPrep LBC vials depending on the system to which the surgery had been randomised. On receipt in the cytology laboratory all samples were allocated a unique identifying number. The ThinPrep samples were processed using the ThinPrep 3000 Processor to produce slides with a printed 14-digit number including the unique identifying number which, after staining with ThinPrep Imaging System stain, were ready to be read on the ThinPrep Imaging System. The use of acetic acid to remove blood from heavily blood-stained ThinPrep LBC samples had to be discontinued as this procedure could affect the validity of the HPV result.

The BD SurePath LBC samples were processed using the BD PrepStain™ Slide Processor to produce slides ready to be read by the BD FocalPoint GS Imaging System. Prior to processing the samples on the BD PrepStain Slide Processor a paper label containing a barcode with the unique identifying number was placed onto the appropriate slide.

All slides were left overnight to dry before being placed into the appropriate imaging system. Both systems produced a print-out of the number of samples processed with any errors incurred during processing; however, the print-out from the BD FocalPoint GS Imaging System could be run only after 120 slides had been processed. The print-outs from both systems were passed to the laboratory co-ordinator to check for errors.

Transporting samples for human papillomavirus testing

The vials from the LBC samples showing low-grade abnormalities were collated at the Manchester Cytology Centre for dispatch to the Specialist Virology Centre in Edinburgh. The samples were anonymised prior to sending by removing the woman’s name, date of birth and NHS number. The identifier used for subsequent interaction between Manchester and Edinburgh was the sample number assigned by the Manchester laboratory.

The transfer of samples was performed according to the United Nations’ (UN’s) regulations governing the packaging of diagnostic and infectious samples UN3373 (packing instruction 650). CitySprint (www.citysprint.co.uk) was the designated courier. The samples were sent on Monday to arrive in Edinburgh on Tuesday and the results of the test sent back to the Manchester Cytology Centre within 4 days. An electronic sheet was sent to Edinburgh with the unique identifying number, date and type of sample.

Processing of samples for human papillomavirus testing

In Edinburgh, samples were accorded an internal sample number for HPV testing. A MAVARIC trial sample identification worksheet and laboratory checklist were completed in the laboratory throughout the testing process. Sample information was entered into a password-protected, bespoke Microsoft access (Microsoft Corporation, Redmond, WA, USA) database.

For the Amplicor HPV MWP test, nucleic acids were extracted from a 1-ml aliquot using a Qiagen BioRobot 9604 in conjunction with the QIAamp 96 DNA Swab BioRobot Kit and a protocol validated in Edinburgh for use with ThinPrep LBC medium. 53 Where weekly sample numbers were small (< 22), nucleic acids were extracted manually using the Roche Diagnostics AmpliLute™ Liquid Media Extraction Kit.

For the HC2 test both ThinPrep LBC and BD SurePath LBC samples were processed according to the manufacturer’s instructions. Initial sample preparation involved denaturation with sodium hydroxide rather than nucleic acid extraction. HC2 is a solution hybridisation assay for the qualitative detection of high-risk HPV DNA (types 16/18/31/33/35/39/45/51/52/56/58/59/68) in cervical samples. It uses an oligonucleotide probe cocktail of 13 probes. Hybrids are captured on the wells of a microtitre plate and detected with an amplified chemiluminescent signal. This assay is FDA approved and CE marked.

A positive sample, i.e. indicating the presence of high-risk HPV DNA sequences, was reported, where a relative light unit/cut-off (RLU/CO) measurement was ≥ 3.0. From 19 February 2008, the protocol was changed to report a positive sample with an RLU/CO ratio ≥ 2.0 to be in line with the NHSCSP sentinel sites protocol. Both these cut-off values deviate from the manufacturer’s recommendation of 1.0 RLU/CO, values below which indicate that the HPV DNA levels were below the detection limit of the assay or absent. The reason for the higher cut-off was to achieve additional specificity without significant loss of sensitivity based on data from the ARTISTIC (A Randomised Trial In Screening To Improve Cytology) trial. 54 From 2 March 2009 any remaining HPV testing was performed in the Manchester virology department along with triage samples from the NHSCSP sentinel sites.

Test data were entered into the local database and results returned to the Manchester Cytology Centre electronically as a Microsoft excel (Microsoft Corporation) password-protected file after each batch run.

Machine set-up

Both companies, Hologic and BD Diagnostics, assessed the site prior to installing the imaging machines. Several changes to the layout of the preparation laboratory and the screening room had to be made to accommodate the installation of the machines.

Training

Staff with varying levels of LBC experience were selected to receive automated screening training. Both companies performed the training, further details of which are provided in Appendix 6. Eight medical laboratory assistants (MLAs) were trained in the handling and maintenance of the imaging systems. Eight cytoscreeners and one chief biomedical scientist (BMS) were trained in the use of the automated microscopes and cell morphology recognition. The laboratory trial co-ordinator and two cytopathpologists were trained in the handling and maintenance of the imaging systems, the use of the automated microscopes and cell morphology recognition for both systems.

Staining

The Becton Dickinson SurePath staining parameters were changed slightly for the study (an additional water wash was added to the process to comply with company recommendations). For the ThinPrep LBC slides, the routine laboratory Papanicolaou (Pap) stain had to be changed to the ThinPrep Imaging System formulation and the Hologic staining schedule had to be followed. The ThinPrep Imaging System formulation stains the cells darker than conventional formulations. The initial proposal was to stain only the trial slides; however, it was recognised that this could cause bias by (a) indicating to the screeners which slides were being read by the automated systems and (b) one of the stains being advantageous in terms of detection of abnormalities. It was therefore necessary to stain all ThinPrep LBC slides received in the laboratory with the ThinPrep Imaging System stain to prevent such bias occurring.

ThinPrep Imaging System stain validation process

In order to validate the ThinPrep Imaging System stain, 100 slides stained with the department’s routine Pap stain (of which 25% were abnormal) were screened. A second slide from each of the 100 samples was made and stained using the ThinPrep Imaging System stain. The slides were then processed by the ThinPrep Imaging System and the 22 FOVs were reviewed using the Hologic automated microscopes. The results of the slides read by the ThinPrep Imaging System were compared with the original diagnoses. The reviewers were blinded to the original diagnoses throughout the validation process. Both Hologic and the departmental validators (two cytopathologists and the laboratory trial co-ordinator) classed the ThinPrep Imaging System stain as not significantly different from the routine Pap stain.

All levels of screening staff manually screened 100 ThinPrep Imaging System stained slides to ensure that they had become accustomed to the new staining process. Slides stained with the ThinPrep Imaging System had to pass the Regional Technical External Quality Assurance. Slides were fed into the first available round and achieved an acceptable result on assessment.

Screening of cytology samples

The slides for automated screening were screened using the review scopes; no marks were made on the slides to indicate any abnormal cells, and the results were entered onto the randomisation list. The list and slides were then passed to another screener for rapid review. The list was removed and passed to the laboratory co-ordinator prior to the slides being placed back into the routine screening in numerical order, thus ensuring that the manual screener was blinded to the result of the automated read. Manual screening (in both arms of the trial) was carried out according to routine laboratory protocols, including the practice of marking areas of interest on the slide. In the paired arm the automated reading was undertaken first, followed by the manual read, and the woman’s management was based on whichever reading was the greater in terms of abnormality.

Blinding procedures

One of the principal reasons for the manual reading-only arm was to blind the screener to whether or not slides had received an automated read. The other main reason for the manual-only arm was to provide a comparison with manual reading in the paired arm in order to be able to demonstrate that manual reading in the paired arm was neither superior nor inferior in terms of sensitivity to that in the manual-only arm. Manual screening was performed in the routine laboratory flow of work by a mixture of auto-trained and non-auto-trained cytoscreeners. This created the potential for the same screener to read the slide both manually and on the automated system; however, owing to the large pool of cytoscreeners performing manual screening the chance of this happening was low.

In order to blind the manual screener to knowledge of which slides had been screened using the automated review scopes, no marks were made on the slides during the automated screen. Routinely in the cytology department any abnormal cells found are highlighted by marking the slide above and below the abnormal cells with a coloured marker pen. The Hologic ThinPrep Imaging system utilises a marker pen on the review scope to mark the FOV after the automated screen has been performed, this pen was removed so no marks could be made. The automated screener could add electronic marks as these could be viewed only when using the automated review scopes.

Once the automated read had been performed the result was added to the randomisation sheet, the sheet and the slides were then passed to another screener to perform a rapid screen. The rapid screener then passed the randomisation sheet and slides to the laboratory co-ordinator, the co-ordinator removed the sheets and placed the slides back into the routine screening in numerical order, again helping to blind the manual screeners to which slides had been screened on the automated system.

Review of discordant pairs

Discordant pairs are defined in Table 5. A list of eligible discordant pairs was produced by the CSEU for the cytology laboratory. A review of the discordant pairs with a known clinical outcome of CIN2+ was undertaken to assess whether or not the discrepant results were due to a location error by either of the imaging systems or an interpretation error by the cytoscreener assessing the FOVs. Two cytopathologists and the laboratory trial co-ordinator reviewed the FOVs (blinded to both the automated and manual results) and recorded their findings on a mismatch proforma (see Appendix 7). A random sample of 10 known CIN2+ concordant pairs was added as a control in order to provide blinding as to whether or not slides were from discordant pairs.

When the results of the review had been recorded on the proforma, the results of the initial and final automated and manual reads plus any histology were entered on to the form. A majority view determined the outcome of each discordant pair. The review of the discordant pairs was to determine whether the FOVs were showing the significant cells. The cytological consensus resolved the discordant results and was used to determine whether or not the slide had been interpreted incorrectly on either the automated or the manual reading. In cases where the reviewers agreed with the negative automated reading it was agreed that the machine had not presented any abnormal cells in the FOVs.

Cytology management

All samples were initially reported as per the departmental/NHSCSP protocols for manual reading, but not authorised. The laboratory co-ordinator then recorded the results of the automated screening (which were recorded on separate proformas to blind the manual screening process) onto the laboratory computer system. In the event of a discordant result the samples were taken to peer review meetings for discussion after being reviewed on the automated system by a checker/BMS and a consensus report produced. All results were reported using the British Society for Clinical Cytology 1986 classification (Table 6). 55 Final reports were issued as described in Table 7.

Colposcopy management

The management of abnormal cytology is shown in Figure 2. Colposcopy was undertaken according to national Cervical Screening Programme clinical practice guidelines. Women with high-grade cytology (moderate dyskaryosis or worse) underwent either a targeted biopsy with subsequent treatment for CIN2+ or an immediate ‘see and treat’ loop excision. Women with borderline or mild dyskaryosis were referred for colposcopy if they were HPV positive. If they were HPV negative they were returned to routine recall. In triaged cases, a biopsy was not mandated in the presence of normal satisfactory colposcopy. CIN2+ was treated by excision, usually loop excision, and CIN grade I (CIN1) would usually be managed conservatively. The study biopsy result was the higher grade in the event of both a targeted biopsy and subsequent loop excision. All histology was read with the pathologist unaware of the trial arm or LBC type and was reported using the World Health Organization (WHO) and the International Society of Gynecological Pathologists CIN classification system. The pathologist was aware of the grade of cytology. The definitions applied to the colposcopy and histology outcomes for the analysis are given in Table 8.

FIGURE 2.

Final management cytology protocol. a, After adoption of the Sentinel Sites protocol only women who were on routine recall were eligible for triage. Women on early repeats and follow-up were managed according to national guidelines. b, After adoption of the Sentinel Sites protocol women who had a negative HPV test were returned to routine recall as amendments had been made to the national screening database to permit this management recommendation.

Transferring data

Data were transferred to the CSEU from a number of sources. Cytological and histological data stored in the Manchester Cytology Centre database (CliniSys Labcentre Laboratory Information System, Chertsey, UK) were downloaded to either a plain text file or Microsoft excel spreadsheet. The file was compressed and encrypted to AES 256 standard using winzip version 11 (WinZip Computing, Mansfield, CT, USA). Finally, the encrypted file was sent to the CSEU by secure file transfer protocol (FTP) data transfer. Randomisation data were also sent from Manchester by the same method.

Human papillomavirus results were sent from Edinburgh by secure FTP, but without encryption. Data on exact ranking and quintile for each slide relating to the BD FocalPoint GS Imaging System were stored on hard disk and also backed up on tape. The hard disk was accessed via the internet by BD Diagnostics and archived. The data on tape were also sent to the company by post. From Erembodegem, the unprocessed data files were passed on to CSEU by e-mail, again without encryption. Encryption was thought unnecessary in the latter stages as they did not contain personal identifiers.

Database development

At the CSEU all the data were stored and processed on a secure Microsoft access database. The database was under the control of the investigators and there was no involvement by either BD Diagnostics or Hologic in the conduct of the study or analysis of the results.

Recording cytology and human papillomavirus results

The data received from the cytology laboratory consisted of the manual reading results, the automated reading results and the final management result (MR). The final MR was the result that determined clinical management (routine recall, triage by HPV test or direct colposcopy referral). The results of the manual readings included up to five readings [the first reading, the rapid review for negative and inadequate first readings, the second reading if required, and further readings by a checker or pathologist/advanced BMS practitioner (AP) for samples with positive cytology]. For each reading, the data received included the test cytology result, whether the screen was full or rapid and the cytoscreener classification (cytoscreener, trainee cytoscreener/BMS, checker, BMS, medic or AP). The data related to the automated readings included the results of the first automated (auto) reading and of the auto rapid review if the first auto reading was negative or inadequate. The data also indicated whether the auto result was used to help determine the final result.

The protocol for determining the FMR and FAR is shown in Figure 3.

FIGURE 3.

Pathways based on actual laboratory application of study.

The following definitions are used:

• The first manual result pre (MR1) and post (MR2) rapid review.

• The FMR, defined as the result of the first reading by a medic or AP, or the result of the last reading that led to the report being signed off if the slide was not seen by a medic or AP (usually a negative finding signed off by a checker or screener as part of the manual process).

• The auto result pre (AR1) and post (AR2) rapid review.

• The FAR, defined as the first medic or AP result from a slide considered as abnormal after the first auto reading or abnormal from the rapid review (post checker), or any negative or inadequate result after the rapid review of negative and inadequate samples that was signed off without being seen by a medic or AP. The three possible pathways are shown in Figure 3. Algorithm A was where the automatic read AR1 was negative and the subsequent manual rapid review was negative; the FAR was therefore also negative. Algorithm B occurred where the auto result after confirmation by a checker was positive, but the FMR was also positive and the auto result was therefore not considered further. Under such circumstances it is assumed that in the real-life situation the slide would have proceeded to be seen by a checker and/or medic and the best estimate of the FAR was therefore assumed to be same as the FMR. Algorithm C was where the auto result (after confirmation by a checker) was positive, but the FMR was negative. Under these circumstances the slide was reviewed again and the FAR was that recorded after this further review, which was also the MR as recorded on the Manchester system.

• The MR was based on the manual result and/or the auto result, whichever was worse, and this determined the woman’s management.

In the paired arm, the further reads by a checker or a medic applied to both the manual and auto; hence, the discordant pairs could arise only from a negative or inadequate read or from NFR.

Collecting histology data

Histology results were linked to the cytology results using patient identifiers from the Manchester Cytology Centre database and dates. The histology result was considered to be related to the cytology if the histology date was between 3 weeks and 12 months after the cytology date. In the case of more than one histology result being recorded during that time period, the highest grade abnormality was used. For samples taken at colposcopy clinic visits, the histology result from that visit was used unless superseded by a further result.

Missing data

Data were missing or unobtainable for the reasons given in Table 9.

Estimating the relative sensitivity using CIN2+ as disease positive

The sensitivity of the FAR from Table 10 = (A + C)/(A + B + C + [D]).

The sensitivity of the FMR from Table 10 = (A + B)/(A + B + C + [D]).

Although D is unknown and the absolute sensitivity cannot be calculated, the relative sensitivity can be calculated as R = (A + C)/(A + B).

The 95% confidence interval (CI) is calculated as [R/y,R × y], where

A calculation for the relative sensitivity was undertaken for both the BD FocalPoint GS Imaging System and the ThinPrep Imaging System in the paired arm.

Relative specificity rates of screening by automated and manual reading

The relative specificity was calculated in a similar manner to that for the relative sensitivity using Table 11.

H is unknown – but a very close estimate can be achieved by assuming that D (CIN2+  not detected by either manual or auto) is 0 so that H = N – [E + F + G + A + B + C], where N is the total number of samples.

The calculations of relative sensitivity and specificity were undertaken for both the BD FocalPoint GS Imaging System and the ThinPrep Imaging System separately.

Further analysis for Becton Dickinson FocalPoint Guided Screener Imaging System and ThinPrep Imaging System involving unpaired arms data and specific data for the Becton Dickinson FocalPoint Guided Screener Imaging System

Additional analyses of secondary outcomes based on the BD FocalPoint GS Imaging System and ThinPrep Imaging System have been performed. A further comparison of the two systems was undertaken using the unpaired data and the combined data. Finally, a further analysis of the BD FocalPoint Imaging System was undertaken to determine the performance of the system with regard to the classification of slides for NFR.

Analysis of manual arm data

The detection rates and PPVs were estimated for the manual-only arm for both the BD SurePath and the ThinPrep LBC systems.

Comparison of paired and manual arms combined

Data from both the paired and unpaired arms were also compared for the two automated tests. Owing to potential confounding factors due to different distribution of the source samples between technologies, these comparisons are restricted to routine samples from women aged 25–64 years. Inadequate rates were examined for both LBC systems. These were also calculated after adjustment for age and reason for test.

Analysis of Becton Dickinson FocalPoint Guided Screener Imaging System ‘no further review’ and quintiles

The results of the ranking of samples by the BD FocalPoint GS Imaging System were also compared with the cytology MR.

The ranking categories are:

• NFR – slides have the highest probability of being normal and may be archived by the laboratory as within normal limits. In total 100 × A/total slides are classified as NFR. The BD FocalPoint GS Imaging System classifies up to 25% of slides as NFR.

• Review – slides are divided into quintiles of which quintile 1 slides have the highest probability of cytological abnormality. The proportion of slides for each quintile with a final histology of CIN1+ or CIN2+ was analysed. CIN2+ was the most important outcome as this was regarded as disease positive in this study. The study examined the relative sensitivity using the NFR category, but also using the cut-off of quintiles 5, 4, 3, 2 and 1 respectively.

• Process review – indicates a problem such as stain out of limits or slide not scanned.

• Rerun – occurs if tray is rejected.

A comparison of the colposcopy outcomes from quintiles 1–5 was also undertaken to examine the CIN2+ rate in each quintile.

Introduction

The aim of the economic analysis and organisational assessment was to compare the productivity and cost-effectiveness implications of automated screening technologies with manually read cytology. Automated cytology has a number of implications for the cytology laboratory, and in particular has productivity implications for cytoscreeners due to changes in slide reading practice. A large element of the economic evaluation related to detailed field work in the laboratory to assess the productivity implications of automated cytology versus manual reading and to assess the broader organisational impact of automated cytology. Changes in laboratory productivity and workload have potential implications for the cost of cytology. In addition, changes in cytology referral rates could affect the total cost per woman screened. To assess the comparative cost-effectiveness of the technologies, a mathematical model was used to assess the long-term cost-effectiveness using cost per quality-adjusted life-year (QALY) gained as an outcome.

Specific objectives of the economic analysis and organisational assessment were as follows:

• to assess the productivity and organisational impact

• to measure costs per slide and per woman screened

• to estimate the cost effectiveness of the alternative technologies.

Measuring productivity and organisational impact

A detailed assessment was made of the productivity implications and broader organisational impact of each automated screening system compared with manual screening. Productivity of laboratory staff, including both cytoscreeners and laboratory assistants, was measured using a number of different approaches throughout the trial. Technical differences between the technologies have productivity implications for both the duration of each activity in the screening pathway and the necessity/probability of undertaking different activities.

In addition to the preparation required for reading slides manually, automated cytology requires ‘loading’ and ‘unloading’ of slides onto either the BD FocalPoint GS Imaging System or the ThinPrep Imaging System. The differences between the two automated systems as described earlier (see Introduction) have implications for the time taken to undertake primary screening. In summary, a number of different factors that could potentially affect productivity were measured during the trial:

• staff time to load and unload automated equipment

• average time for primary screening (time and motion)

• average number of slides screened per day (daily record sheet)

• average workload per year

• average total time per slide for reading [including checking/medic (or AP) review]

• other organisational factors potentially influencing productivity.

Loading and unloading time of equipment in the automated arm

In addition to the preparation time for manual reading, automated cytology slides also need to be loaded onto the automated machines and then unloaded. To determine the additional time involved, record sheets were developed to measure staff workload (see Appendix 1). These record sheets were completed by laboratory staff over a series of batch runs, to estimate the additional time involved by staff loading and unloading samples for both of the automated technologies.

Average primary slide reading time (time and motion)

Automated cytology changes the way in which cytoscreeners read slides. Instead of reviewing the whole slide, cytoscreeners review only specific marked fields on the slide. To compare the staff time involved across the technologies, following initial piloting work, a time-and-motion study was designed. Cytoscreeners recorded timings for reading consecutive slides on a paper form (see Appendix 2). Timings were undertaken by each cytoscreener and measured using stop watches at his or her workstation. Initially, timings were recorded after staff had been reading slides for approximately 6 months. A further, much larger time-and-motion survey was conducted near the end of the trial, when staff had been screening with automated cytology for about 3 years. Timings included the time for reviewing the slide. Within the time-and-motion study, administration times were recorded only in the manual arm. These costs were assumed to be the same across each of the LBC systems.

Average number screened per day (daily record sheet)

While the time-and-motion studies give a valuable insight into the average time taken to read individual slides, an important consideration for cytology laboratories is how this might translate to the number of cytology staff required. Cytoscreeners undertake a number of activities during their working day, and primary screen slides for only up to 4 hours (5 hours including rapid reviews). Within the screening period there are also natural breaks between reading individual slides. Hence, in addition to the time-and-motion studies, a questionnaire (see Appendix 3) was devised to record the cytoscreeners’ overall workload and to help estimate the overall implications of automated technologies for productivity in terms of the actual number of slides screened per day. This survey was undertaken after cytoscreeners had been reading automated slides for over 3 years. The questionnaire recorded the number of hours cytoscreeners work on different activities and number of slides processed over a 5- to 6-week period.

Average total reading time per slide

The total time for slide reading is dependent on a number of factors. Firstly, in the automated arm some slides may not be available for automated reading owing to an ARF and therefore have to be read manually. Secondly, with the BD FocalPoint GS Imaging System, up to 25% of slides are classified by the automated equipment as not requiring review by a cytoscreener – NFR. Furthermore, as well as the time per slide for the primary screening detailed above, automated technologies could potentially affect the rates of referral of slides for ‘checking’ or review by pathologists/APs with time-related implications for these staff.

To allow for these factors, average total reading time per slide for was estimated by adjusting the average time duration of different stages of slide reading activities, with the probabilities of ARF, NFR, checking and onwards referral which were obtained from the clinical trial database held at the CSEU. The average time required for primary screening and rapid review was obtained from both the workload and time-and-motion surveys. Time duration for checking and secondary screening was taken from an earlier study in the same laboratory. 49

Average workload per year (daily record sheet)

To assess the overall workload for cytoscreeners per year and the potential implications for the number of cytology staff, we also estimated annual cytoscreener workloads based on the daily record sheet data. Using this weekly information, and assuming that cytoscreeners work 43 weeks a year, the annual workload was estimated for each technology.

Other organisational factors potentially influencing productivity

Throughout the trial a detailed record was made of any other factors that could potentially influence productivity, such as days of machine downtime or other organisational factors. Utilising the BD FocalPoint Slide Profiler as a stand-alone piece of equipment making use of the ‘quality control review’ and ‘no further review’ options combined with manual screening also has potential productivity implications. We evaluated the time implications of utilising the BD FocalPoint Slide Profiler as a stand-alone piece of equipment combined with manual reading. We estimated the time savings associated with the NFR option and for slide reading by quintile.

The QC method used in the paired arm of the trial was the same as manual reading, which is a rapid review on all negative and inadequate samples. We assessed a potential further option when utilising the BD FocalPoint Slide Profiler as a stand-alone device of dropping the rapid review for routine samples where slides are determined as requiring NFR. We also explored the time implications of not primary screening or rapid reviewing the slides in the lowest quintiles which were least likely to have abnormalities.

Staff satisfaction and preferences could also affect productivity. Following an initial focus group discussion with staff, a questionnaire was developed to assess staff satisfaction with using the different types of automated equipment compared with manual reading (see Appendix 4). Staff preferences between the two technologies were also obtained for different aspects of screening. In addition, staff were asked (1) if they found it easier to concentrate using the automated system compared with manual reading; (2) if work was more challenging using the automated reading system; and (3) if work was more monotonous using the automated reading system than with manual reading.

Cytology laboratory costs

Total costs per slide were calculated by combining the cost of preparation and slide reading equipment with the costs of slide reading.

Cost of preparation and slide reading equipment

The unit cost of LBC cytology test preparation and slide reading equipment in the manual arm covered the following: costs of the LBC slide preparation system (BD PrepStain Slide Processor and ThinPrep 3000 Processor), maintenance, LBC consumables, cost of staff processing time and microscope costs. The number of microscopes required was identified via consultation with laboratory staff and the purchase cost was written off over 5 years. Costing of LBC equipment/consumables was based on 5-year contract lease prices and the assumption that equipment would be used at the recommended annual capacity. The contract prices for manual equipment and consumables were provided by the corresponding manufacturers and based on existing contracts between manufacturers and the NHS Purchasing and Supply Agency. To maintain confidentiality over the contract prices, we have presented the costs in combination with the staff costs of slide preparation.

The cost of preparation with the automated technologies included the same preparation costs as outlined above for manual reading, plus the additional cost of equipment, maintenance and staff time associated with the automated technologies. Equipment costs for both automated technologies were indicative, and as with manual equipment were based on 5-year rental contracts. In the cost analysis, we present costs on the basis of a laboratory processing at the maximum capacity per year for each technology. The indicative prices of the BD FocalPoint GS Imaging System were based on rental of one BD FocalPoint GS Imaging system with five BD FocalPoint GS Review Stations plus full maintenance contracts for 5 years. This system uses existing microscopes and was costed as above.

The indicative price of the ThinPrep Imaging System was also obtained from the manufacturer. This price is based on rental of one ThinPrep Imaging System plus three guided screener workstation microscopes over 5 years based on their recommended annual capacity. With both automated technologies it is necessary to load and unload slides onto the automated machines. The additional staff time for loading and unloading slides was estimated using record sheets as outlined in the productivity section. The cost of staff time was valued using the unit cost of staff time described above.

Costs of slide reading

Data from the productivity surveys were used to determine the grades of staff undertaking the different screening tasks: primary screening, rapid review, checking and secondary screening. To attribute salary cost for each activity to the different grades of laboratory staff, the mid-scale point for the corresponding band in the Agenda for Change salary structure was applied. 58 Salary costs included qualifications and NHS employers’ costs (that is, the employer’s national insurance contribution plus 14% of salary for employer’s contribution to superannuation).

Average staff costs per slide were determined by combining the data on the staff cost associated with each screening activity with the probability of each event in the screening pathway. Primary screening costs in the BD FocalPoint GS Imaging System arm were adjusted to incorporate the fact that some slides would not require primary screening because of the NFR option. Where there was an ARF it was assumed that these slides would be read manually. Further analyses were also undertaken to assess the staff costs associated with NFR and by quintile in the BD FocalPoint GS Imaging System arm. These data were used to estimate the costs of utilising the BD FocalPoint Slide Profiler as a stand-alone device combined with manual reading with or without rapid review for slides identified as requiring NFR. We also explored the cost implications of not primary screening or rapid reviewing the slides in the quintiles which were the least likely to contain abnormalities.

Primary care, human papillomavirus testing and colposcopy costs

Average total cost per woman screened included primary care, cytology costs, HPV testing and colposcopy costs. Probabilities of different screening events and related care were combined with appropriate unit costs. The clinical database was used to determine the final result in each arm to model the costs of downstream events. It was assumed that where there was an inadequate screening result these women would have a further sample taken by their GP. Where the final result was borderline or mild, HPV testing costs were included. Colposcopy was costed according to grade of CIN diagnosed at the colposcopy clinic. Unit costs of primary care, HPV testing and colposcopy costs were determined as follows:

• General practice/community clinic unit costs The unit cost for obtaining a cervical sample using the LBC technique included the time for taking the sample by a doctor or nurse, the cost of the materials and the cost of transportation of the vial containing the sample to a cytology laboratory. As both manual and automated cytology involve the same methods for collecting samples there was no reason why automated technology would change the unit costs in primary care, and so these costs were obtained by reviewing and updating earlier studies.

• HPV testing unit costs The cost of HPV testing includes equipment, consumables and staff costs. Costs were based on the HC2 assaying technique. Equipment costs were based on a manual preparation system as used in the Specialist Virology Centre, Edinburgh. The costing of equipment was based on a 5-year lease cost. Indicative prices for leasing HC2 systems were provided by Digene according to a range of assumptions over volume of sales. However, as the prices were provided in confidence, the unit costs are presented inclusive of consumable and staff costs. For costing purposes it was assumed that each assay run would be at full capacity and all the wells would be used. The amount of time spent by technical staff in operating the system was derived from observational field work at the Edinburgh virology laboratory. Staff costs were then estimated based on the mid-point of the BMS pay rate band. 58 Given the distance between the cytology and virology laboratory, transport costs were also estimated.

• Colposcopy and histology unit costs Unit costs of colposcopy were derived from NHS Payment by results average tariffs. 59 The costs of biopsy, histology outcome and related treatment were obtained from a recent large costing study60 which reported costs of treatment by cytology and histology grades for 600 women with first abnormal cervical screening result who had been recruited from six specialised gynaecology/colposcopy clinics in England and Wales.

Estimating cost-effectiveness

Cost-effectiveness was estimated utilising the within-trial results on the cost per case detected. We also estimated the lifetime cost-effectiveness of alternative technologies utilising a mathematical model.

Modelling beyond the study end points

The aim of the analysis was to compare the lifetime effects, costs and cost-effectiveness (using life-years saved as the primary outcome measure) of using LBC alone compared with using automated cytology screening. The evaluation used the final results of the MAVARIC trial, including both the clinical results and the cost data. As there are no long-term follow-up data on cancer outcomes from the trial, a mathematical model was used to estimate lifetime effects, costs and cost-effectiveness. We used a model adapted from previously published models. 61–64 The model was a Markov simulation model with two components, the first dealing with HPV natural history, progression to CIN and cervical cancer, and the second dealing with screening and treatment. The model provides a comprehensive map of the current screening pathways for managing cytology results and treatment following referral to colposcopy.

The simulation follows a cohort of women, with transitions between states occurring annually according to age-dependent probabilities. The model predicted the lifetime costs and effects of alternative strategies from age 10 years to age 84 years (inclusive). For this analysis the model was adapted to run using 6-month rather than 12-month cycles. Full details of the assumptions and parameter sources are given below and in Appendix 12. The analysis was conducted from a health service perspective, excluding any costs or savings that might be incurred by patients or their families, in line with current UK recommendations. Inclusion of any such costs would be unlikely to materially affect the results or conclusions.

Natural history

The probability of transitions between pre-invasive health states (well, HPV only, CIN grades I–III) and invasive cancer states (stages I–IV) and the probability of symptoms in an unscreened population were based on a previous natural history model, updated to reflect a 6-month time frame as shown in Figure 4. 62,63

FIGURE 4.

Health states defined by the natural history model and the potential transitions between states. IC, invasive cancer.

All transition probabilities were calculated for a 6-month time frame (reflecting screening protocol time frames). The model used data from the West Midlands cancer registry on invasive cancer survival and mortality from other causes. 65 The following assumptions were made: (1) all cases of pre-invasive and invasive cervical cancer begin with a HPV infection; (2) annual cervical cancer-specific mortality 6–10 years after diagnosis is assumed to be the same in the fifth year after diagnosis; and (3) women who survive for 10 years after diagnosis and treatment for cancer are assumed to have the same life expectancy as women in the general population.

Overall cytology results by age

The age range of women who provided the cytology samples is shown in Table 15 by quinquennia. Relatively fewer samples from women aged ≥ 50 years were obtained because screening takes place every 5 years in this age range compared with every 3 years between ages 25 and 49 years. In total there were 25,053 samples from women aged 25–34 years, 22,934 from women aged 35–44 years and 21,231 from women aged 45–64 years. There were 3619 (5.0%) slides from women outside the screening age range in England, 3013 from women < 25 years and 606 from women ≥ 65 years.

Overall cytology results

The breakdown of cytology results for the paired arm in terms of the MR is shown in Figure 8. The proportions of abnormal results by grade are very similar to those for England overall (shown in brackets): borderline 3.6% (3.9%), mild dyskaryosis 2.4% (2.3%) and moderate and severe combined 1.22% (1.2%). 7 The proportions of inadequate results were 2.8% for the manual read and 1.8% for the auto read. The majority of inadequate results were considered so by both methods, but all non-concordant results were negative by the other method.

FIGURE 8.

Management results of samples in the paired arm. Qu.Glan., query glandular neoplasia; Qu.Inv., query invasive.

Automated read failures

The rates of ARFs are shown in Table 16. These have been retained in the analysis; it is assumed that had this occurred in service, the slides would have been subjected to full manual reading. Thus the manual results are used for the auto results: specifically AR1 = MR1 and FAR = FMR. ARFs encompass a number of biological and technical reasons (Table 17). The most common is inability to read scanty or thick cell preparations. The large proportion of ARFs experienced with the ThinPrep Imaging System during 2006 was due to problems with the review scope, which were resolved by Hologic. There was a similar problem to a lesser extent with the BD FocalPoint GS Imaging System. Overall, the ARF rates for the BD FocalPoint GS Imaging System and the ThinPrep Imaging System were 3.11% and 3.99% respectively. When the data are restricted to July 2007 onwards (after the initial problems had been rectified) the average proportion of ARFs was 2.93% for both the BD FocalPoint GS Imaging System (585/19,950) and for the ThinPrep Imaging System (655/22,390).

Cytology results (paired arm)

Table 18 shows the distribution by grade of the cytology samples according to the read. For automated reading it is clear that the process of AR1 to AR2 and FAR does not affect high-grade rates, but did produce a drop in borderline/mild dyskaryosis from 5.3% (AR1) through 6% (AR2) to 4.2% (FAR). When the FAR for BD SurePath was compared with ThinPrep there was a noticeable difference between borderline/mild dyskaryosis combined: 3.92% (BD SurePath) and 4.5% (ThinPrep). Other cytology grades were similar, but there was a slightly higher inadequate rate for ThinPrep (1.94%) than for BD SurePath (1.70%).

When a similar comparison is made for manual reading in the paired arm, the same pattern is seen, with borderline/mild dyskaryosis rates going from 7.5% in the MR1 through 8.1% in MR2 to 5.5% in FMR. This borderline/mild rate is higher than that for auto (5.5% vs 4.2%). When BD SurePath and ThinPrep are compared for the paired manual read there is very little difference across all grades including borderline/mild dyskaryosis (5.5% vs 5.49%), but again there is a slightly higher inadequate rate for ThinPrep (2.98%) than for BD SurePath. The proportion of inadequate results with automated reading is significantly lower than that with manual reading (FAR 1.82%, FMR 2.83%, p < 0.001).

Cytology results (manual-only arm)

Table 19 shows the comparative rates of cytological abnormality between manual reads 1 and 2, final manual read and the MR which in the manual-only arm is identical to the final manual read because there is no automated read to alter the management. The same effect of checking is seen, reducing the combined borderline/mild dyskaryosis rate from 8.2% to 6.0%, but there was a slight increase in the combined moderate/severe dyskaryosis rate from 1.2% to 1.4%. This latter rate is slightly higher than the corresponding MR for the paired-reading arm of 1.3% (see Table 18). A more detailed comparison of final manual reading between the manual-only and paired-reading arms is shown in Tables 22–26.

Comparison of readings: manual and automated

It is relevant to compare the results of readings at various stages in the process not only between manual and auto, but also between manual readings at different stages and similarly for auto. The key comparison is between the final manual reading and the final auto reading, which generated the difference in relative sensitivity. We also present manual results between the arms, first and final readings and final and MRs. These data are shown in Tables 20–26.

Overall human papillomavirus triage results

Table 30 provides details of the HPV-positive rates (according to the LBC platform used) and grade of cytology for the MR in the manual-only arm (equivalent to the FMR) and for the FMR, FAR and MR in the paired arm. Overall there were no major differences in HPV-positive rates between LBC platforms and between arms for corresponding cytology grades. The significance of the HPV-positive rates for the MAVARIC trial outcome is that for borderline and mild dyskaryosis, these represent the rate of referral for colposcopy among triaged women and therefore possible detection of CIN2+. The cut-off for reporting a sample as positive was changed from 3.0 to 2.0 RLU/CO in February 2008; however, only 1% of samples tested had RLU/CO values between 2.0 and 3.0, so the impact on the proportion referred was minimal.

Overall, it can be seen that for the MR of borderline/mild in the manual-only arm, 66% of samples were HPV positive for both BD SurePath and ThinPrep. Within the paired arm the corresponding figure was 63%. The total proportion of samples with an MR of low-grade cytology (borderline and mild dyskaryosis, including those with unknown HPV status) is almost identical between the arms [manual 1476/24,576 (6.01%); paired 2923/48,271 (6.06%)]. For the FMR, there was a slightly higher rate [1476/24,576 (6.01%)] in the manual-only arm than in the paired arm [2642/48,271 (5.47%)]. When the data were restricted to routine samples in the age range 25–64 years, there was a slightly higher proportion of samples with MRs of low-grade cytology in the paired arm: manual 4.31% (821/19,041) versus paired 4.77% (1839/38,522). For the FMR the rates in the two arms were similar: manual 4.31% (821/19,041) versus paired 4.27% (1643/38,522).

Analysis of cytology management

The analysis of actual management following the cytological MRs is shown in Table 31. Most negatives [38,240/43,291 (88.3%)] were returned to routine recall, 4814/43,291 (11.1%) had repeat cytology, presumably as part of a follow-up strategy after an abnormality, and a small number were being seen at colposcopy either because there was clinical suspicion requiring a colposcopic examination or because colposcopy and cytology were part of a follow-up regimen.

Because of the NHSCSP Sentinel Sites protocol being used in Greater Manchester, the majority of borderline/mild cytology was triaged with HPV testing and positives referred for colposcopy. Some borderline abnormalities that were repeat samples were outside this protocol and were referred if persistently abnormal. All women with high-grade abnormalities (moderate dyskaryosis or worse) were referred for colposcopy. There are outstanding colposcopy outcomes for all grades of cytology: 1.2% for low-grade and 3.2% for high-grade cytology.

Colposcopy referral rates

Table 32 shows the proportion of women referred for colposcopy broken down by arm and LBC type, as a result of high-grade cytology and HPV triage of low-grade abnormalities. Of the 72,837 samples included in the analysis, 1000 were taken at colposcopy clinics. Overall the referral rate was 4.7% (3377/71,837). When the data were restricted to routine samples from women aged 25–64 years, the proportion was 4.0% (2292/57,527), 3.9% (735/19,024) in the manual-only arm and 4.0% (1557/38,503) in the paired arm. Between the two LBC systems, 3.7% (1025/27,897) were referred following BD SurePath and 4.3% following ThinPrep cytology (1267/29,666) (p < 0.001) (see Table 32). The reason for this difference is not clear.

Histology results

The numbers and proportion of cases per 1000 of detected histological lesions are shown in Table 33, broken down by grade of lesion, LBC system and trial arm. These data have also been aggregated into CIN2+ and CIN3+ and shown as percentages. However the data are depicted, there was no statistical difference between the detection rate in the manual-only arm and the paired arm for both CIN2+ [398/24,566 (1.62%) vs 707/48,271 (1.46%) respectively; p = 0.10] and CIN3+ [218/24,566 (0.89%) vs 404/48,271 (0.84%) respectively; p = 0.48]. There was a higher detection rate in the BD SurePath samples than in ThinPrep for both CIN2+ [585/35,599 (1.64%) vs 520/37,238 (1.40%) respectively; p ≤ 0.01] and CIN3+ [333/35,599 (0.94%) vs 289/37,238 (0.78%) respectively; p = 0.02]. When the data are restricted to routine samples at ages 25–64 years (Table 34), the rates of CIN2+ are 1.45% (337/23,219) in the manual-only arm and 1.30% (598/45,999), p = not significant, in the paired arm. For CIN3+ the rates are 0.82% (191/23,219) and 0.76% (349/45,999), p = not significant, respectively. The overall CIN2+ detection rates were 1.46% (493/33,784) for BD SurePath LBC and 1.25% (442/35,434) for ThinPrep LBC, p = 0.02. The corresponding rates for CIN3+ were 0.85% (287/33,784) and 0.71% (253/35,434) respectively, p = 0.04.

The relative sensitivity of screening by automated or manually read cytology to detect CIN3+ and CIN2+

For the purposes of investigating sensitivity and specificity, the cytology results were translated into positive and negative outcomes for FMR and FAR.

Definition of FAR positive is a FAR result of borderline or worse, and the woman referred to colposcopy (i.e. if borderline/mild the HPV result is positive). FAR negative is any negative result or where the FAR was borderline/mild, but the HPV result was negative.

Where the cytology result was borderline or mild, but the HPV status is not known, then it is assumed to be FAR positive if the subject was sent for colposcopy. Samples where the women were referred to colposcopy, but no result has been obtained (either due to non-attendance or inadequate result) have been excluded. Samples where either the FAR or the FMR result was inadequate have also been excluded. The definition of FMR positive and negative is equivalent.

The primary outcome of the MAVARIC study is shown in Table 35, where paired comparisons for CIN2+ are shown for the final manual and automated readings. It is clear that there are 52 more CIN2+ lesions missed on auto than on manual reading. These data form the basis for determining relative sensitivity between manual and automated reading. Similar data are also shown for CIN3+ with 18 more lesions missed on auto than manual reading.

When the clinically less significant outcome of CIN1– is considered (Table 36), there were 260 more final auto readings that were negative than final manual readings and for CIN2 or less (CIN2–) the corresponding number was 294. These form the basis for determining relative specificity.

The relative sensitivity based on matched pairs was 0.92 (95% CI 0.89 to 0.95) indicating a statistically significant difference of around 8%. This means that automated reading was less sensitive for manual reading by a margin of 8% in the detection of CIN2+. The specificity for CIN2+ detection was 97.3% (44,564/45,782) for auto readings and 96.8% (44,304/45,782) for manual readings, giving a relative specificity of 1.006 (95% CI 1.005 to 1.007) in favour of auto. This means that automated reading is slightly more specific than manual, but only by a margin of 0.06%.

The corresponding data for CIN3+ include a relative sensitivity of 0.95 (95% CI 0.91 to 0.99), which indicates a statistically significant difference of 5%. This means that automated reading was 5% less sensitive than manual reading in the detection of CIN3+. The specificity difference for CIN3+ is exactly the same as for CIN2+, the specificity for auto reading being 96.8% (44,620/46,085) and for manual 96.2% (44,326/46,085). The relative specificity was 1.007 (95% CI 1.006 to 1.008) in favour of auto reading, meaning that the manual arm was slightly less specific for the detection of CIN3+ and by exactly the same margin as for CIN2+. Clearly, this slight gain in specificity for automated reading is outweighed by the relative loss of sensitivity.

The discordant proportions are shown in Table 37, which shows that FMR-positive/FAR-negative results are almost three times as common as FMR-negative/FAR-positive results in cases of CIN2+ and almost twice as common in cases of CIN3+. When all discordant pairs were counted according to our definition, they accounted for similar proportions of both CIN2+ and CIN3+ (14.9% and 16.1% respectively).

The relative sensitivity of automated versus manual reading for routine screening samples in women aged 25–64 years only is presented in Table 38. Automated reading was in fact 9% less sensitive for CIN2+ relative to manual for routine screening samples, and 6% less sensitive for CIN3+. Again there was a small gain in specificity of only 0.04% and 0.05% for CIN2+ and CIN3+ respectively (Table 39).

The relative sensitivity of screening by the Becton Dickinson FocalPoint Guided Screener Imaging System or ThinPrep Imaging System and manually read cytology to detect CIN3+ and CIN2+

The MAVARIC study was not primarily designed to compare BD SurePath with ThinPrep or the BD FocalPoint GS Imaging System with the ThinPrep Imaging System, but comparing clinical outcomes using these systems was a secondary objective. The study was not formally powered for these secondary analyses and because only half of the CIN2+ lesions followed each of the systems it was likely that the study would be underpowered to detect small differences. The direct comparison of relative sensitivity between the BD FocalPoint GS Imaging System and the ThinPrep Imaging System is shown in Table 40 and is based on routinely obtained screening samples only. For both CIN2+ and CIN3+, the ThinPrep Imaging System had a slightly higher relative sensitivity (0.92 vs 0.90 and 0.97 vs 0.91 respectively), but this did not reach statistical significance (p = 0.53 and p = 0.52 respectively). Relative specificity (Table 41) was slightly greater for the BD FocalPoint GS Imaging System, but not sufficiently to give it an advantage over the ThinPrep Imaging System.

The distribution of discordant pairs according to the LBC platform is shown in Table 42. There were more discordant pairs with the ThinPrep Imaging System, but most of these were associated with CIN1–. There were nine more CIN3+ lesions that were missed with the ThinPrep Imaging System than with the BD FocalPoint GS Imaging System.

The detection rates (positive predictive value) for each category of cytology including the threshold of borderline or greater and mild dyskaryosis or greater

Table 43 shows the colposcopy outcomes in relation to the MRs in the paired arm, while Tables 44 and 45 show the colposcopy outcomes in relation to the final manual and auto results in the paired arm. There are 2411 known colposcopy results with 707 CIN2+, including 404 cases classified as CIN3+. The large majority of CIN3+ (317/404) and about two-thirds of CIN2+ (458/707) were found in women with manual cytology classified as moderate or worse. Among all borderline cytology the CIN2+ detection rate (PPV) was 5.2% (90/1747) and 12.4% (146/1176) for mild dyskaryosis. When these low-grade cytology results were triaged by HPV testing, the corresponding PPVs increased to 13.5% (80/594) and 16.4% (104/635) for borderline and mild respectively. This demonstrates that HPV status is a powerful discriminant of underlying risk irrespective of the category of low-grade abnormality. For the FMR, the PPV for all borderline cytology for CIN2+ (among those with a known colposcopy result) was 13.0% (84/648), for mild cytology 15.1% (138/915) and for moderate or worse 75.4% (450/597). For the FAR the PPVs for borderline are 11.3% (54/476), for mild 15.7% (127/808) and for moderate or worse 76.4% (429/561).

The data in Tables 44 and 45 can be used to determine the additional number of colposcopies required to achieve the added sensitivity (and slightly reduced specificity) of manual reading. If all women with borderline/HPV positive, mild/HPV positive and HPV not known, and moderate+ were referred to colposcopy, 250 additional colposcopies would have been required to detect 47 additional CIN2+, when compared with automated reading. Therefore, these extra colposcopies had a PPV of 19%. Although lower than the overall proportion of CIN2+ among the colposcopy group (707/2411, 29.32%) it is certainly consistent with the rate to be expected for women referred following HPV triage of low-grade abnormalities and would therefore be considered a worthwhile use of resource.

In Appendix 8, the results of triage are shown if they had been based on HPV genotyping as opposed to HC2. Essentially triage based on HPV 16 and/or 18 increased the PPV for CIN2+, but only from 15% to 25%, with a much lower sensitivity.

Analysis of the Becton Dickinson FocalPoint Guided Screener Imaging System no further review and quintiles

The BD FocalPoint GS Imaging System incorporates a ranking element which identifies the samples least likely to have evidence of disease, to the extent that NFR is required; this accounts for up to 25% of the slides. The remaining samples are divided into five quintiles ranging from quintile 1 (most likely to contain abnormal cytology) to quintile 5 (least likely to contain abnormal cytology). The results of the ranking are shown in Table 46. NFR was the ranking in 21.9% (4569/20,882) of slides with just four (0.02%) high-grade manual cytology readings associated with these. Table 47 shows the histological result correlated with the BD FocalPoint GS Imaging System result. Ten CIN2+ and four CIN3+ were detected, which account for only 3.1% of all CIN2+ detected using the BD FocalPoint GS Imaging System (and 2.2% of CIN3+). The proportion of the total number of CIN2+ detected in quintiles 1–5 was 63.6%, 13.7%, 8.9%, 5.5% and 5.5% respectively. It would appear that NFR could be used safely to archive cytology without further reading, which could be labour saving and cost-effective (see Costs and cost-effectiveness of the Becton Dickinson FocalPoint Slide Profiler as a stand-alone device), even without the use of automated reading and the Guided Screener Workstations. If NFR were restricted to only routine samples, < 1% of CIN2+ would have gone undetected.

As shown in Table 48, of the 4910 samples marked for NFR, four were associated with a histological result of CIN3+. These potential false-negatives would not have been identified by rapid review. Of the 26 NFR samples deemed non-negative by the rapid reviewer, the most severe histological result obtained was CIN1.

Histology outcomes of discordant results

The clinical outcomes in women in whom discordant results between manual and auto were found in the paired arm are shown in Table 49. Discordant results included the following matched pairs: borderline or mild/HPV positive and negative cytology; moderate dyskaryosis or worse and negative cytology; and, borderline/mild (HPV not tested) referred to colposcopy and negative. In total there were 52 additional CIN2+ associated with FMR positive/FAR negative than FMR negative/FAR positive (Table 50). Most of these lesions were detected in the borderline/mild HPV-positive category. An analysis of the timings of the discordant pairs revealed that discordant pairs occurred at an equal distribution throughout the duration of the study.

In order to determine whether the discordant readings were due to errors in interpreting the cells as presented in the FOV, or a failure by the automated machine to locate and present the abnormal cells, a rereading of discordant pairs associated with CIN2+ was undertaken. As shown in Table 51, 46/61 cases involving auto negative or auto low-grade/HPV negative were considered to be due to interpretation error, i.e. the abnormality had been presented in the FOVs, but had been missed. This applied to both the ThinPrep Imaging System and the BD FocalPoint GS Imaging System. In one-quarter (15/61), no abnormality was seen on review, suggesting an automated location error. It is important to note that that these ‘missed’ reads on automated reading relate to instances with underlying CIN2+. We have not analysed all instances with similar discordant results where there was no underlying disease.

A sample of the discordant results, where the abnormal cells had been missed owing to automated interpretation error, was rescreened by the review panel in an attempt to determine why the primary screeners had missed the cells presented in the FOVs. The results are presented in Table 52. In the majority of cases the cells were interpreted incorrectly on the automated screening owing to biological limitations within the slide preparation – in nearly a quarter of cases the FOVs contained a scanty preparation of cells, while 16.6% of the slides reviewed were found to have FOVs containing hyperchromatic crowded groups. The remainder of the biological limitations were due to inflammatory cells, pale and small cell dyskaryosis, and blood-stained samples. Another difficulty, alongside biological limiting factors, noted by reviewers was the location of the abnormal cells in relation to the centre of the FOVs. In 17.9% of the cases reviewed, it was thought that the cytoscreeners had overlooked the abnormal cells as they were peripheral to the FOV presented by the automated review scopes. Peripheral cells are not thought to present a problem in manual screening owing to the practice of ‘overlapping’ FOVs, which is lost when primary screeners are restricted to either 10 or 22 FOVs. The practice of screening limited fields on the slide was also thought to hinder the interpretation of the biological limitations as the ability to place the cells in the overall context of the slide is lost.

Productivity

Loading and unloading time of equipment in the automated arm

Medical laboratory assistants completed 34 worksheets recording the times for loading and unloading LBC slides on the automated equipment (17 using the BD FocalPoint GS Imaging System and 17 using the ThinPrep Imaging System). The results show that MLAs on average spent 30 minutes to load and unload 160 slides using the BD FocalPoint GS Imaging System. Using the ThinPrep Imaging System on average they took 26 minutes to load and unload 251 slides. The results (Table 53 and Figure 9) show that the mean (standard deviation) time for loading and unloading a sample using the BD FocalPoint GS Imaging System was 0.10 (0.06) minutes and 0.09 (0.03) minutes, respectively, amounting to a total time of 0.20 (0.07) minutes. The loading and unloading time using the ThinPrep Imaging System was 0.06 (0.04) minutes and 0.05 (0.01) minutes respectively, with a total time of 0.11 (0.04) minutes per slide. Therefore, overall it was quicker to load and unload LBC samples with the ThinPrep Imaging System and this difference was statistically significant (p < 0.000).

TABLE 53 - Duration in minutes [mean and standard deviation (SD)] for loading and unloading LBC slides

Technology	Time to load		Time to unload		Total time
	Mean	SD	Mean	SD	Mean	SD
BD FocalPoint GS Imaging System	0.10	0.06	0.09	0.03	0.20	0.07
ThinPrep Imaging System	0.06	0.04	0.05	0.01	0.11	0.04

FIGURE 9.

Duration of loading and unloading per slide with automated technologies.

Cost analyses

Cost of colposcopy, histology and cancer treatment

The costs of colposcopy, histology and cancer treatment were identified from published literature and are reported in Table 72. Slides that were borderline or mild and above in the final MR were sent for HPV testing. It was assumed that inadequate slides incurred a further sample taking and slide reading cost at the laboratory. The cost per woman also included the cost of colposcopy referral and treatment of CIN where required based on the clinical data.

TABLE 72 - Cost of histology outcome and cancer treatment

Sample cost for automated screening includes loading and unloading cost.

Clinical activity	Cost (£)	Reference
LBC test cost in laboratorya	5.46–5.72	Table 70
Consult cost – GP/nurse visit in community	15.91	Table 64
HPV test reflex costsa	16.75–16.94	Table 71
No CIN	282.76	Martin-Hirsch et al.60
CIN1	432.39
CIN2	590.28
CIN3	625.37
Stage 1 invasive cancer	2874.02
Stage 2 invasive cancer	4590.17
Stage 3 invasive cancer	12,963.53
Stage 4 invasive cancer	13,185.40

The rates of different outcomes that determine the cost per woman are reported in Table 73 based on an analysis of the clinical trial data set. Rates of events are similar between arms. These data reflect the clinical findings of the trial, in that there was a lower rate of referral to colposcopy and detection of CIN lesions with automated technology.

TABLE 73 - Rates of inadequate samples, HPV testing, diagnosis and treatment

Item	Automated arm	Manual arm
Inadequate sample	1.91%	2.99%
Negative	93.63%	91.43%
HPV test	4.29%	5.54%
Colposcopy
No CIN	1.76%	2.08%
CIN1	0.43%	0.49%
CIN2	0.48%	0.54%
CIN3	0.75%	0.78%

The total cost per woman was very similar for manual and automated reading where the same technology was used (Table 74). The overall costs per woman are higher with the ThinPrep technologies either where slides are read manually or with automated screening, reflecting slightly higher referral rates than with colposcopy where no CIN or CIN1 was detected.

TABLE 74 - Total cost per woman screened including inadequate samples/HPV testing/colposcopy/± biopsy (95% CI)

Item	Automated arm (£)	Manual arm (£)
Inadequate sample and negative	20.53 (20.49 to 20.57)	20.49 (20.44 to 20.53)
HPV test	0.72 (0.69 to 0.76)	0.93 (0.90 to 0.97)
Colposcopy	4.97 (4.60 to 5.33)	5.88 (5.49 to 6.28)
No CIN
CIN1	1.85 (1.58 to 2.13)	2.11 (1.82 to 2.41)
CIN2	2.85 (2.45 to 3.25)	3.16 (2.74 to 3.58)
CIN3	4.68 (4.15 to 5.20)	4.91 (4.37 to 5.45)
Total cost per woman, average prices	35.60 (34.79 to 36.42)	37.48 (36.63 to 38.34)

Table 75 indicates that the average cost per case detected between automated and manual screening were very similar. These data incorporate both the total cost per woman including slide reading and downstream costs.

TABLE 75 - Cost per case of CIN2+ and CIN3+ detected (95% CI)

Item	Automated arm	Manual arm
Average cases of CIN2+ per 1000 women	12.31 (11.23 to 13.39)	13.21 (12.09 to 14.32)
Average cases of CIN3+ per 1000 women	7.48 (6.63 to 8.32)	7.85 (6.99 to 8.71)
Cost per case of CIN2+ detected	£2892 (£2720 to £3098)	£2838 (£2676 to £3030)
Cost per case of CIN3+ detected	£4762 (£4378 to £5245)	£4775 (£4400 to £5244)

Figure 10 presents the results of the bootstrapping exercise on the incremental cost per case of CIN2+ detected. These data reflect the uncertainty in the comparative cost and event outcomes by random sampling from the trial data. The majority of results are in the south-west quadrant, indicating that automated screening is less effective in the detection of CIN2+, but is also cost saving. Approximately a quarter of the results are in the south-east quadrant, indicating that, in these random samples of the trial data, automated screening is both cost saving and more effective. A similar picture is seen in Figure 11, where the detection of CIN3+ is used as the outcome measure. In line with the main clinical results there is correlation between CIN detection rates and costs. This is reflective of the fact that where relatively less CIN is detected the costs are lower owing to reductions in treatment costs.

FIGURE 10.

Incremental cost per case of CIN2+ detected for automated reading compared with manual reading (bootstrapped results).

FIGURE 11.

Incremental cost per case of CIN3+ detected for automated reading compared with manual reading (bootstrapped results).

In Figures 12 and 13 the results of the bootstrapped results have been plotted on cost-effectiveness acceptability curves. These figures indicate the probability that manual screening is cost-effective compared with automated screening for different willingness-to-pay thresholds for detecting additional cases of CIN2+ and CIN3+. In the baseline results we have used the average price of manual and automated screening. However, given the uncertainty about prices and the need to blind price differences between the two manufacturers we have also presented curves for the minimum price difference and maximum price difference.

FIGURE 12.

Cost-effectiveness acceptability curve for manual reading compared with automated reading for the detection of CIN2+. Min and max represent the minimum and maximum price difference between automated and manual preparation costs.

FIGURE 13.

Cost-effectiveness acceptability curve for manual reading compared with automated reading for the detection of CIN3+. Min and max represent the minimum and maximum price difference between automated and manual preparation costs.

Given a willingness to pay of £5000 for each case of CIN2+ detected, these data indicate that there is an 80% chance that manual screening is cost-effective compared with automated screening using average prices between the two manufacturers. As detailed previously, there is a high degree of uncertainty reflected in the minimum and maximum price differences between automated and manual screening. The probability of manual being cost-effective rises to around 97% at the maximum price difference and falls to around 25% at the minimum price difference.

In Figure 13, cost-effectiveness estimates are presented using CIN3+ as an outcome measure. This figure indicates greater uncertainty in the cost-effectiveness of manual reading compared with automated reading. Decision-makers would need to be willing to pay an additional £12,500 per additional case of CIN3+ detected to have even a 70% level of certainty that manual reading was more cost-effective than automated screening. With the minimum price difference between automated and manual screening, the likelihood that manual screening is cost-effective falls to just under 50% at a willingness to pay of £12,500 per additional CIN3+ case.

Costs and cost-effectiveness of the Becton Dickinson FocalPoint Slide Profiler as a stand-alone device

Further analyses were conducted on the difference in the average cost of reading the slides that were identified as requiring NFR with the BD FocalPoint Slide Profiler compared with manually reading the same slides (Tables 76 and 77). The average cost per slide including staff and preparation costs was lower when the BD FocalPoint Slide Profiler was used as a stand-alone device utilising the ‘NFR’ option than with manual reading, regardless of whether slides were rapid reviewed. However, slightly fewer cases of CIN2+ and CIN3+ were also identified.

Figures 14 and 15 report the cost per case detected of CIN2+ and CIN3+ for manually reading slides compared with using the ‘NFR’ option on the imager. Manual reading in this case would be cost-effective compared with NFR if decision-makers were willing to pay £2500 per additional case of CIN2+ or £6000 per additional case of CIN3+ detected.

FIGURE 14.

Cost-effectiveness acceptability curve for NFR compared with manual reading for the detection of CIN2+.

FIGURE 15.

Cost-effectiveness acceptability curve for NFR compared with manual reading for the detection of CIN3+.

Further analyses were conducted to compare the cost of manually screening slides compared with using the ‘NFR’ option on the BD FocalPoint Slide Profiler and not reading slides ranked either quintile 5 or quintiles 4 and 5. These results are presented in Tables 78 and 79. Again, the overall cost of not reading slides identified with the BD FocalPoint SlideProfiler as either quintile 5 or quintiles 4 and 5 is lower; however, also slightly fewer cases of CIN2+ and CIN3+ were identified. As shown in Figures 16 and 17 these data indicate that utilising the BD FocalPoint Slide Profiler to identify slides in quintiles 4 and 5 and then not reading them is unlikely to be cost-effective if decision-makers are willing to pay > £2500 per case of CIN2+ detected.

FIGURE 16.

Cost effectiveness acceptability curve for manual reading compared with not reading slides identified as quintiles 4 and 5 for the detection of CIN2+.

FIGURE 17.

Cost-effectiveness acceptability curve for manual reading compared with not reading slides identified as quintiles 4 and 5 for the detection of CIN3+.

Lifetime modelling results

Table 80 shows the predicted lifetime costs and effects of each LBC strategy in a simulated cohort of 10,000 women. Costs and effects were discounted at 3.5% for the first 30 years and 3% thereafter. Modelling beyond trial end points predicts that automated LBC results in a small cost saving over the lifetime of a woman (approximately £12.60 per woman, discounted) and also a small loss in life-years (4.52 per 10,000 women, or approximately 4 hours per woman, discounted; Table 81). The predicted decrease in life-years associated with automated LBC, compared with manual, is primarily driven by the slight loss in sensitivity. If automated LBC was current practice, manual LBC would be associated with an incremental cost-effectiveness ratio of £27,863 per life-year saved. This is above the £20,000 per QALY figure where current NICE recommendations strongly favour adoption, but it is below the £30,000 figure where rejection is favoured. 79

When QALYs are used as the outcome measure, modelling predicted that automated LBC is associated with an increase of 15.83 QALYs per 10,000 women (or approximately 13.9 quality-adjusted life-hours per woman; Table 81) compared with manual LBC. This finding is sensitive to choice of quality-of-life weights, but remained > 0 in all cases examined during sensitivity analysis. The increase in QALYs is driven by a small increase in specificity (which decreases the disutility resulting from false-positives). When QALYs are used as the outcome measure, automated LBC dominates manual LBC as a strategy, as it is both cost-saving and more effective. It should be noted that the quality of life weights for health states were obtained from the international literature and rely on assumptions about the duration of disutility. 67,69,80–84 It should also be noted that there is some evidence from willingness-to-pay studies that UK women express preference for improved sensitivity. These findings have not been formally integrated into a cost–utility framework, but may increase the uncertainty surrounding these results.

Modelling was also used to predict cancer outcomes over the lifetime of a cohort of 10,000 women, assuming current screening recommendations, intervals and compliance. In this cohort of 10,000 women, manual LBC was associated with 69 cancer cases and 12 cancer deaths, and automated LBC with 72 cancer cases and 13 cancer deaths (Table 82) over the cohort’s lifetime.

Discordant results

The review of the discordant pairs was designed to try to explain the reason for automated reading ‘missing’ abnormalities that were associated with CIN2+. The review also included the discordant pairs for which manual reading did not detect abnormalities picked up by automated reading. Low- and high-grade abnormalities were separated in the analysis. In one-quarter of cases no abnormality was seen, suggesting an auto location error, i.e. the abnormalities had not been shown on the FOVs. In a similar review by Halford et al. ,16 discordant readings from a split sample study, 31/37 cases of auto misses were found to contain abnormalities on the FOVs not detected in the initial read, and in the majority of these 31 cases abnormal cells were seen in only 5 out of 22 FOVs. There are reasons why automated reading could result in false-negative reports. Peripheral location of abnormal cells in the FOVs was found in a number of cases, also noted by Halford et al. ’s16 study. This is not a problem in manual screening given the practice of overlapping fields in routine screening practice, which is lost in location-guided screening. The nature of automated screening means that it is more monotonous, a view expressed by several staff, which could result in lower levels of vigilance. The use of new equipment presents challenges for staff used to their own workstation. It is not considered to be a learning curve issue, however, because discordant pairs occurred at an equal distribution throughout the 3 years of the study, during which staff gained considerable experience.

The most recent large-scale evaluation of automation-assisted cervical screening has just been reported from the Scottish Cytology Network for the ThinPrep Imaging System. 37 Around 110,000 samples were randomly allocated to either manual or automated reading using the ThinPrep Imaging System. Samples were not double read to provide direct comparison. The primary outcome measure was based on cytology grade rather than histopathology. This therefore represented a comparison of the effectiveness in detection of abnormal cytology rather than measuring comparative performance in detecting CIN. In Phase 1 of the Scottish evaluation, abnormal slides were read in the ‘Autoscan’ mode and in Phase 2 they were removed and read on a standard microscope. The proportion of high-grade abnormality detected was similar (1.38% manual, 1.45% auto, p = 0.512). Sensitivity was defined as the proportion of abnormals picked up on the first read compared with that after rapid review and checking. By this criterion, the results showed similar sensitivity in both arms (93.7% manual, 91.79% auto, p-value for high grade = 0.09). There were also similar values for low grade (7.5% manual, 7.87% auto). It is noteworthy that there was variation in these proportions of abnormality between the six participating laboratories. For manual reading the low-grade range was 4.48%–9.18% and the high-grade range was 1%–1.8%.

In the MAVARIC study paired arm, the proportion of low grades was 5.5% for manual and 4.5% for the ThinPrep Imaging System, both at the low end of the Scottish range. Similarly the proportion of high grades was 1.27% for manual and 1.14% for the ThinPrep Imaging System, again at the low end of the Scottish range. The authors of the Scottish study reported that their results indicated that automated screening would be safe and more efficient. 37 Cost-effectiveness was not formally evaluated and would depend on the costs of the system. It was not designed to provide the relative diagnostic performance determined in MAVARIC, and as such the two studies are not directly comparable. The abnormality rates are of interest though again not directly comparable because screening in Scotland begins at 20 years compared with 25 years in England and cytology abnormalities are particularly common in the age group 20–24 years.

One of the limitations of MAVARIC is that it was conducted in a single laboratory; however, the routine reporting from the Manchester Cytology Centre is very much in the mid-range of abnormality rates and PPVs for high-grade cytology as reported in the NHSCSP Statistical Bulletin. 6

As stated in Chapter 1 and shown in Table 1, previous studies comparing automated with manual reading have tended to indicate higher rates of cytological abnormality and some have found increased rates of CIN2+. Some of these studies23,27 have been performed simply comparing cytological abnormality rates in consecutive periods of time, and others have used a split sample design,16,24,26 whereby the same sample has been split between a conventional or LBC slide which is manually read and a slide which is subjected to automated reading. Many studies do not use histology as an outcome. 17,18,25,29–32,34,35

Economic analysis

The study provides a detailed comparative assessment of productivity in the laboratory. It clearly demonstrates that primary reading is substantially quicker with automated equipment than a manual approach. We used two different methods to observe changes in primary reading times: time-and-motion studies and workload surveys. The time-and-motion studies indicated that slides could be read three to four times quicker with automated screening, but there was no significant difference in reading times between the two technologies with either manual or automated reading (p = 0.14).

By contrast, the workload survey data estimated longer slide reading times than the time-and-motion study results, suggesting some inefficiency and further administration time not captured within the time-and-motion data. It is likely that these data provide a better reflection of the productivity gains that may be achievable in a real-life setting. These data suggest that eight or nine slides can be read per hour with manual reading, compared with 19 or 20 with automated reading.

Other studies have largely been conducted on the ThinPrep Imaging System and show comparable results, although they use a range of timing methodologies and the comparator is sometimes not LBC but conventional slides. One study26 reported slide reading times for the ThinPrep Imaging System of 3.4 minutes compared with 7.4 minutes for manual reading of conventional slides. An Australian study24 also found that the number of slides read per hour was significantly increased with ThinPrep Imaging System-assisted reading compared with conventional slides. The mean within-reader difference was 7.2 slides per hour. Only one study32 has been identified that compared BD FocalPoint GS Imaging System-assisted screening with manual screening and it was found that interpretation time was reduced by 40%.

In addition to assessing the times for primary screening, we also estimated the overall implications for laboratory staff productivity. With the automated technologies there was a slight decrease in referral for review by checkers and medics. For primary screening the NFR option led to further increases in productivity with the BD FocalPoint GS Imaging System. Over a 7.5-hour working day we estimated an overall increase in productivity for cytoscreeners of between 60% and 80%.

One study29 found slightly higher productivity increases with the ThinPrep Imaging System-assisted method, with an estimate that the rate of slides screened was typically doubled over an 8-hour day. Another study26 found that the ThinPrep Imaging System-assisted screening led to a 27% productivity gain when compared with manual screening with LBC, a smaller gain than observed in our study.

With automated screening and reductions in primary screening time, the average proportion of a cytoscreener’s time spent on rapid review increases, which has a significant impact on cost. Potentially, further cost savings could be made with automated screening by changing the rapid review protocols. The BD FocalPoint GS Imaging System also flags up at least 15% of all successfully processed negative or inadequate slides for QC review.

MAVARIC has produced unbiased and comparable productivity estimates across manual and automated technologies. The study has compared both BD FocalPoint GS Imaging System and ThinPrep Imaging System technologies with their manual counterparts. The slides were blinded between arms and therefore led to unbiased results, which indicate that use of automation in screening in the UK can reduce the average time taken to process a slide. The key area for savings in time is the primary screen.

The results of the staff satisfaction study indicate that staff prefer manual reading to automated technologies. Some physical and ergonomic discomfort was noted particularly with the ThinPrep Imaging System, although subsequently there has been some redesigning of the technology to address this. Another issue highlighted was the monotony of automated reading, although some cytoscreeners noted that manual and automated reading both had elements of monotony.

Comparative data on the cost per slide indicate that the additional costs associated with the automated equipment are offset by savings in the costs of staff time. For confidentiality purposes the price of equipment was blinded between the two manufacturers, but with one manufacturer the additional equipment costs were more than offset by time savings and therefore automated reading became cost saving compared with manual, whereas with the other manufacturer the additional costs were not completely offset by staff time savings. Averaging across these indicative prices, automated screening was slightly less expensive per slide than reading slides manually once staff savings were taken into account. However, these estimates are sensitive to the price of the equipment and, where the maximum price difference was used between automated and manual reading, overall it cost more to read slides using automated equipment. It should be noted that these price estimates are based on automated machines operating at maximum capacity. As the volume of slides required to operate at full capacity is higher than observed in the NHS Cancer Screening Programme, national implementation of automated cytology would require careful consideration of the need for amalgamation of existing laboratories or alternative ways of configuring or delivering services, in order to maximise efficient use of the technology. In addition, the costs of training were covered by manufacturers and it is unknown if this would be the case if automated technology were rolled out nationally. Assessment of the overall cost per woman screened including downstream costs associated with treatment of CIN, colposcopy and HPV testing indicated very similar costs between manual and automated screening from each manufacturer.

Within-trial analysis of the main trial results indicated that there is an 80% chance that manual reading is cost-effective compared with automated reading (using average prices between the two manufacturers), given a willingness to pay of £5000 for each additional case of CIN2+ detected. These results were sensitive to the price of automation, and at the minimum price difference between technologies decision-makers would need to be willing to pay £8500 per additional case of CIN2+ detected for manual reading to remain cost-effective.

Further analyses evaluated the use of the BD FocalPoint Slide Profiler as a stand-alone device with manual reading, either used to identify slides requiring NFR with or without rapid review, or not to be required to read slides in the quintiles with the lowest risk of abnormal cells. Our results indicated that when using the equipment in this way, cost savings were generated; however, slightly fewer cases of CIN2+ and CIN3+ were detected. With the NFR option only, manual reading would remain cost-effective if decision-makers were willing to pay £2500 per case of CIN2+ detected. Again, utilising the BD FocalPoint Slide Profiler to identify slides in quintiles 4 and 5 and then not reading them is unlikely to be cost-effective if decision-makers were willing to pay at least £2500 for each additional case of CIN2+ detected by manual reading. These analyses have not been applied to NFR for routine samples only.

The results of the lifetime modelling of the cost-effectiveness of automation compared with manual reading show uncertainty about the relative cost-effectiveness of automation compared with manual reading. If automated LBC was current practice, manual LBC would be associated with an incremental cost-effectiveness ratio of £27,863 per life-year saved. This is above the £20,000 per QALY figure at which current NICE recommendation strongly favour adoption, but it is below the £30,000 figure above which interventions are likely to be rejected on cost-effectiveness grounds. 79 One-way sensitivity analysis indicated that these results are highly sensitive to the relative test performance between manual and automated reading with estimates ranging from £11,881 to £36,229 per life-year saved.

Quality-adjusted life-year estimates were also derived from modelling, and these indicated that automated reading might produce a small QALY gain due to the difference in specificity and potential disutility associated with ‘overtreatment’ of lesions that might regress. This finding remained the same in all options explored in the sensitivity analysis, including when minimum values were used for disutility and relative test performance. These QALY results should, however, be treated extremely cautiously, as the empirical evidence on utilities came not from the trial but from the international literature. 67,69,80–84 In particular, the true duration of disutility for women associated with overtreatment of pre-invasive cervical cancer lesions is difficult to determine. In particular, more data are required on the true value and duration of the disutilities (reported in Table 96) associated with treatment for CIN and colposcopy referral: in the model these apply for the 6-month cycle in which the event occurs. It may be that disutility from a false-positive result is shorter. Further studies from the UK, evaluating women’s overall preferences between cervical cancer screening strategies with comparatively higher sensitivity, at the cost of lower specificity, indicated an overall preference for comparative gains in sensitivity when traded with lower specificity. 86,87 While we have performed extensive one-way sensitivity analysis on the modelling results, we have not performed a probabilistic sensitivity analysis. The modelling exercise suggests, however, that the key area of uncertainty for drawing more affirmative conclusions on the true cost-effectiveness of automated compared with manual reading rests with the need for improved understanding and empirical research on the quality-of-life implications and women’s preferences for trading for improvements in sensitivity.

A published systematic review14 provided an analysis of automation in cervical screening programmes in the UK. This review of cost-effectiveness studies indicated strong limitations in the evidence used to populate previous models and therefore lack of certainty about any conclusions. Our results have significantly reduced the uncertainty relating to the costs of automated compared with manual reading, but substantial uncertainty remains concerning lifetime quality-adjusted survival.

Chief investigators

• HC Kitchener, Clinical Principal Investigator, School of Cancer and Enabling Sciences, University of Manchester.

• S Moss, Statistics and Epidemiology, CSEU, Institute of Cancer Research, Sutton.

Trial co-ordinators

• R Albrow, School of Cancer and Enabling Sciences, University of Manchester.

• J Mather, Manchester Cytology Centre, Central Manchester University Hospitals NHS Foundation Trust.

Epidemiology/Statistics

• R Blanks, CSEU, Institute of Cancer Research, Sutton.

• G Dunn, Health Science Research Group, University of Manchester.

Data management

• L Gunn, CSEU, Institute of Cancer Research, Sutton.

• E O’Brien, CSEU, Institute of Cancer Research, Sutton.

Cytopathology

• M Desai, Manchester Cytology Centre, Central Manchester University Hospitals NHS Foundation Trust.

• DN Rana, Manchester Cytology Centre, Central Manchester University Hospitals NHS Foundation Trust.

Virology

• H Cubie, Specialist Virology Centre, Royal Infirmary of Edinburgh.

• C Moore, Specialist Virology Centre, Royal Infirmary of Edinburgh.

Health economics

• R Legood, Health Services Research Unit, London School of Hygiene and Tropical Medicine; Health Economics Research Centre, University of Oxford.

• A Gray, Health Economics Research Centre, University of Oxford.

• Z Sadique, Health Services Research Unit, London School of Hygiene and Tropical Medicine.

This study was conducted under the guidance of a Steering Committee. The independent members are as follows:

• Professor David Torgerson, Independent Chair, Alcuin Research Resource Centre, Department of Health Sciences, University of York.

• Dr Maggie Cruickshank, School of Medicine, University of Aberdeen/Department of Obstetrics and Gynaecology, Aberdeen Maternity Hospital.

• Dr Karin Denton, Department of Cellular Pathology, Southmead Hospital, North Bristol NHS Trust.

The study Data Monitoring and Ethics Committee comprised the following members:

• Professor Paula Williamson, Chairperson, Centre for Medical Statistics and Health Evaluation, University of Liverpool.

• Dr John Smith, Department of Histopathology, Sheffield Teaching Hospitals NHS Foundation Trust.

• Mr Patrick Walker, Department of Obstetrics and Gynaecology, Royal Free Hampstead NHS Trust.

Cost-effectiveness modelling beyond the study end points was performed using a model previously developed by a group at Cancer Council New South Wales (NSW), Australia and adapted for this project in collaboration with Dr Rosa Legood. We are grateful to Dr Karen Canfell, Jie Bin Lew, Megan Smith and Robert Walker at Cancer Council NSW for their assistance in accessing the model, conducting the lifetime modelling and drafting the document sections relating to this.

We are grateful to Yvonne Hughes, Laboratory Manager, for her co-operation and effort in accommodating the MAVARIC study in the Manchester Cytology Centre.

We acknowledge the use of a BD FocalPoint GS Imaging System provided free of charge by Source Bioscience (formerly Medical Solutions) for part of the study.

We thank Qiagen for providing substantially discounted HC2 kits.

Stain validation

Prior to training the ThinPrep Imaging System stain had to be validated by two cytopathologists and the laboratory trial co-ordinator (see Chapter 2, ThinPrep Imaging System stain validation process).

Training

The training took place over 3.5 days and comprised:

1. Presentations and an introduction to the review scope.

2. Review scope training over 1.5 days which comprised three modules – two training modules (10 slides in each) followed by a test module (25 slides). The results of the review scope training are provided in Table 85. This session was attended by the laboratory trial co-ordinator, two cytopathologists, one chief BMS, one senior cytoscreener and seven cytoscreeners.

3. Training in the use of the ThinPrep Imaging System; this 1-day session covered guidance on loading and unloading the machine, maintenance and troubleshooting. This session was attended by the laboratory trial co-ordinator and seven MLAs.

Hologic were satisfied with the training results, and positive feedback was given by those who had taken part in the training.