Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study

Demand for emergency and urgent care

Emergency and urgent care is provided by a range of services, including emergency services (999 ambulance service, emergency departments), urgent-care services [general practitioner (GP) out of hours, minor injury units, walk-in centres, urgent care treatment centres, NHS 111] and in-hours general practice (requests for same-day appointments and telephone advice). There is widespread concern about rising demand for urgent and emergency care services. In England, national statistics show that in March 2015 there were 1.76 million first attendances at emergency departments (including minor injury units and walk-in centres), 543,172 ambulance responses to 999 calls and 1.08 million calls answered by the NHS 111 telephone service. Four years later, in March 2019, these figures had risen to 2.17 million, 730,000 and 1.32 million, respectively,1 and NHS England estimates that each year there are 85 million same-day urgent care contacts with primary care. 2 These demand increases are not wholly explained by population increases or an ageing population. 3

With demand rising year on year and increasing difficulties in recruiting and retaining a sustainable workforce,4 it is questionable if emergency and urgent care services have the capacity to deal with future demand increases. There are also concerns that some of this demand is a result of clinically unnecessary service use, for example where people choose a higher-acuity service level than is clinically necessary5,6 and could be diverted to lower-level care. Developing and improving the delivery of emergency and urgent care has been a key policy issue for a number of years. In 2013, the NHS England Urgent and Emergency Care Review7 was published, which set out to address ongoing concerns about the increasing and potentially unsustainable pressure faced by emergency and urgent care services. Underpinning the review was the premise that care should be provided ‘as close to home’ as possible by helping people with urgent care needs to get the right advice, in the right place, first time. This emphasis has continued, with emergency and urgent care provision identified as one of the four areas for further development in the 2017 NHS Five Year Forward Plan. This includes an explicit commitment to the introduction of an online version of NHS 111 to help people navigate access to urgent care. 8

Developing remote access to emergency and urgent care

Over the last two decades, a number of different organisational interventions have been implemented in England to attempt to manage demand for emergency and urgent care services. A cornerstone has been the introduction of telephone advice and triage via NHS Direct, subsequently replaced by NHS 111. This is a telephone service to provide clinical assessment and direct people with non-urgent or low-acuity non-emergency health problems to the most appropriate service, or to provide self-care advice for their clinical need. Despite these services’ apparent success in terms of receiving large call volumes, they have had little impact on reducing the demand for emergency services. 9 NHS England data show that, in 2017/18, NHS 111 answered 14,995,168 calls, an average of 46,082 per day, and this is expected to continue increasing. The outcomes of these calls were that 21% had ambulances despatched or were recommended to attend an emergency department (ED), 60.7% were recommended to attend primary care, 4.6% were recommended to attend another service and 14% were given information or self-care advice. 1 The Five Year Forward View8 policy has prompted significant efforts to further enhance the NHS 111 service by increasing the clinical assessment capacity and supporting more integration with other services. It also highlighted the potential for some lower-urgency calls to be handled through online services, which may relieve some of the pressure on the telephone service by diverting some activity, make more efficient use of the workforce, support demand management and reduce costs. An online service also fits with the broader aspirations of exploiting technological innovations to improve health-care delivery and offer alternative access systems in a world where digital solutions have transformed the way people live. This has been done successfully with the introduction of smartphone application (app)-based services to help people manage specific conditions such as diabetes. 10 More broadly, with service provision, as it is currently delivered, reaching saturation, it has been suggested that strategies to reduce patient health-seeking behaviours and increase self-management should be a focus for the NHS. 11

In response to this policy, four pilot online NHS 111 services were introduced in different geographical areas across England in 2017. A useful description of these services is provided in a recent discussion,12 but, briefly, they comprise online and app-based platforms that provide symptom checkers and advice on what users should do next, either self-care or contact another service. Early results from the pilot study reported that NHS 111 Online uptake across all pilots varied between 2% and 15%, but on average was approximately 6%. However, it was hoped that uptake would increase over time to a similar level to that reached in a similar Australian service, where calls to the equivalent telephone triage service decreased by 33% 3 years after the service was introduced. 13

During 2018, the service rapidly expanded to cover the whole of England. In addition, a decision was taken to use only one of the original four platforms across the country. This is the online application developed from the NHS Pathways clinical assessment system used by the NHS 111 telephone service. The service can be accessed via a web-based application or the NHS app and allows users to work through a set of symptom-based questions. It then provides advice on what service to access themselves, options for further telephone assessment with NHS 111 clinicians, out-of-hours GP or other services, appointments with services or self-care advice.

Need for evaluation research

The principles behind introducing an online service as an access point to urgent care in the NHS are based on an expectation that it can make access easier, direct people to appropriate levels of care and divert some activity away from the NHS 111 telephone service and, hence, slow increases in demand for that service.

However, there is also a risk that this new system may increase overall demand, duplicate health-care contacts, change the pattern of service use through differences in triage outcomes, remove some of the ‘gatekeeping’ that the telephone service provides, or provide unsafe advice. The uptake of an online service may be influenced by a lack of consideration of the many and varied reasons why patients chose particular services, including access to primary care; perceived urgency, anxiety and the value of reassurance from emergency-based services; views of family, friends or health-care professionals; convenience; and the perceived need for ambulance or hospital care, treatment or investigations. 14

A recent systematic review of digital and online symptom checkers for urgent health problems undertaken by the University of Sheffield found little evidence to suggest that digital and online symptom checkers are harmful to patient safety, but this may be because this outcome was reported by small, short-term studies only. 15 Deficiencies have been reported in the diagnostic and triage capabilities of symptom checker algorithms that would potentially be used in a digital 111 platform. 16 The systematic review identified that research is needed to investigate the pathways followed by patients using a digital 111 service and to identify whether or not the overall number and level of contacts with the health system can be reduced without the quality and safety of patient care being affected. 15

This research has been designed to meet a clear need set out by NHS England to evaluate and establish the current and future potential impact of an online service on the existing NHS 111 telephone service and wider urgent care system. There is a commitment to providing a national NHS 111 Online service, but robust evidence is needed to inform future plans about what the online offering should look like, and the likely impact on service use and associated costs, and to identify any potential unintended consequences that need to be addressed as the service continues to develop.

Conceptual framework

To our knowledge, there is no established and agreed theoretical framework that underpins the delivery of urgent care, but we have used three conceptual approaches to guide and interpret this research:

• Our recent extensive evidence review of models for delivering urgent care3 found a clear consensus that evaluation research concerned with the provision of urgent care services needs to take an emergency and urgent care system-wide perspective, rather than focus on a single service. This research uses an emergency and urgent care system model.

• The provision of an online NHS 111 service is one element of the broader NHS England emergency and urgent care strategy set out in the 2013 Urgent and Emergency Care Review7 and is relevant to the principles set out in that review of providing access to care that is ‘right first time’, providing better support for people to self-care and providing more care closer to home in local communities. More specifically, enhancing the NHS 111 service by improving integration with other services to provide efficient patient care pathways, increasing clinical assessment and adding an online access gateway are critical to the concepts set out in the NHS England Next Steps on the Five Year Forward View8 to help manage demand for urgent care. A key objective to achieving this is the development of online triage services that enable patients with urgent health-care problems to enter their symptoms and receive tailored advice or a call back from a health-care professional. Conceptually, therefore, the population of potential users of NHS 111 Online is expected to be similar to that of those who currently use the NHS 111 telephone service. More broadly, the addition of an online NHS 111 service fits with the NHS Digital Technology strategy to harness the potential of digital technology in ways that can empower and support people to take control of their health and care. 8 The call for this research is embedded in a very clear policy need, and these underlying urgent care policy principles have been used to frame the questions and design of this study and will be used to support the interpretation of the findings.

• NHS 111, whether a telephone or an online service, provides a point of access for people who have health problems that they perceive to be urgent. The operating model of using a symptom-based approach to assessment assumes an immediate need and uncertainty about what to do rather than simply seeking information about available services. This demand for and use of urgent care services is driven by the health-seeking behaviours of a population, but these have been poorly understood. This issue has been addressed in two recent National Institute for Health Research (NIHR)-funded studies on sense-making strategies and help-seeking behaviours associated with use of urgent care16 and a population study on the drivers of the demand for urgent care. 17 These studies were unpublished when this study was being designed but the researchers were available to help with the interpretation of this study’s findings.

Aims and objectives

The primary aim of this study is to evaluate the impact of providing an NHS 111 Online service to access urgent health-care advice and signposting to services on the telephone-based NHS 111 service. The secondary aim is to explore the potential effects on the wider urgent care system and the implications of providing a national NHS 111 Online service. The objectives are to:

1. update and summarise the evidence on digital and telephone-based services for accessing urgent care, building on existing systematic and rapid evidence reviews

2. measure the impact of the NHS 111 Online system on contacts with the NHS 111 telephone service, and estimate the effects on other services in the emergency and urgent care system and on NHS 111 services in the future

3. explore and compare in detail the characteristics of users of the NHS 111 Online service and the NHS 111 telephone service, service processes, patient care pathways and user experience and satisfaction

4. assess the practical issues associated with the implementation of the new service and the workforce implications of any changes to the overall NHS 111 telephone service and, specifically, the impact on clinical assessment teams

5. estimate the cost–consequences of implementing an NHS 111 Online service for the overall costs of the combined online and telephone 111 services and model the potential cost effects for the emergency and urgent care system.

The NHS 111 service

The NHS 111 service has been in operation since 2010, with the core function of providing a free-to-use telephone access service for people with urgent health-care problems that is available 24 hours per day, 365 days of the year. The service provides health information and an assessment service, which uses the NHS Pathways system to ask callers a series of questions about their problem and then direct them to an appropriate service using a local Directory of Services (DoS), provide telephone clinical advice or provide self-care advice. The initial assessment is conducted by a non-clinical call adviser and, where appropriate, the call can be passed on for further assessment by a clinician (usually a nurse).

As it has developed, the NHS 111 service has grown to include a secondary function of providing a single point of access for urgent care services, including out-of-hours services, to create a fully integrated urgent care access, clinical advice and treatment service. Integrated urgent care services have access to a wide range of general and specialist clinicians. They offer advice to health professionals in the community and to the ambulance service, manage patients over the telephone who have called NHS 111 and, in some cases, provide appointment booking services and face-to-face management of patients in treatment centres, the patient’s residence or another location, if required. Although a national service, NHS 111 is commissioned by Clinical Commissioning Groups (CCGs) using a variety of providers, including ambulance services, social enterprise organisations and commercial organisations. The geographical footprint of providers is very variable, ranging from large regional services covering multiple CCGs to small areas of two or three CCGs. The range and scope of integration with other services also varies by area and by commissioning arrangements.

The introduction of the NHS 111 Online service has essentially added another arm to the NHS 111 service by providing an alternative access point for people who need information and advice about what to do when they have an urgent health problem.

How does NHS 111 Online work?

The NHS 111 Online service has been designed so that people can get medical help or advice using a smartphone, laptop or other digital device. Users of the service can ask questions about their symptoms, get advice about what to do and where to go and, if needed, get further advice from a nurse, doctor or other trained health professional. The service can be accessed via a website (https://111.nhs.uk/) or the NHS App (https://digital.nhs.uk/services/nhs-app) for smartphones and tablet devices.

The online version of NHS 111 uses an adapted version of NHS Pathways to replicate the questions that are asked by the health advisers in the NHS 111 telephone service.

The implementation of the NHS 111 Online service has occurred in three phases:

• Phase 1 – people accessing the service can answer questions about their health problem and the results direct them to the appropriate service using the local DoS.

• Phase 2 – in addition to advice on appropriate service, users can, where needed, book a clinical call-back from the telephone NHS 111 service or integrated urgent care provider.

• Phase 3 – in addition to 1 and 2 above, users can receive a call back from services such as out-of-hours GPs, dentists, urgent treatment centres and other services, where the technology links have been put in place to accept referrals from the online service.

There are some differences between the online process and the telephone process. These are described in Table 1.

As with the telephone service, NHS 111 Online is a national service, but the outputs of a user’s interaction with the central NHS Pathways assessment system are customised and linked to the provider of the telephone and integrated urgent care service in their postcode area. So, for example, if a user is advised to go to an emergency department or other service, a list of local available services and opening hours is provided. If a call back with a clinician is booked, then this is passed to the NHS 111 service that covers their area.

Some early adopters of the NHS 111 Online service implemented the three phases sequentially, but areas that went live later implemented a full phase 3 service from the outset. The service was available nationally from December 2018. We spoke to nine NHS 111 provider sites, which identified the following main features of the set-up and implementation phases:

• Setting up and testing technical links for Personal Demographics Service matching. When a user of the NHS 111 Online service enters their details (date of birth, full home address, home telephone and emergency telephone number) for a call back, a Personal Demographics Service look-up automatically identifies the patient’s GP practice and NHS number. If a full match is found, this populates a document that is sent into a call queue at the NHS 111 service so that the clinical adviser has access to the patient’s details. If a full match is not found, the patient is allocated an individual patient identification number for the NHS 111 call. When the clinical advisor calls back, they clarify personal details in order to find the full match. Rates of automatic Personal Demographics Service matching vary between sites.

• DoS profiling so that the NHS Pathways disposition at the end of the NHS 111 triage session maps to the right service from which direct referrals (for call backs or appointments) can be made. The ability to make direct referrals to other services differs by site, depending on the local services available, but can include dental services, urgent treatment centres and GP out-of-hours services. The correct technical links (electronic transfer and e-mail requests) were also put in place. The range of services to which online users can be referred is likely to increase as the integrated urgent care hubs expand and incorporate more specialist services, such as mental health, end-of-life care and pharmacy.

• The process of project set-up and commissioning of the online service appeared to be relatively straightforward, particularly for the later sites, which described how some of the teething problems had been resolved by earlier sites. Project set-up phases were short, with the most resource-intensive work reported to be developing, testing and re-testing the DoS to ensure that all pathways ended in the correct dispositions. The technical links for referrals were tested robustly to make sure that they worked correctly.

• Sites reported having a clear plan of the actions required to set up the online service and clear instructions of responsibilities set out by NHS Digital. Discussions between providers and commissioners were uncomplicated, although some site leads reported that commissioners were concerned that the introduction of the online service may lead to increased demand. Site leads reported that detailed descriptions for dealing with clinical governance and clear guidance were provided by NHS England, which helped to oversee the robust testing phase.

• The pilot phase and early roll-out of the NHS 111 Online service involved significant work in adapting and testing NHS Pathways questions to make them as user-friendly as possible. The tone and language of questions were adapted to maximise how easy these would be for the general public to understand. The question development process aimed to ensure that there was as accurate a match as possible between the wording of the online questions and the scripted questions read out by the call advisers on the telephone service.

• The work required to implement the online NHS 111 service was absorbed into existing job roles and none of the site leads reported needing any additional resources or staffing to get the sites up and running. Some site leads expressed concerns about potential future cost implications if call volumes increase as use of the online service exapands due to the increased length of clinical adviser time for online NHS 111 users.

Database search strategies

Digital and online symptom checkers

The search strategy used in our recent systematic review15 was replicated. Search terms included symptom checker(s); self-diagnosis; self-triage; online, web based, electronic, digital, app; online diagnosis; web based triage; and electronic triage.

The following databases were searched: MEDLINE (via Ovid SP), EMBASE (via Ovid), the Cochrane Library (via Wiley Online Library), and CINAHL (via EBSCOhost). Searches were limited by publication date from 1 January 2018 to 22 May 2020.

Review process

The same process was used for both reviews. Titles and abstracts of references identified from database searches were scrutinised by one member of the research team and assessed for relevance using the inclusion and exclusion criteria. A randomly generated 10% sample of studies from the database searches was double sifted by another member of the study team. A single reviewer assessed full papers of potential inclusions, with a 10% random sample assessed by a second reviewer. Studies where there was uncertainty were agreed by consensus by the two reviewers.

Data extraction was carried out directly into summary tables, rather than using detailed data extraction forms. We used a simple broad template to summarise the key characteristics from each included paper, including the study design used; the population and setting; the main purpose and objectives, including the outcomes measured; and the key findings and conclusions. A formal quality assessment of inclusions was not carried out.

Findings were synthesised in a narrative summary that included commentary on the quality of the included studies. The main focus was on identifying any key new evidence that changed or strengthened the conclusions of the previous reviews.

Telephone triage

The results of the review sifting process are given in Figure 2. Among 46 full-text reviews, 22 papers were included for data extraction and narrative summary. Eight papers were duplicates included in our previous 2015 review and were not considered again. There were three review papers. Of the 19 inclusions presenting primary evidence, four (21%) each were from the UK and Australia, two (10.5%) were from the USA and nine (47.5%) were from the rest of Europe. By broad study design type, there were six (31.6%) observational studies, four (21%) mixed-methods studies, three (15.8%) survey studies, two (10.5%) each of cohort and qualitative studies and one (5.8%) each of modelling and uncontrolled before-and-after designs. The majority of studies were conducted in an urgent care setting (13/19, 68.4%), four (21%) were conducted in primary care and one each was conducted in an emergency department and an emergency medical services setting. A description of each of these 19 studies is provided in Appendix 1, Table 26.

FIGURE 2.

The PRISMA flow diagram for telephone triage.

Summary of findings

Of the three review articles identified, one19 was a review of 10 existing systematic reviews about telephone triage in urgent care settings. Seven of these systematic reviews were included in our own 2015 review. 3 As a result, the conclusions about the same broad themes concurred with our earlier work; that is, there is variation in the measured estimates of appropriateness of triage decisions depending on the definition used. Compliance with advice is variable and higher for emergency care (ED) or self-care than for primary care; patient satisfaction is generally high; overall decisions tend to be safe but higher risk for more serious conditions; and there is some evidence of impact on service utilisation, but this is mainly for primary care. Evidence is weak about the impact of telephone triage on access and costs, showing potential for only modest savings. One review20 focused specifically on telephone secondary triage of low-urgency ambulance (emergency medical services) emergency calls. Findings have been consistent that this process can divert suitable calls away from an ambulance response, although this rate varies from 52% to 83% depending on the setting and call type. The process has also been found to be safe, with very low reports of adverse events, high levels of patient satisfaction and the potential for cost savings through the reduction of unnecessary hospital admissions. The final review paper21 examined the evidence on the safety of triage decisions made by clinical and non-clinical assessors. It found that nurses had higher levels of appropriate referral rates (91%) than non-clinicians (82%) and were most likely to be cost-effective, although different triage systems were used. The authors compared nurses with non-clinical emergency medical dispatchers and concluded that the latter group were less effective clinical decision-makers, but the authors also acknowledged that this might not have been a useful comparison as the two groups perform different functions.

The 19 primary research studies identified addressed key outcomes relevant to telephone triage.

Digital and online symptom checkers

The results of the review sifting process are given in Figure 3. Among 18 full-text reviews, eight papers were included for data extraction and narrative summary. Three were review papers and five presented primary evidence: one each from Hong Kong and Australia and three from the USA. A description of each of these eight studies is provided in Appendix 2, Table 27.

FIGURE 3.

The PRISMA flow diagram for online and digital symptom checkers.

Of the three review papers, one was a systematic review of digital interventions for parents of ill children to enable decision-making for self-limiting infections. Three studies were included, with two triage services reporting sensitivity of 84% and 93% when compared with nurse triage decision for ED intervention and none of the studies demonstrating a reduction in the use of urgent care services. 41 A scoping review42 of self-diagnosing digital platforms synthesised 19 inclusions, although more than half were conceptual or opinion articles, with primary research representing specific conditions rather than urgent care. The authors identified significant research gaps in the use of digital interventions and the complexity in the number of applications and diseases considered, and that more research is needed to understand user experience. The third systematic review43 focused on digital triage for primary care problems and synthesised the same evidence included in our review,15 and concluded that evidence is limited, with more evaluation needed.

Three of the primary research papers were simulation studies using patient records to create vignettes to test symptom checker applications. One study tested five symptom checkers for hepatitis C and HIV (human immunodeficiency virus) infection and found that they had poor diagnostic accuracy; only 20% of infections were identified at the first diagnosis and 55% of the checkers did not identify either infection at all, with substantial variation between applications. A correct emergency care triage decision was more accurate for hepatitis C (59.7%) than for HIV (35.6%). 44 A study using case records of ED patients compared the accuracy of two symptom checkers with the ED triage category and found accuracy of 74% and 50%. Sensitivity for emergency cases was 70% and 45%, with low negative predictive values (43% and 24%), leading to the conclusion that the symptom checkers were not a suitable alternative to ED triage. 45 A study in Australia46 investigated the performance of 36 public access diagnostic and triage symptom checkers using medical condition vignettes. Diagnostic symptom checkers listed the correct diagnosis first in 36% of cases (range 12–61%) and in the top 10 in 58% of cases. Triage applications provided the correct advice for 49% of tests and were more accurate for emergency and urgent care advice (63% and 56%, respectively) than for non-urgent or self-care (30% and 40%, respectively). The authors concluded that the quality of symptom checkers was variable and that triage applications were generally risk averse, often recommending higher levels of care than were needed. 46

An experimental study47 comparing the accuracy of three symptom checkers for cough in an outpatient setting found that their diagnostic accuracy was poor when compared with physician assessment reference standard, with a 26.2% difference in correct diagnoses. A recent paper48 described the rapid introduction of a digital self-triage and scheduling tool for COVID-19 integrated into a primary care system. Users enter their information, and people who are asymptomatic receive information and advice is provided, whereas people who are symptomatic are triaged to four categories from emergency care to self-care and, where needed, directed to appointment scheduling or a telephone hotline. In the first 16 days of use, the tool’s sensitivity in detecting the need for emergency-level care was 87.5%, and there were early indications of reductions in unnecessary face-to-face visits, telephone calls and messages. 48

Data collection

Data on NHS 111 calls and online contacts were collected between October 2010 and December 2019 and between January 2018 and December 2019, respectively. The NHS 111 calls data were obtained from the NHS 111 Minimum Data Set, Time Series, to December 2019 and accessed online (URL: www.england.nhs.uk/statistics/statistical-work-areas/nhs-111-minimum-data-set/nhs-111-minimum-data-set-2019-20/) in January 2020. 49 The NHS 111 Online data were provided by NHS Digital. Call data provided monthly counts at the NHS 111 area code level, whereas online provided daily, individual-level data. As call data were at a summary level only, analyses of both data sets were conducted at this level. Both data sets provided variables for all calls and contacts; triaged calls and contacts; calls and contacts passed for clinical advice; and the final disposition for the call/contact as the advice about which service to contact or attend, or where no service was needed.

The call data were made up of 71 NHS 111 area codes and the online data were classified by both Sustainability and Transformation Partnership (STP) and CCG areas, which were mapped to 38 of the NHS 111 area codes. For historical call data, some of the older codes have been merged into newer area codes. These 38 area codes were the potential sites for our analysis. However, because of the way the NHS 111 telephone service is commissioned, geographical coverage of different service providers means that either the area codes can be made up of multiple STPs or some STPs can be split over two area codes. Because the NHS 111 Online service was introduced at STP level, this meant that not all online area codes could be included, as some STPs had different start dates. In addition, for the interrupted time series analysis, 1 full year of NHS 111 Online data were required; therefore, any area codes where the online service had not been operating for at least 1 year were removed. For consistency, the call data were capped at 2 years prior to the introduction of the NHS 111 Online service, so each area code had a minimum of 36 months of data. This meant that there were 18 area codes remaining for the analysis; however, these areas are spread across the country and represent a range of geographical, service size and provider types, so we are confident that they are a reasonable representation. It was unfortunate that the area where the NHS 111 Online service was first introduced (Yorkshire and the Humber) and, therefore, had the longest post-implementation time could not be used. This is because the NHS 111 telephone call data are reported for Yorkshire and the Humber as a single area, but there are 22 CCGs and the NHS 111 Online service became live at different times in these CCGs, meaning that we could not account for the time point of change in the interrupted time series models. A list of the 18 sites included is provided in Appendix 3, Table 28.

Outcomes measured

The primary outcome was the impact of introducing the NHS 111 Online service on the number of triaged calls to the original NHS 111 call service. Triaged calls are those where a call advisor assesses the call using NHS Pathways and excludes calls that, for example, just provide health information with no assessment.

The secondary outcomes were to explore the impact of NHS 111 Online on the total number of calls, the number of clinical call backs and the effect on the relative proportions of final dispositions and hence on other related services.

These primary and secondary outcomes can be split into those which affect the NHS 111 telephone service only and those that have an impact on the wider NHS system.

NHS 111 telephone service only:

• triaged calls

• total calls answered – including non-triaged calls for health information, and those where the caller terminates before triage

• clinical call backs – calls referred to clinical advisors for further assessment.

Wider NHS system:

• emergency ambulance referrals or advice to contact 999

• advice to attend an emergency department or other urgent care treatment facility

• advice to contact primary care

• advice to contact with community service

• advice to attend another service

• no recommendations to attend any service and self-care advice.

For calls to the telephone service, there is a specific data variable recording the number of calls passed for and receiving a clinical assessment. This is less clear for the NHS 111 Online data, as clinical call-backs are managed outside the online platform. The online data provided two data variables, a clinical call back variable (offered and sent or offered and failed to send) and the final disposition (advice) code (DX code) at the end of the questioning. There were 11 DX codes for ‘speak to a clinician from our service immediately’ as the final disposition. If a patient had both a clinical call back variable and a relevant DX code, they were assigned as having a clinical call back. The online clinical call backs were added to the telephone call clinical call backs as the outcome for this analysis.

The secondary outcomes, which look at the impact on the wider NHS system, were created in a similar way to the clinical call back outcome. The outcome was made up of a combination of the dispositions (DX codes) from both the call and the online data. For example, the outcome of all ambulance referrals is made up of all the ambulance dispositions in the call data plus all the ambulance dispositions to call 999 in the online data.

Statistical analysis

All analysis was conducted using R, version 3.6.3 (R Core Team, 2020; The R Foundation for Statistical Computing, Vienna, Austria).

Interrupted time series

To model whether or not the introduction of the NHS 111 Online service has had an impact on the monthly number of calls, an interrupted time series was used. However, unlike with conventional interrupted time series, a dose–response model was used. This means that instead of the number of calls being modelled as a function of the time after the launch of the online service, it was modelled as a function of the number of online contacts that month. The dose–response model provides an estimate of the reduction or increase in the number of telephone calls per online contact. The model also included some systematic components: an underlying time trend, a step change for when NHS 111 Online was introduced and ‘fixed’ seasonal effects (four levels: December–February, March–May, June–August and September–November).

As each area code had different start dates for the introduction of NHS 111 Online, each was modelled separately and a meta-analysis was used to determine the overall effect (see Meta-analysis). Given that there were 18 different area codes and a range of outcomes to model, we decided to use the same model for each site and outcome, but different models were used as a sensitivity analysis.

To determine which model to use, four area codes were chosen independently by two statisticians (RS and RJ) to test the models. Both statisticians chose the same four sites (Hertfordshire, Milton Keynes, North East and Nottinghamshire) as these represented areas with large to small numbers of calls. The models tested were Poisson or negative binomial, and whether or not the model was autoregressive (AR) was also examined.

To determine whether an AR model was appropriate, the primary outcome was differenced to remove the general upwards trend (see Appendix 3, Figure 19) and the autocorrelation function and partial autocorrelation function plots were investigated. It was agreed for all four models that an AR model was not needed but there may be some seasonality. Seasonality would be accounted for with the season variable which was prespecified to be included in the model.

As the primary outcome variable was the number of triaged calls to 111 each month, these are count data, so either a Poisson or a negative binomial model was considered. After fitting both the Poisson and the negative binomial models, the model output for the Poisson model showed that the data were overdispersed. Given this, the negative binomial model was chosen over the Poisson model. The autocorrelation function and partial autocorrelation function plots were investigated again and it was confirmed that an AR model was not needed (see Appendix 3, Figures 20 and 21).

The final model used for the analysis was:

where the outcome is the number of calls to the NHS 111 telephone service each month, time is a linear variable 0, 1, 2, . . ., dose is the number of NHS 111 Online contacts for each month, step is a binary variable that is coded 0 before the introduction of NHS 111 Online and 1 afterwards, and season is a fixed variable that represents the four seasons in the year.

For sensitivity analyses, an AR(1) model and a non-linear model with a non-linear term for time were used.

Both of these sensitivity models were applied to all sites and outcomes. The Isle of Wight was one of the 18 area codes included in the analysis. Owing to the size of the Isle of Wight, with small call volumes and an atypical urgent care service configuration, we have included a further sensitivity analysis from which we have excluded the Isle of Wight.

All three of these models included a log-link function. For the linear and non-linear negative binomial model, the glm.nb function of the Modern Applied Statistics with S (MASS) package was used. 50 For the AR model the tsglm function of the tscount package was used. 51

Primary outcome (triaged calls)

The primary outcome for the analysis was the number of triaged NHS 111 calls. Triaged calls are all the calls that were triaged which excludes both calls which were abandoned and those not triaged. Figure 4 presents LOESS (locally estimated scatterplot smoothing) plots for the four area codes in which the model was chosen.

FIGURE 4.

The LOESS plots of the number of triaged calls and online contacts for the four test area codes. Blue, triaged NHS 111 calls; orange, online NHS 111 contacts. (a) Hertfordshire; (b) Milton Keynes; (c) North East; and (d) Nottinghamshire.

Figure 5 presents the interrupted time series model for the four area codes for the primary analysis method [linear negative binomial with no AR(1)].

FIGURE 5.

The interrupted time series plots for the four test sites. Solid blue line, interrupted time series model; dashed blue line, null model (no intervention); solid dots, triaged NHS 111 calls; hollow dots, online NHS 111 contacts. (a) Hertfordshire; (b) Milton Keynes; (c) North East; and (d) Nottinghamshire.

Meta-analysis

Below are the meta-analysis results of triaged 111 NHS calls for the primary analysis method for each site and then for each sensitivity analysis overall.

Negative binomial generalised linear model

Figure 6a shows the forest plot of results for the primary analysis. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.013 (95% CI 0.996 to 1.029; p = 0.127). This means that, on average, for every 1000 online contacts, the number of calls to the NHS 111 telephone service that are triaged has increased by 1.3% (95% CI –0.4% to 2.9%). However, this result is not statistically significant.

FIGURE 6.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of triaged calls to the NHS 111 telephone service. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 71.5% (95% CI 54.1% to 82.3%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 6b shows the forest plot of the main analysis method and various sensitivity analyses. Excluding the Isle of Wight has little effect on the estimate. Including a non-linear term for time increases the standard error and lowers the estimates, but the overall conclusion remains the same. The AR(1) model provides similar incidence rate estimates and CIs.

Secondary outcomes

Total calls

Total calls is all calls offered to NHS 111, reflecting how many people attempted to contact the service. Figure 7a shows the forest plot of total calls for all area codes for the primary analysis method. The x-axis is showing the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.008 (95% CI 0.992 to 1.025; p = 0.313). This means that, on average, for every 1000 online contacts, the number of calls to NHS 111 has increased by 0.8% (95% CI –0.8% to 2.5%). However, this result is not significant.

FIGURE 7.

Forest plots showing the effect of introducing the NHS 111 Online service on the total number of calls to the NHS 111 telephone service. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 68.0% (95% CI 47.7% to 80.4%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 7b shows the forest plot of the main analysis method and various sensitivity analyses. Excluding the Isle of Wight has little effect on the estimate. Including a non-linear term for time increases the standard error and decreases the incidence rate ratio; there is now a 3–4% decrease in calls per 1000 online contacts, but the overall conclusion remains the same. The AR(1) model provides similar incidence rate estimates and CIs.

Clinical calls

Clinical calls are calls or online contacts that resulted in a disposition of call back from a clinical adviser. For the telephone data, the number of actual clinical assessments is recorded. For the online data, the disposition reflects a request for a call back but not whether or not this call back actually took place.

Figure 8a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 0.946 (95% CI 0.901 to 0.993; p = 0.025). This means that, on average, for every 1000 online contacts, the number of clinical call backs decreases by 5.4% (95% CI –9.9% to –0.7%). This result is considered a statistically significant effect, suggesting that, on average, the online 111 service has caused a reduction in the number of NHS 111 contacts (calls and online) that result in a clinical call back.

FIGURE 8.

Forest plots showing the effect of introducing the NHS 111 Online service on the total number of call backs or requests for call backs from a clinical advisor. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 94.5% (95% CI 92.7% to 95.9%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 8b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate. The non-linear model also has little effect on the estimate and CIs. The AR(1) model provides slightly smaller incidence rate estimates but the result is no longer statistically significant (p = 0.148).

Ambulance dispositions

One disposition at the end of an NHS 111 contact is to recommend an emergency ambulance response. Within the telephone service, this is facilitated by NHS 111 sending the request directly to the local ambulance service. For NHS 111 Online contacts the disposition advice is to call 999. The outcome for this analysis is the number of ambulance dispositions for both NHS 111 calls and NHS 111 Online. Figure 9a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.067 (95% CI 1.035 to 1.100; p < 0.001). This means that, on average, for every 1000 online contacts, the number of ambulance dispatches potentially increases by 6.7% (95% CI 3.5% to 10.0%) if all online contacts follow this advice. This result is considered a statistically significant effect, suggesting that, on average, the online 111 service has the potential to result in an increase in the number of ambulance dispatches overall.

FIGURE 9.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of ambulance dispositions. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 89.8% (95% CI 85.4% to 92.8%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 9b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate. The non-linear model also has little effect on the estimate and CIs; the estimates have decreased slightly, and this is similar for the AR(1) model.

Emergency department dispositions

The outcome of this analysis is the number of ED recommendations for both NHS 111 calls and NHS 111 Online. Figure 10a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.050 (95% CI 1.010 to 1.092; p = 0.014). This means that, on average, for every 1000 online contacts, the number of recommendations to attend has increased by 5% (95% CI 1.0% to 9.2%). This result is considered a statistically significant effect, suggesting that, on average, the NHS 111 Online service has caused an increase in the number ED recommendations overall.

FIGURE 10.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of recommendations to attend ED. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 94.4% (95% CI 92.4% to 95.8%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 10b presents the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate, and this is similar for the AR(1) model. The non-linear model changes the direction of the effect; however, this result is no longer significant (p = 0.110).

Contact with primary care

Being recommended to primary care can be a suggestion that a patient speak to or attend various different services. Patients can contact primary care (GP and related services) or they can be recommended to contact a pharmacy or dentist (community care). The analysis for this section looks at primary care and community care separately.

Primary care only

The outcome of this analysis focuses on the number of primary care recommendations for both NHS 111 calls and NHS 111 Online. Figure 11a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.051 (95% CI 1.027 to 1.076; p < 0.001). This means that, on average, for every 1000 online contacts, the number of primary care recommendations increases by 5.1% (95% CI 2.7% to 7.6%). This result is considered a statistically significant effect, suggesting that, on average, the NHS 111 Online service has potential to result in an increase in the number of primary care contacts overall.

FIGURE 11.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of recommendations to contact primary care. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 84.3% (95% CI 76.4% to 89.5%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 11b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate and similarly for the non-linear model and the AR(1) model. The non-linear model has slightly smaller estimates but is no longer statistically significant (p = 0.168).

Community care (dentist and pharmacy) only

The outcome of this analysis focuses on the number of dentist and pharmacy recommendations from both NHS 111 calls and NHS 111 Online. Figure 12a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.042 (95% CI 1.005 to 1.080; p = 0.027). This means that, on average, for every 1000 online contacts, the number of dentist and pharmacy recommendations increases by 4.2% (95% CI 0.5% to 8.0%). This result is considered a statistically significant effect, suggesting that, on average, the NHS 111 Online service has the potential to result in an increase in the number of dentist and pharmacy recommendations overall. This result is very similar to the primary care result.

FIGURE 12.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of recommendations to contact a dentist or pharmacy. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 86.5% (95% CI 80.2% to 90.9%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 12b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimates, and this is similar for the AR(1) model but the non-linear model has slightly larger estimates.

Attend another service

Another of the various dispositions at the end of a 111 contact is to be recommended to attend another service that is not ambulance, ED or primary/community care. The outcome of this analysis is the number of recommendations to attend another service for both NHS 111 calls and NHS 111 Online contacts. Figure 13a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.216 (95% CI 1.115 to 1.327; p < 0.001). This means that, on average, for every 1000 online contacts, the number of recommendations to attend another service increases by 21.6% (95% CI 11.5% to 32.7%). This result is considered a statistically significant effect, suggesting that, on average, the NHS 111 Online service has increased the number of recommendations to attend another service overall.

FIGURE 13.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of recommendations to attend or contact another service. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 94.9% (95% CI 93.2% to 96.2%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 13b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate, and this is similar for the non-linear model and the AR(1) model. Both the non-linear model and the AR(1) model have smaller estimates but are still statistically significant.

Not recommended any service

The final of the various dispositions at the end of a 111 contact is to be recommended not to attend any service. The outcome of this analysis is the number of recommendations not to attend any service for both NHS 111 calls and online. Figure 14a shows the forest plot of clinical calls for all area codes for the primary analysis method. The x-axis shows the incidence rate ratio per 1000 online contacts. The overall incidence rate ratio per 1000 online contacts is 1.034 (95% CI 1.002 to 1.067; p = 0.035). This means that, on average, for every 1000 online contacts, the number of recommendations to not attend any service increases by 3.4% (95% CI 0.2% to 6.7%). This result is considered a statistically significant effect, suggesting that, on average, the NHS 111 Online service has the potential to increase the number of recommendations to not attend any service overall.

FIGURE 14.

Forest plots showing the effect of introducing the NHS 111 Online service on the number of recommendations to not attend any service. (a) Estimated effects for individual areas and the overall average effect from the primary analysis (negative binomial GLM), heterogeneity I² = 87.1% (95% CI 81.1% to 91.2%); and (b) average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. GLM, generalised linear model.

Figure 14b shows the forest plot of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate, and this is similar for the AR(1) model. The non-linear model has smaller estimates and the result is no longer statistically significant (p = 0.474).

Limitations

There are some limitations to these analyses. First, because of the rapid roll-out of NHS 111 Online as a national service, we were not able to use an experimental design with a control arm. This means that we cannot establish if the effects we have found are the direct result of the introduction of the new service or if they would have happened anyway because of other factors influencing both the NHS 111 telephone service and the wider emergency and urgent care system.

Second, the compromise of using the NHS 111 minimum data set aggregated data rather than patient-level data for the telephone service meant that we were able to successfully match only 18 of the 38 potential area codes to NHS 111 Online data, so we have not been able to establish a national estimate of impact. However, we are confident that the 18 sites included are representative of different geographical areas, activity volume and provider types across England to allow reasonable inferences.

Third, we evaluated the impact of the NHS 111 Online service during the early stage of its implementation, and at the sites we used it had been in operation for only 12–18 months. We have therefore estimated system impact based on the ‘dose’, in terms of contacts with the new service, present at that time. As the service becomes more widely understood by the public and as contacts increase, it is possible the impact may change and any subsequent assessment of impact would be more robust.

Finally, as discussed above, we have considered only recommendations for care, not actual care accessed, which might be quite different and will depend on how people use the service and their subsequent utilisation of other services: both those in the emergency and urgent care system and those in routine primary care. This makes it difficult to estimate how much new demand there may be. This is discussed further in Chapter 10.

Data collection

Four data sets were used for this descriptive work.

Descriptive statistics

The NHS 111 MDS provides aggregated data only, so the Yorkshire and the Humber CUREd data were used as the source of the characteristics of the telephone user population, which were compared with the characteristics of those who used the online service.

Descriptive analyses comparing the final dispositions of both online and call data could be compared within our data set alongside overall summaries of the characteristics of the online data population for the 18 area codes.

For the more detailed analysis of the NHS 111 Online population, including the reasons for contacts, univariate analysis was used to describe the frequencies and distributions of single variables. Bivariate and multivariate analyses were used to compare different variables.

Demographics

Table 3 presents the demographic data for both the NHS 111 telephone call (January–December 2016) and 111 Online (January–December 2019) contacts for Yorkshire and the Humber. These data allow a comparison of the characteristics of those who choose to call 111 and those who use 111 Online.

For both calls and online, there were more female users; however, among online users this percentage is higher. The biggest difference was in age. Those who called NHS 111 tended to be older, whereas those in the 16–35 years age group used NHS 111 Online most often, at 61% of the population. A higher percentage use NHS 111 during the week than at the weekend for both calls and online, this percentage being higher for online. Online contacts had a higher percentage for all dispositions except for those recommended to not attend any service. However, this may be because of differences in how final disposition is recorded in each data set. Because of this, the number of callers offered clinician assessment cannot be included. In telephone data the disposition after clinician assessment is recorded, whereas in the online data this is an end disposition.

We have also compared the NHS 111 Online service dispositions with telephone call data for each of the 18 study sites used in the analyses in Chapter 4 (see Report Supplementary Material 1). Overall, the pattern of dispositions is the same, with ambulance, ED and contact primary care dispositions higher among the NHS 111 Online population. However, these comparisons should be treated with caution as the Yorkshire and the Humber data may not reflect individual site differences.

NHS 111 Online service only demographics by area code

Tables 4 and 5 present the characteristics of those who use NHS 111 Online in each area code. The percentages with each characteristic tended to be similar for each code, with almost 90% of online users overall in the 16–75 years age group. Less than 2% of contacts with the service were in the ≥ 75 years age group. Over 60% of contacts were made during daytime hours (08.00–20.00) and proportionally more were made during weekdays, although this is spread over 5 days rather than the 2 or 3 days for weekends and bank holidays.

NHS 111 Online contact characteristics

Triage completion and duration

These analyses were conducted using the early NHS 111 Online data set for January 2018–July 2019 (470,008 cases). We examined the method of contact and the completion rates of online triage sessions. The most common method of accessing the service was using a smartphone (77.8%). Less commonly, a desktop computer (15.3%) or a tablet (6.9%) was used. Figure 15 shows the rapid increase in service use during the first 18 months of the service and the mode of access.

FIGURE 15.

Monthly contacts to NHS 111 Online.

Table 6 shows the number and proportion of contacts where the triage was completed and session times.

TABLE 6 - Summary of NHS 111 Online contact completion processes

	Number (%)
Completion status
Not completed	25,807 (5.5)
Completed	444,261 (94.5)
Call back
No call back attempted	419,224 (89.20)
Call back offered but failed to send	657 (0.13)
Call back offered and sent	50,187 (10.67)
Completed triages during session
Not completed	25,807 (5.4)
1	394,292 (83.7)
2	50,751 (8.6)
≥ 3	9218 (2.3)

Most contacts (83.7%) were completed in a single triage session. It is possible to reach a disposition and then go back through the questions, answer them again and arrive at a different disposition. Of those completing more than one session, 8.6% used two sessions and a small number (2.3%) used three or more sessions. Eleven per cent of contacts were offered a call back from a clinician. Figure 16 presents the distribution of triage and session times and shows that the majority of triages were completed within a much shorter time, with the majority completed within 6 minutes.

FIGURE 16.

Distribution of NHS 111 Online triage duration. Triage 1, triage completed in 1 session; triage 2, triage completed in two sessions; triage ≥ 3, triage completed in three or more sessions.

Figures 17 and 18 show the distribution of contacts by hour of day and day of week. There are two peaks of activity during the day, 9.00–10.00 and 18.00–19.00, which coincide with the working day. Although proportionally more of the total contacts occur on weekdays, the distribution shows there is more activity at weekends.

FIGURE 17.

Distribution of contacts by hour of day.

FIGURE 18.

Distribution of contacts by day of week. 1 = Monday.

Limitations

We have not been able to undertake the in-depth analyses of patient characteristics and case-mix differences that we had planned, as we were unable to obtain detailed patient-level data for the telephone population. In particular, we could not compare the reasons for contacting the services to determine if there were different levels of acuity. We did use our own existing data to enable some comparisons, but these data are older and for a single area and so are limited in terms of generalisability.

Settings

NHS 111 Online user experience survey data were obtained from NHS Digital. Data were collected for all users who accessed the service between September 2019 and May 2020. We did not place geographical exclusions on these data, collecting user data from across England.

NHS 111 telephone user experience data were obtained for two geographical areas, Yorkshire and the Humber, and Hampshire, which were provided by Yorkshire Ambulance Service and South Central Ambulance Service, respectively. Data were collected during the same time period in both areas (between October 2019 and February 2020).

The questionnaires

Providers of both the NHS 111 Online service (NHS Digital) and NHS 111 telephone services undertake regular monitoring of user experience using questionnaires. Individual service providers determine the content of these questionnaires, the methods of delivery and frequency of this monitoring. Data are collected using various survey methods, which include post, telephone, online and text message.

All NHS 111 providers are mandated to ask a series of five questions, with closed responses. These form part of the MDS. The MDS questions and response sets were lifted, verbatim, from a questionnaire developed by members of the research team in an earlier evaluation of the NHS 111 telephone service:59

• How helpful was the advice given by the 111 service?

• Did you follow the advice given by the 111 service?

• Seven days after the call to the 111 service, how was the problem?

• Overall, how satisfied or dissatisfied were you with the way the 111 service handled the whole process?

• If the 111 service had not been available, would you have contacted another service about your health problem?

Respondents were asked to answer most of the questions using a closed response set. Some questions allowed for a free-text comment to allow respondents to elaborate on their closed response. For example, if a respondent indicated that they did not agree with the advice provided, they were asked to explain this further in a free-text box.

Additionally, we wanted to understand if, and why, some users were ‘channel shifting’, that is, using the online version of NHS 111 instead of the telephone service. We constructed a question in consultation with NHS Digital to be asked of online users. We adapted this question for use with telephone users to allow us to understand if there were instances when 111 users preferred telephone to online assessment (as well as vice versa).

Survey administration

Users of NHS 111 Online are invited to complete a questionnaire directly after their clinical assessment (the ‘live’ survey). Users completing the live survey are invited to input their e-mail address if they agree to receiving a second survey (the ‘follow up’ survey). The follow-up survey is administered approximately 2 weeks after their clinical assessment. In the study settings, telephone users received a postal questionnaire, which may have arrived a number of weeks after their contact with the service, which may have implications for recall among those users.

Questionnaire data were collected by each NHS 111 provider. An anonymised data extract was forwarded to the research team for analysis.

Respondent demographics

The demographics of patients were collected consistently across all providers (Table 11). Telephone and online respondents differed in age group (p < 0.001), ethnicity (p < 0.001) and presence of a long-term condition (p < 0.001). There was less evidence of a difference between the groups with respect to sex (p = 0.009).

There were fewer older NHS 111 Online users (i.e. aged ≥ 75 years) among the respondents. Overall, the proportion of respondents identifying themselves as ‘white British’ was higher than the population estimate for the UK (82%; Census 201160). This was particularly evident in the telephone user group. Overall, around half of the survey respondents had a long-term condition. This proportion was higher for online users (51%) than for telephone users (43%).

Previous use of NHS 111

Respondents were asked about their previous use of the NHS 111 service that they had contacted (e.g. telephone users were asked about their previous use of the NHS 111 telephone service). Given that NHS 111 Online is a relatively new service, it was not surprising to find that 93% (n = 1897) of respondents reported using this service for the first time. Sixty per cent (n = 446) of telephone users reported using the NHS 111 telephone service for the first time.

Perceptions of intended behaviour

Stakeholders are often interested in what people say they would have done had a service not been in operation. We should be mindful that this is a hypothetical question and that a perception of intention may not reflect actual behaviour. Service users were asked if they would have contacted another service had NHS 111 not been available (Table 12). Owing to differences in the question wording, some users were asked to tick one option only, whereas some users ticked as many as applied to them. We analysed the responses of respondents who indicated that they would have contacted a single service, rather than multiple services.

A small proportion of NHS 111 Online users reported that they would have made contact with high-acuity services such as 999 ambulance (1%) or an ED (3%); 43% reported that they would not have contacted another service if NHS 111 Online had not been available.

When considering differences between telephone and online users, we found that respondents using the telephone service were more likely to report contacting a higher-acuity service (e.g. the 999 ambulance service, ED or general practice) than those using the online service. Users of the online service were more likely to report that they would not have contacted another service if NHS Online had not been available. These differences persisted even after adjusting for age, gender, ethnicity and long-term condition (p < 0.001).

Helpfulness of, and compliance with, advice

Respondents were asked how helpful they found the advice given by NHS 111 (Table 13). This question was not asked of users of the NHS 111 telephone service in Yorkshire and the Humber.

Three-quarters (75%) of online users felt that the advice given had been helpful.

When considering differences between telephone and online users, telephone users were more likely to report that they found the advice to be ‘very helpful’, even after adjusting for age, gender, ethnicity and long-term condition (p < 0.001).

Respondents were asked about whether or not they had followed the advice given by NHS 111 (Table 14). NHS 111 Online users were provided with a dichotomous response set: ‘complied’ or ‘did not comply’. Telephone users were given three response options: ‘full compliance’, ‘some compliance’ and ‘did not comply’. To enable binary logistic regression analysis, we compared those reporting ‘full compliance’ with those reporting that they ‘did not comply’ with the advice.

Just over two-thirds (67.5%) of online users reported complying with the advice provided. When considering differences between telephone and online users, telephone users were more likely to report that they fully complied with the advice, even after adjusting for age, gender, ethnicity and long-term condition (p < 0.001).

Those using the online service were asked for the reasons why they had not followed the advice given. Respondents were asked to identify all of the reasons that applied to them (Table 15).

Around one-quarter (24%) of respondents reported that they had been unable to follow the advice. Similar proportions had wanted to wait and see if their symptoms changed, or had not wanted to waste the time of the service they were advised to contact. Around 1 in 10 of respondents could not get an appointment with the service they were advised to contact. A small number of respondents (4%) did not understand the advice.

Respondents who answered that they had been unable to follow the advice were invited to explain their answer further in a free-text comment; 55 responses were analysed. Over half of those who commented (n = 32) said that they could not access the service they had been advised to contact by NHS 111 Online. This was predominantly in relation to GP care, with a minority not able to access dental care. Some respondents said that they had been advised to contact a GP during the weekend, when their own GP surgery was closed. Four respondents had been unable to follow the advice given because of difficulties in arranging transport to get to the service advised by NHS 111 Online.

Satisfaction

Respondents were asked a number of questions that could indicate their satisfaction with NHS 111 (Table 16). Fifty per cent of NHS 111 Online users indicated that they were ‘very satisfied’ with the service; when considering differences between telephone and online users, telephone users were more likely to report that they were ‘very satisfied’, even after adjusting for age, gender, ethnicity and LTC (p < 0.001). Fifty-seven per cent of NHS 111 Online users reported that they were ‘very likely’ to recommend the NHS 111 Online service to family and friends; when considering differences between telephone and online users, telephone users were more likely to report that they were ‘very likely’ to recommend the NHS 111 service to family and friends, even after adjusting for age, gender, ethnicity and LTC (p < 0.001).

Just over half of NHS 111 Online users (56%) reported that they were ‘very likely’ to use the online service again. Telephone users were not asked this question. A total of 313 respondents (9%) indicated that they were ‘unlikely’ or ‘very unlikely’ to use NHS 111 Online again. Among these respondents, 60 free-text comments were analysed: 22 attributed their response to feeling dissatisfied with the information or advice offered by NHS 111 Online; eight respondents referred to a problem with the call back from NHS 111; and five respondents preferred the NHS 111 telephone service or another service where they could speak to someone.

NHS 111 Online users were asked if they had got everything that they wanted from the service. NHS 111 telephone users were not asked this question. Most respondents (84%, n = 3140) reported that they had received at least some of what they wanted from NHS 111 Online; 16% (n = 588) of NHS 111 Online users reported that they ‘got none of what I wanted’.

NHS 111 Online users were presented with a series of statements about the process of using NHS 111 and were asked whether they agreed with these statements (Table 17). We selected to dichotomise at ‘strongly agree’ versus all other options because research has shown that people who state that they are ‘satisfied’ (equivalent to, or similar to, ‘agree’ with positive statements) can identify some areas for improvement with the service, whereas those who state ‘very satisfied’ (equivalent to ‘strongly agree’) can see no room for improvement. 61

NHS 111 Online users indicated that there was room for improvement across each of the statements but particularly with regard to the relevance of questions and the helpfulness of the advice and information given. Only one process question was asked of telephone users: relevance of questions. This question was included in the questionnaire administered to Hampshire users only, but worded slightly differently (‘The questions asked by the 111 service were relevant’). When considering differences between telephone and online users, telephone users were more likely to report that they felt that the questions were relevant, even after adjusting for age, gender, ethnicity and LTC (p < 0.001).

Seven-day health status and use of services

Respondents were asked about any improvement in their health 7 days after their contact with NHS 111 (Table 18). Around one-fifth of users reported that they felt ‘completely better’ 7 days after their contact with NHS 111. When considering differences between telephone and online users, telephone users were more likely to report that they felt that their health problem had improved or was completely better, even after adjusting for age, gender, ethnicity and LTC (p < 0.001).

Respondents were asked to identify the services that they had contacted in the 7 days after their contact with NHS 111 (Table 19). Data from telephone users in Hampshire were excluded because of differences in the phrasing of this question.

One-third (31%) of NHS 111 Online users reported making no contact with a service in the 7 days after their contact with NHS 111. A small proportion (1%) of NHS 111 Online users made contact with the 999 ambulance service, and 7% reported visiting an ED.

In terms of differences between telephone and online users, NHS 111 Online users were less likely to report contacting the 999 ambulance service (p < 0.001), visit an ED (p < 0.001) or make any contact with a health service (p < 0.001) in the 7 days after contacting NHS 111. These differences remained even after adjustment for age, gender, ethnicity and LTC (p < 0.001).

Channel shift

We were interested to know why respondents had chosen to contact NHS 111 Online rather than the NHS 111 telephone service. A list of response options was provided. Respondents were asked to select one response (Table 20). NHS 111 Online users were given the opportunity to expand on their response by providing a free-text comment.

Over one-third (35%) of those using NHS 111 Online reported that they did not want to use the telephone service; 1114 respondents expanded on their answer by providing a free-text comment. Reasons included that the problem did not feel urgent enough; they did not want to waste NHS resources; they had difficulty speaking or preferred not to speak to someone (owing to either the health problem or anxiety about using the telephone); or perceived that there would be a long wait to speak to someone on the NHS 111 telephone service.

One-quarter of NHS 111 Online users (26%) reported not thinking about using the telephone service. Around 1 in 10 online users (11%) had tried to access the telephone service but found that it had taken too long for their call to be answered and so they had used NHS 111 Online instead. A small proportion (3%) of NHS 111 Online users had already spoken to the telephone service. Free-text comments suggested that many had gone on to use NHS 111 Online because of an actual or a perceived lengthy wait for a call back. A similar proportion (2%) of NHS 111 Online users were using the telephone and online service simultaneously. Fifty-three respondents expanded their answer by providing a free-text comment. Comments focused on delays in getting through to the telephone service: respondents felt that they would get a quicker response using NHS 111 Online.

Just over one-quarter (27%) of respondents who used the NHS 111 telephone service had not been aware that NHS 111 Online existed.

There were apparent differences when comparing telephone and online users (p < 0.001). However, there are difficulties in interpreting these differences. Almost one-quarter of NHS 111 Online users reported an ‘other’ reason for using the online instead of the telephone service; 709 respondents expanded on their answer by providing a free-text comment. A wide range of reasons were reported in these comments. These included difficulties in communication (owing to either their health problem or an underlying anxiety about using the telephone); taking a personal decision not to ‘burden’ the NHS 111 telephone service (particularly during the COVID-19 pandemic); perceptions of long waiting times to access the telephone service; receiving a recorded message to use the online service after calling NHS 111; and perceptions that the problem was not urgent enough to warrant a telephone call to NHS 111. These comments were similar to those provided in response to ‘I didn’t want to phone NHS 111’, so it is probable that we have underestimated the proportion of NHS 111 Online users not wanting to telephone NHS 111.

Limitations

The main limitation is the representativeness of the respondents, as white British people are over-represented when compared with population norms. There were also proportionally more respondents in the older age groups than in the user age profiles described in Chapter 5. This is particularly true of the NHS 111 Online respondents, as the largest proportion of its users are younger. It is possible that the user groups under-represented in the survey may have different views about the experience and might have made different choices about subsequent health contacts. The response rates to the surveys were low and therefore subject to responder bias, which means that the experiences of under-represented groups are not reflected in our findings.

The last 3 months of the survey period coincided with the beginning of the COVID-19 pandemic. NHS 111 rapidly adapted to provide triage and signposting for suspected COVID-19 symptoms, with information added to the NHS 111 Online website and an information message introduced for callers to the telephone service, including how to access the online service. This will have influenced some users’ reasons for contacting both services and created contacts from people who may not otherwise have used the service. Where possible, we have highlighted in the results findings that were specifically attributed to using the service for assessing COVID-19 symptoms, but it is possible that the experiences and responses reported may be different between users contacting the service about COVID-19 and those contacting for other urgent health problems.

Design

We undertook a qualitative interview study with recent users of the NHS 111 Online service.

Patient recruitment

We undertook semistructured interviews with recent users of the NHS 111 Online service. Interview participants were recruited following the completion of two online user questionnaires administered by NHS Digital: one immediately after use of the service and one 2 weeks later. On the second of these questionnaires, a web link to our project web page was provided, which invited survey respondents to read about and consider taking part in the interview study. After reading this information, potential interview participants were invited to complete a short online form, registering their interest. The form asked for their name, their contact details (telephone/e-mail address) and the date of their NHS 111 assessment. A researcher contacted these individuals, by telephone, within 1 week of completing the form (where possible), with the intention of undertaking the interview shortly after. Therefore, interviews took place at least 2 weeks after the individual’s contact with NHS 111 Online. Potential participants were given the opportunity to ask any questions about the study prior to deciding whether or not to take part. Interviews were booked over the telephone. Written confirmation of the interview date and time, a copy of the patient information sheet, and the consent form were sent to the interview participant by e-mail or post.

Data collection

Interviews took place over the telephone between December 2019 and April 2020. Informed consent was recorded at the start of each interview. The topic guide was developed based on our research objectives and discussion at our project management group. The topic guide covered general use of online services and health services, previous use of NHS 111 (both telephone and online), their experience of using NHS 111 Online, any preferences for using the online service ahead of the telephone service (and vice versa), and the likelihood of them using the online service in the future (see Report Supplementary Material 2). One researcher (JL) undertook all of the interviews. Interviews were digitally recorded and lasted between 24 and 55 minutes. Participants were sent a £20 high-street shopping voucher on completion of an interview.

Sampling

We planned to undertake up to 40 interviews. At the start of recruitment, we took an unstratified approach to sampling, interviewing all service users who had expressed an interest in participating. This resulted in a predominantly female, middle-aged, ethnically white sample, who reported having received a call back from NHS 111 after their online assessment. After completing 20 interviews, we took a more selective approach to recruitment, targeting younger service users and those reporting that they had not received a call back from NHS 111.

Data analysis

We audio-recorded each interview and transcribed it verbatim. Transcripts were checked for accuracy. We uploaded the finalised transcripts and summaries into NVivo 12 (QSR International, Warrington, UK), which was used throughout to facilitate data management and retrieval.

We analysed all interviews using framework analysis. 62 First, we familiarised ourselves with the interviews by reading a sample of transcripts. Second, we developed a thematic framework based on our research questions, the topic guide and reading a sample of transcripts. The thematic framework consisted of descriptive themes, including positive and negative aspects of the experience of using NHS 111 Online, compliance with advice, propensity to use NHS 111 Online again, and recommended future development of the service. Third, we coded all of the transcripts to the framework, adding further emergent themes and subthemes throughout the process of coding transcripts. One researcher (JL) undertook the coding of transcripts.

We (JL and EK) read extracts of transcripts within a theme, considering connections between themes and refining the thematic framework.

Demographics of interview participants

Data on the age groups of interview participants were collected. Around half (n = 15) of the participants reported being in the 45 and 64 age groups. Younger people aged under 35 years, whom we were particularly interested in recruiting, were represented in the sample by seven participants.

Two-thirds of interview participants were women (n = 22), and most interviewees reported their ethnicity as white (n = 28).

Most made contact with NHS 111 Online with regard to themselves (n = 30); one made contact in their capacity as a carer and one did so on behalf of a friend/family member.

Two-thirds of participants used the service during a weekend or weekday out-of-hours period (n = 21) and the remaining participants used the service during the day on a weekday (n = 11).

Participants described a range of health problems that were the focus of their contact with NHS 111 Online. A number of participants described symptoms that could be linked to COVID-19, with six contacting NHS 111 Online specifically because they felt that they were displaying signs of COVID-19. A further 12 participants described respiratory problems, such as breathing difficulties, cough, sore throat, and flu-type symptoms. Localised pain was described by a number of participants (n = 9), including dental, leg, kidney, groin and chest pain. The remaining four participants described health problems that were not shared by others: bleeding during pregnancy, skin problem, thumb problem and eye redness.

User experience and views

Recommendations for the future development of NHS 111 Online

As a relatively new service, participants offered suggestions as to how the NHS 111 Online service could develop in the future, and identified issues regarding current service delivery that might be changed.

Limitations

The detailed interviews with service users provided a rich source of insights into how people experience this new service, but there are some limitations. As with the survey reported in Chapter 6, the most important limitation is about representativeness. Participants were predominantly older, female and of white ethnicity. Although we attempted more inclusive sampling, it was difficult to engage the interest of those from other ethnic groups and from younger people, who are much more frequent users of the online service than those represented in the interviews. As the participants were identified from users who had initially completed the NHS 111 Online user survey recruitment was influenced by any respondent bias present in this survey.

The latter stages of recruitment for the user interviews coincided with the early stages of the COVID-19 pandemic. Concerns about this new and very specific potential health problem are likely to have affected the reasons for contacting the online service and the responses to the questions. It also means that some users who may not have previously contacted the service chose to use it as a result of national publicity. Nevertheless, the questions about experience remained relevant to main objective of this work package and provided some useful insights. In particular, using the online service to minimise use of other NHS services at a time when they were becoming increasingly under pressure was an important consideration for some of these later participants.

Recruitment procedure

Participants were recruited via a key contact at the NHS 111 sites. We asked sites to circulate an invitation letter, together with the information sheet to participants who had the role of health advisers, senior health advisers, clinical advisers, senior floor walkers, specialist clinical advisers or other management, as well as those with any other roles that the sites felt would be relevant. Participants were asked to contact us (FS) directly if they wished to take part. Our purposive sampling strategy aimed to recruit staff in a mix of roles, at a mix of sites and with different lengths of experience.

Recruitment took place between November 2019 and June 2020. After a low initial response rate, we submitted an amendment to the study ethics committee to allow us to provide participants with a Love2Shop voucher as a thank-you for taking part. We then resubmitted the invitations in February 2020. Sites reported low recruitment as a result of significant pressures on the service. Owing to the procedure agreed with ethics and to GDPR (General Data Protection Regulation), we were reliant on site leads sending out the invitations.

Recruitment was low during January because of the pressures that the system was under, and recruitment started towards the end of January, but was then halted because of the COVID-19 pandemic. Sites reported that they could no longer take part in non-COVID-related research owing to the pressures on the system.

The implementation process was rapid, and rollout occurred before our research was undertaken. We were unable to obtain sufficient information about the implementation process to enable us to apply a generic implementation framework to our analysis.

Data analysis

Telephone interview schedules were developed initially following conversations with the project management group and were discussed with patient and public involvement representatives prior to submission for ethics approval. Some additional questions were added following discussions with site leads for the data gathering exercise to understand the models of service delivery.

Interviews were transcribed verbatim and entered into NVivo version 12. Data were analysed using thematic analysis according to the principles of Braun and Clarke. 63 Interviews were read and re-read by Fiona Sampson. A sample of transcripts were read by Emma Knowles and Jaqui Long and a coding framework was developed. Analysis was based on a number of a priori themes: impact on workload/resource implications, impact on wider services, challenges to implementation, perceptions of how people use NHS 111 Online, value of the service/role in the urgent and emergency care system, future development and refinement, and safety. We also developed themes around trustworthiness and belief, knowledge and awareness, and integration.

Knowledge and awareness of NHS 111 Online

NHS 111 staff demonstrated a lack of awareness of, or misconceptions about, how NHS 111 Online worked. With the exception of those from commissioners or senior management, answers about the online service were often speculative and demonstrated low levels of understanding about, for example, how users receive call backs from the telephone service or how online referrals work. Staff spoke in uncertain terms when describing how the system works, using terms such as ‘I think’ or ‘I assume’. Even team leaders did not always fully understand how the online service worked and was integrated into the rest of the integrated urgent care system. For example, this clinical team manager describes how the system works, including their perception of how most online interactions are referred into the clinical queue:

Yeah, so if, if a, when they fill in the online form, I, I mean I’ve gotta say I’m not 100% about it, but, erm when they fill in the online form, if they need a, an appointment for a dentist, err they come through to us if it’s a small timeframe. If they need primary, primary care, they, they come through to us, pain management, not quite sure about pain management actually. Erm, no I don’t, know, I don’t think they come through to us. Any ambulance disposition they’ll get advice to call 999, but they also come through to us. Erm, any, you know, dispositions that are ‘speak to clinician’, from operator-generated ones, will come through to us. Erm, so, ya know, like you say the, the online will give its own information, but there’s very few, as far as I can tell, that don’t end up being safety-netted by ending up in the queue.

SH01 clinical team manager

Erm no, the NHS online is a lot shorter version than questions that we have to ask. We have to go through a full assessment, it’s not actually–, obviously because I’ve used it myself at one point, it’s not actually a full assessment of what we would normally ask in a telephone.

SH10 health adviser

We haven’t had a full briefing on what the system does or how close it is to pathways. It seems when I’ve done one example of something, it seemed to come out fairly similar to on [telephone] pathways.

SH04

Most of the interviewees had not received information from their managers about how the NHS 111 Online service worked. Many had found out about the service by chance, for example when a caller to the telephone service reported having used the service, when the online call back requests started ‘popping into’ their call queues, or when they received queries from other providers who had received a referral from an online user. Some participants reported that they had received an e-mail from their manager telling them that the online service was going live and to expect some referrals from this, but they had been given little detail about how this was to happen:

We actually, the first knowledge I had really was from a patient to say, you know, we’ve done this, we’ve done the online 111 service (OK). So I’ve not really had any background into it. And then I asked, obviously, because I thought, well, something I hadn’t actually heard of it.

SH08

No to be honest. I don’t know much and it’s not something that is spoken about, not even in the training I: [. . .] It’s not something oh by the way people can do their assessments online (OK). It’s not mentioned, not something that we talk about.

SH06

Engagement with NHS 111 Online was low; some participants had had a ‘play around’ with the online service to see how it worked and others had not looked at it at all. One participant reported having used it for a genuine problem because of the length of time that they had known they would have had to wait for the telephone service. This lack of knowledge about NHS 111 Online and how it worked may have had an impact on participants’ perceptions of the impact of the online service on their workload. Similarly, staff were unclear about the purpose and role of NHS 111 Online in the wider urgent and emergency care system. Whereas some referenced the online service as being intended to reduce demand for the telephone service, others talked about it as offering an additional, rather than a replacement, service:

So do you feel that the online service, from what you know, has worked as anticipated?

I never knew what the goals were. That’s what I’m saying. They never implemented it like they do others things and pathways.

SH05

Well I think from my perspective it hasn’t reduced the amount of activity going through, but we didn’t necessarily set out to reduce activity. What it has done, of course, is to have allowed certain individuals to, whoever wants to use the service, to possibly use it as a precursor to dialling 111 because people like to get the same info– like to be reassured by getting the same information or same answers from different sources (interviewer: right), whether it’s Google or anything else. So people do use the service before they dial 111.

SH14

Impact of the introduction of NHS 111 Online on the workload of NHS 111 staff

Overall, participants perceived a limited impact of the introduction of NHS 111 Online on their workload or job roles. This was attributed in part to the small numbers of patients being referred to the telephone service for call backs, with staff each seeing a handful of calls per shift. However, there were three ways in which the NHS 111 Online service was reported to have had an impact on the workload of the telephone service: (1) increasing clinician time required for online referrals; (2) increasing health adviser time when users of the online service called the telephone service to clarify advice given online and (3) responding to other services requesting information about an online referral.

Impact on wider services

We specifically asked about the potential impact on wider services, in part to understand whether we should speak to staff and stakeholders in those wider services. Interviewees did not perceive that stakeholders would have seen any impact of the introduction of NHS 111 on any wider services, or would be able to differentiate between whether people had used the NHS 111 telephone or NHS 111 Online services:

And the problem is this, it’s difficult to differentiate whether it’s online or [telephone] advice to be honest with you. I assume there’s probably an element of both. It’s just impossible unless we were to sit down and actually do survey of how many come through, it’s be impossible to know.

SH02

There were some speculative concerns that people would use NHS 111 Online as a shortcut to clinical advice, as they could manipulate the questions to get through to a clinician or to get an out-of-hours GP referral.

Perceptions of channel shift

Participants across all sites did not present any consistent perception of an increase or a decrease in demand for telephone service or evidence of channel shift from telephony to online, describing the overwhelming increase in call volumes and limited volume of online calls being too small to make any impact. Participants described how demand had increased significantly for the telephone service, in line with demand for face-to-face urgent and emergency care services, amid a significant ‘winter crisis’. This increase in demand meant that it was difficult to attribute any changes to the introduction of the online 111 service:

Well the workload, I mean, it’s hard to separate it out at the moment. Our workload at the moment is colossal. I remember seeing such an increase in calls coming through out of, through 111 we were up on average about 65% just in the last month. Because I haven’t checked the figures on 111 Online, it could be that that is increasing as well, my feeling is that it is, certainly in the last 12 months onwards in reviewing figures monthly, they are going up each month, the number of users online you know, like I say whether it’s a mix, it’s just so busy at the moment.

SH03

Well we have seen call volumes go up. So of course that’s very hard then to say well they’ve gone up because of online 111. Or they’ve gone up because they would have gone up, as a natural. I mean there may be some piece of work that will come through to us and let us know at some point. But at the moment we don’t understand it, the impact.

SH07

Some participants felt that the online service had the potential to reduce telephony demand, and used the fact that people were using NHS 111 Online (e.g. when receiving calls from out-of-hours providers chasing online referrals) as evidence of some channel shift. One user described evidence of increasing volumes as an indication of new activity:

So, you say the, it should have value in terms of taking pressure off call centres?

Yeah, definitely.

Do you see that happening at the moment?

Yeah, I mean it’s hard to tell because obviously the, our pressures go up and down so it’s hard to tell like an exact reduction but, there’s definitely people using the online service because the calls are coming on to our queue. So, even if it’s only a small reduction it’s definitely working it’s just scaling it up hopefully.

SH09

Some staff, particularly senior staff and commissioners who were involved at the implementation phase, reported that initial concerns about a potential increase in demand for the telephone service following the introduction of NHS 111 Online appeared unfounded. However, others expressed concern that the online service had induced demand, rather than shifting demand, evidenced by people using both the online and the telephone services:

I think the early aim was to try and get people to not have to phone through to us but also have more of an idea what’s happening, what NHS 111 does as well. I also think it was set up to help other health advisors to not have as much of an influx of calls, but I think that doubles that problem. I think they do use NHS 111 Online and then they call up say, ‘Well, NHS 111 Online told me this’ and we’ve told them something else. Then they’re wanting to either complain or to complain about 111 Online and it’s just a massive circle that wouldn’t need to be there if they just called 111 up, we just went through an assessment and then we would have got the appropriate care to them.

SH10

So we have an increase in volumes that are coming through to us that need clinical call backs, we did, hoped that we would see a sort of channel shift and that we would see a reduction in the number of cases coming through via the telephones. And potentially offset the costs associated with increased activity with our clinicians by reduced activity with the call handlers but we’ve not seen that, it’s just been new activity I think. In fact our, our telephony calls are going up and up and up. So yeah it has been impactful in terms of there’s extra stuff for people to do, I think previously, I think it’s just new medium. It’s a bit like when 111 was introduced you know, new people use it rather than old people changing their behaviours.

SH03

Again, perceptions of channel shift or changes in demand were based on personal experience rather than data, and many participants revealed a lack of information coming from management with regard to the volume of calls from online. Some expressed a desire for further information about online user volumes as well as telephony volumes and the interaction between the two services, as this could help them to understand demand and the role of the online service:

Well I, we haven’t had anything in our, like, higher management and we haven’t had any thing it’s not you know, you’re not in our like staff (yeah), we get like statistics like every week it’s never been put in there which I don’t know if they could do, it might work.

Yeah would you find that useful?

Yeah ‘cause then at least then we get to know like how many are still using the phone 111 service (yeah) and the online together (yeah), going to doctors and the out-of-hours service err just so we can get a guide of, I don’t know, just like good to know as well.

SH11

What, what we’ve seen is a growth in that activity anyway on the on– to, we’ve seen a growth in online and we’ve seen a growth in 111 usage anyway. And, but we’re not able to say there’s a– I don’t think we’ve got anything that tells us there’s a (pause) we haven’t got enough information or the right information to tell us how that, how the two things are, two elements of the service are interacting (Interviewer: right) at this point in time.

SH14

Role and value of NHS 111 Online

Safety concerns

Future development of NHS 111 Online

Participants felt that NHS 111 Online had an important and increasing role in future NHS services, with online health advice seen to be the ‘the way things are going’. Although there was no consensus about whether or not the increasing use of NHS 111 Online would reduce the workload of the telephone service, some participants felt that the online model was vital, particularly given difficulties of NHS 111 staff recruitment and retention:

It has to, well, the answer, the answer really is that it, it– given the challenges we have around workforce, then it has to feature very heavily going forward in an integrated service offering, alongside the telephone-based model, or even video-based model.

SH15

Participants felt that there was a need to improve public awareness of NHS 111 Online, particularly by using a realistic advertising campaign to show the public what online and telephone NHS 111 can and cannot do. In addition, further work and education to convince staff of the value to the online service to enable them to recommend the service to patients contacting the telephone service.

It needs more airplay. And we need more education at the coalface so that we can actually sell it with conviction to family and friends really. But certainly I can’t see the take-up in the role that I have and certainly I think my colleagues would agree. So when I say that I’m not saying it loosely (yeah), you know, I just think it should be, you know patients should be made more aware of it really. Like on the advert. Well, have you thought about instead of ringing us, why don’t you make your referral online? [. . .] And actually with the call volumes that we have, that would be something that’d really help. It would reduce our calls considerably.

SH05

I’ve not noticed any impact at all on the call volumes. To be honest, in my opinion, I don’t think it’s been advertised very much because most of the people that I talk to, like staff and public, don’t know about the online service. So, I don’t know whether or not that’s something that’ll come with time or, I suppose it’s similar to when we started, that hardly anyone knew and then word of mouth and that it gets round. But, I’d say most of the population don’t know that there’s an online version.

SH09

Participants also felt that it was important for staff to have an understanding of how NHS 111 worked so that they could understand the system and promote it to users. Improving the intelligence and data available to staff would also help them to understand demand and channel shift. There was also a suggestion that improved data on pathways of care would help staff understand how to improve the service and pathways of care:

So how about the NHS 111 Online, do you see that, how do you see that developing?

I don’t know. I can’t really offer anything on that because I don’t know enough about it. Having never used it and had no access and no training about it, I don’t know.

What training do you feel, if any, would be useful?

Well it would be useful if we’d all sat and perhaps had a dummy run on one. If they’d got a test server where we could all go on and see how it works, see the kind of information it’s asking. See how it relates to what we do (OK). I mean we spend a lot of time on our pathways training for running the assessment, does the online system kind of do the same? Is it asking those kind of questions? Does it have the same algorithm built into it? So if somebody was on the phone to me and at the same time doing the online service, would we be doing the same steps? (Yeah OK) That would be interesting to know (right). And now you’ve made me think about it so I’ll have to ask that question at work! [laughs]

SH07

But we’re not able, we haven’t got the analytical ability to be able to go back and actually go back and look and see whether that was the right, the right thing to do. So, in my view, we need more research about the pathways of care that people are on.

SH15

Other suggested improvements included continued refinements to Pathways to improve the online algorithms and improved direct booking of appointments (e.g. in-hours GP appointments, repeat prescriptions). Improved and transparent communication between NHS Digital and individual sites and commissioners was suggested as important to future service development and particularly in providing services to meet the needs of local populations:

I do think, I do think that it’s interesting that different areas work in different ways. Because of 111 Online being a national service and following all the national guidelines, NHS England guidelines, national disposition, it is challenging for, for me, as a co-ordinator in the [area], because we have lots of nuances, I have commissioners that want a different ring back time or they want somebody to go to this area rather than that area and that’s outside the standard. [. . .] Inner-city [area] is very different to rural [area] and the needs of the patients in those two different areas are very different, so commissioners want different things. And that’s not something online can really adapt to, it has to be the national guidelines and I understand that. But I do hear that as a frustration.

SH16

So I think the, so the, I think the challenge for us is– in a sense it’s been set up, that the, the computer algorithms will be amended by NHS Digital. Are NHS Digital involving commissioners in that? Not to, not to any great. So, and, though every impression that I get from those events I’ve been to with NHS Digital is that actually the patients are quite happy with the service, and as I’m sitting here today talking to you I’m not aware of any issues (I: right). I’m not certain that that’s the case.

(SH15)

Impact of COVID-19 on perceptions of NHS 111 Online

The majority of interviews took place prior to the COVID-19 pandemic, and the substantial role that NHS 111 played in managing demand for health care during the pandemic meant that we were unable to finish recruitment during this period of the study. We undertook two interviews after February 2020 in which participants suggested that the role of NHS 111 Online may have changed during the pandemic, in terms of both increased awareness of the online service and the value of the service. One participant suggested that the online service was more trusted after COVID-19, and that the demographic using the service had changed, perhaps because they saw the value of the service once they had understood how many cases were going through the online service, and also possibly because older people were now more familiar with technology:

I do see it continuing in the future because I think the culture of technology has changed for lots of different generations now because of Zooming and Microsoft Teaming, and people that were technophobes that feel more comfortable it, and accessing stuff online.

SH16

There were suggestions that the increased role of NHS 111 Online in managing demand during the COVID-19 pandemic may have increased the level of respect for the service:

So I think it’s different, I think in answer to your question, I think the perception of how it’s used is different, I think it’s more trusted. You know the advertising of the politician saying on national news, go on 111 Online trust it, use it, let be the guide that tells you what to do next if you’re concerned or needing more information.

SH16

Limitations

There were some limitations to this work. Despite sites writing to up to 200 staff at each site, we were unable to recruit sufficient staff to undertake the interviews. Recruitment of staff to studies has been recognised as increasingly problematic in recent years because of increasing pressure on the system. 64 In addition, the lack of awareness about the online 111 system may have acted as a barrier to staff participating in the interviews. The impact of COVID-19 on the NHS 111 service also limited staff’s ability to take part in research.

Data

The proportions of each service used are taken from the analysis of user surveys in Chapter 6. The only additional service use data used in the model are the proportions of patients offered a clinical call back from the 111 service, which are 17% and 12% for the telephone and online services, respectively. The source of the call back data is the NHS 111 MDS, and our analysis is reported in Chapter 5.

The unit costs of each service were primarily taken from standard sources, using 2018/19 prices. The exceptions to this are the unit costs for NHS 111 telephone, NHS 111 Online and clinical call back, as there are no standard reference costs for these. The estimated costs of the online service were provided by NHS Digital. Costs of the telephone services are more complex as the services are commissioned locally through competitive tendering processes and the costs are commercially sensitive. Via our personal contacts we obtained the likely costs from two CCGs and from NHS England, which are confidential, and used these to provide an estimated cost (Table 22). The cost of ‘other service use’ was derived using proportions of different services indicated in survey responses (see Table 22).

NHS 111

In the simplest terms, people with a health problem use an urgent care triage service because they have two questions: (1) what is wrong with me?; and (2) what do I need to do? There is now a choice of ways to access NHS 111. Some people who might have used the telephone service can now choose to use the online service instead (substitution); some may use both services, using one to confirm the decision received from the other (duplication); and some who would not have used the service at all might use the online service because it is quick, convenient and instant or because it provides a more accessible way to get help, for example where language is a barrier to using the telephone service (new demand).

We have taken the approach that NHS 111 is a single service that now has two points of entry and, hence, two streams of activity that can be pooled. The primary outcome was the impact on activity to the NHS 111 telephone service for every 1000 contacts with the online service, that is, whether or not the trend in demand for the telephone service changed following the introduction of the online service. Using pooled data from 18 sites, we found that the online service had little impact on the number of triaged and offered calls to the NHS 111 telephone service during the first year of operation, indicating that the workload for this service did not increase or decrease as a result of the introduction of NHS 111 Online. This suggests there has been no substantial shift to using the online service instead of the telephone service. There was variation between individual sites, indicating that in some areas activity for the telephone service has increased (duplication) but in others it has decreased (substitution). However, the changes were very small, indicating that the majority of contacts with the NHS 111 Online service are ‘new’ activity from the perspective of the total number of people accessing the NHS 111 service for advice. The interviews with staff suggested that there had been no obvious shift following the introduction of the online service, at least in terms of reducing call volumes; this varied between sites, but staff had experienced calls from people who had used the online service and then called the telephone service to confirm the advice given online (duplication). There was also some evidence of duplication in the survey of, and interviews with, service users.

Some contacts with NHS 111 result in a clinical assessment as a result of either the call being passed to an NHS 111 clinician as part of the telephone triage or a request for a clinician to call back (from an NHS telephone service clinician) an online contact. This creates the possibility of additional work for clinicians. Our analysis found an overall reduction in clinician contacts after the introduction of the online service. However, we are uncertain about whether this is a genuine effect, perhaps because users of the online service have lower-acuity problems, or because we have not accurately estimated the number of clinician call back requests from the online service data. The experiences of staff did not indicate any substantial increase in clinician workload, although staff cautioned that this may no longer be the case if the volumes of contacts with the online service continue to increase. Staff also reported that not all requests for a clinician call back take place, as patients do not always answer when they are called. NHS 111 staff also reported that clinician call backs took longer for online users than for telephone callers, but we were unable to test this using our data.

Emergency and urgent care system

Currently, NHS 111 is primarily a triage and signposting service and does not provide direct care, although it can facilitate care by, for example, arranging appointments. Of course, the integrated care hubs that are increasingly connected with the NHS 111 service do provide care, but, for the purpose of this evaluation, the signposting component is the focus. Each contact results in a disposition – a recommendation – about which service to contact or attend (unless no service is needed) and, hence, results in the potential use of an emergency or urgent care service. We assessed what effect the combined activity from the NHS 111 telephone service and online service has on the overall number of recommendations to contact or attend different services following the introduction of NHS 111 Online. We found that recommendations to attend one of the main emergency and urgent care services (emergency ambulance, ED, primary care and community services) increased by between 4.2% and 6.7% for every 1000 NHS 111 Online contacts. The biggest increase, of 21.6%, was for other services outside the usual emergency and urgent care system, such as opticians and sexual health clinics, but these accounted for a very small number of contacts. However, this does illustrate the increasing breadth of services that are becoming available as part of integrated urgent care systems. Overall, the findings suggest a potential net increase in demand for emergency and urgent care services, a finding that has been reported previously when new services have been implemented. 56–58 However, this analysis reflects recommendations for care. The extent to which this translates into actual utilisation of services depends on patient factors, including their original intentions about service access before they used NHS 111 and the actions they take afterwards. There are a number of possible scenarios, and the user surveys and interviews provide some insights into user intentions and actions:

• Some people may not have considered using any service – 43% of NHS 111 Online users reported that they would not have contacted another service if NHS 111 Online had not been available, compared with 5% of telephone service users. This represents potential new demand, although some may have still chosen not to access a service. Some of this ‘new’ demand might also have been from people who had previously found it difficult to access services and for whom the online option provided a more accessible entry point. This is an important distinction in avoiding the interpretation that all ‘new’ demand is ‘unnecessary’ demand.

• Some people may have intended to use a service but used NHS 111 Online to check and confirm this action. Any subsequent service contact would be not additional utilisation (although it might be for a different level of service) but an extra step added in their care pathway. Some may decide not to access a service, resulting in a potential saving. The survey did not ask this question directly, but 32.5% of NHS 111 Online users reported they did not comply with the advice given and half of these did not access any service. Interviewees who had used the service did describe using the service in this way.

• Users who contact NHS 111 because they are uncertain about what to do can follow the advice and use the recommended service, decide to use a different service (which may be at a higher or lower level than recommended) or decide not to contact any service at all. Only 4% of telephone service users did not comply with advice, compared with 32.5% online users. In the 7 days after the contact, 31% of NHS 111 Online users used no service compared with 16% of telephone service users. There were some reports in the user interviews that recommendations for higher-level care, such as an emergency ambulance, were perceived as inappropriate and unnecessary.

It is entirely plausible that introducing this additional access point for emergency and urgent care will create some extra demand; an additional 2 million contacts were created during 2019. The integrated findings show that, although there is potential for demand to increase, it is unlikely to be of the scale suggested by the interrupted times series analyses, as a substantial proportion of the NHS 111 Online user population limit their use of services. One objective of the NHS 111 service from its inception has been to try to reduce some of the unnecessary demand on emergency (999 ambulance and ED) services,7 so additional demand would be counterproductive. The available data do not allow us to determine the clinical appropriateness of NHS 111 Online dispositions to emergency care. However, the evidence from this study suggests that only a small proportion of service users subsequently access emergency services. Previous research has also demonstrated a degree of risk averseness within the NHS Pathways assessment system, with GP review judging that 73% of ED dispositions are suitable for lower-level care. 28 It is, therefore, likely that potential increases in emergency service use are less concerning than they first appear. There is, however, a population of people now accessing services who would not have done so if the NHS 111 Online service was unavailable, representing potential new demand. The findings from the surveys and interviews also indicate that there may be a case-mix difference between the telephone and online populations, with overall lower acuity in the online population and the impact of new demand more skewed towards primary care and community-based services.

Demand

Previous research has shown that the introduction of new services and access points for emergency and urgent care has created an increase in and, therefore, new demand for services, and it is likely that adding another new service has had the same effect. However, the findings of the previous research were based on data of actual utilisation of other services in the emergency and urgent care system. Our analyses were limited to estimating demand using service recommendations, and the survey results have shown a gap between recommended and subsequent service use. Better patient-level data that links NHS 111 service use to subsequent service utilisation is needed to properly estimate the impact of NHS 111 Online on the emergency and urgent care system. This would also allow a better understanding of the relationships between new demand, service substitution and service duplication. However, there is a barrier in that the current configuration of NHS 111 Online would not support this as, unless a user requests a clinical call back or is provided with an appointment, the only personal information collected is age and postcode, which would be insufficient to link to health records.

User characteristics

The lack of patient-level data prevented us from examining more closely the characteristics of the two NHS 111 populations. Understanding these characteristics, including the types of problems for which people contact each service, may provide better information for users when they have to choose whether to use NHS 111 Online or the NHS 111 telephone service.

Triage

The NHS Pathways system is well established and has been adapted to operate on a remote digital platform. There are clear advantages to having a single interoperable system across both NHS 111 services. Our research has shown that there is a gap between recommendations from the triage system and the action people subsequently take. For the online service, the proportion of contacts with a higher service level disposition of emergency ambulance or ED was similar to the telephone service, but uptake was much lower. Within the telephone service, calls with an initial disposition of ambulance or ED are reviewed by a clinician (revalidation) to reduce unnecessary ambulance dispatches and ED attendances. With the online service, this option is less straightforward. In some areas, assessments that arrive at a ‘call 999 ambulance’ disposition will offer a call back, but, at the time of our study, this was not in place in all sites. However, it is unclear if this is still recorded as an ambulance disposition in the online service data. The number of ambulance dispositions may reduce as there is further integration and clinician call backs increase. Nevertheless, it was clear from responses to the survey and interviews that some patients are themselves, to some extent, ‘revalidating’ these recommendations by choosing not to access these services. All triage systems where there is no face-to-face contact have a level of risk averseness, and this is certainly a feature of digital systems where there is no human interaction. 15 Further scrutiny of the online triage process to reduce overtriage would likely improve confidence in the service for users and reduce potential unnecessary demand on emergency services.

Clarity of purpose

Despite sharing a common name (NHS 111), ‘NHS 111 Online’ and ‘NHS 111’ are different services, and although some people may be able to use either service successfully others will have a preference for one or the other. Choosing the ‘wrong’ option increases the risk of duplication. There is a clear value to providing an online service; one-third of NHS 111 Online respondents used the service because they did not want to contact NHS 111, and it also met a need of groups of the population who prefer services delivered online rather than over the telephone. Users also appreciated the value of the online service during the COVID-19 pandemic as they did not want to make unnecessary contacts at a time when the NHS was under severe pressure. However, at the moment the online service does not seem to meet some users’ expectations. If a service meets expectations, then people are more likely to use it again and recommend it to others. If one objective is ‘channel shift’ – people using the online service instead of the telephone service – then it may be helpful to provide more explicit information to users about what the online service can provide and when to use it. To support more effective use of the service and improve user experience and satisfaction, some further development of the online service could result in more benefits, including cost savings, as we have shown that these can be realised only if there is substitution of services. Finally, the importance of an integrated urgent care system cannot be overemphasised. Some dissatisfaction arises from users being unable to action the recommendations of the online triage in terms of appropriate service access. If these do not match up, then people will continue to self-present to higher-level services, which they perceive to be always open, even if this level of care is not needed. 17 The NHS has committed to developing integrated urgent care services8 and has published a detailed service specification describing what this should look like. 71 Historically, NHS 111 has been a separately commissioned and delivered service that refers callers to other, external services, such as GP out of hours. Individual services have formed more integrated partnerships, but there is now a national ambition to develop the ‘provision of a functionally integrated, 24/7 urgent care access, clinical advice and treatment service’. 71 This will be staffed by multidisciplinary teams that include GP, pharmacy, mental health, dental and palliative care clinicians with NHS 111 (telephone and online) as part of the integrated urgent care service as the ‘front door’ to access. This means that users should expect their problem to be managed and resolved in a single encounter, described as a ‘consult and complete’ model of service delivery rather than ‘assess and refer’. The successful implementation of integrated urgent care services will be a central component of maximising the potential benefits of the NHS 111 Online service and improving efficiency and patient experience. Clarity about who should use each service given the possible differences between the telephone and online populations will improve efficiency and contribute to reducing the complexity of navigation that current users experience.

Potential improvements

Evidence from the user surveys and interviews and staff interviews provided some suggestions of factors that could improve the online digital process and management of contacts. These were:

• Further refine the questioning. Some users found questions difficult to navigate, and the need to provide binary yes/no answers did not help them convey their problem. Where people had multiple symptoms this was difficult to manage as they had to choose one.

• Provide images or diagrams to help better understand the questions (examples were pictures of rashes so the user could choose the nearest one to their problem, or a diagram of a body to indicate what is meant by ‘lower back’).

• Provide more options for a ‘human’ interaction either through an online ‘chat’ function (which people are familiar with from other service industries) or the option for anyone to request a clinician call back (rather than the current system when this is offered only for some contacts where a ‘speak to clinician’ disposition is reached). This does have implications for clinician workload.

• The starting point for this study is when a user has accessed the online service. However, there is a step before this, not within scope, which is how people navigate to find the NHS 111 Online service. This may be simple, such as putting ‘NHS 111’ into a search engine, or people may first access the NHS website or app and use this to navigate to the online service. Further exploration of this process would help better understand how easy, or how difficult, people find it to access the service.

• Provide more information about locally available services and when they are open. This is already a feature of the NHS 111 Online service and depends on integration with the local DoS. It may be that some early survey responses reflected use before these were all fully integrated, but there may be other types of services of interest to users that are not currently included.

NHS 111 telephone service staff made some suggestions of improvements to make the process more efficient. These were:

• Provide better information to the public about what the NHS 111 Online service can offer and when to use it.

• Streamline the process of triage for online referrals so that they are similar to telephone referrals (particularly if reimbursement and targets are involved). This would reduce longer call times that may be a disincentive for staff.

• Allow NHS 111 telephone service to have access to online referral case notes.

• Include local commissioners more in the development of services.

Service expansion

This evaluation presents findings from the early-implementation stage of the NHS 111 Online service. There is a tension between the need to provide early evidence on the impact and effectiveness of a new service and the limitation that this may not reflect what happens when a service has developed and matured. Use of the NHS 111 Online service increased dramatically during this study. There were just over 2 million contacts during the first year of national provision (2019), the time frame for this work. NHS Digital data (not in the public domain) recorded 7.45 million contacts for the first 10 months (January–October) of 2020. Of course, this was in part fuelled by the COVID-19 pandemic. In response to this, the NHS 111 Online service was rapidly adapted to provide a digital symptom checker for coronavirus, and between March and November 2020 this was used over 3.5 million times. 72 It is a striking example of how digital technologies can be utilised to provide a functional, national service triage service capable of managing huge volumes of contacts in a short time in response to a public health crisis. It is impossible to estimate what the growth in use of the online service would have been without the pandemic, although use was clearly already increasing rapidly. The high profile of the COVID-19 symptom checker will undoubtedly have increased awareness of the NHS 111 Online service and have driven growth. It is also possible that some people who might not have considered using NHS 111 Online but used the symptom checker will now use the service again in the future. It was also clear from the surveys and the interviews that service users valued a digital alternative that could deflect some of the pressure from other NHS services. Both of these factors may influence future behaviour about choice and decision-making when accessing urgent care. This means that the combined populations of NHS 111 telephone and online users will be substantially greater in the future than the one we have examined and the impact may already be changing. It also has significant implications for the NHS and reinforces the need to take steps to try to maximise the channel shift from the telephone service to the online service and minimise duplication of service use and growth in new demand. This will become even more important if the NHS 111 First programme73 – in which people are encouraged to call NHS 111 before attending an ED– is expanded nationally. The combined NHS 111 telephone triage, NHS 111 Online and NHS 111 First populations will create a huge cohort of people with expectations of a rapid response to their urgent health problem.