Developing new ways of measuring the quality and impact of ambulance service care: the PhOEBE mixed-methods research programme

Article history

The research reported in this issue of the journal was funded by PGfAR as project number RP-PG-0609-10195. The contractual start date was in June 2011. The final report began editorial review in November 2017 and was accepted for publication in November 2018. As the funder, the PGfAR programme agreed the research questions and study designs in advance with the investigators. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PGfAR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Anne Spaight reports grants from East Midlands Ambulance Service NHS Trust during the conduct of the study. Steve Goodacre is a member of the Health Technology Assessment (HTA) Clinical Trials Board, HTA Funding Boards Policy Group and HTA IP Methods Group. Helen Snooks is a member of National Institute for Health Research HTA and Efficacy and Mechanism Evaluation Editorial Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2019. This work was produced by Turner et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2019 Queen’s Printer and Controller of HMSO

Background

The NHS provides emergency care for a diverse population of patients who need medical care for a wide range of conditions and have different levels of urgency. For many patients, the first point of contact with the NHS is when they request help from the ambulance service.

Demand for ambulance services has been steadily increasing for many years. In 1974, ambulance services in England responded to 1.5 million emergency (‘999’) calls per year. By 2016/17 this had increased to > 6 million responses to almost 10 million calls. 1 In the past, the main purpose of the ambulance service was to respond to serious emergencies and transport patients to the nearest hospital emergency department (ED). As the number of calls has grown, so has the type of health problems people call 999 for.

Ambulance services now provide care for conditions that range from life-threatening emergencies, such as heart attacks, stroke and serious injury, a wide range of illnesses and problems associated with chronic disease and long-term conditions, to relatively minor illness or injury. In response to these changes, ambulance services have also adapted and developed.

In 2005, the Department of Health and Social Care policy document Taking Healthcare to the Patient: Transforming NHS Ambulance Services2 recognised that patients who call 999 should receive care that not only is timely but also best meets their clinical needs. Patients with serious medical emergencies and injury still need a fast response, early treatment and good clinical care at the scene, followed by transport to hospital. However, research evidence also showed that, for some patients with less urgent problems, a trip to the ED was not always necessary and their needs could be better met by providing the care at home or in a community service. 3

As a result of the 2005 report, and subsequent broader policy initiatives focusing on delivering care closer to home for urgent problems and developing expert centres for specialist care such as stroke units and major trauma centres for emergencies,4 there have been a variety of innovations in types of response provided by the ambulance service. These types of response now fall into three main categories:

1. More detailed clinical telephone assessment of some 999 calls by nurses or paramedics so that patients who do not need an emergency ambulance can be provided with self-care advice or referred to the right service – ‘hear and treat’.

2. Development of paramedic skills including advanced paramedic practitioners so they can treat minor illness and injury at home and, when needed, refer to other services for follow-up care, for example community falls services – ‘see and treat’.

3. Improving prehospital assessment and care pathways so that patients who need specialist care can be taken straight to the best facility (e.g. a stroke or heart attack centre) and other patients taken to the nearest ED – ‘see and convey’.

The changes in the way that ambulance services provide care have contributed to broader efforts to improve emergency and urgent care by developing a more ‘joined up’ system approach in which ambulance, accident and emergency (A&E), and community services work in partnership to ensure that patients can access and receive care in the right place and at the right time. 4

These changes also need to reflect the NHS principles set out in the Next Stage Review,5 which are to provide high-quality services through improving effectiveness, safety and patient experience. If these aims are to be achieved, then the assessment and monitoring of the quality of services and the wider system in which they operate become important tasks. 6 This means that we need ways of measuring how well ambulance services are performing, in terms of both the services they are delivering and the impact that these services have on the patients that they care for.

Quality assessment and quality improvement can be achieved only if we can find ways of routinely and consistently measuring these important aspects of patient care so that we can identify where care is good and where it needs to be improved. Measuring performance and quality of care is never easy but is particularly difficult for ambulance services, as they provide care to such a diverse group of patients and those patients will be in their care for a relatively short period of time only.

For many years, the quality of ambulance service care has been mainly assessed by measuring how quickly they respond to 999 calls – response time performance. This is not unique to the UK and has been the predominant performance or quality measure for Emergency Medical Services (EMS) internationally. 7 Response time has been used as a proxy measure because of the relationship between speed of response and survival following out-of-hospital cardiac arrest – the faster the response, the more likely a patient is to survive. 8 However, cardiac arrest accounts for a very small proportion of 999 calls only, currently 0.6% in England,9 and research evidence shows that, for other calls, the speed of response has little impact on survival or outcomes. 10–12 Response time also tells us nothing about what clinical care was given or the impact of that care on patient outcomes so, overall, it is a poor measure of service quality for the vast majority of patients who request an ambulance.

The need to develop better ways of measuring ambulance service performance and quality of care, particularly the effect on patient outcomes that are relevant to all people who use the service, not just a few, has been recognised for some time. This need was a key recommendation of Taking Healthcare to the Patient: Transforming NHS Ambulance Services2 and also a high priority in a UK Delphi study assessing priorities for prehospital care research. 13 Despite this recognition, a review of outcome measurement in prehospital care found few research studies investigating this area and found that most published literature focused on discussion of the need to develop measures rather than any solution to the problem. 14

In England, some progress has been made with the introduction of a set of 11 ambulance service quality indicators adopted in 2011. 15 Having a set of indicators, rather than just a single measure of response time performance, that had relevance to all calls was a big step forward, as was the inclusion of a small number of clinical indicators designed to measure the care provided for some key conditions like cardiac arrest, stroke and heart attack. 16 However, the focus remained on processes [what the ambulance service did in terms of response times, type of service (e.g. how many calls are managed using ‘hear and treat’) and providing treatment for a small number of conditions] rather than how care affected patients. It is also fair to say that these indicators or measures have mainly been developed by health professionals and academics so may not include aspects of care that are important to patients.

There has been a major problem holding back the development of better ways of measuring how well ambulance services provide care to the population who call 999 and that was the lack of information available to them about what happens to patients and their outcome once they had left ambulance care. If the only information they have is their own data about how they respond and what they do then, it is not surprising that these are the only things that were measured. These data are also incident rather than patient based, which makes it difficult to monitor related calls for the same patient. However, if ambulance service information about patients could be linked to, for example, hospital information, so that they have a more complete picture of what happens to patients and their outcomes, then a better assessment could be made of the impact of care and the benefits it produced. Creating linked data sets can also potentially add value by creating comprehensive information sources that can be used to provide feedback on outcomes and improve performance and practice, but this was not something that was being done routinely.

This programme was designed to try and resolve these problems by bringing together the two key themes of:

1. developing more meaningful ambulance service performance and quality measures that reflect the key principles of quality, safety and effectiveness of care

2. creating linked data sets that can combine ambulance service, hospital and mortality data to provide the information needed to support better performance and quality measurement.

Even with a linked data set that includes patient outcomes, measuring impact is problematic. Outcomes can be influenced by a whole range of factors in addition to ambulance care received, such as a patient’s age, illness severity or hospital care. One way of overcoming this is to develop performance measures that can take account of these intrinsic and extrinsic factors by using risk or case-mix adjustment models. These have been successfully developed and used for research and audit in settings such as trauma17 and intensive care18 and have shown they can make an important difference to improving processes and outcomes of care.

Aims and objectives

The aim of this programme was to develop new ways of measuring the impact of care provided by ambulance services. This could support quality improvement by providing information to monitor and audit service performance and assess the impact of change through service evaluation. The programme was called PhOEBE (Prehospital Outcomes for Evidence Based Evaluation).

Its objectives were to:

1. review and synthesise the research literature on prehospital care outcome measures and use consensus methods to identify a small set of measures relevant to the NHS and patients for further development

2. create a data set linking routinely collected prehospital (ambulance service) data, hospital data and mortality data to provide outcome information

3. use the linked data to develop the measures identified in objective 1 by building case-mix adjustment models that could potentially be used to assess ambulance service performance and detect change over time with repeated measurement

4. explore the practical use of the linked data set and the case-mix adjustment models to measure the effectiveness and quality of ambulance service care and assess how they can be best used to support quality improvement strategies.

Programme design

The programme was designed to be conducted in four linked stages or workstreams (Figure 1).

1. Workstream 1: a review and synthesis of the evidence on ambulance service-related performance and quality measures to produce a list of potential measures. Then use consensus methods to assess and prioritise these measures and to identify a small number that are relevant to different stakeholders, including patients and NHS staff, research academics and patient representatives for further development.

2. Workstream 2: linking ambulance service call information and patient information from the care record completed by ambulance clinicians with routine ED and hospital information [Hospital Episode Statistics (HES)] and national records of patient deaths. This creates a single data set that follows what happens to each patient who makes a 999 call.

3. Workstream 3: explore and develop case-mix-adjusted models for processes and outcomes in patients attended by the ambulance service using the linked data. This allows us to assess whether or not case-mix adjustment is needed to improve the usefulness of process or outcome measures, and their potential as indicators to measure quality and performance between services or within services over time.

4. Workstream 4: testing the risk adjustment models to assess if they can be used to measure effectiveness and quality. This looks at how they might be used in practice and explores with users (patients and staff) how useful they are and how they might be best implemented in the NHS. In addition, the linked data can be used to estimate the costs of different types of ambulance response.

FIGURE 1.

The PhOEBE programme overview. APC, admitted patient care; CAD, computer-aided dispatch; ePRF, electronic patient report form; ONS, Office for National Statistics.

Figures 1 and 2 provide a summary of the four workstreams and how they are linked together.

FIGURE 2.

Workstream 1. PPI, patient and public involvement.

The programme was overseen and supported by a project management group that included all of the programme collaborators, the research team, representatives from the two ambulance services taking part and a public involvement member. We also had a project steering committee comprising key members of the management group and external advisors representing ambulance services, the College of Paramedics, NHS commissioners and emergency care research. The programme was carried out over the 6-year period of 2011–17.

Public involvement

Public involvement made an important contribution to the PhOEBE programme. At the outset we formed a public involvement reference group of three members who provided substantial support and input to each stage of the programme, both individually and by providing links to other relevant external groups. Our public involvement reference group not only contributed advice but also co-produced some of the programme work and outputs. We describe the patient involvement work in more detail under Patient and public involvement below.

Changes from the original proposal

During the course of the programme we encountered a major problem during workstream 2. The data linkage component was contracted to the then Health and Social Care Information Centre (HSCIC) (now NHS Digital). NHS Digital held the central HES data and also provided a trusted data linkage service. We planned to use this service to link the ambulance service, hospital and Office for National Statistics (ONS) mortality data. The plan was to obtain the first set of linked data by the end of year 2 and a second set in year 3 (2013–14).

During 2013, major data security issues arose at NHS Digital and, as a result, there was a major review and restructuring of the organisation. This meant that no data were released for (any) research use for almost 2 years. These issues are described in more detail in Creating a linked data set but the overall impact was that, because of these external delays, we did not receive a linked data set until October 2016 – 4 months after the original expected end date of the programme. We were fortunate to obtain a 1-year extension to the programme, which enabled us to complete the workstream 3 work to develop the risk-adjusted measures, albeit within a much reduced time frame.

The delays meant that we were unable to conduct much of the planned work for workstream 4, principally because the second linked data set that we had intended to use to test and validate the measures developed in workstream 3 arrived too late and we had no time left to complete this. The implications of these delays and the difficulties in obtaining and managing the linked data are discussed in more detail in Discussion and conclusions.

Systematic searches and review of related research evidence

We conducted two systematic searches to review, assess and synthesise the research literature for existing and potential process and patient outcome measures for prehospital care. These were not conventional systematic reviews in that we were not appraising evidence of the effects of prehospital care. The aim was to identify all measures that had been used to assess the impact, quality and safety of prehospital care as well as potential and as yet untested measures, using systematic searching and evidence synthesis strategies. We conducted two reviews so that we could examine both policy literature and primary research evidence.

Review 1: documentary analysis of policy documents

The first review was designed to identify actual and aspirational quality and performance measures of ambulance and prehospital care. We used a comprehensive search strategy to search four electronic databases: MEDLINE, Scirus, Scopus and Google Scholar (see Appendix 1, Table 15). We also searched relevant websites such as the Department of Health and Social Care,19 National Association of Emergency Medical Services Physicians20 and NHS Confederation. 21 We supplemented the searches with our own extensive archive from previous related research studies. Any policy documents produced by national, regional or professional organisations or agencies were included, but these were limited to those in the English language published between 2000 and 2011 to ensure relevance. Searches were conducted in August 2011. The results of the searches are given in Figure 3.

FIGURE 3.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart for review 1. Reproduced from Moher et al. 22 © 2009 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart for review 1. Reproduced from Moher et al. 22

References were screened by six members of the research team. We screened 319 potential references, assessed 72 full-text papers and included 36 documents.

Double data extraction of included references was carried out by the same six researchers and the measures identified were classified using established frameworks of health-care quality (structure, process and outcome23 and timeliness; efficiency; effectiveness; safety; patient centredness and equity24).

Of the included references, the majority were discussion documents. Some were specific to ambulance services (also known as EMS), setting out the case for the inadequacies of using response times as a performance measure and the need to find alternatives, but stopping short of providing specific alternatives. Others were strategy documents for the management of specific conditions (primarily stroke, coronary heart disease or major trauma) that contained a section on prehospital management with suggestions for potential quality measures, for example time from a call to arriving at a specialist unit for stroke patients.

Of the documents describing performance measures in use or suggestions for measures, these documents were, unsurprisingly, dominated by time measures. After time measures, the most common measures were also process related, mainly recording what ambulance clinicians did to either assess patients or provide treatments. Service measures included types of response (e.g. if a paramedic was sent), how accurately a clinical problem was identified at the time of the emergency call and how calls were managed (e.g. the proportion managed at home and taken to hospital). There were few examples of patient outcome measures and this category was dominated by survival from cardiac arrest. Several documents supported the need to measure patient experience and satisfaction and some identified relief of symptoms, such as pain as important, but there were no examples of methods to do this routinely through quality measures.

Review 2: systematic literature search and synthesis of primary research studies

For the second review, we conducted a systematic literature search and synthesis of longitudinal studies, audits and evaluations of ambulance services (or EMS) performed at a local, regional or national level. The aim was to identify potential performance and quality measures that may have been used to assess differences between, or change in, the delivery of ambulance service care in primary research projects. Some of these may not have been considered as routine measures but could potentially be developed and adapted for this purpose.

We conducted a systematic search of five electronic databases: MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ISI Web of Science and The Cochrane Library (see Appendix 1, Table 16). This was supplemented with references identified in review 1, hand-searching of included studies and articles from our own relevant archive. Any relevant research study that had investigated ambulance service delivery and care from a service perspective and incorporated some measurement of change was included.

We were aware that there was an enormous amount of related research literature for specific patient groups, particularly cardiac arrest and trauma, which could potentially overwhelm our search. Much of this research is about the clinical management of patients and its effectiveness. We therefore excluded studies for which the primary aim was to assess a specific clinical intervention or if it was a descriptive study. The key focus here was comparative research and the measurement of change. Included studies were limited to those in the English language and published between 2000 and 2011. Searches were conducted in October 2011.

As in review 1, references were screened by six members of the research team. For included studies, data extraction was completed using a two-stage process. As review 2 yielded a larger number of studies than review 1, double data extraction was carried out for a 10% random sample of references. For the first stage, six reviewers extracted descriptive information on study aim, population, setting and the main process or outcome measures used. For the second stage, three reviewers carried out a more detailed data extraction on each measure identified including the type of measure, how it was measured and how it was reported.

We screened 5088 references by title and abstract, reviewed 257 full-text references and included 139 references. We identified 136 different measures that were recorded 483 times in the included studies. The results of the searches are given in Figure 4.

FIGURE 4.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart for review 2. Reproduced from Moher et al. 22 © 2009 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The PRISMA flow chart for review 2. Reproduced from Moher et al. 22

We classified measures into three broad groups: service (operational) measures, patient management measures and patient outcomes (see Appendix 1, Boxes 5–7). The largest group was service measures (41%), which mainly included a large number of time-interval measurements. This group also included call handling, skill level of response and type of response (e.g. transported or not transported). The patient management group accounted for 29% of measures and these were mainly concerned with clinical procedures and interventions, such as assessing symptoms and condition and treatment provided (e.g. drugs given, splinting, defibrillation and oxygen therapy).

It also included decisions about where to take patients (i.e. to the nearest ED or to a specialist hospital) and subsequent hospital measures, such as length of stay or where a patient was discharged to. The third group of patient outcomes included 30% of the identified measures but this was dominated by the single measure of survival (or mortality), which accounted for more than half of the measures. The reason for this is that there were many different end points for measuring survival ranging from < 1 day up to 5 years. A small number of functional measures were recorded, which included quality-of-life measures, physical disability and cognitive (brain) function. Some examples of the types of measures included in each category are provided in Table 1.

TABLE 1 - Example measures identified from systematic evidence review

Category	Measure
Clinical management	Accuracy of call-taker identification of different conditions (e.g. cardiac arrest, heart attack, stroke, low-urgency calls suitable for nurse advice)
	Proportion of people with diabetes mellitus treated at home
	Accuracy of paramedic diagnosis, for example agreement of on-scene and final hospital diagnosis
	Compliance with protocols and guidelines (e.g. triage or transport protocols)
Whole system	Completeness and accuracy of patient records
	Frequency with which ambulance staff administer treatments (e.g. inserting breathing tubes, heart monitoring, oxygen therapy, defibrillation)
	Proportion of all calls that receive an ambulance response with patients who are not conveyed to hospital/other health facility
	Volume and nature of complaints
Patient outcomes	Survival at different time points after the event. For example, in hospital, 24 hours, 7 days, 30 days, 90 days, 1 year
	Health/quality-of-life status
	Proportion of patients left at home who have a contact with any emergency/urgent health service within 72 hours
	Pain measurement and symptom relief

Example search strategies for each review and a table of the categorisation of the 136 included measures are provided in Appendix 1. Further details on the findings and individual measures identified in the two reviews and the final list of combined measures identified from both reviews is provided in a supplementary file available at www.sheffield.ac.uk/scharr/sections/hsr/mcru/phoebe/reports. 25

An update of the searches in 2016 did not reveal any new measures that need to be considered. In our original proposal we said that we would conduct a third systematic review of any tools or instruments that we identified that had been specifically constructed to measure performance for prehospital care. However, we found no relevant instruments and the only composite measures were already validated general tools to assess, for example, quality of life [EuroQol-5 Dimensions (EQ-5D)] or functional outcome (Glasgow Outcome Score); therefore, this review was not conducted. The results of both reviews were pooled and a list of potential measures was constructed using the three broad groups described above. This list provided the focus for the next stage of workstream 1 – the consensus event to begin identifying and prioritising measures for further development.

Delphi survey

The consensus event allowed us to begin to prioritise the large number of candidate measures and reject some measures that were agreed as not important. For the next stage, we used another consensus method – a Delphi survey to further rate and prioritise measures. At this point we considered not just what could be measured but also how this might be done.

A total of 67 measures were included and these were categorised into the same three groups: patient outcomes (n = 25), whole service measures (n = 32) and clinical management measures (n = 10). The number of items was larger than those considered in the consensus event because, at this stage, we included time measures and began to develop more explicit, discrete descriptions of potential indicators. For example, where a broad principle such as accuracy of dispatch decisions was used for the consensus work, this was refined into multiple descriptions for specific conditions or call types. Potential measures were presented in a survey that enabled responses to be completed and returned electronically. Participants were asked to consider each measure and score their level of agreement on a scale of 1 to 9 (strongly disagree to strongly agree) using the statement:

This measure (either on its own or within a set of measures) is a good reflection of the quality of care provided by ambulance services and is likely to be a good indicator of the quality of the 999 ambulance service care pathway.

Participants were able to suggest additional indicators for inclusion. Responses to round 1 were recorded and the median score was calculated for each measure. This was followed by a second round during which revisions to measure descriptions were made following suggestions from the first round. Participants were provided with their own and the group median score and asked to score the measures again on the same 1–9 scale.

There were 23 participants who completed the round 1 form and 20 completed round 2 with an overall response rate of 74%. As in the consensus event, the participants represented a wide range of service provider and professional viewpoints, and most UK ambulance trusts. Some participants had also participated in the consensus event. Scores from round 2 were recorded and median scores calculated. A large number of measures scored highly, so a median score of 8 was used to discriminate between measures, with 20 (67%) participants ranked as high scoring.

We intended to include patient and public participants in the Delphi survey but our PPI reference group thought that the level of technical detail would make meaningful participation difficult. Instead, we held a separate event for PPI participants so that the concepts and measures could be explained and discussed in a face-to-face format.

Using a similar format to the previous consensus event, measures were presented for each of the three categories and small group discussions held. Participants then used an electronic voting system to rank each measure. Eighteen PPI representatives attended the PPI workshop and represented a range of people, including young people and vulnerable groups. The results of the PPI event were added to the results of the Delphi survey for the final stage of this workstream.

Two published papers26,27 are freely available and describe in more detail the methods and results of the consensus event and Delphi survey,26 and the co-produced event created with our PPI reference group to complement the Delphi study. 27

Patient and carer views of ambulance service care

At the outset of the PhOEBE programme we were aware that little research had been done to investigate the aspects of emergency ambulance service care that are valued by people who use the service. This includes patients but also their carers, who may be the person who makes a 999 call asking for help. To address this we conducted a qualitative study in which we interviewed people who had recently used the ambulance service in one of our study services. Ethics approval for the study was sought and gained from the National Research Ethics Service Committee East Midlands – Northampton (Research Ethics Committee reference 12/EM/0022) on 23 February 2012. During 2012, we talked to 22 patients and eight of their spouses (n = 30) using a semistructured face-to-face (n = 18) or telephone (n = 14) interview. We felt that it was important to explore the processes and outcomes of care that were important to ambulance users and we wanted to ensure that we captured issues that were relevant to the range of ambulance users, not just those with a life-threatening condition. We therefore included patients and carers who had called for serious problems requiring transport to hospital, those who had an ambulance crew attending but who were managed at home, and those managed by telephone advice. In the first part of the interview, we explored positive and negative aspects of their ambulance service experience and this was followed with questions about what they valued about the service and how performance might be measured. Interviews were recorded, transcribed and then analysed using framework analysis. An initial thematic framework was developed and then interviews coded to these themes, adding new ones as they emerged. A thematic map was constructed related to issues participants valued.

Participants in our study, regardless of clinical condition or level of ambulance service response received, valued similar aspects of their prehospital care experience. Users were often extremely anxious about their health and the outcome they valued was reassurance provided by ambulance service staff to alleviate the anxiety, fear or panic that they experienced at the time of calling an ambulance. They also valued reassurance that they were receiving appropriate advice, treatment and care, and this was enhanced by the professional behaviour of staff, which instilled confidence in their care, communication, waiting times for help (i.e. a short wait), and continuity during transfers. These features are themselves a consequence of the ability of call-takers and ambulance clinicians to competently recognise what the problem is and deliver appropriate advice and care and so implicitly reflect good-quality care. A timely response was valued in terms of allaying anxiety quickly. Participants valued the experience that they had, not just with ambulance crews who attended them but also the call-takers when they made their 999 call.

The interviews with users highlighted very clearly that, regardless of the actual clinical problem, the ability of the emergency ambulance service to allay the high levels of fear and anxiety felt by patients and their carers was crucial to the delivery of a high-quality service. Measures developed to assess and monitor the performance of emergency ambulance services have predominantly focused on actions such as response times or treatments provided. However, it was the more human interactions with the service that users recalled and described, and which could be included in the development of ambulance service patient experience measures. We used the findings from this study to add context to the description of ‘patient experience’ as a potential measure within the consensus work. Although it was recognised as important, it was acknowledged that measurement of patient experience is a longer-term objective outside the scope of the programme.

The qualitative study has been published as an open-access peer-reviewed journal article. 28

Final selection of measures for further development

The reviews and consensus work allowed us to consider a large number of potential ambulance service performance and quality measures, and to determine which were considered important to a range of end users. The final stage was to select from this list a small set of measures that could reflect the range of perspectives (service measures, patient management and patient outcomes) and take account of the broad population of people calling 999, not just a few with specific conditions.

The final set was selected using an expert panel drawn from our programme management and steering groups. The panel comprised 13 members and included representatives of the research team (reflecting research, statistics, ambulance service clinicians, PPI, emergency medicine) and external expertise from a further emergency medicine consultant, consultant paramedic and commissioner. We assessed all measures considered in the consensus work26 to avoid missing potentially important measures that did not feature highly in the rating exercises. Each measure was rated using a set of criteria that considered, for example, how highly it ranked in the consensus meetings and Delphi survey, the population it applied to, feasibility and availability of data, relevance to ambulance care, importance, meaningfulness and whether or not an item was already being measured. A score was derived for each potential measure using these criteria and the final set selected using these scores and expert judgements so that the set as a whole provided a balanced assessment of the different aspects of ambulance care considered to be important. The full set of criteria used and 56 measures assessed is available in Appendix 2 (see Table 17).

For two measures, survival from an emergency condition and accuracy of call identification, we had to identify a set of relevant conditions, as not all 999 calls were appropriate. We had previously conducted some consensus work as part of a study to develop emergency care system indicators and in this work identified a set of 16 emergency conditions [with relevant International Classification of Diseases, Tenth Edition (ICD-10),29 codes] that were considered appropriate to include in the indicators. We therefore used this same set of validated conditions for this work, including only patients with this diagnosis at discharge from hospital or as a cause of death. 30 The 16 emergency conditions are listed in Box 1.

The final set of six measures selected for further development, included in workstream 3, is shown in Table 3. We initially included two further measures in this list. First, the compliance of ambulance clinicians with protocols and guidelines for specific conditions. The current ambulance service Clinical Quality Indicators31 for England already include a measure of compliance with expected care bundles for a small number of conditions. The purpose of this measure was to explore whether or not the availability of linked data and better information on patient outcome could be used to improve this indicator. However, the problems in obtaining the linked data and reduced time available to develop the performance measures meant that we had to exclude at least one intended measure and, as this measure already exists at least in part, we decided to concentrate on new measures. Second, we included a measure of mortality in patients with urgent problems, that is, those who have a low risk of dying. However, the lack of information on final diagnosis for patients not admitted to hospital made it impossible to identify all relevant patients. Instead, we took a different approach with this measure and explored the use of a structured judgement review process to identify potentially avoidable deaths.

Summary

Workstream 1 encompassed a number of related activities. The evidence reviewed revealed a large number of potential measures although many were variations on a single theme, such as time. The consensus work allowed us to consider this broad range of measures from a number of different perspectives. In particular, there was strong patient and public input including use of a novel approach to meaningful participation in the consensus process. The final set of measures for further development represented the potential to provide a broader and more balanced view of ambulance service care. These were relevant to all people who used the service rather than the current focus on single processes, such as response time or smaller populations with important but more specific conditions (e.g. cardiac arrest). The qualitative study produced new and important primary research evidence in an area that has not been well studied and revealed important insights into patient perceptions that were poorly understood. We found that:

1. Previous quality measures and performance indicators were dominated by time measures and accounted for over one-third of identified measures.

2. Outcome measures were dominated by varying durations of survival or mortality, spanning the range from admission to hospital to up to 5 years post admission, in a small number of longitudinal primary research studies.

3. Measures of accuracy were most frequently voted as essential, followed by measures (including pain) that reflected patient experience.

4. Patients felt that addressing anxiety and providing reassurance were important. This applied to the call process as well as face-to-face interaction with ambulance clinicians.

Workstream 1 produced a set of candidate measures potentially suitable for further development as indicators of ambulance service quality and performance. Development required an information source that brought together details of what happened to patients at the time of the incident and after their ambulance service contact. This was the focus of the next workstream.

Introduction

Health-care information relating to a single person is often held by different services and is usually unconnected. In addition, different systems may be event rather than person based, so the same person can have multiple and different unconnected event records. The purpose of linking data is to match data from different information systems, bringing them together to create a single record of events for an individual person.

Having access to linked health data provides a real advantage for assessing health-care quality and performance. Patient pathways often involve multiple service providers or service contacts. If we base our assessments of how good care is on information from a single health provider or service, this provides only a ‘partial view’ of quality and performance and does not capture the range of services or complex care pathways available in today’s health care. 32

This is important for the ambulance service as, although they are a key service providing immediate help to people with an emergency or urgent health-care problem, very often this is a relatively short component, being only the first step in a longer set of contacts with different parts of the health-care system. In most cases, the impact of ambulance service care may not be obvious until further along the episode of care and this is particularly true of patient outcome information.

The availability of linked patient information enables important outcomes to be measured. In addition, making better use of the routine information collected along an episode or pathway of care for a population of patients, such as those who call 999, means that it becomes possible to monitor and compare processes and outcomes of care over time. Although linking information from different parts of the health service into a single patient record might seem obvious in the digital age, this is still not routinely available.

There have been previous attempts to link ambulance service data with ED data in the UK33 and Australia. 34 In both of these previous studies, data linkage was achieved but only after problems with data quality, finding suitable patient identifiers (IDs) and developing statistical matching processes had been overcome. Within the PhOEBE programme, our aim for workstream 2 was to revisit this problem and attempt to create a data set that linked routinely collected health service and national mortality information for individuals who used the 999 emergency ambulance service. The objectives were to:

1. develop data linkage processes that are acceptable to patients, data processors and data controllers, and comply with information legislation

2. obtain the necessary research and data approvals

3. link routinely collected ambulance service information about the 999 call and the clinical care given to patients with routinely collected hospital information and national mortality information, using a third-party data processor (NHS Digital)

4. create a new information source that provides a single record of the emergency care pathway for each patient contacting the 999 ambulance service.

Types and sources of information included in the linked data

We used five different types of information from three sources to create the linked data set. A full list of data and variables included in the linked data set can be found in a supplementary file at www.sheffield.ac.uk/scharr/sections/hsr/mcru/phoebe/reports. 25

Study services and planned data collection periods

Two ambulance services in England took part in the programme: East Midlands Ambulance Service NHS Trust (EMAS) and Yorkshire Ambulance Service NHS Trust (YAS). When the PhOEBE programme started in 2011, the use of the ePRF was not widespread among ambulance services. EMAS had ePRF coverage for > 80% of the population who it served and YAS for one small operational area (about 15% of the service population). In our original plan we intended to create two linked data sets for each service (four data sets in total) for two time periods in year 2: July –December 2012 and January–June 2013. Delays in obtaining the right permissions meant that this moved by 6 months so the final data linkage periods were January–June 2013 and July–December 2013. Two separate time periods were used as the intention was to use the first data sets to construct the performance and quality measures in workstream 3 and the second sets to test the measures in workstream 4.

Data permissions

Linking different sources of health data together must be done in a way which is ethical and secure, acceptable to patients and service users and meets the requirements of the Data Protection Act36,37 and other relevant legislation. A central principle of data-linking studies is that individual concerns about the use of personal information must be balanced against the research benefits for the general population, so measures to manage risk and safeguard personal health information must be in place. 38 We therefore had to obtain a number of relevant permissions and put in place the required information governance and secure data management processes before we could request and obtain our linked data. Patient identifiable data were needed to enable the processes that linked ambulance service, hospital and mortality data. However, no identifiable data were transferred or processed outside the NHS and, therefore, no patient identifiable data were retained in the final linked data set we used for our research.

We obtained the following permissions:

• NHS research ethics approval – as elements of the process required patient identifiable data, approval was sought and gained on 12 July 2012 from the NHS health research authority through the National Research Ethics Service Committee East Midlands – Derby (Research Ethics Committee reference 12/EM/0251, Integrated Research Application System project number 84751).

• Confidentiality Advice Group – approval is required from this group (previously the National Information Governance Board) where research studies wish to use personal-identifiable patient information without consent, for purposes other than the direct care of patients and where it is not possible to use anonymised data or to seek patient consent. In this project, seeking individual consent was not feasible as the number of patients was very large (the individual ambulance services respond to > 400,000 999 calls per year) and we anticipated that some patients would have died. Approval was confirmed on 17 August 2012.

• NHS Digital Data Access Request Service – permission is required from the NHS Digital Data Access Advisory Group to process and receive data. As part of this approval process, the legal basis for accessing the data, information governance and security arrangements, including data storage systems, and whether or not the project has a purpose beneficial to the health system is assessed.

The process of applying for data permissions and approvals proved to be very challenging and time-consuming. When we started this process, the HES data were held by NHS Digital. This organisation was also engaged to provide the data linkage service. We initially obtained the necessary (at that time) approvals in 2012 and began the process for obtaining the first set of ambulance data for linkage in January 2013. However, shortly after this a number of serious internal problems at HSCIC meant that there was a major reorganisation into what is now NHS Digital. During this period no data were released from NHS Digital. The reorganisation also meant that new approvals processes were put in place and the data permissions process had to begin again. This was completed in May 2015 and it was then a further 4 months before we received any NHS Digital data. Additional work and data approvals were then required owing to poor match rates for some patient groups, meaning it was not until October 2016 that we received the first adequate data set required for the workstream 3 work. Figure 5 provides a summary of the timelines and processes for the data linkage work.

FIGURE 5.

Timetable of data permissions and processes for obtaining linked data. CAG, Confidentiality Advice Group; DoB, date of birth; DSA, data sharing agreement; IG, Information Governance; NIGB, National Information Governance Board; R&D, research and development; UoS, University of Sheffield.

Creating the linked data sets

To create the linked data that we needed for the programme, a number of steps were needed to bring the different types of information together. Within NHS Digital, processes are already in place to link HES records and ONS mortality data. The new task for this project was to link ambulance service electronic records with these subsequent health records.

The first step was to retrieve the relevant information from ambulance service CAD and ePRF records. The starting point was all 999 calls received in the relevant time frame. Some calls were excluded at this point such as attendances with no ePRF, interhospital transfers, calls passed to other ambulance services and duplicate calls for the same incident. The exception was ‘hear and treat’ calls, defined as those calls that received input from a clinician (nurse or paramedic) but which have no ePRF record as no ambulance is sent.

The following stepwise process was then followed:

• Yorkshire Ambulance Service and EMAS selected and extracted the study data sample, based on all included ambulance service contacts within the specified time period.

• The study ambulance services linked the CAD and ePRF data (except ‘hear and treat’ calls) for all selected ambulance service contacts and produced a linked data set in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA). These data contain a large number of variables recording details of the patient, call processes, response provided, clinical assessment and treatment.

• The ambulance services assigned a unique ID code to each individual patient record.

• The ambulance services created a version of the data set that contained only the clinical data from the ePRF, non-identifiable emergency call and dispatch information from CAD and the unique ID number. This anonymised file, in the form of a password-protected Excel spreadsheet, was sent via secure encrypted e-mail to the research team at the University of Sheffield.

• The ambulance services created a second version of the data set that contained only the variables required for data linking including patient identifiable data. These included, for example, date, time and location of incident, patient name, date of birth, address, hospital attended, the unique ID number, and (when available) NHS number. For cases for which there was no NHS number available, these were traced by NHS Digital. This data set was sent to NHS Digital as a password-protected Excel spreadsheet via NHS Digital’s secure electronic file transfer system.

The next step was to link the ambulance service data with HES and ONS mortality data. This was undertaken by NHS Digital using its data-linking algorithm. This was a deterministic linkage of NHS number, sex, date of birth and postcode using a series of progressive steps39 to match the same information in one data set with that in another. When the NHS number was unavailable, we used NHS Digital’s NHS number-tracing service to look up NHS numbers using date of birth and patient name. NHS digital linked ambulance data with a large number of variables from the HES A&E, HES patient admission and ONS death records so we could identify all patients who subsequently attended an ED, were admitted to hospital or died. The unique patient ID provided by the ambulance service was retained in this linked data set. After all possible records were linked, NHS Digital removed identifiable data and, when necessary, replaced it into a pseudonymised variable, for example date of birth was transformed into age. The de-identified data were returned to the research team using the same secure transfer processes.

The final step was for the research team to re-link the clinical and CAD data provided by the ambulance services with the HES and ONS data provided by NHS Digital, using the unique ID number contained in each data set to produce our final linked data set. Figure 6 shows the data flow processes used for workstream 2.

FIGURE 6.

Data flow processes.

Because of the delays in obtaining linked data, we were unable to obtain the intended four complete data sets in our original plan. The first best-quality data received was that created for EMAS data for the period January–June 2013 in October 2016. We did subsequently obtain linked data for YAS for the same period and also the linked data for both EMAS and YAS for the second period of July–December 2013. However, given the time needed to then process these data sets into formats needed for the programme research, we were unable to use them within the time available. These data will be available for further research but the description below of data processing and the number of cases included in the linked data used in this programme was confined to the first EMAS data set we were able to fully utilise.

Processing the linked data

Data were housed on a secure virtual machine and read for processing into R (The R Foundation for Statistical Computing, Vienna, Austria), which is an open source programming language and data management software programme for statistical computing. The processing involved data cleaning and standardisation to create variables required for the study, for example calculating time intervals.

A full list of the variables included in the data sets from each information source, a detailed description of the processes for requesting and returning data, the technical specification of the linkage algorithm and a description of how each data package was created are provided in the supplementary file at www.sheffield.ac.uk/scharr/sections/hsr/mcru/phoebe/reports. 25

Cases included in the final data set

In our first complete EMAS data set, 83% (154,927/187,426) of patients in the sample were successfully traced and their records linked. Unsuccessful traces were due to missing or incomplete patient ID data from the CAD or ePRF records, so these could not be linked to subsequent health-care information or deaths. However, subsequent re-contacts with the ambulance service were identifiable using a unique HES ID generated for each patient. Figure 7 shows the numbers of cases and proportions of cases included and traced and Table 4 shows the numbers and proportions of calls traced at each tracing step.

FIGURE 7.

Numbers of cases traced and included in the linked data set.

Linkage of data for patients with an ePRF (‘see and treat’ and ‘see and convey’) was high, leading to a high overall (> 82%) match rate for the PhOEBE programme data sample (all calls attended with an ePRF and ‘hear and treat’). However, data-linking success for different patient groups was variable because of differences in the quality of data recorded for different types of patients. In particular, linkage rates for ‘hear and treat’ patients were very low. This is because at the time that the data were recorded, it was not standard practice for date of birth to be recorded on the CAD system. Date of birth was a key part of the NHS Digital data-linking algorithm and without this information the algorithm produced a non-match. As the CAD data system was the only data source available for ‘hear and treat’ patients, this resulted in an initial match rate of zero. A final linkage rate of 23.7% was achieved through searching subsequent and previous ambulance attendance data for additional linking information for ‘hear and treat’ patients. This potentially introduced a bias into the sample as the ‘hear and treat’ patients who were matched within our sample were those that had previously contacted and been seen by the ambulance service. These patients were more likely to be sicker than the ‘hear and treat’ patients in our sample where linkage was not possible. We assessed whether or not there were differences between patient characteristics for those with linked or unlinked data and found little evidence of differences for those discharged at scene or conveyed to ED. We did, however, find differences for ‘hear and treat’ patients; for example, linked ‘hear and treat’ patients were older than patients with non-linked data. This was most likely because older people were more likely to have had other contacts with the ambulance service.

Summary

We were able to develop data linkage processes that were acceptable to patients and the public and to data controllers, met with data legislation and were technically possible.

Although it was technically possible to link the data within the context of a research project, the complexity and time-consuming nature of data approvals, obtaining linked data and processing that data means that this is not feasible for individual ambulance services to undertake this routinely at present.

We found the following:

1. For cases involving patient contact with ambulance staff it was possible to link ambulance, hospital and mortality data for > 85% of ambulance calls.

2. Much lower rates of linkage were possible with ‘hear and treat’ calls, resulting in a potentially biased sample. This also made it more difficult to accurately establish consequent events such as re-contacts with other parts of the urgent-care system.

3. Recording date of birth was essential for linking data sets and ambulance services could improve this in future for data processing.

4. We were able to define the steps and processes required to link ambulance, hospital and mortality data for future research studies, assuming that the regulatory requirements remain unchanged. Future data linkage for evaluation could be achieved more efficiently through data-sharing agreements between ambulance services and hospitals with linkage performed at an NHS organisation if there were sufficient resources and expertise to do this.

The completion of the data linkage work allowed us to proceed to the next activity: exploring the use of case mix and building statistical models to measure the six indicators identified in workstream 1.

Introduction

The objective of this workstream was to use the linked data created in workstream 2 to explore the development of the performance and quality indicators identified in workstream 1. For each indicator, we could simply measure the related process or outcome (e.g. discharge from hospital with no treatment or survival from an emergency condition). However, there could have been factors other than receiving ambulance service care that influenced the processes or outcomes. Some of these are intrinsic to patients, for example age or type and seriousness of the health problem they call for. Others will be related to extrinsic or other NHS factors, such as when the call is made (day or night), incident location or hospital attended. Only considering the crude values of a measure means that inaccurate and unhelpful judgements might be made about the performance or quality of ambulance service care. For example, we could calculate the survival rate for all patients receiving an ambulance response in two ambulance services. In service 1, the rate is 95% and in service 2 it is 85%. We might then judge that service 2 is providing poorer care. However, the patients attended by service 2 may be much older or the proportion of calls for serious emergencies (rather than minor urgent problems) may be higher and, hence, the risk of dying will also be higher in these patients. The lower survival rate may therefore be a consequence of these case-mix differences and not ambulance service care.

For this reason we considered the effect of intrinsic and extrinsic factors by building case-mix adjustment models for some indicators. This allowed us to assess whether or not an indicator needed case-mix adjustment, if it did what characteristics or variables were important and needed to be taken into account, and if it was possible to build models that are statistically robust enough to be useful using currently available information. Developing each indicator using statistical techniques to include case-mix adjustment that allows patient and other differences to be taken into account may provide more accurate and realistic measurement so that differences between services or within services over time can be detected.

Indicator development methods

We used a complex, stepwise statistical approach to the development of each indicator. These are described in detail along with the comprehensive results of the analyses for each indicator in a supplementary file available at www.sheffield.ac.uk/scharr/sections/hsr/mcru/phoebe/reports. 25

For each indicator, the basic approach was to develop a ‘model’ that predicted the outcome (or process) from characteristics of the incident and the patient. For simplicity, only the outcome is used in the following description.

Summary of indicators developed

Tables 5–11 provide a summary of each indicator and how they were constructed. Table 12 compares response times for urban and rural calls. The abbreviations used for data sources and case identification are described in Box 2.

Indicator 1: mean change in pain score

Indicator 2: proportion of serious emergency conditions correctly identified at the time of the 999 call

Indicator 3: response time

Indicator 4: proportion of decisions to leave a patient at scene (‘hear and treat’ and ‘see and treat’), which were potentially inappropriate

Indicator 5: proportion of patients transported to emergency department by 999 emergency ambulance, but who were discharged to usual place of residence or care of general practitioner without treatment or investigation(s) that needed hospital facilities

Indicator 6: proportion of ambulance patients with a serious emergency condition who survive to admission, and to 7 days post-admission

Summary of the indicator development

We have examined the case-mix factors that influence the outcome of the six performance indicators that were derived in earlier workstreams as being important and potentially useful in assessing the performance of ambulance services in England. The results of these analyses, summarised in Table 11, suggest that most of the indicators do need to be adjusted for case mix before being used to make ‘fair’ comparisons. The indicator for mean change in pain score was not improved by risk adjustment. The indicators for accuracy of correct identification of emergency conditions and re-contact rates for patients left at home or managed by telephone did not quite reach the statistical significance for a good predictive model, but nevertheless appeared to be useful for monitoring changes over time and could be improved with some further work to improve precision. The best indicators, in terms of their predictive ability, were those for transporting potentially unnecessary attendances to ED and survival from a set of emergency conditions.

The advantage of this set of indicators is that it provided a more complete picture of ambulance service care than the current focus on a single process (response time) and clinical outcomes from a small set of acute conditions (cardiac arrest, stroke and heart attack) that comprise only a small proportion of ambulance service workload. The PhOEBE programme indicators are more inclusive as they represent patients with both emergency and urgent conditions, consider patient- as well as process-related outcomes and have expanded the measurement of survival to a broader group of patients than just cardiac arrest.

The models fitted as part of the development of case-mix-adjusted indicators are an indication of what models may need to be used to enable ‘fair’ comparisons to be made between services, areas or time periods. As the models were fitted using data from one ambulance service, they may not be generalisable to all ambulance services in England. Therefore, before the indictors could be used to compare and monitor performance within and between all ambulance services, the models would need re-fitting using national data.

We identified one other potential indicator at the end of workstream 1 – survival rates for patients who call for urgent conditions – as we would expect the risk of dying to be low in this patient group and it could tell us something about unexpected deaths, which are an important aspect of patient safety. We did construct a case-mix-adjusted model for this indicator. However, identification of patients with one of the specified urgent conditions relied on having an ICD-10 code that was available only for patients who were admitted to hospital or died. After discussion with the programme steering group we abandoned this indicator as it was recognised that including only admitted patients or those who died, and who were therefore more likely to be sick, introduced a serious bias. A substantial number of patients with urgent conditions will be managed by ‘hear and treat’ or at scene and these cases were not included in the model as we could not determine if they had one of the urgent conditions. Instead, we conducted a separate piece of work that allowed us to look at potential unexpected deaths by reviewing all deaths based on calculated risk of dying from our linked data. This is described in the next section.

Patient safety in prehospital ambulance care: exploratory structured judgement case record review study

Introduction

Patient and public involvement is recognised as an important component of high-quality health services research in the UK and internationally,49 and was an integral part of our programme:

The long-running PhOEBE project has had PPI at its heart from the beginning.

Maggie Marsh, PhOEBE programme PPI reference group member

Patients offer unique insights and knowledge of a clinical condition or experience of care that researchers may not possess. Our PPI reference group helped the research team to focus on meaningful and relevant issues, improving the overall quality and credibility of the programme. As the PhOEBE study was a publicly funded programme that was focused on improving ambulance care, it was crucial that patients and the public were involved, helping to improve the NHS and people’s health-care outcomes and experiences, moving from being ‘mere users and choosers to being makers and shapers of health services’. 50

Reference group

A PPI reference group was created at the outset of the programme to independently consider the PPI issues and perspectives relevant to the programme and advise the research team. We aimed to optimise PPI input by identifying innovative ways to involve patients and the public.

The PhOEBE programme PPI reference group included three lay members: Maggie Marsh and Dan Fall from the Sheffield Emergency Care Forum (which is a PPI group representing patients and the public in prehospital emergency care research in Sheffield and across the UK) and Andrea Broadway-Parkinson, a freelance Disability Consultant, patient and public perspective ‘champion’ and ‘expert patient/advisor’ working with YAS NHS Trust, focusing on patient safety, experience, patient voice and involvement strategies. Details are provided in Box 4.

These three PPI members had previous experience of involvement in ambulance-related research or ambulance service patient involvement. This was necessary given the complexity of programme. However, an important part of their role was to ensure that there was wider PPI involvement and participation at key points during the work by drawing on much wider networks, including hard-to-reach and vulnerable patient groups.

Recognition of the role of public representatives also grew in the PhOEBE programme. Initially, PPI representatives were acknowledged in reports or included in group authorship as members of a steering or advisory committee. Increasingly they were recognised as co-authors of publications and co-presenters at conferences. Dan, Andrea and Maggie’s work on the PhOEBE programme represented a broad spectrum of involvement activities from consultative to event co-design, co-presenting and as co-authors on research outputs.

The PPI reference group benefited from having a dedicated member of the research team act as PPI co-ordinator to support its activities across the increasingly complex and multiple workstreams within the programme.

Co-ordination and activities

Key components of the PPI co-ordinator’s role were to:

1. provide an accessible single point of contact

2. build and sustain relationships across the PPI reference group and research team

3. act as an advocate for the needs and wishes of the PPI reference group

4. translate the research and research team’s wishes back to the PPI reference group

5. respond to PPI reference group lay members’ input and ideas to maximise co-production and adoption of general collaborative principles and practice

6. learn more about PPI-related issues in order to transfer learning to other research projects and other external research partners/stakeholders.

The PPI reference group developed its own Gantt chart to clarify the PPI tasks and related timetable, roles and responsibilities. This guided planning of PPI reference group meetings and agendas, and enhanced integration of PPI into management meetings and vice versa, which created a sense of a proactive rather than reactive PPI function:

The contributors’ roles and requirements must be spelt out from day 1 to ensure everyone knows their place in the structure.

Dan Fall

A terms of reference document was co-written with reference group members, setting out the purpose of the group, responsibilities of members, general approach to working, accountability of members and co-ordinator as well as a mission statement.

The specific responsibilities of the group are outlined in the terms of reference document:

1. To review all patient-facing aspects of the planned project including writing lay summaries in the initial funding application and ethics applications from a patient and public perspective, and to suggest any changes that might be made.

2. [One member] to sit on the project steering and management committees, attend all committee meetings, feedback PPI-related activity to the committee (with support from the PPI co-ordinator from the research team when this was required) and report back to the PPI reference group.

3. To help review summaries of research findings for a lay audience.

4. To plan for and help disseminate research findings and wider PPI/engagement activities.

5. To develop and deliver key PPI-specific activities (e.g. consensus events, focus groups, workshops and conference events).

6. To maintain communication with INVOLVE and other panels or conferences relating to lay involvement in research.

7. To work with the PPI co-ordinator in any other PPI-related tasks as appropriate.

The PPI reference group meetings lasted 2–3 hours and were held at The University of Sheffield School of Health and Related Research approximately every 3 months with communication by e-mail in between. The meetings were attended by at least two PPI reference group lay members (MM, DF or AB-P) and the PPI co-ordinator, project administrator and members of the research team as required.

One PPI member (DF) acted as a link and attended the Project Management Group and Study Steering Committee when possible. This mechanism helped to ensure a lay perspective on significant decisions within the project was considered:

My role in PhOEBE has been to participate within both the PPI reference group and also to be the PPI connection into the management and steering group of the project. This has been a 5-year project where I have been intended to have a day per month for input, contribute to the PPI element of the project and also to be the PPI presence within the core of the research team.

Dan Fall

The PPI reference group and wider patient and public contributors played an essential role in the PhOEBE programme, helping to steer and define measures for further development reflecting both service provider and public perspectives. The group contributed to the qualitative study in workstream 1 by commenting on the study protocol, patient information, consent forms and interview schedules results and interpretation.

The PPI reference group members and members of the Sheffield Emergency Care Forum were active participants in the initial interview study and subsequent consensus workshop and PPI event. The members also provided invaluable help in recruiting members of the public via local PPI networks as well as personal and professional contacts. Their most innovative work came with the co-production of a PPI workshop to support the Delphi study and ensure a patient and public lay input into the final choice of measures for development in workstream 3:

I had the inspiration to increase [PPI] to a manageable number, perhaps 20, of lay people to deliberate, choose and vote on their preferences of the measures in a new consensus day, closely working with the research team to bring this to fruition.

Maggie Marsh

The broad aim was to develop a more interactive way to listen to those who used and cared about ambulance services beyond a mere ‘tick-box’ exercise, and also to meet the requirements of the PhOEBE research programme:

It seems that within the concept of involving the public and patient representatives within academic research there is first an idea that the subject matter must be understandable by lay people. This needs to be a priority issue at the outset of any project. If genuine lay people are being used there is a high chance that the subject might be very alien.

Dan Fall

This resulted in a paper, co-written by PPI reference group members, which was published in Health Expectations,27 after editorial advice to ensure that the ‘voice’ of the PPI reference group was clearer in the paper. This was a real success for the PPI voice in the PhOEBE programme. The PPI reference group has also presented posters51,52 about its work on the PhOEBE programme at both an INVOLVE and a 999 EMS Research Forum conference, with other conference and public event attendances pending.

As the PhOEBE programme drew to a close, the PPI reference group was instrumental in the creation of another ‘first’ for a research programme with the development of an animation video to support dissemination of the programme findings. Group members have invested a huge amount of time, effort and enthusiasm in this activity and their experiences will be shared at the INVOLVE conference in November 2017. The animation video can be viewed online. 53

There is no doubt that the PhOEBE programme has presented some real challenges and difficult concepts that have tested our PPI reference group. It has risen to this challenge and found innovative and new ways of ensuring that the views and needs of those who are the users of our ambulance services have been truly embedded in the programme. Its work has not only benefited this programme but has more broadly generated new knowledge and methods for meaningful PPI in research that will hopefully influence future research studies.

A more detailed narrative about the PPI contribution to the programme, co-produced by our reference group, is provided in a supplementary file at www.sheffield.ac.uk/scharr/sections/hsr/mcru/phoebe/ppi.

Introduction

In our original programme proposal and plan the primary intention of this final workstream (workstream 4) was to conduct further analytic work to test and validate the case-mix-adjusted indicators developed in workstream 3. However, as we have described earlier in this report, the substantial delays that occurred in obtaining and creating the linked data set meant that we were unable to conduct this work even with the extension period we had been granted. Instead, we have considered the conceptual issues related to the indicators we have developed, their value as ‘good’ indicators of performance or quality and their potential advantage over existing measures.

In our proposal, we also set out an objective to use the linked data to assess the costs associated with different types of ambulance response. We have been able to conduct this work and this is reported here.

What is a good performance indicator?

As part of the PhOEBE programme, we developed candidate indicators to assess the performance of emergency ambulance services. The process of development has included wide consultation with patients, ambulance service, commissioners, policy-makers and researchers. We have also calculated the indicators and examined their statistical reliability and whether or not case-mix adjustment makes them a potentially more reliable way of assessing performance or quality. To decide whether or not to recommend these indicators for inclusion in a national assessment framework, we have also considered wider questions about what characteristics a good indicator should exhibit.

There is a wide literature on performance indicators, some of which is focused on what makes a good indicator and how to choose and develop them. We have examined some of the publications issued by UK organisations, as well as a comprehensive review, to try to identify a ‘checklist’ of the characteristics of good indicators. This literature is not comprehensive or based on a systematic search as we are only trying to achieve ‘saturation’ of the ideas about the criteria that good indicators should possess.

Assessment of measures for the PhOEBE programme

Each of the measures developed for the PhOEBE programme were assessed against the six key criteria for good indicators by a small group of five ‘experts’, drawn from the programme steering committee, who assessed the indicators from the perspectives of health-care commissioners, clinical academics, ambulance providers and statisticians.

Each of the six key criteria for good indicators was broken down into several subcomponents (based on those identified in the literature). The PhOEBE programme’s indicators were assessed against each of the subcomponents by our expert group and the results were collated (Table 14). The PhOEBE programme’s ‘toolkit’, which set out the definitions and construction of each indicator, was provided to each of the panel to facilitate the assessment.

On the whole, all the measures met or partly met most of the criteria. This means that the measures are potentially good indicators of ambulance service quality and performance.

One of the panel members frequently felt that they required more information than was available to assess the measures, as shown by the high number of times they used the ‘do not know/unable to assess’ option. Our expert panel agreed that all the PhOEBE programme measures are relevant to people. This was also identified through the large amount of consensus work undertaken by the study in workstream 1. There were only 5 cases out of 510 ratings that were rated as ‘does not meet’. Four of these cases related to the survival from emergency conditions measure and there was some uncertainty about whether this measure is remediable or deconstructable. There was also some uncertainty about the reliability of the pain measure, which may be because the pain scales used to assess pain are subjective.

Summary

Overall, the set of indicators created appeared to provide an inclusive picture of different aspects of ambulance service care. They have relevance to different patient groups – for example, response time includes all calls, pain to a substantial proportion, accuracy of call problem identification and survival to the most urgent cases – and separate but complementary measures for patients who are and are not taken to hospital encompass all service users. Combined, they also fulfil the different domains of quality set out by the Institute of Medicine framework23 and some are relevant to more than one dimension. For example, response time relates to timeliness and efficiency; re-contact rates and unnecessary attendances relate to effectiveness and patient centredness (providing the most appropriate clinical response for patient need) and also safety; and change in pain score and survival are patient-centred outcomes that are relevant to a much larger proportion of the 999-call population than existing measures (which have focused on small and specific groups, such as cardiac arrest). There is no direct measure of equity but the ability of the indicators to measure potential differences across different geographical areas or populations can address this dimension.

In this respect, the set of measures or indicators that we have explored has achieved our objective of developing measures that can tell us something useful and valuable about the service and care delivered to all patients who contact the ambulance service and that consider both processes and outcomes that are important to a range of stakeholders with different perspectives.

Exploring the use of case-mix adjustment has revealed that, for some measures, there are patient and environmental factors that exert effects on processes and outcomes that are independent of the response and care provided by the ambulance service. Adding case-mix adjustment to performance or quality indicators takes these factors into account and allows a more meaningful and precise assessment of the impact of ambulance care that crude unadjusted measures, as currently used, cannot convey. This is particularly important if comparisons were to be made between different services or areas within services. It is also important for single-service comparisons over time. If there are changes in the 999 population, for example as a result of shifting demographics with an increasing proportion of elderly patients, and the effect that this has on processes and outcomes is not controlled for, there is a real risk that apparent changes in performance or quality will be erroneously attributed to poor or deteriorating ambulance service provision when this may not be true.

The PhOEBE programme has created a potential set of ambulance service performance and quality indicators that fulfil the requirements of good indicators and the dimensions considered important to good-quality health care. How they might be used in future is explored in more detail in Discussion and conclusions, Implications for practice.

Estimating the cost impact of alternative types of ambulance response

Summary of programme achievements

We set out an ambitious programme to explore and test the potential use of case-mix-adjusted measures to assess ambulance service performance and quality. We have achieved most of the objectives we set out in our original proposal. To fulfil the first objective of identifying potential indicators for development we used a systematic approach to the identification of potential measures and indicators by conducting and triangulating different evidence sources. This included collating and summarising the existing evidence on process and outcome measures from both policy and primary evaluation research and new primary research on patient views of ambulance service care. Consensus methods allowed us to rate and prioritise potential measures using an inclusive, interdisciplinary approach ensuring that a range of views and perspectives were considered in the final choice of measures or indicators for further development. The final set of measures identified for development reflected a range of important processes and outcomes that, together, provide the potential to make a balanced assessment of ambulance performance and quality that encompass all users of the emergency ambulance service. This could be achieved with a relatively small set of just six indicators that have relevance to patients, providers and commissioners. The six indicators were reduction in pain, accuracy of 999-call triage, response times, inappropriate non-conveyance to hospital, unnecessary transport to hospital, and survival for 16 emergency conditions.

The second objective was the development of a linked data set that could capture processes and patient outcomes for the whole episode of care, including hospital care, related to each individual call, not just the prehospital component. Despite considerable difficulties and delays we have, for the first time in England (to our knowledge), created a comprehensive data set that provides information on ambulance service response and patient care, hospital events and mortality outcome for > 150,000 individual 999 calls. For the third objective on indicator development, the primary purpose was to assess whether or not case-mix adjustment could improve the measurement of processes and outcomes, whether or not this was feasible and what characteristics might be important. We found that for four of the indicators (i.e. triage accuracy, inappropriate non-conveyance, unnecessary conveyance and survival) case-mix adjustment did identify patient and environmental factors that influenced the process or outcome being measured. This confirmed the value of taking this approach if measures are to be useful and meaningful in detecting changes that are a consequence of ambulance service care rather than other influences that are outside the control of a service. In particular, the significant effect that individual hospitals have on some outcomes was confirmed and highlights the difficulties of separating the impact of ambulance service care from that delivered by other parts of the health-care system. This means that the utility of the indicators and what they can usefully be used to compare varies. For the outcomes that are solely within the control of the ambulance service (i.e. triage accuracy, decisions not to convey to hospital and survival to arrival at hospital), the indicator could be used to make both within- and between-service comparisons. When other system factors (e.g. the hospital) have an influence, the indicator would be most useful for measuring within-service changes over time. The indicators also revealed that rates for some of the indicators were relatively low; for example, the rate of inappropriate decisions to leave patients at home was between 5% and 10% across CCGs. Survival to admission to hospital was 90.5–97.3% for 16 emergency conditions, which confirmed that a very small proportion of 999 calls are for immediately life-threatening conditions. This does bring in to question the value of using survival (or mortality) rates as an indicator of ambulance service quality.

The fourth objective was to test the indicators on a separate data set and further assess the validity and utility, but delays in obtaining the data meant that we were unable to meet this objective.

The creation of the linked data also allowed us to make a comprehensive assessment of the costs associated with different types of ambulance response and provide a method, using the indicators, for estimating the potential cost savings that could be made if decision-making about conveyance to hospital could be improved. This is the first time that calculation of costs for the complete episode of care, including contacts and re-contacts with the wider health system, has been possible.

The programme has been underpinned by a comprehensive component of PPI despite some of the complex concepts involved. Most people understand what the ambulance service do; understanding performance measurement, why it is needed and how it should be done is more difficult. By embedding PPI within our project, we were able to prioritise a potential set of ambulance outcome and performance measures, which have considered the preferences of professionals involved in prehospital ambulance care and also factors that are important to patient and public representatives. The animation developed with our PPI representatives will help disseminate key messages from this study around the way ambulance services have evolved and how their performance is measured to the general public.

Challenges and limitations

Clearly the biggest challenge to the programme has been the delays encountered in workstream 2 and the excessive time taken to obtain the linked data. Although we recognise that it was unfortunate that our programme coincided with a particularly difficult period for NHS Digital and that we had no control over these events, there have been substantial consequences. The related issues took so long to resolve that we had not received any data by the time the programme was due to end in May 2016 – 5 years after the programme had started. We were mindful that the National Institute for Health Research had invested considerable funds in this programme, so to have finished at this stage would have been wasteful of the money already invested. With careful management of the budget and allocating staff to other projects during the periods when we could make no progress, we were able to ask for, and we were granted, a 1-year no-cost extension to the programme. This means that we were able to complete the work planned in workstream 3, but this put considerable pressure on the team as we condensed > 1 year of planned work into 8 months. The delays do mean that we were unable to complete some of the work that we had planned in order to further test and validate the case-mix-adjusted indicators. We were also unable to re-assess their usefulness in a ‘real-world’ setting, in terms of both their relevance to the ambulance service and wider emergency and urgent-care system. It has also meant that of the four planned data sets, we have been able to process and use only one.

An important consideration of linked data sets is completeness. In this programme, although the overall linkage rate was high (> 82%), this was variable across different patient groups and was very low for the ‘hear and treat’ group because of the quality of recorded data about this patient group. This potentially introduces bias into our sample and our analysis. We assessed the patient characteristics of those with linked and those with unlinked data for all of the study patient groups and found very little evidence of difference for those that were conveyed to ED or seen and treated at the scene. However, the ‘hear and treat’ patients with linked data were older than patients with non-linked data and this limits the generalisability of these findings across the whole ‘hear and treat’ population.

There are other broader challenges and limitations that have arisen from our experiences in obtaining the linked data that have implications for the future use of the set of indicators that we have developed. First, the process of creating the linked CAD and ePRF data within the study ambulance services into a format that could maximise the number of calls traced at NHS Digital proved to be more complex and time-consuming than we had envisaged. Having worked through the process, lessons have been learned that would make this easier in the future but it does require a substantial investment of time to complete this work. Ambulance service information systems that enable linkage within and across services are still variable in terms of implementation, adoption and capability and trying to improve this is a work in progress. Second, even assuming that the NHS Digital data linkage service is operating normally, it takes time for tracing, matching and processing. There are also natural delays because NHS Digital’s own processes for cleaning and checking HES data means that this is not available for matching for several months. This means that using this type of approach is unsuitable for performance measurement that is near to ‘real time’. Data will always be several months old before it is useable. Third, once the linked data were obtained, the processing required to transform it into a format suitable for conducting the indicator model development was also complex and time consuming. Again, having done this and refined the processes needed, this would be less onerous to replicate but the resource needed to carry out this work cannot be underestimated. These challenges are not unique to England. Recent work in Australia linking health data across different jurisdictions found the same problems, that is, the processes remain costly and time-consuming. 61 However, the challenges are not insurmountable. In Scotland, an urgent-care data set linking NHS 24, ambulance service, primary care out of hours, ED, hospital and mortality data has been developed and contains complete patient pathway records from 2011. 62

We were fortunate to have research funds to support this work. We also requested a very large number of variables for the linked data from both the ambulance service and NHS Digital. Having constructed the case-mix-adjusted indicators and identified which variables are needed, a much smaller data set would be needed in future to replicate these measures, which would simplify the process. Nevertheless, without considerable investment or better data sources that are more easily linked, it will not be feasible for ambulance services to routinely create linked data for performance and quality measurement in the short term.

The construction of the case-mix-adjusted indicators was a complex process requiring multiple iterations to identify the most suitable predictive model. For some indicators, there were issues with missing data that limited the value of the model. This was particularly true for cases that could not be matched, which then reduced the number of incidents that were included in a model and potentially introduced bias into the measure. We also had to make some assumptions that need to be taken into account when interpreting the measures. For example, when measuring re-contacts it may be that some subsequent hospital attendances or admissions are justifiable and the consequence of a worsening problem and the original decision to not take a patient to hospital was correct at the time it was made. The indicator cannot tell us which individual decisions are ‘right’ or ‘wrong’, and we cannot assume that all re-contacts are the result of an incorrect decision – there will be some natural variance. The value of the case-mix-adjusted indicators is that they calculate a rate and it is variation in this rate that potentially provides a signal that the decision-making is changing over time, or varies between areas in the same service. However, there are limitations. The models fitted as part of the development of case-mix-adjusted indicators are an indication of what models may need to be used to enable ‘fair’ comparisons to be made between services, areas or time periods. We have demonstrated that direct risk standardised rates can be calculated to make comparisons between performance in different geographical areas within the ambulance service and how funnel plots can be used to identify outliers. As the models were fitted using data from one ambulance service, they may not be generalisable to all ambulance services in England. Before the indictors could be used to compare and monitor performance within and between all ambulance services, the models would need re-fitting using national data and this may lead to changes in the structure of models as well as the model coefficients. Furthermore, we have examined only the influence of case-mix factors that have been measured and that were available to us. There may be other factors that we have not explored that matter, and so before these indicators are used it would be important for policy-makers and other stakeholders to consider whether or not they believe that the case-mix-adjusted indicators do give a fair reflection of an ambulance service’s performance.

We have been unable to test the ability of the case-mix-adjusted measures to detect changes or differences, as was our original intention, which limits the usefulness at present. Further work needs to be conducted to establish if the models can be replicated with different data sets and to test whether or not they are sensitive enough to measure important changes or differences.

Implications for practice

The development of more useful and outcome-based performance and quality measures has been an issue of international interest and importance for a long period of time. Despite this, little progress has been made beyond the measurement of simple process measures such as response time in most countries. 7 The most commonly measured outcome has been survival from out-of-hospital cardiac arrest, which is relevant only to a very small proportion of calls (in England, this is currently 0.6% of 999 calls). 9 The UK has used a combination of system and clinical indicators for a number of years, but the limitations of predominantly process measures and an overemphasis on response time performance has been acknowledged. 16

A recent Delphi study identified potential measures from existing literature that could be used as key performance indicators for prehospital care and captured many of the same items that we found in our own reviews63 but the prioritised measures comprised single items. Little attention has been paid to case-mix adjustment with only one US study addressing this issue,64 although no outputs in terms of validated indicators have been identified. The advantage of the set of indicators that we have developed is that they are composite measures, for example including 16 emergency conditions in a single indicator, and incorporate case-mix adjustment where we have identified that this is important. Although they need further testing, there are real potential advantages in measuring a small but inclusive and comprehensive set of indicators. However, as we have shown, most of these indicators require linked data and the issues encountered in this programme suggest that it is not feasible to create the necessary data sets in a routine and timely way using current systems.

The problems associated with the creation of linked data sets and their use in research are a consequence of the much broader and more difficult policy issues concerned with use of technology and data. In England, the Five Year Forward View65 acknowledges the shortcomings of previous attempts to streamline and connect information across the NHS and there is a clear intention to address this issue with the formation of the National Information Board. Within this is an explicit statement of the need to bring together different data sources to support quality improvement and research. 65 The NHS Digital data and information strategy also sets out ambitious objectives to create better data and access to data that can support service planning and delivery and measurement of quality and performance. 66,67 Access to data is particularly challenging issue. Internationally, the opportunities that the use of ‘big data’ can potentially bring to improving and transforming health care are recognised but critical issues remain around how data are collected, stored and used. 61,68 In particular, important factors around regulation, privacy and the need to reassure the public that their data are used legitimately and safely still need to be overcome. A policy analysis by Heitmueller et al. 68 highlights how policy and legislation may help or hinder change in digital information gathering. We have described our experiences within the context of one programme of research but the solutions to overcoming the problems we encountered will need changes that require revisiting these broader issues at a national level.

We have considered the short- and long-term implications for practice and how the indicators we have developed might be used.

In the short term:

1. Two of the indicators require ambulance data only and so could be implemented. The indicator that explored alternative ways of displaying response time performance has already had an impact. In 2015, NHS England embarked on a programme of work to improve emergency ambulance response performance – the Ambulance Response Programme (ARP). 9 Members of the PhOEBE programme research team have also evaluated this programme. The main changes tested in the ARP were allowing additional time for call assessment and a revision of the 999-call categories to better reflect urgency and response needed. As part of this programme, there was also a revision of the current ambulance system and clinical quality indicators. The PhOEBE programme helped this work in two ways. First, we utilised the outputs of workstream 1 to support a workshop that reviewed the current quality indicators. Second, the work on the response time indicator has been adopted and the 75% within 8 minutes response time target has been replaced by reporting mean and 90th percentile response times for each of the new call categories. 69 The ARP changes were approved by the Secretary of State for Health in July 2017 and implemented in all ambulance services in England by November 2017.

2. The indicator measuring mean change in pain score also requires ambulance data only. In the ARP indicator review, pain management was identified as an important outcome. The national ambulance clinical indicators are currently being reviewed and there is now scope to include the pain measure within this revision for some specific condition types.

In the long term, progress will be dependent on resolving the broader issues around provision of and access to data. Current ambulance quality indicators use aggregated data. A basic principle for the indicators that we have developed is that they require individual call data to provide the information, such as demographics, condition and outcomes, needed to construct the indicator. Should data linkage become a simpler process, there would still be a burden on individual services to support the process by providing call-level data. One potential solution would be to design and support a standardised way of collecting individual call- or patient-level data that could be warehoused centrally. This could be a similar process to the recently developed Emergency Care Data Set that now provides a standard data repository for EDs. A national data set could potentially be a more efficient way of handling the individual records needed for data linkage. It would also provide an important resource for research that could also reduce pressure on individual ambulance services as they are increasingly asked to provide data for research studies but have limited capacity to support these requests.

If a national ambulance data set were created that could be linked easily to the Emergency Care Data Set, then it would be possible to use the indicator developed to measure the rate of unnecessary ED attendance in conveyed patients as this does not require inpatient or death data.

The three indicators measuring accuracy of call assessment, re-contacts and admissions for patients who were not transported, and survival do all need complete linked data. It will not be feasible to measure these indicators routinely until simpler, more efficient and timelier processes for linking data can be found. A national ambulance data set could in part facilitate this. It would also enable indicators to be measured centrally rather than separately by individual services, which would improve efficiency. If better and more meaningful performance and quality indicators are to be implemented then it also needs to be recognised that these are also more complex to measure and will need the resources necessary to support data collection and management, and analysis.

The mortality review provided a useful exercise and is a potential method for reviewing patient safety. Using the linked data to identify deaths and assess risk of death provided a structured and systematic way of sampling cases for further review. The process is resource intensive and not suitable as a regular monitoring tool but could be used on a periodic basis to add another dimension to quality improvement.

The case-mix-adjusted indicators may also have value as outcome measures for new research projects in which, for example, change in service delivery is being evaluated. If an indicator is relevant then the data collection requirements and methods for construction of the indicator are already specified or could be modified. A future national, patient-level ambulance service data set that has interoperability with related data sets in other parts of the urgent-care system, similar to that provided in Scotland, could enable the development of more sophisticated measures of performance and quality, based on those described here.

Recommendations for research

The different studies conducted as part of the PhOEBE programme open up a wide range of potential future research:

• The measures prioritised through the consensus studies should be further developed, validated and examined to investigate their importance, validity, feasibility, relevance and sensitivity to differences in services, service changes and quality improvement efforts in practice.

• New measures, such as patient-related experience measures, should be developed based on our understanding of what is important to patients using ambulance services.

• Further work could investigate how to use the indicator to monitor performance over time and the frequency with which the indictors could be calculated statistically and feasibly.

• The existing and future data sets linking ambulance, hospital and mortality data could be used to investigate the effects, safety and costs of different pathways and processes for a variety of clinical conditions and patient outcomes.

Conclusions

We identified and prioritised, through systematic reviews of the literature followed by a series of formal consensus processes with a wide range of stakeholders, a set of potential ambulance service quality measures that reflect the preferences of both services and users. We also created a comprehensive linked data set providing information for individual calls that extends beyond the prehospital component of care, although this proved to be a complex and time-consuming process. Six candidate indicators were developed using case-mix adjustment and, of these, four were found to need adjustment to make fair comparisons. Hospital was found to have a substantial effect on the process or outcome for two indicators, which means that these are suitable only for use at an individual service or system level. Other indicators could be used to make comparisons between regions or services. The complexities of both creating linked data and constructing the indicators means that, at present, they are of limited value as it would not be possible to measure them routinely. This indicator set, or subsets of indicators, could potentially be used to compare ambulance services or regions or measure performance over time after further testing to establish their utility when used in practice. Substantial improvements will be needed in both the collection and management of ambulance data and the mechanisms for linking data across services if more complex and comprehensive outcome-based measures are to be adopted for routine monitoring and quality improvement.

Contributions of authors

Janette Turner (programme co-lead, co-applicant) co-led the design of the programme and was substantially involved with all aspects of the programme, and led the drafting of the final report and synopsis and the assimilation of report components and appendices.

A Niroshan Siriwardena (programme co-lead, chief investigator, corresponding author) made substantial contributions to the conception, design and analysis and interpretation of the data.

Joanne Coster made substantial contributions to the conception, design, and acquisition, analysis and interpretation of the data, including leading the consensus studies and workstream 2, gaining approval for data linkage and accessing data, contributing to the systematic reviews and drafting these sections of the report and the sections on what makes a good indicator and the assessment of measures.

Richard Jacques made a substantial contribution to the analysis and interpretation of data through conducting the statistical analysis of data and developing the risk adjustment models.

Andy Irving made a substantial contribution to the acquisition, analysis and interpretation of the data through facilitating PPI in the study, undertaking the Delphi study and leading the writing of the PPI section of the report.

Annabel Crum made substantial contributions to the acquisition, analysis and interpretation and provided data management support.

Helen Bell Gorrod made a substantial contribution to the analysis and interpretation of data by leading on the economic analysis and drafted this section of the report.

Jon Nicholl made a substantial contribution to the conception, design and interpretation of the data, and advised on all aspects of the programme, particularly the development of statistical models and good indicators.

Viet-Hai Phung made a substantial contribution to the systematic reviews and the qualitative research through acquisition, analysis and interpretation of the data.

Fiona Togher made a substantial contribution to the acquisition, analysis and interpretation of the qualitative data through undertaking fieldwork, analysing data and writing a publication.

Richard Wilson made a substantial contribution to the design acquisition and analysis of data, particularly for the systematic reviews.

Alicia O’Cathain made a substantial contribution to the design, acquisition, interpretation and analysis of data by leading the qualitative interview study and participating in the consensus events.

Andrew Booth made a substantial contribution to the design, acquisition, analysis and interpretation of the systematic reviews by designing and leading this aspect of work.

Daniel Bradbury made a substantial contribution to the acquisition, analysis and interpretation of data for the systematic reviews and was involved in the consensus work.

Steve Goodacre, Ronan Lyons, Helen Snooks and Mike Campbell made substantial contributions to the conception and design of the study, provided methodological advice and contributed to the analysis and interpretation of data and the development of the indicators.

Anne Spaight, Jane Shewan and Richard Pilbery made substantial contributions to the analysis and interpretation of the data and the development of the indicators and also helped access to ambulance service sites and data.

Daniel Fall, Maggie Marsh and Andrea Broadway-Parkinson made substantial contributions to the design, analysis and interpretation of data through their role as PPI representatives for the study. They also took part in the consensus research and contributed to writing the PPI sections of the report.

Mike Campbell made a substantial contribution to the design of the study and to the analysis of data and the risk-adjusted models for the indicators.

All of the authors were involved with drafting and/or revising the report, approved the final version and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Publications

Coster J, Turner J, Siriwardena AN, Wilson R, Phung VH, O’Cathain A, et al. Prioritising Outcomes Measures for Ambulance Service Care: A Three Stage Consensus Study. In Society for Social Medicine 57th Annual Scientific Meeting, 11–13 September 2013, University of Sussex, Brighton, UK.

Coster J, Turner J, Siriwardena AN, Wilson R, Phung VH. Prioritising outcomes measures for ambulance service care: a three stage consensus study. J Epidemiol Comm Health 2013;67(Suppl. 1):A33–4.

Coster J, Turner J, Wilson R, Phung VH, Siriwardena AN. Prioritising Prehospital Outcome Measures with A Multi-stakeholder Group: A Consensus Methods Study. In International Forum on Quality & Safety in Health Care, 16–19 April 2013, ICC Excel, London, UK.

Phung VH, Booth A, Coster J, Turner J, Wilson R, Siriwardena AN. Prehospital Outcomes for Ambulance Service Care: Systematic Review. In Making an Impact: what is new in emergency prehospital care research? 27 February 2013, Novotel, Cardiff, UK.

Phung VH, Coster J, Wilson R, Booth A, Turner J, Bradbury D, et al. Prehospital Outcomes for Evidence-Based Evaluation (PhOEBE): A Systematic Review. In World Congress on Disasters and Emergency Medicine, 28–30 May 2013, Manchester Central Convention, Manchester, UK.

Phung VH, Coster, J, Wilson R, Turner J, Booth A, Siriwardena AN. Systematic review of prehospital outcomes for evidence-based evaluation of ambulance service care. Prehosp and Disaster Med 2013;28(Suppl. 1):S102.

Turner J, Coster J, Wilson R, Phung VH, Siriwardena AN. What outcome measures should be developed for prehospital care? Results of a consensus event. Prehosp and Disaster Med 2013;28(Supp. 1):S89.

Coster J, Irving A, Turner J, Wilson R, Phung VH, Siriwardena AN. How Should we Measure Ambulance Service Quality and Performance? Results from a Delphi Study. In 8th European Congress on Emergency Medicine, 28 September to 1 October 2014, Westergasfabriek BV, Amsterdam, Netherlands.

Coster J, Turner J, Irving A, Wilson R, Siriwardena AN, Phung VH. How Should We Measure Ambulance Service Quality and Performance? In International Conference on Emergency Medicine, 11–14 June 2014, Hong Kong Convention and Exhibition Centre, Hong Kong.

Coster J, Turner J, Irving A, Wilson R, Siriwardena AN, Phung VH. Moving on from Response Rates: Linking Patient-level Ambulance Data to ED, Hospital and Survival Data to Assess Quality and Performance. In International Conference on Emergency Medicine, 11–14 June 2014, Hong Kong Convention and Exhibition Centre, Hong Kong.

Turner J, Coster J, Wilson R, Siriwardena AN, Phung VH. Developing New Ways of Measuring the Impact of Ambulance Service Care. In 8th European Congress on Emergency Medicine, 28 September to 1 October 2014, Westergasfabriek BV, Amsterdam, Netherlands.

Togher F, O’Cathain A, Phung V-H, Turner J, Siriwardena A. Reassurance as a key outcome valued by emergency ambulance service users: a qualitative interview study. Health Expect 2015;18:2951–61.

Coster J, Jacques R, Turner J, Crum A, Nicholl J, Siriwardena AN. New indicators for measuring patient survival following ambulance service care. Emerg Med J 2017;34:e4.

Coster J, Siriwardena AN, Turner J, Jacques R, Crum A, Nicholl J. Multi-method development of new ambulance service quality and performance measures. Emerg Med J 2017;34:e2.

Crum A, Coster J, Turner J, Siriwardena AN. Creating a linked dataset to explore patient outcomes after leaving ambulance care. Emerg Med J 2017;34:e6.

Coster J, Irving AD, Turner JK, Phung VH, Siriwardena AN. Prioritizing novel and existing ambulance performance measures through expert and lay consensus: a three-stage multimethod consensus study. Health Expect 2018;21:249–60.

Irving A, Turner J, Marsh M, Broadway-Parkinson A, Fall D, Coster J, Siriwardena AN. A coproduced patient and public event: an approach to developing and prioritizing ambulance performance measures. Health Expect 2018;21:230–8.

Siriwardena AN, Akanuwe J, Crum A, Coster J, Jacques R, Turner J. Preventable mortality in patients at low risk of death requiring prehospital ambulance care: retrospective case record review study. BMJ Open 2018;8.

Turner J, Jacques R, Coster J, Nicholl J, Crum A, Siriwardena N. Development of risk adjusted indicators of ems performance and quality (PhOEBE programme). BMJ Open 2018;8.

Data-sharing statement

Owing to the conditions attached to original ethics agreements, there are no data available for wider use. All queries should be submitted to the corresponding author in the first instance.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, CCF, NETSCC, PGfAR or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the PGfAR programme or the Department of Health and Social Care.

Can we risk adjust?

Each measure was entered into a Microsoft Excel spreadsheet and the assessment against each criterion entered enabled a composite score to be obtained. Figure 8 shows an illustrative example of the scoring sheet.

FIGURE 8.

Assessment of potential measures.

For the next stage, the results of the criteria assessment were considered and measures included or excluded using the following criteria.