Using patient experience data to develop a patient experience toolkit to improve hospital care: a mixed-methods study

A scoping review of sources of patient experience feedback in the UK

We conducted a scoping review comprising academic databases, grey literature databases and websites, and supported this with our own knowledge from the field and that of our study steering group. We also hand-searched citations contained within returned documents. We identified surveys from the existing reviews34,35 and then conducted our own search of academic databases to update and focus on the UK only. We used grey literature and websites to identify other types of PE feedback that we knew, because of their non-validated status, were not likely to be found in academic journals, but more likely to be discussed in ‘guidance’ documents and commentaries. We adopted a scoping review method, and not a systematic review, because flexibility of search terms within grey literature was paramount to enable as wide a range of PE feedback to be returned. Comprehensibility of sources available in the UK, although important, was secondary to our aim of developing a characterisation system and categories that we anticipate could be applicable to other types as they emerge. We were informed by a five-step framework for conducting scoping reviews40 as shown in Table 1.

Developing a list of ‘defining characteristics’

We established a consensus team to develop a list of 12 key descriptive characteristics to help understand the role of different feedback types in QI. This list is provided in Table 2. The team comprised the principal investigator (PI; Professor in Psychology of Healthcare), four health services researchers (one psychologist, two social scientists and one sociologist), two design researchers (concerned with the presentation and usability of patient feedback), one health-care improvement specialist and one patient involvement facilitator. The list developed iteratively through the following stages:

• The PI first used the evidence referred to above, combined with own knowledge of QI and PE, to produce an initial list of nine characteristics and presented this to the consensus team.

• The consensus team then added a further four characteristics to make 13 characteristics.

• One researcher (RP) attempted to use this list to characterise six of the types returned through the review, finding that twelve of the characteristics worked effectively and only one did not so it was removed. This was ‘whether or not the feedback related only to specific patient groups’, which was not possible to ascertain from descriptions of the types.

This list of 12 characteristics was then presented to the study steering group to ensure that it made sense beyond the consensus team. This process led to clarification of the definitions and potential variability (character options) of each characteristic as listed in Table 2.

Five broad headings (i.e. nature of data obtained, mode of feedback collection used, availability of feedback, perspective captured and defined role in QI) were used to group the 12 characteristics.

Assigning ‘characters’

These characteristics were then applied by the researcher RP to all returned types of PE, which generated ‘raw’ descriptive categories (see Appendices 1–4). These were checked by two other members of the team before finalising.

Four categories of types of patient experience feedback

We then used our characteristics list to further analyse, understand and subdivide the descriptions contained in these ‘raw’ tables. As well as enabling us to provide a more nuanced presentation of the distinctions between PE types, this process led us to more indicative titles for the categories than those we used as appendices titles. The refined categories and their subcategories are shown in Box 1. The distinctions that we make between them are now described, highlighting potential implications for roles within improving PE.

There were 17 types of feedback that fitted into the first category of ‘Hospital-initiated quantitative surveys’. 22,38–40,42–58 Common to almost all of these types of feedback is that the data are predominantly quantitative, initiated by hospitals, targeting patients and not carers, and involve a significant delay (caused by processing) in providing information back to the organisation. However, closer inspection reveals a distinction between those that are mandated for high-level organisational use (for whole organisation or whole A&E or whole maternity departments) at regular but infrequent intervals, and those that are offered as voluntary tools for use as and when an organisation decides. The former most clearly exhibit accountability features: providing organisational-level data (within parameters defined and initiated by the organisation) and validated to make generalisations and comparisons (between organisations or over time) when conducted for large samples. On the other hand, with the exception of one survey, Hospital Care & Discharge,49 the voluntary surveys can be applied at any level at a timing to suit, or are especially designed for use within a local service or specialty, such as the Intensive Care Experience (ICE) Questionnaire,55 without prescribing regularity. Unlike the mandatory surveys, only some are clearly validated. Potentially, these more flexible surveys that elicit local-level information offer more scope for informing or monitoring local improvement of PE. Only one survey, Your NHS Patient Survey Wales,46 does not conform neatly to this subdivision. This survey is strongly recommended for use, not mandated, and is designed for use at any level. This implies more flexibility, and that perhaps it has been designed to inform or monitor local improvements as well as to provide accountability.

We call the second category ‘Patient-initiated qualitative feedback’ and include 12 feedback types here1,9,41,46,59–65,75 that exhibit common traits: they provide qualitative data (applicable to any level of the organisation), are initiated by patients on an ad hoc basis (whenever they choose to) and the feedback is available to the organisation quickly (referred to as in real time). The concept of validity is not applicable because all data are provided on a case-by-case basis. Within this category, the significant distinction is between those types that are formally supported, which could be because they are mandated to do so (e.g. by complaints,46,60,61,75 concerns9,62–64 and NHS choices),9 or because they choose to adopt a system (e.g. to set up a ward-based Facebook page or buy into iWantGreatCare65 to organise their feedback). Other types have no supporting system in place and include informal feedback (e.g. compliments, thank-you cards) that is received but not perceived as data requiring attention or processing. We include a caveat here because some hospitals could have more formal systems for handling these (we know anecdotally that this happens) but this is not widely acknowledged or articulated as a process. This subcategory also includes websites external to the organisation [e.g. Facebook, Twitter, Mumsnet,41 Google reviews] where patients/carers may upload feedback, but there is no guarantee that this will be viewed by hospital staff. Other less well-known sites could also exist on the internet. Care Opinion1 currently spans both subcategories: it is offered as a formal system of data management for a fee, if hospitals choose to adopt this. If not formally adopted, the platform could still be used by patients to upload feedback that may or may not be viewed by the hospital.

In summary, this category offers a different kind of ‘data’ than that offered in category one and, therefore has a potentially different role within QI. In category one, feedback offers evidence-based scope for use in benchmarking and monitoring of organisational trends. Category two feedback provides more local-level information that would not be valid for use in that way. It exhibits some characteristics (e.g. nuance, specificity) that suggest potential use within local QI processes, especially problem identification. Currently, feedback within this category is, however, presented largely on a case-by-case basis and not as collated data ready to use. This makes its proposed role as a data source more tentative than the surveys of category one, and we return to this issue in Discussion.

We name the third category ‘Feedback & improvement frameworks’, and this includes seven types of feedback66–72 with some common, defining features: feedback is exclusively qualitative, can be collected for any level of service by a variety of qualitative research methods with a varying degree of prescription in this regard. Interviews are common but focus groups, observation and shadowing all feature here. All types elicit rich data that take time to process. All have a defined role within QI, albeit to a varied extent. Feedback collection is initiated by staff, but, in striking contrast to surveys (which cover issues deemed important to organisations about their service delivery), qualitative methods are used in ways designed to tap into patients’/carers’ authentic voices. Like category two, the data elicited are qualitative but they differ significantly owing to embedding within qualitative methodology, which elicit rich (collated) data sets ready to use.

The nature of the third category can be explored further using two subcategories. In the first subcategory are two types of feedback that focus primarily on eliciting authentic voice (Emotional Touchpoints66 and Discovery Interviews). 67 Their associated guidance makes reference to the use of feedback to make changes, but this aspect is not covered in detail. The second subcategory includes frameworks for linking the collection of feedback (still attempting to tap into patients’/carers’ authentic voices) to QI techniques. With reference to QI, these frameworks do not offer data suitable for accountability (nothing generalisable for large samples). They do offer information appropriate for the QI process (problem definition and monitoring) but the specific ways in which they do this varies considerably. Some advocate linking feedback directly into the continuous learning process [Patient Journey = AR,68 Kinda Magic = links to metrics collected separately,69 Fifteen Steps and Always Events = mainstream QI approaches such as plan–do–study–act (PDSA)]. 71,72 Both EBCD and accelerated experience-based co-design (aEBCD) recommend collecting qualitative feedback of impact to assess perceptions of how the service has changed and also suggest collecting other measures about the change, for example, cost-savings to a service. Three also stand out for the way they use feedback as data for problem definition. Within EBCD/aEBCD and Always Events, feedback is interpreted together with staff and patients/carers in a process of co-design so that contextual meaning informed by those who work in the service can be added. This is an interesting category that appears to push the boundaries of how we consider feedback as data within QI. We return to this observation in the Conclusion section.

Finally, we identify a fourth category of miscellaneous, ‘Other’, with two types of feedback:73,74 FFT73 and HowRWe,74 which do not fit into any of the above three categories. They are both surveys that hospitals can initiate, asking standardised questions, but unlike those surveys in category one, they are not designed to capture large numbers of data (lots of questions) infrequently, but instead they are very short and designed to be used more frequently, potentially providing a more continuous flow of PE feedback. The FFT has only one question and HowRWe has four. They both allow qualitative comments to be added and they can both be applied to any type of health-care setting; however, this is where their similarities end. Most significantly, FFT is mandatory in England, whereas HowRWe is a voluntary tool and is therefore much less widespread. The data arising from the HowRWe standardised questions are validated to provide comparable data over time and between areas, whereas the data arising from the FFT standardised questions are not. Potentially, the HowRWe tool has more obvious potential for the measurement and monitoring of trends within QI over time and between areas than FFT. The comments provided by both tools could be used within this process. These comments can be likened to the data arising from the feedback types included in category two: qualitative and context specific. However, just like these data types, comments are not provided as collated data ready to be used and the steps to enable them to be used as data are not specified.

Hospitals currently have limited access to data that can potentially help to inform and monitor local patient experience improvement

Of the mandated PE feedback types available, none of these would appear immediately suitable for informing and monitoring a local improvement process (e.g. ward level). Mandated feedback currently comprises quantitative survey data [the national inpatient surveys for whole organisations,22,42–46 A&E,43,47 maternity departments,43,44 complaints and liaison service data,9,59–64,75 one form of online feedback (NHS Choices)9 and the FFT results]. 73 As explained above, mandated surveys are most suitable for accountability purposes but do not provide locally relevant data that are accessible to those who need it, in a timely manner35,76 that would be required for informing and monitoring the QI process. The qualitative, locally applicable information collected via the FFT test, mandated for England, offers potential within the QI process;25 but this proposal is also fiercely questioned,26 described as a laudable ambition thwarted by the quantitative rating system that currently forces hospitals into achieving acceptable response rates at the expense of considering and utilising the qualitative comments effectively. There is interest in the increased use of other mandated feedback (from complaints and liaison services) by coding and theming into data sets. 27 However, there are challenges to these proposals,28 relating to system practicalities (collation of case-by-case complaints), the nature of the story told (complex and difficult to code) and availability (often infrequent and inconsistent in style). In short, seen from a QI perspective, mandatory PE data (national surveys, FFT, complaints/concerns and NHS Choices) currently appear to offer little ready-to-use data, with respect to informing and monitoring local PE improvement.

The potential for other types of feedback to help inform and monitor local improvement process is not yet clear

Other types of feedback are available should hospitals wish to use them. Hospitals could use voluntary surveys of category one, these offer more granular data and can be used more flexibly if analytical capability exists. 38 Similarly, there are proposals that online comments could be used more: they are context-specific, qualitative, and provide almost instantaneous information. Some online platforms such as Care Opinion1 and iWantGreatCare65 are being developed and offered to hospitals for this purpose, and some hospitals/wards are establishing Facebook pages as a dedicated place to collect feedback. As well as supporting use of these formal platforms, there is an emerging interest in harnessing ‘the cloud of patient experience’ from social media that exists in informal ways (e.g. Twitter, Facebook, Google). 77 All of these proposals warrant further exploration. We suggest that the process of using PE data within the improvement process needs further conceptualisation in itself before we can judge the comparative value of these different feedback types and we use some observations on category three feedback to develop this proposal.

Beyond metrics: what we can learn from ‘feedback and improvement frameworks’

Category three feedback significantly differs from category two feedback despite them both being qualitative. Each category three framework is concerned with eliciting the authentic patient voice using in-depth qualitative methods. Two (EBCD/aEBCD and Always Events), overtly attempt to develop shared meanings (staff and patients/carers) from the feedback. This indicates a shift away from the notion of patient/carer feedback as a static metric (objective data) that can be used to directly state what should be improved, and then be assessed again to measure impact. Within EBCD/aEBCD this is described as a co-design and co-creation process involving techniques that aid critical, collective reflection. 25,78

We propose that the way in which EBCD/aEBCD and perhaps other such frameworks use the patient voice to inform and to monitor progress is variable and tentative, and that the commonalities between these ways are ripe for further exploration. Of relevance to understanding this further is the arrival of the term ‘soft intelligence’ within QI more broadly, in which value is placed on understandings gained through everyday interactions and caution is urged with respect to reducing such insights to metrics. 79 When it comes to making decisions about how the seemingly ever-growing stream of PE feedback should be used, these broader conceptual developments about how the patient/carer voice can inform change appear extremely relevant and could add much to traditional concepts of QI as articulated by the ‘Three faces of Measurement’31 over 20 years ago.

Owing to the flexible approach taken to search terms, our scoping review may not have revealed all potential feedback types available in UK hospitals. In addition, because of the occasionally subjective nature of these search terms, a repeat exercise by others may not yield exactly the same results. This is also true of the characterisation and categorisation exercises in which some subjective decisions were made. In some cases there was ambiguity and we used our characteristics list as a sensitising framework rather than an absolute.

Data collection

The fieldwork took place between February and August 2016. The University of Leeds ethics approval was secured in October 2015. All participants gave written, informed consent. Ward staff took part in focus groups and management staff took part in individual in-depth interviews. Ward staff mostly represented opportunistic sampling and management participants were sampled for maximum variation. Ward staff predominantly encompassed senior and junior nursing staff, support workers and the inclusion of allied health professionals in some of the focus groups. Management participants were drawn from a range of roles occupying middle- and senior-level hospital management, such as PE managers or heads of PE, matrons, heads of nursing (and their deputies), research leads, medical, quality, risk, governance and performance directors. The bulk of interview participants worked directly in or managed PE teams.

Seven focus groups and 23 individual interviews were conducted. Focus groups ranged from three to seven participants and two management participants were interviewed as a dyad. The average length of an interview was 55 minutes and 45 minutes for a focus group. In total, 50 participants took part in this qualitative study. Two topic guides were devised; one for the data collection from ward staff and another for management participants. Headline topic guide questioning was derived from the literature. Focus group questioning centred on what types of PE feedback the participants received, how they engaged with it and responded to it, and where/how it fitted in with their everyday clinical work. Interview questioning explored the different kinds of PE feedback available to the trust and how these were generated, prioritised and managed at the level of the ward, directorate and whole organisation. The formats of the topic guides and that of interview questioning were flexible to allow participants to voice what they considered to be important. All focus groups and interviews were conducted face to face in staff offices, digitally recorded and then transcribed by a professional transcriber. All participants gave written, informed consent to take part in the study. Author B collected all interview data. Authors A, B and C all collected focus group data. All are experienced qualitative health researchers with doctorates in their respective fields.

Analysis

Authors A and B took the same five interview transcripts and each independently developed a provisional descriptive coding framework. These five transcripts were chosen as those that were representative of the whole interview data set in terms of spread across the trusts and general content. The same exercise was repeated for the focus groups, albeit with three transcripts. Authors A and B held an intense analysis session where they met (along with author D) to discuss the differences and similarities in their coding frameworks, although there was general parity among them. Author A then returned to the selected transcripts and immersed herself in the data in order to devise an overall meta coding framework that would allow for data from both the interviews and focus groups to be coded together. This meta coding framework sought out themes on a conceptual level rather than a descriptive level; that is, rather than simply describing what the participants discussed, author A looked for the differing ways in which PE feedback was approached conceptually across the participants involved in both methods. Differences and similarities were identified, with Author A noticing that participants discussed the topic at different levels, with the management interviewees tending to view PE feedback in a macro manner (both explicitly and implicitly) and the ward staff focus group participants viewing it in a micro manner. The meta-level coding framework was checked with author B for representativeness and accuracy. After slight modification, author A then coded all transcripts and some subthemes were modified as coding progressed. Author A conducted further interpretative work in order to write up the findings. Initially, we began by conducting a classic thematic analysis90 but realised that this was not sufficient for our needs, as thematic analysis often relies on portraying a descriptive account of participants’ narratives. Instead, we conducted a high-level conceptual analysis. The analysis was wholly inductive and, as such, we did not structure it on any existing theoretical frameworks.

Setting the scene: what are the sources of patient experience feedback?

All participants were able to name a wide variety of the types of PE feedback that they had encountered and interacted with in their professional roles. This took the form of formalised written sources such as the FFT, complaints and compliments, thank-you cards, Patient Advice and Liaison Service (PALS) communication, patient stories, NHS Inpatient Survey, local surveys and other initiatives such as ‘You Said, We Did’. Senior leaders within the organisations spoke about Care Quality Commission (CQC) inspections and the use of social media as outlets for feedback, although ward staff paid less attention to these sources of data. When first asked to discuss PE feedback, ward staff spoke about the more immediate, direct, ‘in the moment’ verbal feedback from patients on their ward that they received in an impromptu manner during the course of a shift. This often took the form of patients complaining verbally (in an informal manner) about their care or the environment to the clinician caring for them or to a more senior staff member. Conversely, it also included spontaneous thanks or praise given in an interpersonal exchange. In this study, we focus on formalised sources of PE feedback and we discuss factors surrounding their effective use, as per our key areas of interest and research brief. However, it should be acknowledged that informal feedback was often used by ward staff in a timely way to improve the experience for the needs of a particular patient.

What is impeding the effective use of patient experience feedback?

We chose to focus on the factors that are impeding the use of feedback rather than an account that paid equal attention to the factors that were assisting it. Although there were certainly instances where individual personnel and small teams had instigated processes and ways of working that were beneficial, these accounts were localised and not of sufficient importance to most participants about the topic at hand. Furthermore, attempts to improve issues identified in feedback sometimes led to unintended consequences, which further problematised an already complex and fraught task. When participants talked in positive terms about PE feedback, they often spoke of idealised situations or what they would like to see happen in the future rather than what was currently happening in practice. Overwhelmingly, the participants interviewed across the data set pinpointed significantly more negative factors within their current working practices when trying to use patient feedback than positive factors and this is, therefore, where we place our analytical attention.

There is a clear division between a macro and a micro understanding of how participants discussed PE feedback within their health-care organisation. Management participants commented on feedback and the use (or not) it serves at the level of the organisation, whereas both ward staff and management pinpointed the problems at a micro level with the function and usefulness of the individual data collection sources.

Recommendations for change

There should be an organisational emphasis placed on the principle that all feedback collected must have the ability to be meaningfully utilised by those providing front-line care. Otherwise, it becomes unethical to ask patients to provide feedback that will never be taken into account. An immediate concentration on quality over quantity is important, with a strategic focus that takes the priority off the collection of data and onto its utilisation. Secondly, ward staff need to understand the formalised sources of feedback that they are receiving from their patients before they can begin to use them. There are two approaches here, possibly complementary, but both are difficult to achieve within the current NHS climate. One is that significant work needs to be undertaken upstream by PE teams to relay the data to ward staff in an accessible, straightforward and engaging manner. Another would be for a proportion of ward staff to be given robust training in how to understand and act on the feedback that they receive from their patients. This should encompass both qualitative and quantitative analytical techniques and QI methodologies. One without the other is futile and allows staff to see only half the picture. The macro influences the micro here because if PE teams were not overwhelmed by a volume and multiplicity of data sources while simultaneously underprovided with analytical resource, then this could potentially be accomplished.

At the level of the organisational structure, teams that have been tasked with a narrow focus on individual sources of data (e.g. FFT, PALS, complaints) should be working more closely together, with a strategic emphasis placed on learning across the organisation from the variety of feedback sources. This does not necessarily require extra resource but a firm commitment to different ways of working that aim to understand the big picture instead of paying attention to the treadmill of targets and metrics per individual data source. If, as the data indicate, a significant number of senior health-care leaders believe in the importance of PE feedback, then it is important that PE feedback is valued as a priority and effectively incorporated into management agendas and practice.

Strengths and limitations

To our knowledge, this is the first study that has paid significant attention to the system-level macro factors that are inhibiting the use of PE feedback. Other commentators10,11,37,87 have noted some of the micro-level factors that we have identified here but not how they interact with structural issues, which further compounds the issue at hand. A limitation may be the way that we have chosen to concentrate explicitly on the problems surrounding the use of PE feedback, owing to the emphasis that participants themselves placed on this aspect. It could be that a more traditional barriers and levers style write-up (seeking to pay equal attention to problems and solutions) may have uncovered different, or more worthwhile, suggestions for change.

Co-design and co-designing

Co-design refers to the collective creativity of designers and people not trained in design working together in the design development process. 95 In a co-design scenario, there is a duality of roles for all participants. The designers are designers in the process, contributing their professional expertise to process and ‘product’, and they are facilitators, guiding participants through the design activities. In this case, the designers are also researchers. The co-design participants are simultaneously ‘client’ (or end users), stakeholders within the context of use and ‘junior’ designers taking on some of the creative process. This specific project included the additional roles of co-design participants who were action researchers and health services researchers.

There has been an increase in design methods and approaches within health-care development and research. UK organisations such as NESTA96 and the Design Council97,98 have produced reports that discuss the need for, and benefits of, using design methods and involving patients and wider stakeholders in the development of public services. In their 2004 RED paper for the Design Council, Cottam and Leadbeater98 discuss using design to help transform public services, citing chronic disease as a case study. In this report they argue for an end to top-down approaches and an embrace of co-design. 98

With the NHS calling for new empowered patient relationships, and design theorists such as Manzini99 and others100 believing that design is important in shaping new societal futures, the design profession appears well placed to answer this call. Furthermore, it is indicated that these ‘Design-led participatory approaches, help the NHS think differently’. 97 Muratovski101 discusses the changing face of design, the ever-complex problems that designers face and the movement away from ‘product creation’ to ‘process creation’. This can be seen within most of the papers within the Chamberlain et al. 100 review, which use user-centred, participatory and co-design methods and cite the use of designed physical artefacts and health informatics as key outputs. The diverse range of design-led activities and responses found within the review include exhibition, environment and service design solutions. This encouragingly supports the potential of design-led activities in the health-care environment with the potential to play an active role in health agendas and within interdisciplinary research teams.

Prototypes and prototyping

The design process is based on experiential learning cycles. In these, participants in the process (usually designers) create real ‘things’ that enable people to respond, interact and react. This creates new knowledge and understanding about a specific ‘problem’, ‘context’ and proposed ‘solution’. For a designer, the process of drawing or making something is not always to transcribe ideas from their heads, but as a means of orchestrating a conversation with themselves and others. Externalising emergent thoughts and making them tangible allows designers to extend their thinking, distributing it between conception and perception simultaneously. 102 When others are invited into this ‘conversation’, the materiality of drawings or prototypes makes it easier to share and develop knowledge in a common language, unbounded by barriers between disciplines or hierarchies. The mock-ups and prototypes created play a vital role in facilitating the conversation between co-design participants as well as between the co-design group and others beyond the co-design group. The process of making, either collaboratively or as an independent enquiry, elicits deeper forms of knowledge such as tacit, behavioural or experiential knowledge. The result can elevate research findings into meaningful, impactful outcomes that are sensitive to the ‘messy’ reality within which they hope to contribute.

These externalised tangible ‘things’ are called prototypes. In the health-care innovation landscape the word ‘prototype’ is sometimes applied to PDSA cycles of improvement. In this context, they are usually planned quite carefully to be carried out at a specific point in the future. This differs to their use in the design world where a designer spontaneously sketches or ‘mocks-up’ models using materials available within the immediate vicinity, ‘exploring’ an idea and how it might become real.

Design iterations and cycles

When mapping out a design process to develop a specific idea, strategic thinking is applied to a programme of successive prototypes that aim to build the designer’s knowledge about what will work for the target user(s) in context, and what will not. Over a programme of prototypes, these intend to explore what is technologically feasible, acceptable and desirable to the user, or economically viable (in production, set-up and running costs). For example, some prototypes may test specific functional features of the proposed solution whereas others may explore form and aesthetic appeal. Additional functions that prototypes might fulfil are to ‘challenge’, ‘provoke’ or ‘catalyse’ in relation to assumptions or perhaps imagination. Successive prototypes, the cycles of gathering feedback about them and designing responses to the feedback, are termed design iterations or design cycles.

A description of workshop 1 (Bradford)

One of the biggest challenges in a co-design process is how to engage people effectively. If this fundamental aspect is not addressed appropriately, then issues of dominance, hierarchies, not accessing genuine experiential knowledge, or tacit professional knowledge are more likely to distort the process. One of the inherent features of co-design is the ‘production’ of something: this production can also refer to the creation or the sharing of knowledge. However, knowledge is an intangible concept that can mean different things to various stakeholders participating in the same process. Using co-design practices to problematise what we mean by knowledge and the sharing of knowledge can be a useful activity.

In workshop 1, the ‘LEGO Serious Play’ methodology103 was used to stimulate dialogue about the participants’ experience and expectations of patient feedback through the medium of 3D model-making. LEGO Serious Play methodology is used to enable participants to access subconscious and tacit knowledge, surfacing awareness of the way one has done things automatically in the past that was not explicit. It also helps to make ideas tangible, which can reduce assumptions. 104 The ‘LEGO Serious Play’ process gives all participants the space to make something physical, then talk about what they have made and why. This inherently gives all participants a platform to contribute and allows them to make their contribution with respect to the thing that they have created. Using the artefact as the vehicle for discussion and a form of media can protect people’s contribution from criticism by others and can help to level hierarchies and empower people. LEGO and the LEGO Serious Play Methodology was used to facilitate creative/participatory ways of involving people in the toolkit research and improvement work (Figure 1).

FIGURE 1.

Examples of LEGO Serious Play exercises for workshop 1.

Workshop 1 outline

The aim of the first workshop was to identify and explore the current experience of patient feedback data with workshop participants. At the end of this workshop we hoped to have further insight into:

• What forms ward staff are used to receiving patient feedback data in and what forms would they like the feedback in? (Where and when would they like to receive it? How often? etc.)

• How to get from data to action?

• How might action be recognised and recorded/celebrated?

The workshop was conducted according to the following sequence of activities:

1. Introduction to the project and research to date – a research team summary of the project to remind the participants of the aims behind the workshops.

2. What is a co-design experience? – scene setting to the workshops and the co-design concept.

3. LEGO Serious Play – a brief introduction to LEGO Serious Play and its use in this context.

Participants were divided into table groups of six or seven people to form a mix of trust sites and roles and they were asked to undertake the following introductory exercises:

• Skills building – getting all the participants familiar with using the bricks and building to create metaphors. This also introduced the idea of full participation from everyone.

• Individual model building –

  • Team 1: to focus on the what/when/where and how of PE data.

  • Team 2: to focus on how to get from data to action.

  • Team 3: to focus on recording and recognising the impact of PE data.

• Landscaping – models were placed in groups to look for themes and links to form the basis of areas for ideas to move forward.

• Recording and sharing – participants used word cards (placed on the models) and cameras to record the models and what they mean. Feedback was given to the table by individual participants, then summarised table by table to the rest of the group.

After the completion of the first workshop, the outputs from the workshops were examined by the project researchers and the research design team alongside the documentation about how the groups got on with the activities (taken on the day, in note form). The findings were analysed to identify areas and patterns of concern and interest that could be used to frame the second workshop. Responses to the questions posed to each of the three teams were summarised in the following diagrams (Figures 2a, 2b and 2c). These were shared with the participants at the start of workshop 2.

FIGURE 2.

Responses to question (a) what might you need to use patient experience data?; (b) give your vision for how patient experience data would be given to you; and (c) how would you recognise that patient experience data has been used? D, data forms; E, environments; P, people and relationships.

A description of workshop 2 (Leeds)

As mentioned, an analysis of the information collected from workshop 1 was used to structure the content and exploration of workshop 2. In this second workshop, we examined how three identified common areas of interest could be used to begin to think about both simple and innovative ideas for effectively using the patient feedback resources to hand. The three common areas that formed the basis of the enquiry in workshop 2 were:

• the different types of data available (data forms and uses)

• the people who use and create PE data (people and relationships)

• the places in which this data are used (environments and timing).

Workshop 2 outline

This began with a brief recap and a sharing of the findings from workshop 1. The workshop 2 exercises were then introduced. These exercises were used to give the participants the opportunity to explore each of the areas identified. Dividing the participants into three groups, we took turns to investigate each of the themes, collecting information through a series of participatory research design exercises. All the participants were given the opportunity to rotate and contribute to each theme. To make the conversation context-relevant we suggested that on this occasion people remain in their own trust groups (patient representatives were given the opportunity to choose to stay with a particular theme or to move round with a trust). Different activities were set up on three tables and groups rotated after 40 minutes. The activities are described below.

Table A theme: thinking about different data forms and their uses

On this table, two activities were designed to generate a conversation and to collect feedback on people’s experience with existing forms of patient feedback and typical uses for this on their ward.

Exercise 1: what forms of data are you familiar with and what are the positive values of having access to this data?

Using a template, participants were asked to make a list of different types of PE data/information that people have seen or used and write alongside this any positives of having access to PE data in this form (Figure 3). Participants were initially asked to work in pairs, before their responses were collated onto one list per table; 15 minutes were allotted for the exercise.

FIGURE 3.

Thinking about forms of data and their uses template.

Exercise 2: ideas for using and sharing patient experience data in different forms

In this exercise participants were asked to think about different PE data formats and consider what new ways this information could be used on the ward through both formal and informal means. The participants were given blank pre-printed card templates and stickers of different data forms that represented different types of patient feedback (Figure 4). A new card was completed for each type of feedback form identified. After selecting a feedback form participants were asked how the feedback could be used, what effect this might have and what resources might be needed to implement the use of the data. Along with instructions, the following notes were printed on each card to help guide the participants:

Think about the form – How it arrives on the ward and is shared with people? – How people might use it? – When – where, etc.?

How might it be adapted? Might it work better on a screen – on a mobile app – in a folder – a conversation?

Is it a quick win or will it need a change in culture?

There were 25 minutes allotted to the exercise.

FIGURE 4.

Cards and stickers for describing how different types of patient feedback might be used.

Table B theme: thinking about people and relationships

The focus of this theme was to think about how we could encourage people to better record, communicate and make use of PE data. Again, two exercises were designed to facilitate this conversation and collect information.

Exercise 1: recognising and understanding roles and responsibilities – What are the positive values of the community and how it works?

In this first task, participants were asked to make a list of all the different people’s roles that they encounter in a typical day. Then they were asked to add a list of the positive aspects of how people currently work in and inhabit the ward (Figure 5). Again, participants were initially asked to work in pairs, and then collate examples onto one list per table. There were 15 minutes allotted to the exercise.

FIGURE 5.

Thinking about people and their relationships template.

Exercise 2: ideas for making use of our relationships and community

In exercise 2, we asked participants to think about people and relationships, and to consider in what new ways PE data/information might be shared, acted on, evidenced, recognised and celebrated on the ward. Using cards and stickers representing a cross-section of service provider and service user communities, participants were asked to choose a person/role and fill in a blank card with an idea of how they might share PE data, what this sharing might achieve and what resources might be needed to make this happen. Participants were encouraged to think of different types of interactions and roles; as an individual, one person to another, one person to a group, as a group, etc. (Figure 6). Along with instructions, the following notes were printed on each card to help guide the participants:

Think about the relationships between people? Times when people meet? Different times of the day? Different roles and responsibilities? Different types of interaction between people? Is it a quick win or will it need a change in culture?

There were 25 minutes allotted to the exercise.

FIGURE 6.

Cards and stickers for thinking about the different patient feedback stakeholders.

In these exercises, participants were asked to think about how we could make our physical environment more responsive to PE data to help promote ownership of PE data by staff and to celebrate its effective use. Participants were encouraged to reflect on the use of existing spaces and activities: handovers, ward meetings, displays, noticeboards, suggestion boxes, etc.

Exercise 1: what spaces do we use in our work? What are the opportunities? What are good spaces?

In this first task, participants were asked to make a list of all the different spaces they move through/use in a typical working day, then add a list of the qualities of the spaces identified (Figure 7). Participants were initially asked to work in pairs before collating examples onto one list per table. There were 15 minutes allotted to this exercise.

FIGURE 7.

Thinking about patient experience data and the environment template.

Exercise 2: ideas to make better use of our current spaces

In this task, participants were given cue-card pictures of typical health-care spaces and asked to discuss and write ideas on how and where PE data might be shared, acted on, evidenced, recognised and celebrated in these spaces, as well as adding or suggesting any missing spaces (Figure 8). Participants were encouraged to think about where we currently see evidence of patient feedback in our environments and whether this feedback was from a locally generated or a formally collected source. The participants were also asked to consider if their ideas represented ‘quick wins’ or required some form of technological intervention and a change in culture. There were 25 minutes allotted to this exercise.

FIGURE 8.

Cue-card pictures used to facilitate a discussion about PE data and the ward environment.

Three primary themes came out of the data from workshop 2: ‘Forms of data and their uses’, ‘People and relationships’ and ‘Environments’.

A description of workshop 3 (Harrogate)

After scene-setting and introductory talks, the participants were introduced to the principal activity of the day, which was to work through the first toolkit prototype. A number of copies of the toolkit folders were handed out, which allowed small teams from each trust to work through the material. There were 40 minutes allotted to this exercise.

The participants were told that the prototype was informed by ideas and suggestions from the first two workshops and their subsequent evaluations. They were asked to complete the toolkit by working through different activities and considering the questions in context of their own wards or environments. As they worked through the toolkit, they were asked to make notes on the processes: how easy or useful or difficult they found the activities within it (using a different coloured pen or post-it notes to differentiate between comments and the completion of the activity). Groups also gave suggestions for resources that they felt might help the completion. Members of the research team were on hand throughout the workshop, to observe and assist in the collection of feedback.

After working through the toolkit, the participants were asked to consider the experience and give feedback on the following questions. Feedback was either written on the worked through copy of the toolkit or collected by project researchers.

Feedback questions and prompts included:

• How could they see this working in your setting?

• How would they capture the activity – summary? As a team and to share?

• Were they able to follow the steps and activities?

• Do the steps and questions make sense?

• Did they like the form – did they have any suggestions about another way of presenting the material?

• Is it similar to other PE feedback activities they have done in the past?

• Are there any missing sections – (changes in the order) or materials such as communication materials required?

Feedback: prototype v1, action research phase 1

As described, the toolkit is intended to provide a framework and process guidance to enable ward teams to work with PE data. The aim is to drive changes and improvements aimed at improving patients’ experiences. Importantly, version one aimed to ‘walk’ the user through a process of using the PE data, understanding key messages from the data, identifying opportunities for improvements and then structuring a ‘project’ to make and test the proposed improvement.

The feedback from prototype version one, used in the first AR phase by the action researchers, highlighted several high-level tensions or conflicting interests and a number of secondary issues and cross-cutting themes. There was also one specific highlight noted: the single-side A4 sheet flow chart that outlined a summary of the process the toolkit was walking users through. This had been immensely useful to both the action researchers and the ward staff at numerous points throughout AR phase 1. Everyone felt that this was an incredibly useful feature of the existing content and it should be maintained and enhanced in subsequent prototypes.

The high-level tensions were:

• The way that the toolkit was used as a document versus a facilitated and peopled process.

• The depth of information in the toolkit. The tension between overall guidance versus specific detail.

• The ward teams had a focus on ‘current patients’, whereas the majority of PE data were from ‘previous patients’.

• Ownership – bottom-up versus top down. Who owns the process and who owns the toolkit? A ward leader, a ward team or a central support department (improvement team or PE team)?

• ‘Idealisation’ of toolkit use in context; the ward staff were taken off the ward for the initial toolkit development. When this was tried out on the ward, it was realised that there were elements that would never be used on the wards.

Secondary or cross-cutting themes were:

• ‘Skills gap’ – there is a ‘gap’ in the skill base of the ward teams that is essential to utilise PE data, regardless of this toolkit.

• Ward capacity – there is limited ward capacity and competing priorities for staff to do the basic care and nursing essentials in their role profile.

• Mindset – the toolkit needs to impart a positive mindset to improvement as a ‘core activity’. This includes being open-minded to feedback from patients, reflective, considering small and achievable changes and seeing improvement as a continuous process.

• Appearance – the finished product needs to ‘sell’ it, look ‘easy-to-use’ and not ‘burdensome’; for example, the physical size (i.e. being too big) was seen as a barrier to engagement.

• Format – needs to be ‘punchier’ and bring the topic and process ‘alive’; the folder would be a useful repository for shared learning and resources ‘collected’ or ‘developed’ along the way.

• Content – not overwhelming, user specific (multiple different users?).

Iterative prototyping process

Based on the feedback summarised in Feedback: prototype v1, action research phase 1, a design process was outlined that sought to address or query, perhaps even challenge, the high-level tensions and secondary themes. The resulting ‘steps’ or iterations in the process are shown in Figure 11, which shows an illustration of the various prototypes.

FIGURE 11.

An illustration of the various prototypes developed in the iterative prototyping process of this toolkit.

The first step was to strip out the content completely, keeping only the ‘backbone’ of the flow chart. Various improvements on the flow chart were explored to address the issues relating to ‘Appearance’ and ‘Format’. Content was then gradually reintroduced throughout subsequent prototypes, in a tiered form to explore what content was most important and who it would be useful to. The next stage was to purposively explore issues relating to who the end user(s) might be and who might ‘own’ or control the process. The next prototype explored issues relating to the ‘Skills gap’, ‘Ward capacity’ and ‘Mindset’. The final two prototypes explored issues relating to content.

A detailed description of the design iterations and related prototypes (v2–v5) is included in Report Supplementary Material 1. This includes a description of feedback from each preceding AR phase, changes between previous prototypes and ‘current’ prototype, what the ‘current’ prototype is seeking to establish and how it was presented to ‘users’ in the subsequent AR phase. The overall process is shown (see Report Supplementary Material 1).

The final prototype (vFinal) following all design iterations and feedback is shown in Figure 12. This version embodies a toolkit that will be used by a facilitator to assist ward teams in going through a process of collecting and using PE data to make ward changes that improve their patients’ experiences. It includes a refined flow chart of a six-phase process that a facilitator will lead them through. Each phase is structured into a three-tier hierarchy of detail such that the facilitator can share more or less detail with the ward staff depending on the level of understanding sought by the ward staff.

FIGURE 12.

Final toolkit prototype.

It is suggested that toolkit variations that could be requested, ordered or downloaded included:

• full toolkit and resources bound together

• toolkit and resource packs bound separately (with or without external folder)

• flow chart on its own (in various sizes)

• ‘high-level’ detail for ward teams (either in a pack or unbound)

  • – flow chart and ‘Russian doll’ level 1 for each phase

  • – full toolkit on its own (bound) without resources.

Product

As a product, this toolkit has to work across a variety of contexts in acute and community health-care settings. There are also significant variations in the ‘staff’ users. The final design has been robustly developed with a variety of potential users and contexts. Full evaluation still remains to determine its impact, ease of use and its transferability to other contexts not involved in the development. As it stands, it is not without its limitations. It requires a skilled and trained facilitator with specific knowledge about eliciting, collecting, collating and interrogating PE data to present such data back to ward staff and assist them in identifying relevant themes for improvement. This person also facilitates a ‘design’ process of generating ideas for improvement, testing them and scaling them up.

However, initial signs and early feedback from the ward staff, patient representatives and QI staff that were involved in the project is extremely positive and the early evaluation signs indicate some improvements. Attributing these specifically to the toolkit (as opposed, for example, to the increased staff awareness arising from involvement in the project) is harder to say.

Process

One design method currently being applied in health care is the EBCD methodology. 78 It was first used in 2005, by HCPs who looked to design in response to creating a more patient-centred NHS. 105 Co-design methodologies were used to develop a replicable free online toolkit. However, this case study is an example of disciplines outside design using design methods without including designers in the activity. This has led to incomplete use of EBCD where process steps that are perceived to be too challenging or unfamiliar are omitted. Because of this, EBCD is often criticised by designers106 for its limited tangible service improvement and its lack of ideation tools, and is often described as ‘design like’ rather than designerly. Challenges and questions have been raised about the level of innovation actually achieved.

The approach used in this work to design and co-design the toolkit was both novel and challenging in a positive way. Starting with a co-design process led by designers to develop the first prototype, it was followed by an immersive, AR phase of testing and developing the initial prototype before closing with four iterative cycles of prototype evolution and ‘user’ feedback; this process has never, to our knowledge, been utilised before. In the final implementation of a health-care intervention, context is often influential in determining whether or not interventions are successfully and sustainably adopted and used. This means that interventions need to be context-sensitive and adaptable. The approach described here was purposively deployed to enable the research and design team to respond to the wide range of context variations described above and it enabled them to design and develop a toolkit that would work across a number of these contexts.

The co-design approach used to generate the initial prototype, working with staff from six different ward settings across three different hospitals, utilised designerly methods of ‘making’ to enable all participants to contribute equally. We used LEGO Serious Play as the underpinning protocol and LEGO as the media for making as it has, in our previous experiences, proven to be an accessible approach and medium to deal with complex topics like this. This inclusion and attention to equity ensured that we elicited details about variations across contexts and, to an extent, these were catered for in prototype v1. The workshops that this co-design phase was structured on were appropriately conducted in a neutral venue, taking the ward teams, patient representatives, service improvement staff, researchers and designers away from their usual contexts of work.

The immersive AR phase tested v1 in different ward contexts and, through the experiential knowledge co-created with the ward teams, insights about the v1 toolkit were generated, challenging some of the assumptions that had been made by the co-design participants in the workshops.

The final iterative prototyping phase was challenging in that it was often difficult to get representatives of the ward staff and patient and public involvement (PPI) representatives together with the researchers and designers to explore the practical and perceptual added value of proposed design changes. These dialogues were often facilitated by the action researchers, using physical prototypes to illustrate various design modifications and alternatives. This led us to consider the notion of boundary objects with regard to the prototypes and further, with regard to the action researchers.

Susan Leigh Star107 is credited with proposing and defining boundary objects, describing them as objects ‘which both inhabit several intersecting social worlds and satisfy informational requirements of each’. She went on to suggest that boundary objects were vague, had strong cohesive properties and were flexible and recognisable across cultures. Henderson108 paraphrases this to describe boundary objects as agents that socially organise distributed cognition.

It is useful to clarify the scope of what an ‘object’ is. In the context of boundary objects, an object may or may not be a physical artefact or thing. It could be a computer program, a space, a theory, a drawing or even a person. The object is something people (or, in computer science, other objects and programs) act towards and with. 109 The notion of boundary objects has been studied further with specific reference to ‘products’ and to issues of knowledge ‘translation’ or ‘transformation’. 110,111

In the context described in this chapter, the discussion about what a boundary object is, is ongoing. For example, in this project the boundary object is classed as the PET prototypes. However, it could be considered that the boundary object expanded beyond this to include the action researchers; or perhaps the action researchers became a part of each prototype. The specific definition of what the boundary objects were in this project is not necessarily important. What is important is that in the field of health-care innovation, in which it is often difficult to host face-to-face co-design workshops and in which it is so important to consider context, boundary objects become a useful concept to consider prototypes and perhaps people or spaces that help to transfer the prototypes between stakeholder groups.

This process raised several interesting questions for the design researchers:

1. For the nurses and midwives involved in the co-design phase – did taking them off their wards and out of their context of professional practice for those initial co-design workshops make them think about their practice in an idealised way, building in design features to the toolkit based on assumptions of how it would be used in idealised practice?

2. Did the immersive AR phase ‘uncover’ these idealised assumptions?

3. The challenge of enabling ward staff to participate in face-to-face co-design activities is increasing as pressures on their time grow. Did the immersive AR phase and the subsequent physical prototype iterations work sufficiently as boundary objects to enable ward staff to engage sufficiently in the later stages of the design process?

4. The transition of the toolkit end user from ward staff to facilitators was based on an emergent realisation (from the first AR phase) that there was a knowledge and skills gap in the ward teams that was vital to following a process that utilised PE data. A question arises for us about an unintentional bias that may or may not have been introduced to this transition by the action researchers who gradually found themselves acting as facilitators of the toolkit in the first AR phase to ensure that projects got started.

5. During the prototyping phase, the action researchers initially began by facilitating the sharing of the prototypes with the target users in their context of use. However, their role gradually shifted to one of primary user. This was because of a recognition that, no matter how the toolkit was designed, there was a gap in ward team ability, capacity or capability to use the toolkit: a gap that the action researchers could fill. When this was recognised, and a toolkit ‘facilitator’ was accepted as the new target user, the designers began designing with, and for, the action researchers as representatives of facilitators who would use the toolkit with the ward staff in their context. This toolkit facilitator role was being prototyped at the same time as the toolkit was. How much these were two distinct prototypes or actually one emergent prototype, and whether or not this made one boundary object or two interwoven boundary objects, remains a point of intense discussion within the team.

Our original ‘heuristic statements’

Table 4 summarises each of our original heuristic statements, indicating how they were articulated in PET prototype 1. Further detail about each (including its location in existing theory) is then provided.

Participation

We were committed to the principle of participation and democracy that, as modelled by Reason and Bradbury,126 is central to AR. This meant seeking to involve as many as possible of those groups of people whom this research sought to serve and who had experiential knowledge of the area of study, as co-researchers. At the funding stage, senior staff from three hospital trusts had committed to allowing two clinical teams from each to take part, and the process of nominating which clinical teams (we refer to hereon as ‘wards’) had already taken place. We then ensured that all teams were represented at the co-design workshops so that they could take part in ‘owning’ the PET that their teams were to test and refine. A mix of different levels of nursing staff attended the workshops (matrons, ward managers, midwives, senior sisters, staff nurses and health-care assistants) and, at the final workshop, were formally invited to take part as co-researchers (‘practitioner co-researchers’ and ‘patient co-researchers’) in the AR. As well as ward staff, we invited those in relevant organisational roles (e.g. from PE teams) to the co-design and subsequent AR. We recognised that others from wards or from elsewhere in the organisation could join as the process developed and we wished to retain this flexibility to a changing context throughout.

We also invited members of the public with an interest in each of the hospitals to join in the co-design and then the AR. Although not currently in hospital, these people were viewed as potential patients who could advocate on behalf of current patients. In the co-design workshops, we linked one ‘patient’ with each ward, with a view to maintaining this relationship throughout the AR.

Finally, we invited co-researchers who could bring some form of generalised expertise and we called these ‘advisory co-researchers’. These were members of the wider research team and two improvement specialists. We maintained the title ‘action researchers’ for CM and RP as they performed the co-ordinating role. Table 5 provides a summary of all co-researchers (n = 64) who worked alongside CM and RP over the course of the AR. Our expectations of the levels of collaboration from each of our co-researchers remained flexible and pragmatic, balancing our desire for participation with our respect for their time constraints. 127

Action research cycles

The heuristic statements were tested and refined through cycles of AR that took place on the six individual wards. We framed these using the cyclical model of AR for organisational change128 that outlines four iterative stages: (1) construction of a shared understanding of an issue that people have come together to address, (2) planning actions appropriate to context, (3) implementing based on these plans and (4) evaluating whether or not intentions have gone to plan, with a view to revising the shared construction. Such cycles are informed by a broader theoretical understanding, which in the model is termed ‘context’. This means an understanding of why the issue is important, and what changes could potentially be made.

In our case, this understanding of ‘context’ is captured in our original heuristic statements, which we took into each ward. We then worked with individual teams to apply these to their clinical area and to plan, implement and evaluate successes. The ward-based AR cycles developed at different speeds and with different foci, so our conceptual understanding of the heuristic statements was influenced not only within but also iteratively between each area, with learning from one situation being brought to inform another. The cycles were split into two phases: between February and September 2017, we worked mainly with heuristic statements 1–5, and between September 2017 and March 2018, we worked mainly with statements 6–8. We used a concept that we called an ‘action research hub’ to maintain a sense of collective research across the six wards. Hub meetings took place at the beginning, middle and end, with the mid-point signifying a pause to collate and review learning so far, with a view to informing all ward cycles for the remaining months.

It is important to note our approach to ‘evaluation’ within these ward-based cycles, as AR is often critiqued for its failure to do this aspect well. 129 We focused on ‘evaluating’ the different heuristic statements that we derived from PET prototype 1 to assess if they made sense and to improve their articulation within a refined version. This took place using reflective methods outlined in Story-dialogue technique. An evaluation of how the PET as a whole enabled teams to work with PE feedback and how/whether this led to improvements was the subject of a separate evaluation study (see Chapter 6).

Story-dialogue technique

We used the story-dialogue technique130 to reflect on practice throughout the cycles. Story-dialogue technique is a form of evaluative questioning designed to generate new knowledge about what happens in practice and from these insights, decide on responsive actions. Box 2 provides an illustration of the nature of these questions, which we adapted for purpose.

Broadly, we used the technique in two main ways. First, it guided our own action researcher reflections where CM and RP kept reflective journals after each meeting or other significant encounter. 131 These journals played a dual purpose in informing the evaluation (see Chapter 6) as well as our AR. We therefore agreed with the evaluator a set of prompts for all entries that addressed the overlapping needs of the evaluation and those of the story-dialogue technique. In total, CM and RP made 119 journal entries.

Second, we periodically prepared an account of what we (CM and RP) collectively thought about one or more heuristic statements and used a variety of methods to share and explore our reflections with others. We made pragmatic decisions about what would work for different co-researchers at different times. These included written invitations, which received three responses, three focus group discussions at the mid-point hub meeting with a total of 19 participants, five group discussions back in the wards with a total of 12 participants, and two individual interviews. Within each of these settings, we provided each co-researcher with a written synopsis of our understanding of one or more statements of relevance to them and asked them to comment from their perspectives. We recorded and transcribed all those involving verbal responses.

Consultation exercises linked to the co-design

Running in parallel to the second half of the AR were co-design activities (in addition to initial workshops), as indicated on Figure 13. These comprised consultation exercises on two significant reiterations of the toolkit product, organised at each of the three participating trusts, which as many of the co-researchers as were able to come, attended. They also comprised development meetings at which the research team met with the design team to interpret and utilise the consultation feedback. Although these co-design activities focused primarily on the format of the product (how it looked and how it could be used), these discussions were open-ended enough to explore some of the theories behind our evolving heuristic statements. Therefore, it is important to acknowledge their contribution, albeit informal, to our developing ideas.

Rapid thematic analysis at the mid-point

At the mid-point, a rapid thematic analysis was conducted to inform PET prototype 2 in which we scanned the reflections for key findings, shared and checked these at the mid-point hub through the focus groups, then summarised them for the design team.

Rapid thematic analysis towards the end point

Prior to the end-point hub, a further rapid thematic analysis was conducted in which we scanned the reflections for key findings and provided these to the design team who produced an (almost) final version of the toolkit that they could share and discuss with co-researchers at the final hub.

Full thematic analysis at the end point

When all activities with co-researchers were complete, a full thematic analysis was conducted of all documented data (the reflective journals along with all transcribed accounts of others’ reflections). The action researchers CM and RP first did this individually and then came together to discuss and create a synthesised account of what their heuristic statements had changed and developed into. This is documented in Our action research story and, as it is still considered as provisional, will be the basis of future discussion and development with co-researchers beyond the formal research programme.

What happened in each ward?

Although it is not our aim to provide an in-depth story of the journey of each ward using the PET, we do provide a snapshot of key activity in Table 6 to serve as a reference point from which to understand how and why our heuristic statements changed. The table summarises the main messages contained within patient feedback received for each ward and an overview of PE work that they subsequently undertook. More detail of what was achieved in each ward, along with implications for PE, is provided in Chapter 6.

How and why our heuristic statements changed

In this section, we document how our original heuristic statements were revised as a result of AR. We mainly worked with statements 1–5 in the first phase and statements 6–8 in the second phase. Table 7 summarises these revisions and their articulation in the final version of the PET. Here, we combine the two so that we have seven final statements. The AR revealed the details about what they imply with respect to the steps and activities that are outlined in the final PET. Notably, we concluded that most activities require substantial facilitation and support for hospital teams at all levels. We highlight these requirements as we explain our revised statements.

A multidisciplinary approach: an ideal exposed

Originally, we all agreed that PE initiatives needed input from all types of health-care staff, but interestingly, all those attending the workshops and, therefore, those who became the leaders and core members of AR teams for the wards, were nurses. We learnt that by situating the PET at ward level this meant that nurses automatically took on the leadership role for it even though we had not specified this, because they ran and managed the wards. Staff from other disciplines tended to have a more transitory relationship with the wards. In every case, one or two leaders for the initiative self-nominated (ward leader plus one other in some cases) and committed to inviting others into the process as required.

In fact, we learnt that it was beyond our capability, in all cases except one, to invite others in from non-nursing disciplines to work with the ward leaders on PE. It was not that staff did not work with other disciplines regularly (they had daily ward rounds and other such meetings with doctors and allied health professionals), it was that the PE remit did not appear to be appropriate material for these forums. These existing forums focus on immediate patient care, so we sought out other places where multiple disciplines may be able to hear feedback, reflect more broadly and tackle issues together, but here we had no success. In B (i), C (i) and C (ii), concerns about communication with doctors were evident in their patient feedback, but, although efforts were made in B (i) to escalate these to the multidisciplinary care quality meeting with doctors, the forum did not meet regularly enough to enable this. In C (i) and C (ii), the nurses could not easily envisage any existing multidisciplinary team forum that would prioritise these concerns (focusing instead on issues deemed higher priorities such as patient safety or staffing cover). So, to bypass this problem, the nurses chose other issues to work on that they felt could be solved by them, rather than require a multidisciplinary team.

Only in B (ii) (elderly rehabilitation), in which there was an already thriving multidisciplinary approach to whole-patient care, were we able to truly engage other disciplines. In this case, their PE feedback advocated the need for more social activities, and this clearly supported their care objectives for rehabilitation, so they were able to progress this issue across disciplines (e.g. occupational therapy, physiotherapy and nursing).

Although success in engaging across disciplines was limited, we did have some more success in engaging with different levels within the nursing hierarchy, although the ways in which more junior staff engaged varied according to the ward leaders’ management style. Where the ward leader already supported regular team meetings [B (ii)] or training away-days [A (ii)], involvement opportunities were greatest. However, even in these cases, it appeared unusual for junior staff to think outside their defined roles to develop new areas of PE work (an issue that appeared to be at odds with the spirit of creativity and participation that we were trying to inspire), leaving many issues in the hands of the already overstretched ward leaders. In all other cases, few ward team meetings took place, so staff involvement beyond our meetings with ward leaders was extremely limited. As action researchers, we had to take a flexible approach, continually seeking to identify and develop connections across the ward teams and between departments or disciplines, treading a fine line between posing potential alternative structures, roles and activities and not being perceived as disruptive of a ward manager’s established systems.

In the main, we remained narrowly focused on our relationships with the ward leaders and realised how difficult it is to carve spaces for the collective working that had been envisaged in PET prototype 1. Tellingly, the exercises that had been included to support a multidisciplinary and team approach were left uncompleted in all our wards and, therefore, do not feature in the final PET, which advocates a more dispersed method of engagement led by a facilitator seeking out and developing connections whenever and wherever possible.

Patient and public involvement and collecting current feedback: combining two ideas successfully

We initially saw the two aspects of ‘PPI’ and ‘collecting current feedback’ as separate ambitions within the PET. In fact, there was a natural combination of these two ideas. At first, co-researchers had agreed that having patient representatives to work with staff teams was essential, but that their specific roles would develop in accordance to need. When the AR began, it became evident that there was a real thirst from staff for feedback from their current patients in a format that they did not already have. They wanted credible (informative and up-to-date) information that would help them to identify actual changes that could be made (see Triangulation of different feedback: a false promise for more on this). Patient representatives in four of the wards felt able (with support) to address this thirst and talk to patients and relatives on behalf of the staff to gather feedback.

Experimentation took place on A (ii), B (i), B (ii) and C (ii) to develop a model of patient representative interviews. In this, a prompt sheet of four open questions was used to very flexibly guide a conversation with a significant proportion of patients and relatives (approximately half of the patient population over approximately one week) about what was important to them with respect to their experience on the ward. Patient representatives were then involved in presenting key concerns back to staff teams. On wards A (ii), B (ii) and C (ii), the representatives also revisited the wards to gather follow-up feedback about the PE initiatives undertaken. Staff wanted this feedback, but patient representatives also benefited from ‘seeing the fruits of their labours’, and whether or not changes had been appreciated by patients. Where this model was not adopted [(C (i) and A (i)], the primary interest of the ward managers (at least at the start of the project) was on handling existing feedback more effectively, and this combined with the patient representatives’ interests in contributing in other, more advisory ways.

Where this model of patient representative involvement in data collection did take place, staff were extremely impressed with what it provided them with. They knew it revealed only a snapshot (not statistically representative) of PE but they valued it as a way of getting detailed, relevant feedback in a useable form. Staff also felt that patients were more likely to be open to a patient representative than they would be to a uniformed member of hospital staff. We observed that feedback was not necessarily revealing things that staff did not already know, but that hearing it directly from their patients gave them impetus to make changes. In B (ii), there had been many staff changes and, therefore, a period of instability prior to the project, and several remarked on how this had helped them refocus on patients’ needs. In A (ii), the ward manager explained that in busy, pressurised wards, it is possible to let standards of compassion and communication slip; periodically hearing the direct patient feedback serves as a reminder of the need to keep these standards high.

Success of the model was the result of a combination of staff desire for this type of feedback with a patient representative who was willing, skilled and with appropriate personal attributes to undertake this sensitive role. With both aspects in place, it would appear that the intended ‘representative’ role plays out and that they can draw attention to marginalised groups’ needs. In B (i) the representative was vocal about the needs of those patients who were seen as medical ‘outliers’ in a surgical ward. In C (ii), the representative helped staff to understand preferences of visitors to patients from the south-Asian community. In A (ii), the representative championed the needs of relatives or carers who could also be as elderly, vulnerable and lonely as the patients. As action researchers, we worked with trust volunteer teams to support these representatives appropriately and helped to nurture the supportive working relationship between patient representatives and staff throughout.

Triangulation of different feedback: a false promise

Led by the heuristic statement around triangulation, we set out to achieve a ‘combined’ analysis of different types of feedback to present to teams. There was a sense of anticipation at the outset that this step in the toolkit would allow teams to get access to all the feedback that exists about them and to view it in an accessible format. In fact, we learnt that this was highly problematic. We found that the NHS in-patient survey data did not contain any information about individual clinical areas, with the exception of the Maternity In-patient survey, which did contain broad themes, albeit quite out of date. Within FFT data, we found little use for the quantitative ratings because these were, in the main, consistently high, thus indicating no sense of trend or themes. We focused, therefore, on the qualitative comments and found that, where quantities were high [over 1000 per month in A (i)], these could be helpfully categorised into topic areas (e.g. communication, medication). However, these lacked detail about contexts and nuance, and they required staff to fill in the substantial gaps with their own knowledge; therefore, the patients’ own perspectives remained weak. Although we could at least derive topics within this high turnover area, the development of topics from existing data were impossible almost everywhere else, either because patient turnover was slow [B (ii)] or because there was a long gap between collection and the comments being available to senior ward staff. Feedback about a ward from other sources (complaints, PALS, social media) was either sporadic or specific to situations and times, so that combining with FFT into categories and themes was, at best, difficult, and at worst, misleading. Placing one-off events into categories gave the impression of generalised themes when this was not necessarily the case, causing unnecessary upset when presented to one ward team.

In four wards, patient representatives collected additional feedback from current patients and these interview responses were added to the mix of information available. Although all these types of feedback did need to be considered together, the concept of triangulation does not explain what we did to achieve this. Rather than cross-referencing different sources to arrive at an objective instruction, validated by its occurrence in multiple sources, we introduced an interpretive process for assessing the content of all the types of feedback that we had collated. This involved discussion with staff from PE teams, or senior ward staff who had a knowledge of typical issues and the frequency at which these arose for the ward. We then organised and prepared these into three or four topics for discussion that could be presented to the clinical teams. For each clinical area, this involved CM and RP, the patient representative who had collected the feedback and, in Trusts A and B, members of the PE teams. Although the ward leaders were invited to this, they did not have time to take part. We found that there was no capacity within any of our three trusts for PE teams to collate and organise multiple feedback sources ahead of these discussions. As action researchers, we brought research skills to this endeavour (collation, qualitative interpretation) and we saw little capacity to perform these roles among the trust teams.

Patient experience team involvement: limited resources, other priorities

In essence, this heuristic statement was about what can be achieved at the ward level if help is brought in by those who work centrally for the organisation. We had high hopes that PE teams would be able to help our clinical teams to access and interpret their feedback. With the drive of our co-researchers, PE teams did give us access to existing feedback for each ward if they had it. Although they held NHS in-patient survey data, FFT data, complaints and PALS, they did not routinely hold other types of data that we were interested in using (e.g. they did not subscribe to information feeds from social media sites such as Care Opinion), so we searched these out ourselves. The extent to which they were able to provide collated and easy-to-use data depended on their current PE data systems. In one trust these were quite advanced and in the other two they did not exist and we took on the role of creating user-friendly formats.

The PE teams and others (drafted in as required) did fulfil some important tasks for the ward teams. In Trusts A and B, PE teams helped interpret feedback ahead of sharing with staff teams. In Trust A, they also attended several ward meetings, with a QI specialist eventually taking over this role from the PE lead. This specialist help was hands-on; in A (i), he created the leaflet that the team required for their patients, taking it through the lengthy trust approval processes on their behalf. In both A (i) and A (ii), teams said that they would like the input of additional ward volunteers to help address some of the patients’ and carers’ emotional needs that had been evident in the feedback. They wanted volunteers who could chat or run activities. The QI specialist tried to help by enlisting the help of the voluntary service team. However, this was not achieved within the duration of the project, as there were not enough volunteers available and the recruitment process was lengthy. In Trust B, we were able to ask the Trust Volunteer Co-ordinator for similar help with the recruitment of volunteers for B (ii) and this was successful.

In all cases, these central support staff stressed caution in presuming that this level of help would be available beyond the confines of this discrete study. Although PE teams and QI staff were clearly interested (and passionate) about PE on the wards, they often stated that they had alternative organisational-level priorities related to governance (e.g. complaints management), or simply that they had too few staff to work at this level of intensity with multiple ward-based teams.

Reflection and planning: emotive and complex

This heuristic statement sought to articulate the need for a reflective process through which staff teams could engage with and make sense of their feedback. In understanding this process further, we learnt that it is the varied and complex nature of messages contained within PE feedback that warrants such a reflective approach. Often feedback does not contain a clear critique of what staff are or are not doing, or indeed what they should or should not do, but it contains an expression of patients’ and relatives’ emotional needs: loneliness [A (ii) and B (ii)], anxiety about the staff’s busyness [A (i), A (ii) and B (i)], anxiety about the future [B (ii)], anxiety about baby and learning baby skills [C (ii)], anxiety about relatives [A (ii) and B (ii)] and feeling vulnerable [B (i) and C (i)]. This type of content needed consideration alongside more practical concerns (e.g. noise or information provision). We recognised that some of the issues presented were deemed by ward staff as within their control to address, but some were considered outside their remit of influence.

We found that presenting this type of complex, emotion-rich feedback to staff was initially difficult. Conversations could feel defensive (‘that is not our fault’, or ‘we know all that but it’s the way things are’), could contain expressions of sadness (‘I just wish we could spend more time with them’), annoyance and frustration (‘there will be no money to help with that’). This process needed sensitive facilitation, recognising the emotional nature both of patient feedback and of staff responses to it. One ward manager admitted that she thought that it required great bravery on the part of teams to commit to hearing what their patients really felt. Dips in levels of energy within the meeting rooms were evident as staff took on board, either vocally or quietly, the implications of patients’ comments. Sufficient time was needed for staff to make sense of the comments and to suggest what could make things better. Teams then needed support in breaking down a variety of possible actions and we found it helpful to categorise these for them into quick wins (e.g. move the phone to reduce noise), ideas that needed a team approach (e.g. introduce social lunches), actions that needed to be escalated straight to the ward manager (e.g. addressing a problem with specific agency staff) or escalated to other departments within the hospital (e.g. requesting additional ward volunteers). We encouraged small but realistic steps within limited resources.

When teams were able to carve out some time and space, it was possible for such team sessions to become playful and creative, especially after the nervousness of the initial presentation of feedback was over. We found that many ward environments were not conducive to this process: there was little communal or meeting space or the only available space, usually the ward manager’s office, was small and continually in use. Some teams were more able than others to create time and space: A (ii) included reflective feedback meetings into their staff training days away from the ward, and B (ii) legitimised this activity within their existing multidisciplinary team meetings. In all other cases, we were only able to conduct this exercise with one or two senior staff.

Applying improvement methods: no textbook guide

Our co-researchers agreed at the outset that the Institute for Healthcare Improvements (IHI)’s model for improvement,123 including concepts such as PDSA, measurement and impact, could be usefully applied even if no one quite knew how. The IHI model’s principle of problem definition (having a clear knowledge of what issues are for patients and the resulting changes that teams would like to make) guided us as we facilitated all teams to understand and plan changes from their feedback, influencing our introduction of the distinction between categories of actions described above.

We found that PDSA cycles were then appropriate for the category that involved teams changing an established practice or introducing something new. These PDSA cycles enabled them to take a broad issue [e.g. patients need more social activities in B (ii)] and from this identify a simple change that they could make (daily communal lunch). We used PDSAs to start small and immediately (four patients, tomorrow), checking and addressing stumbling blocks (e.g. inappropriate table) before extending more widely. This approach worked well in A (ii) and B (ii) where changes were spread to include many members of staff and patients. In other cases, it revealed the impracticality of initial plans; in C (i), sisters did not have time for afternoon PE rounds with all patients and in C (ii), senior midwives realised that their staff would not automatically use a welcome leaflet in all admissions processes, and that this would require engagement and prompting. In these four wards, we deemed PDSA successful in slowing down the ever present desire that we observed within ward teams to ‘roll out’ changes quickly, and in ensuring that processes were reliable before doing so. We used some formal monitoring of their activities to support their PDSAs, using tally charts or staff reflection sheets to capture progress. These were useful when completed but they could not be relied on, as any additional paperwork was viewed negatively. Often the same information could be captured more successfully through verbal conversations, but these required additional ‘checking-in’ visits from us.

In B (i) and A (i), the PDSA approach was less feasible. In A (i) (which was an emergency department) the numbers of people involved (staff and patients) were so great, and the issue of waiting times was so intensely felt by staff, that incremental PDSA cycles were considered to be too slow an approach by the ward team, especially as they foresaw even greater pressures as winter approached. Instead of PDSA, established means of communication that reach large numbers of staff (e-mails, team briefs) were chosen to spread the new initiative before reliability was fully established, losing the opportunity to ensure effectiveness. In another ward, C (i), the ward manager stopped PDSA at the problem-definition stage because of staffing issues and a cancellation of her non-clinical time availability.

We found that, although we facilitated the formal PDSAs, other activities arose spontaneously. Actions that staff had labelled as a quick fix (e.g. ordering soft closing bins) happened at unpredictable times depending on organisational factors (e.g. money suddenly being made available) and related ideas were tried without prompting. This was particularly evident in B (ii) where what staff called a ‘cultural shift’ of focusing on social needs of patients led to two staff volunteering to lead exercise classes on their days off. Because of these emergent changes, the notion of measuring impact of any single change (in this case, lunches) was complex.

Throughout, we were aware that the terminology of measurement (before and after improvement) that had been brought in from co-researchers’ understandings of QI, did not quite reflect what we were doing. Originally, feedback had not provided a measure of a PE state to be improved, but rather had revealed a collective mood/feeling of current patients. The issues raised had at times not all lent themselves to improvement (i.e. remove the problem) but instead some form of response that often involved talking more to patients. We did conduct repeat exercises that were able to pick up on how patients perceived staff responses to their needs, or indeed if their needs had changed. In B (ii) and A (ii), where significant changes had been made, a follow-up exercise did reveal some positive feelings towards the staff initiatives put in place. However, in A (ii) particularly, there were also clear signs that this patient group still had the same needs, and that attending to PE was an ongoing and daily process; patients and their relatives remained elderly and anxious and staff remained busy. Ensuring due care with respect to communication was appreciated, but it was not a need that could be removed and ticked off.

Facilitation of QI was deemed by our improvement specialists as supporting ‘the art of the possible’ (indeed this terminology seems to reflect more the creative and adaptive approach that we developed throughout), using concepts of systematic QI in a particular way that used quantitative measurements where they were helpful [e.g. in B (ii) we kept a tally of how many patients went to communal lunch], but used qualitative interpretation to assess change (a follow-up review of topics of importance to patients).

Celebration: do not wait until the end

Our original intention for celebration put forward a fairly linear conception of how ward teams would receive feedback, make changes and assess impact, leading to the need to celebrate at the end to mark the progress made. As we did not know how many times they would go through this process, we anticipated that perhaps they would celebrate more than once. In fact, we learnt that celebration should not be conceived only as a ‘round-up’ activity but rather there was something to celebrate at many stages, and this ongoing approach to recognition and appreciation was good for morale for all involved. Without exception, when we collated PE feedback for each ward we were struck by the number of compliments and accounts of ‘exceptional’, ‘caring’, ‘competent’ and ‘excellent’ care, combined with a recognition that staff provide this level of attention while being extremely pressurised and busy. When feedback was shared with the team, these messages were conveyed first and we presented them with certificates that they could display in their wards. Many members of staff, particularly the ward managers, were visibly moved by this recognition (stopped in their tracks in fact) and it appeared to help them to relax and settle in to the process of reflecting on their feedback as a whole.

Throughout the AR, we found other opportunities to recognise and appreciate effort. In A (ii), the team of health-care assistants had embraced a new role for themselves within their ‘PE rounds’, so they received individual certificates. In B (ii), where the whole team had played their parts in improving social activities for their patients, we knew that they had a good success story that could be shared more widely. We helped them apply for a regional award to publicly acknowledge their efforts. Less formally, as facilitators, we also sought to acknowledge (taking pictures where possible) the small but important changes that they were making [e.g. B (ii)’s tablecloth and flowers for the social lunches]. This aspect linked well to the adaptive, supportive QI approach that we were providing as facilitators: searching out opportunities to maintain and improve engagement and interest. Interestingly, the notion of a final celebration in each ward did not take off; in some cases, the initiatives did not really reach an end point, but they wanted to keep going. In some cases, they were just too busy to eat the cake!

Multiple data sources: are they worth the effort?

The need to recognise the different types of information contained in PE feedback has been noted previously. Murrells et al. 54 distinguished between transactional and functional information (what happens to the patient and the ways in which it happens) and relational (how they are treated human to human). Entwistle et al. 38 advocates for more appropriate responses to handling and responding to ‘relational’ information than the reductionist approaches that currently dominate. She argues that procedure-driven, standardised approaches such as surveys (collection) and checklists (to guide response) are too narrow. She builds on this134 to explain why person-centred care and the needs of individuals are lost when PE is approached in this way. Our AR has revealed the thirst that ward teams have for all types of information. They want to get information from patients about things that can be quickly fixed, but they also want to understand how their patients feel so that they can develop more appropriate ways of relating to them, and they want to share this beyond the senior staff who currently receive it.

We learnt that all of these types of information can be found within feedback gained through a single route; our patient interviews provided sufficient functional, transactional and relational information within them to enable staff to make changes, potentially bypassing the need for ward teams to engage with other data sets (e.g. FFT), or to attempt aggregation and triangulation exercises with other available data. It is likely that other processes of direct engagement with current patients, similar to our patient representative interviews, could also elicit quality feedback. Where available, we did use existing, routinely collected feedback to supplement the interview findings. Where ward managers had good knowledge of its content, having kept informal track of its changes over time, they were able to bring this insight into discussions. More often however, routinely collected data were relegated in importance within these discussions because of their sporadic and patchy nature. Indeed, even for A (i) (the emergency department), where quantities of FFT and PALS were sufficient to develop topics for discussion from these data alone, the collation and organisation of this feedback was extremely time-consuming, and is likely to be prohibitively so outside a research project.

A ward-level intervention: how far can this go?

We discuss this question with reference to the Feedback Response Framework of Sheard et al. 15 This framework proposes that in order for changes to be made, there needs to be a local (ward-based) desire to respond to patient feedback, which is aligned with sufficient hierarchical structures of the right people with the right roles (structural legitimacy). There also needs to be sufficient organisational drive (readiness) to prioritise and enable changes to be made. Through AR we found that nursing staff at the ward level generally do have the desire to respond to patient feedback, but we have also learnt that if it comes in a helpful format, and with sensitive facilitation, it can be intensified to become an impetus for change. The PET facilitated this by ensuring that staff received credible, ward-level, current information about their patients and then great effort was made to generate responses together so that they owned and participated in them. It was clear that it is at the ward level that staff (particularly nurses) can know, care for and relate to patients; and these staff are central to making change. However, other disciplines (medical staff and allied health professionals) and other supporting staff (e.g. PE teams), all of whom have ‘homes’ not on the ward but in other parts of the organisation, must be aligned to support this impetus for change, something we found was not easily achieved.

As action researchers we were able to provide limited assistance in strengthening structural legitimacy. We enhanced the relationship between ward teams and the PE team in Trust A who were able to progress volunteer recruitment and leaflet development on behalf of the ward teams. This also happened in Trust B, with respect to volunteer recruitment. However, we could not create structures that were not there, such as the multidisciplinary team forums required to engage medical staff. Neither could we enhance organisational readiness, but we did note how this differed by trust. Trust A was the most willing to engage in testing new roles for PE and QI teams to assist in this work, as it clearly fitted their corporate agenda most closely. There is a role for facilitators working as we did, but they would need to have legitimacy (permission) themselves to fully help ward-based teams to navigate hierarchies. 89 It has been noted that such boundary-spanning work appears extremely difficult to achieve in everyday work,135,136 suggesting that organisations must fully support teams to work with the PET if they want them to be successful in their responses to the broad range of PE feedback that they are likely to receive.

Working with patient experience feedback: pushing the boundaries of quality improvement

At the outset, our collective assumption was that PE feedback could be used within a broad QI framework of diagnosis, change and impact assessment and our AR revealed three significant things in this regard. First, the nature of the feedback that patients provided, although containing other types of information, was highly relational. Significantly, anxiety, loneliness and uncertainty were recurrent themes (all issues that require a relational response) usually involving time to talk, listen and empathise. These are issues of the nature described by Cohn124 and Pflueger125 as not helpfully reduced to objective measures for use within improvement cycles.

Second, our approach to handling the complex stories provided by patients bore more resemblance to the concept of ‘soft intelligence’ processes (seeking and interpreting soft data)79 than to the use of metrics in improvement. In our approach, very open methods were used to talk to patients to obtain ‘untamed, unpredictable and spontaneous’ insights and these were used to stimulate sense-making, encouraging as many staff as possible to participate in a collective process that was challenging and, at times, discomfiting. Feedback was not so much used to instruct (to tell staff what needed to be changed), but to disrupt collectively held assumptions about what it was like on a ward and to make staff question their role, or potential roles, in making things better.

Third, we found that assessment of impact did not involve the revisiting of one specific metric. In fact, in an approach described by Øvretveit137 as a break from more traditional linear QI models, staff did not decide on only one specific change, but on a number of related actions. It suited our staff teams to do this because they had been presented with complex stories, which demanded an array of related responses (e.g. social lunches, volunteer recruitment), not only discrete issues that demand discrete responses (e.g. noisy nurses’ station). As well as this making it hard to decide what to measure for impact, we also found that the nature of some of the issues tackled was such that they never go away (e.g. patients may always have anxieties and staff will always be busy). Feedback serves to remind and reignite empathy (as soft intelligence) not to measure the impact of a specific change, but to periodically disrupt the complacency that can lead to poor practices. 79

Importantly, we did not however throw the QI ‘baby’ (any attempt to systematically track progress) out with the ‘bathwater’ (recognition that traditional measurement concepts are limited). We could effectively use PDSA as a systematic approach to learning and informed action, which as Reed and Card138 argue is one of the few QI tools that gets to the ‘crux of change’ process on the ground. The important skill is to balance the use of soft intelligence that is best left intact and not reduced to simple metrics, with the use of simple metrics that can be used to help understand progress along the way.

Beyond inanimate toolkits: can we find facilitators?

Across all steps of this PE intervention, we have found that progress relied on the support of a facilitator to lead, connect people, support patient representatives, organise reflective meetings, obtain and organise feedback, celebrate, facilitate priorities and support ongoing improvement work using appropriate techniques adapted to context. On top of these, there is the less tangible activity of bringing a positive sense of ‘the art of the possible’ to teams that are often bogged down and have low expectations of the changes that could be made.

As action researchers we were able to step into this facilitation role, but outside a research project it is not at all clear who would. We know that PE teams are overstretched,80 and that currently there are few QI staff within hospital trusts, as this is a relatively new professional grouping. 139 Our AR approach (rooted in participative and reflective techniques) has served us well in grounding the intervention within a theory of change akin to emerging participative approaches to QI. 140,141 With respect to QI for PE, it also shares participative grounding with EBCD. 30 What AR has unashamedly done for us is to place the person (i.e. the facilitators themselves) at the centre of the intervention so that it becomes more than a toolkit document, but a facilitator’s guide to supporting this work. The QI coaching models in health care142 may provide opportunities for articulating such a role outside a research context.

Research questions

1. What were the main factors that enabled progress on the wards?

2. To what extent and how were barriers to the improvement work overcome?

3. Did PE improve on the wards, as assessed through a quantitative measure?

Type of process evaluation

The use of process evaluation methods has increased in recent years amid recognition that in-depth, theoretical understanding of interventions is required if their full potential is to be achieved. 149,150 The method is commonly used alongside RCTs to understand divergences in outcomes when interventions, already fully developed and standardised, are delivered across different sites. 151 This approach was not suitable here because PET prototype 1 was in its developmental stage, being implemented and refined through AR. Establishing prospective measures of fidelity, dose and reach, as is common in process evaluations to evaluate implementation processes,151 was therefore not possible. As such, the evaluation did not strictly adhere to existing process evaluation guidance but adopted a flexible, data-driven approach to theory development, incorporating aspects of program theory evaluation147 and abductive research theory. 152 The quantitative component, discussed in Quantitative data collection and analysis, deployed statistical process control (SPC) methods to analyse the survey data chronologically over the implementation period.

Constructing a logic model for the Patient Experience Toolkit intervention

The initial stage of the research involved constructing a logic model for the PET intervention, commonly used in process evaluations to represent the ‘program theory’ of interventions. 147 The evaluator (TM) immersed themselves in relevant literature, including improvement literatures, previous studies undertaken by the research team and the study protocol. An initial program theory for the intervention was subsequently formulated and represented in the logic model. This specified the intervention components, including resources and activities, which were expected to bring about effects, contextual enablers and barriers (termed ‘moderators’), which were expected to shape the project work and possible outcomes, ranging from proximal ‘mediators’ to more distal outcomes. An initial version was shared with the research team and the wider steering group and revised based on their feedback. This was then developed over the course of the study through an ‘iterative process’147 in which the model contents was tested and refined in relation to themes emerging in the data. The final logic model is presented in Logic model for the action research project.

Data collection

The methods of data collection were specifically tailored to account for the status of the intervention as being both implemented and refined through the AR. To track the intervention’s development, the action researchers kept detailed reflective diaries for each ward based on a prompt created by the evaluator. They made 119 diary entries in total. A log of major activities and project meetings was also kept for each ward by the action researchers. In addition, TM undertook participant observations of project meetings and kept detailed ‘pen portraits’ for each ward, used previously by the research team. 89 These included a story of each ward’s progress, summaries of the reflective diaries/participant observations and ‘analytic memos’,153 whereby TM reflected on emerging themes for later analysis. Qualitative interviews with key stakeholders at the halfway point and at the end point (n = 17) were also conducted. These were tailored to stakeholders’ roles in the project and they included patient representatives (n = 4), HCPs (n = 9) and members of PE teams (n = 4). The data were transcribed and fully anonymised. Wards were assigned number codes to ensure anonymity. These were 1, 2, 3, 4, 5 and 6. Staff members and patient representatives were assigned the same number as the ward they were attached to, while trusts were assigned a code of A, B and C.

Data analysis

A coding matrix was created in a Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) spreadsheet at the halfway point of the project to organise and summarise the qualitative data to aid its analysis, following the Framework Method. 153 This was then applied to the data for the remainder of the study. The matrix included columns for each ward and four rows for the logic model categories (resources, activities, moderators and outcomes), which served as descriptive themes for organising the data. Although this framework was deductively derived, it had the status of Miles and Huberman’s ‘start list’154 and new themes were incorporated when appropriate. The final coding framework included seven core themes: the project as a whole, intervention resources, activities/mechanisms, the process of implementation, people and relationships, moderators and outcomes. Each theme included sets of subthemes that were either informed by the initial logic model contents or had been identified in TM’s ‘analytic memos’ in the pen portraits. These themes could be descriptive or conceptual and often included contrasting participants’ views on a topic. There was also some overlap between themes. For example, the prominence of references to the people and relationships involved in the project led to a new theme being introduced, but these references could also be coded as a subtheme of mechanisms when a person or group were identified as contributing to outcomes.

The coding framework was analysed in order to refine the logic model. Where a successful outcome was identified on a ward in the outcomes column, factors that enabled this apparent success could be traced by examining other columns for that ward and vice versa with non-optimal outcomes. The content of the initial logic model was refined accordingly, going through eight iterations. However, towards the later stages of the project, TM questioned whether the logic model could accurately represent the intervention’s logic. Created for all wards combined, the logic model was observed to be failing to capture how the intervention was taking on a different form on the wards. Hence, in a final data theming stage, TM deployed the techniques of abductive research to interrogate the logic model and the coding framework. An abductive approach was deemed appropriate because it seeks to creatively theorise unexpected research findings that have emerged in empirical data. 152 The approach has been used previously by the research team, in the context of patient safety improvement projects. 15 Here, TM, in frequent discussions with LS, formulated various meta-level themes that sought to convey the core functioning of the intervention, addressing research questions 1 and 2 set out above in Research questions.

Quantitative data collection and analysis

A validated Picker PE survey50 was selected for the quantitative component of the evaluation and 12 questions were included in the final survey because of their relevance to hospital in-patients. The survey began in the week commencing 10 January 2017 to allow for a baseline of 8 weeks prior to the start of the project on the wards. Data collection ended in the week commencing 15 April 2018, 4 weeks after the end of the improvement work. A research team consisting of five people collected the data. Each member was assigned a ward and wards were mostly visited on the same day and time each week to minimise possible confounding variables related to the person and day of collection. Because SPC methods do not use traditional statistical significance tests but adjust to the available data set, the recruitment target was not based on a statistical power calculation, but rather what was feasible and sufficient to create a functioning SPC chart in order to analyse change over time. We therefore chose a weekly sample of 15–25 surveys. It was also anticipated that having 20 surveys would provide a reasonable number of responses across the six wards on any individual question/statement. However, because the ward environments are very different, recruitment rates were expected to differ, because of factors such as patient throughput and patient capacity to engage. Hence, although 20 surveys per week was the overall aim, researchers were asked to recruit up to six participants on each visit so that the wards with a higher recruitment rate would compensate for the wards with a lower recruitment rate. A total of 1028 participants were recruited over the course of the project, amounting to a weekly average of 15.5 participants. The main factors that prevented the target of 20 from being attained were very low patient throughput on some wards, meaning that on a visit it might not be possible to recruit any new patients, and mental capacity issues that prevented some patients from being recruited. Survey responses were therefore unevenly spread across the wards. Table 8 details the number of participants recruited per ward.

Once collected, the data were analysed using SPC charts. The SPC method is increasingly used in health care, often as part of an intervention to provide practitioners with real-time feedback. Here, it was separate from the intervention, used to assess whether or not PE improved on the wards during implementation. A core assumption of the SPC method is that variation exists in all performance data, the task being to identify variation that is out of the ordinary and, therefore, has a ‘special cause’. To that end, data are displayed chronologically in SPC charts, enabling various statistically derived decision rules to be applied visually. The charts feature three main lines: the mean (sometimes the median) and an upper and a lower control limit. The control limits set the statistical range within which data can be expected to fall under stable conditions. Data points outside these limits indicate special cause variation, as do unusual data patterns: these include trends of 6 consecutive data points (going up or down) or rows of 8 data points that are not consecutive but are situated above or below the mean. 155

A common use of the method is to see whether an intervention improves performance across a measure of interest. An instance of special cause variation can imply that the intervention has ‘worked’. If a trend or row is identified, a ‘step change’ is entered into the chart whereby the mean and the control limits are recalculated to reflect the change in the data. Performance will stabilise around this new level if the improvement is sustained. 156 We deploy this approach in Quantitative findings, presenting two SPC charts. The first of these presents, as initially intended, the survey data for all wards combined over the study period. The second uses the findings of the qualitative research to inform a secondary analysis of the survey data.

Not necessarily the toolkit document

The PET prototype 1 took the form of a large, turquoise ring-binder. Inside this were an introductory page to the project and a one-page flow chart outlining the process designed for HCPs to go through to successfully act on patient feedback. Following this, each step of the process had a specific section that outlined how it would proceed. Because one aim of the AR was to refine this process, deviations from it were expected but nonetheless it was still anticipated that PET prototype 1 would be used by the ward teams. Each ward had a copy that was presented to the ward managers by the action researchers who explained to them that they would be working through the prototype over the course of the year. However, PET prototype 1 was rarely consulted in meetings from that point onwards. Only the one-page flow chart was used as a device for the action researchers to introduce newcomers to the project.

Interviews with the ward managers at the halfway point revealed a significant degree of scepticism about PET prototype 1 (ward managers 3, 4, 5 and 6). Some of this was directed at its form, with some ward managers viewing it as too large (ward managers 3, 5 and 6). Others highlighted stages that they thought were not required. A guiding principles exercise, designed to heighten awareness of PE among staff, received criticism (ward managers 3, 4, 5 and 6). One ward manager who had not been able to attend the co-design workshops saw the whole process as too time-consuming, stating:

This has clearly been worked out by somebody who has had time and time is something I do not have.

Ward 3, interviewee 1, ward manager 3

Ward manager 1 liked the initial prototype but, even then, it did not feature prominently in the meetings that they were involved in. They also recognised that other resources had been developed for use instead of it:

You can use the toolkit to think ‘right, how do we get from x to y’ and it helps you with that doesn’t it? Whether we’ve used it fully as they want us to use it is another matter. That’s what you’re going to ask me next isn’t it? [Laughs] It’s very possibly not happened but I think it has happened on other bits of paper.

Ward 1, interviewee 2, ward manager 1

Other ward managers similarly thought that PET prototype 1 had been superseded by resources developed by the action researchers, referring to patient feedback posters and flow charts as evidence of them using the toolkit (ward manager 4, 5 and 6). These resources had not superseded the toolkit but they had been developed as part of their wards’ PDSA cycles, reflecting the ward managers’ distance from PET prototype 1.

For the action researchers, PET prototype 1 was an occasional theme of their reflective diaries, with critical commentary on both its form and the process contained within it. Similarities could be noted between the action researcher and ward manager accounts, including concerns about its size and the guiding principles exercise. During the later stages, PET prototype 1 did become central to the AR as a document to be critiqued as the action researchers, the design team and the ward teams focused their attention on developing it based on the learning that they had acquired. Here, having a concrete toolkit document appears to have been useful for the research, if only in the sense that inadequacies could be identified to be avoided in later iterations.

People and relationships

The involvement of a diverse group of committed research participants was pivotal to the progress of the AR project. The study was located at a major research institute in the north of England, involving a dedicated and experienced research team and a steering group. The action researchers were academic researchers and often sought advice from the PI, the research programme manager and the two improvement specialists who sat on the steering group. In addition, before the project began and over the course of its first phase when the prototype was being developed, the action researchers established close relationships with the patient representatives, the ward teams and the corporate staff of participating trusts.

The research institute’s established PPI networks and links to the volunteering services of participating trusts meant that patient representatives were available and they could take on a vital role in the project. Some worked closely with the ward staff, collecting ‘live’ patient feedback on wards for discussion at meetings (patient representatives 1, 3, 4 and 6). Some brought with them knowledge of the role and had relevant work experience that meant they were particularly suited to be working closely with staff and patients on the wards. Likewise, the ward staff played a pivotal role and great care was taken by the action researchers to develop and maintain relationships with them. Not only was their involvement crucial for improvements to be made, but the action researchers viewed the trusting relationships built up with staff as crucial to enable often highly emotional patient feedback to be discussed. Similarly, efforts were made to develop relationships with corporate staff. The PE team of Trust A was not significantly involved in the project but did provide some support after the action researchers actively sought to develop a relationship with them. In Trusts B and C, relationships between the action researchers and local PE teams were strong throughout.

The participatory nature of AR meant that these relationships assumed even greater significance than they would have in a standard research project. Corporate staff, ward staff and patient representatives were engaged with as co-researchers and were recruited into the project as such. The action researchers also took the participatory ethos of AR very seriously, frequently discussing its implications for their practice. This is significant as it may have shaped how the staff engaged with the project.

Indeed, while staff engagement ebbed at times, it was generally high throughout and the trusting relationships forged between the action researchers and staff may have helped to maintain engagement through any rocky patches. Where staff engagement did ebb, this seemed to be correlated with staffing pressures. At different points of the project, ward managers referred to staffing pressures as a reason for planned meetings not going ahead (ward managers 2, 3, 4, 5 and 6) or needing to be put on hold for a period (ward managers 3, 5 and 6). Some members of corporate staff believed that the action researchers were too cautious and should have pushed the ward managers to prioritise the project during these times. However, this was a minority view. Others expressed the opinion that the action researchers’ approach, of respecting the wishes of ward staff when they wanted to pause participation, may have been beneficial in the long run. Certainly, ward managers interviewed at the halfway point appreciated how the action researchers engaged with them. Ward manager 2 stated that they liked developing something ‘straight out of the box’ rather than being told to implement something developed elsewhere. Ward manager 3 spoke of how they liked that problems identified in patient feedback were not just seen as errors that required to be acted on but could be interpreted and explained through dialogue. Ward manager 4 spoke of how it was empowering for staff, which was important as they were set to leave the ward:

It’s nice for the staff to be able to identify areas that they want to improve on because ultimately, and I have said this the whole way through, I’m not a permanent fixture here in terms of the ward manager and I will hopefully very soon be back to my original ward so they need to feel empowered to make changes themselves once I’m gone. So you know it has been nice to work collaboratively that way so that the staff feel that actually this is something that they can do themselves for the patients, they don’t need somebody telling them that this is what you need to do and it’s nice for them to be part of the decision-making process.

Ward 4, interviewee 6, ward manager 4

The facilitation role

Research participants overwhelmingly agreed that the action researchers’ facilitation was more important to the project work than PET prototype 1. One of the improvement specialists had fully anticipated this, expecting that the evaluation would produce evidence to back up their view that toolkit documents are insignificant relative to the facilitation role required to support them:

If I’m honest and being blunt I have always been cynical that a toolkit is what is needed. I don’t think the NHS needs any more toolkits and that is based on my experience of working with front-line teams over many years where no matter how good the quality of the content within a toolkit, I have never known any front-line team member or team use it and generally speaking it sits on a shelf. So I was never convinced that in order to help teams improve patient experience what we needed was a toolkit . . . You have got to facilitate it and support it in a collaborative way whatever it is you are trying to do.

Interviewee 10, improvement specialist

Many of the core facilitation tasks were not anticipated beforehand but emerged as the project unfolded. On finding that none of the wards had patient feedback available to them in a usable format, the action researchers themed existing data and/or made arrangements for more to be collected. One ward had sufficient survey data available (ward 5) while ‘live’ data were collected on the others (wards 1, 2, 3, 4 and 6). It was mostly the patient representatives who collected the ‘live’ data but the action researchers collected them on ward 2 when the patient representative could not attend. This was then organised into topics by the action researchers, patient representatives and some corporate staff in what was referred to as a ‘collective, interpretative process’. The action researchers facilitated these sessions and subsequently created patient feedback handouts for each ward based on the discussions. Once presented to ward teams, the action researchers then guided them through PDSA cycles to prioritise issues and implement and evaluate changes. It was anticipated at the start that knowledge of QI methods would be crucial here and both action researchers attended courses to increase their knowledge of the field. In addition, outputs of the PDSA cycles, such as posters and leaflets, had to be designed for use on the wards, requiring additional skills on behalf of the action researchers. It was observed that front-line HCPs did not have the time or the skills to undertake these tasks.

An additional, emergent facilitation activity was the more intangible effort to encourage ward staff to believe that they could make changes despite the pressure that they were under. Strategies included, for example, focusing on ‘quick wins’ first to show what could be done or celebrating existing examples of good PE. The action researchers referred to this as encouraging a positive sense of the ‘art of the possible’. Finally, the facilitation sometimes involved escalating issues when they required the input of other hospital departments. Over the course of the project, the action researchers contacted a range of actors as part of the ward action plans, including PE teams and volunteering services. Escalation included procuring door stoppers and bin silencers to address noise at night (ward 3), securing volunteers for wards (wards 4, 5 and 6) and clearing project outputs through relevant trust authorities (wards 1 and 6). Newly established connections could also be drawn on when problems arose. Some ‘escalation’ activities were also carried out by PE teams, although ward manager 3 was adept in this regard, contacting their information technology (IT) and estates department as part of the project work.

The ward staff expressed gratitude for the facilitation provided by the action researchers. The patient feedback handout was a popular item. Some ward managers had shown it to their staff, reporting that it had helped boost staff morale (ward managers 5 and 6). Others had used it to escalate issues (ward manager 3). In one ward (ward 4) that had gone through a period of uncertainty at the beginning of the project because of a change of leadership and some organisational changes, the project was seen by the ward manager and a qualified member of staff as having been empowering for staff and as having improved ward culture and team working. The patient representative for the ward observed a new sense of enthusiasm on the ward. In another ward (ward 1) where the project work had required a change to staff routines, the action researchers had provided a ward manager with assistance over several months to help them ‘roll out’ the change. The ward manager reported that this had improved their understanding of QI and even their management skills. Similarly, three ward managers saw the new connections formed through escalation as a positive outcome of the project (ward managers 4, 5 and 6). While these connections made ward-level improvements possible, they also opened opportunities for further improvement work in the future:

By being part of this project it’s opened more doors so if we needed more support or we needed something else implementing we have got that. There is an individual for each area where you can tap into a resource and just say ‘look, it isn’t working, we need this, this and this’. Like XXXX [local PE team member] is wanting to look at different things. He has come into XXXX to observe and bring some fresh ideas.

Ward 5, interviewee 3, ward manager 5

A possible issue with this expansive facilitation role is that ward teams may have left tasks for the action researchers, as they preferred to do as little as possible. A related concern is that the action researchers’ extensive facilitation may have minimised staff’s exposure to the intervention, blunting its potential effects. A patient representative shared this concern, warning that the significant changes that had occurred on their ward could be undone once the action researchers withdrew, given their prominence in the project. Yet, without the facilitation provided as part of the PET intervention, it is questionable whether or not improvements would have occurred at all, particularly given the extent of staffing pressures on the wards (see The significance of staff pressures). The action researchers were also wary of doing too much for staff and could be seen sometimes pushing back at their requests if they seemed unreasonable or if staff could do tasks themselves. This issue should, therefore, be seen as a tension rather than as a fundamental weakness of an expansive facilitation role.

Organisational support

The project work received significant organisational support from the three participating trusts, although the level and nature of the support differed between them. Each trust had a PE team that provided most of the support, although in Trust B the PE Team was spread over research and risk departments. Trust C integrated the project into their own work on PE. Other actors who provided some input to the project were Heads of Nursing, matrons and personnel from volunteering services, medical illustration, information technology, estates and communications.

Organisational support took a number of forms, ranging from making patient feedback available to direct participation in project meetings. The attendance of a PE lead at ward meetings was critical to the project’s advance on ward 6, as the ward manager expressed initial reluctance to get involved. The fact that a prominent member of corporate staff attended the meetings may have served as a signal of Trust C’s commitment to the project and opened up new opportunities for collaboration, which was appreciated by the ward manager. The PE teams also had a significant role to play when the project encountered difficulties related to staff pressures. The most notable example here occurred on ward 5, when all project meetings had to be cancelled because the non-clinical time of staff was pulled centrally by Trust C. With the action researchers having to withdraw from the field, they reached agreement with the local PE team to continue supporting the ward’s improvement work. The local PE team produced a patient information leaflet and took it through a lengthy clearance procedure involving the trust’s communications and medical illustration departments.

The involvement of PE teams meant that organisational learning emerged as a main outcome of the project. The two PE teams which were heavily involved spoke of how they learnt about the support that is required for front-line staff to successfully work on PE feedback. On seeing the potential of having patient representatives to collect ‘live’ feedback, one of the PE teams planned to replicate their role across Trust B. The new connections forged between the PE teams and the wards also make future collaboration possible. A PE team member from Trust C welcomed this, emphasising positive impacts of the project beyond the immediate use of the PET:

This came about as a direct result of Rose and Claire’s involvement in XXXX. Their involvement kick-started how XXXX were looking at themselves and how they could improve the service to patients. And now that I have got a relationship with XXXX the matron we are beginning to look at other things we can do. So although the toolkit might not be at the forefront of our thinking, all these other things wouldn’t have happened without their involvement. I don’t know how you capture that other than to say it sparked this interest and wider thoughts about what could be done.

Trust C, interviewee 15, PE team member

The significance of staff pressures

A common view among research participants was that the overarching context of resource constraints and pressures on staff limited the full potential of the project. The action researchers and the patient representatives were frequently concerned about the well-being of staff and efforts were made to ameliorate the worst effects of the pressures (see Creative and pragmatic implementation). However, despite the resourcefulness of the action researchers and ward staff in working around the pressures, full implementation of the PET could not always be achieved.

Indeed, although establishing prospective measures of fidelity, dose and reach was not possible because the PET was not fully developed at the start of the study (see Type of process evaluation), it is possible to retrospectively assess how each ward did, now that the process embedded within the PET is fully developed. The final PET iteration contains six stages, including stages for forming a ward team, gathering data, reflecting on it, prioritising issues, making changes and measuring impact. All wards got to the stage of making changes but only three wards completed the final stage of measuring impact: wards 1, 4 and 6. This does not imply higher or lower engagement with the intervention, as each ward context presented different opportunities and challenges. It is notable that these wards had less acute staffing pressures, reflected in fewer and shorter pauses in their participation in the project: ward 6 paused participation for 4 weeks over the summer period because of shortages of qualified staff but returned to full participation shortly after, whereas wards 1 and 4 fully participated throughout. By contrast, the three wards that did not complete the PET had significant pauses lasting longer than 3 months because of staffing pressures (wards 3 and 5) and a combination of staffing pressures and a change of leadership (ward 2). In addition, the wards that fully implemented the PET mostly did better at forming a core ward team consisting of a large group of staff. Ward 4 already had multidisciplinary team meetings that the action researchers were invited to attend and the ward manager of ward 6 incorporated the project into their ‘away days’, thereby involving a considerable number of nursing staff. Ward 1 is the exception among this group, as the project was led by two ward managers but they worked together as a team. By contrast, ward 5 had severe staffing pressures that prevented large numbers of staff from being involved, whereas the ward managers of ward 2 and 3 mostly worked alone. Their desire to retain sole ownership of the project may be linked to staffing pressures, as they did not want to add more work onto already intense staff workloads. Ward manager 3 was apologetic about this, recognising that it may have prevented the project from achieving its full potential:

I’m worried about my role in the project because I have felt like I can’t give it the commitment that it probably needs. I certainly haven’t been able to get a team of people together, not that they are not interested but it is the time commitment to get to the meetings.

Ward 3, interviewee 1, ward manager 3

This failure to form a multidisciplinary team on some wards complicated PET’s implementation. On ward 2, the improvement work effectively ended when the ward manager left for another position because no one was available to take over from them.

Creative and pragmatic implementation

A defining characteristic of the intervention was how key components of it could be adapted, via the action researchers’ facilitation, to some of the challenges and opportunities (i.e. ‘moderators’) present in the different ward settings. Progress using the PET (if not always full implementation) hinged on its creative and pragmatic implementation.

Logic model for the action research project

Figure 14 presents the final logic model, developed as the project progressed. The logic model serves as a complement to the theoretical assumptions explored in Chapter 5 and the final PET iteration. The key intervention resources are identified to the left of the diagram. The PET is included alongside the skills, leadership and motivation of ward teams, including ward staff and patient representatives, who are central to PET’s implementation. The wider supporting mechanisms are also identified, including the facilitation role and organisational support. More detail of what the facilitation entails is provided in the intervention activities section, along with a list of additional intervention mechanisms, notably participation, reflection, feedback and action planning/QI cycles. These are mechanisms that are recognised as affecting change in wider improvement literatures. The detail as to how they manifest through the PET and how they can be adapted to PE are found in Chapter 5. Contextual factors that shaped the intervention, whether enabling it or serving as a barrier, are listed in the moderators section of Figure 14. These have also been distinguished between those that were external to the ward settings and those that were internal. To the right of the diagram are the outcomes of the intervention, distinguished between proximal ‘mediators’ and more distal outcomes. While the ideal, distal outcome of a fully embedded PE system (with sustained improvements to PE occurring) was not achieved on any ward during the year we worked with them, the various proximal outcomes listed in the logic model could be identified across the participating wards. Furthermore, the list of moderators included in the model provides insight into what would have to be in place for the PET to achieve such an ideal.

FIGURE 14.

Final logic model. PEIT, Patient Experience Improvement Toolkit.

Finally, the methodological implications of the project are listed at the bottom left of Figure 14. The evaluation lends weight to the action researchers’ conclusions regarding the benefits of AR’s participatory approach and the centrality it assigns to the facilitator role within the intervention. The need to creatively apply QI to PE was also a core finding of the AR that the evaluation again lends weight to (see Chapter 5).

Weaknesses

A weakness of the evaluation concerns the stage of the PET’s development. The final iteration has changed significantly from PET prototype 1, now being targeted at potential facilitators rather than HCPs. This may enhance its potential utility but testing is required among this group and the question of how to achieve success at scale requires further consideration. At least some of the action researchers’ learning regarding how to implement the PET will be tacit and, therefore, difficult to capture and transfer across people and settings.

In addition, the AR project took place in the English NHS between February 2016 and March 2017, a time of considerable organisational pressures. The evaluation findings may become less relevant if these pressures were to lift or the PET was implemented in a setting where they are less evident.

Finally, the survey analysis using SPC methods has several limitations. The baseline of 8 data points is small for the method as 20 data points is generally advocated,155 and the absence of a control makes it difficult to attribute causality between interventions and outcomes. In addition, a limited number of data meant that a detailed, ward-level analysis was not possible and the sheer complexity of the intervention made it difficult to annotate the SPC charts, adding to the difficulty of attributing causation.

Current patient experience feedback is not fit for purpose

Measuring PE can serve dual goals for ‘selection’ and ‘change’. 164 In the first instance, where performance on the measures is made public, together with comparisons with other providers, patients or their families can use the information collected to select where they would prefer to receive care and this, in turn, might serve to encourage higher performance on these measures. These data can also serve a reputational goal and feed into external review processes, such as those of the CQC in England.

In the second case, organisations, teams and individuals can use the feedback to motivate a change in how care is delivered. In our work, we identified a tension between the collection of data that serves the selection function and that which serves the change function. This is a finding reported in other studies that demonstrate a chasm between management and front-line staff whereby data are collected at an organisational level, but without a change plan and engagement of front-line clinicians this rarely leads to improvement. 92 For example, of the 38 different types of feedback we identified in our scoping review (see Chapter 2), none of the mandated measures were considered suitable for informing and monitoring local improvement. The data lack local relevance and they are not accessible in a timely manner. While the qualitative data collected via the FFT have the potential to inform QI, the current focus within the three hospitals we worked with on achieving acceptable response rates meant that using the data for change was rarely considered. Moreover, the number of data available to teams and the value of them varied hugely. In our A&E there was an abundance of data, but staff struggled to collate and use these data effectively. In an integrated care unit, turnover of patients was so slow that they felt that they had too few data to work with. Either way, the generic positive feedback, with many of the comments of the type ‘staff did a great job’, meant that there was little to guide improvements. The ethical implications of patients being asked to complete measures that are not then used to make improvements were noted by some. In general, our findings help to explain earlier contentions that there is little evidence that patient feedback about their experience of care has led to improvements in the quality of health care. 10

Our qualitative work (summarised in Chapter 3) also revealed that organisational structures within hospitals mean that teams are tasked with a narrow focus on an individual source of data (e.g. FFT, PALS, complaints). This impeded learning at both an organisational and a local level. We recommend that ideally these hospital functions should work more closely together. If this is not possible then there should, at the very least, be a strategic emphasis on learning across the organisation from the variety of feedback sources. In addition, of note from this qualitative work was the absence of a narrative about the value of PE data beyond the specific purpose of improving PE. Two systematic reviews, first Doyle et al. 7 and then Anhang Price et al.,165 have demonstrated that higher levels of PE are associated with greater patient adherence, best practice clinical processes, better safety culture and lower resource use, yet this is not articulated by those working in health care, at managerial level or at the front line.

The process of co-design

In Chapters 4 and 5 we describe the co-design of a PET and the subsequent implementation of this toolkit in six different hospital departments using AR. In both designing and implementing the toolkit, we identified tensions, some of which echo ideas that had already arisen from the scoping and qualitative work. Before discussing these, it is important to reflect on the methodological tensions that we ourselves experienced when using co-design within a research project. Co-design aims to bring staff and patients together to reflect on their experiences and to develop together improvement priorities and ideas for change. However, within a research project, inevitably, the intervention design phase comes at a point after the research team have spent some time understanding the nature of the problem and considering potential solutions. This might mean that they have an important understanding and perspective, and yet staying close to co-design processes means that the voice of the researcher is not a central one. Those involved in co-design within research projects may need to develop a means by which they can optimise the co-design process by utilising this knowledge without risking the inherently user-centred nature of the co-design process. In the existing project, this compromise was achieved firstly by involving researchers as partners in the co-design process and secondly through meetings between the co-design team and the research team before and after each workshop. These meetings allowed for the earlier research findings to feed into ideas about how to structure the workshops and what fundamental conversations were important to encourage during these workshops.

Another important finding that has implications beyond this particular co-design project is the potential for the idealisation of the work context by staff involved. When staff are taken off the ward (or other care environment) and given some time and space to be involved in co-designing an intervention, which later they will deliver in the hustle and bustle of a busy ward, they are not necessarily able to anticipate the difficulties that they will face. Alternatively, they may ignore these challenges because they are fearful of admitting to them in a group setting, particularly in a group that includes patients. This has important implications for co-design. Staff may be engaging in designing an intervention for an idealised environment. For example, although staff in the co-design process suggested the need to bring a multidisciplinary team together to work on the PE project and to agree a set of principles in an initial meeting, this did not happen in practice. In practice, this stage was not perceived to be actively moving the project forward. It was also perceived to be unrealistic when multidisciplinary teams rarely came together at all for any other purpose.

In co-designing and implementing the toolkit, an interesting trade-off was acted out between the need for detail, because staff did not necessarily know how to collect useful feedback, how to measure for improvement, etc., and the need for brevity. Feedback from the teams was that the appearance of the product needed to ‘sell it’ and that it should not look like a major task to undertake. It needed to be punchy and bring the process alive and yet still contain all the useful resources that had been collated along the way. These conflicting aims meant that, in the final solution, the detailed toolkit was designed for use by the person facilitating the process and an overview was developed to introduce teams to the key steps in the process.

Data for improvement?

As we moved into the co-design phase we came to understand more about the huge volumes of PE feedback that staff are exposed to, but that are not in the same form as that required by external and governing bodies. Staff receive written and verbal compliments, they receive grumbles and complaints, they receive smiles and hugs, chocolates and cards from patients and families. In fact, dealing with this informal feedback, good and bad, is what they do; it is embedded within their work. In thinking about PE, therefore, nursing staff are focused on their current patients, those that they are caring for at the time. This is in direct contrast to the focus of the hospital management who produce PE feedback reports often based on the experience of previous patients who were cared for weeks, months or even years before. This disconnect may help to offer some explanation for the lack of engagement of staff with the PE feedback reported in previous studies. Although this feedback provides staff with a ‘feel’ for how patients and their families are experiencing the service, staff did recognise that it did not provide useful information that would guide improvement. In some of the wards we worked with, there was scant PE feedback available that could be used for improvement. The action researchers, who were able to respond creatively and flexibly to address this gap, worked with patient representatives to collect new information for the purposes of this project.

The emotional response to patient feedback

Throughout the co-design process, the importance of considering positive feedback was a central narrative. Staff expressed a concern about the relentlessly negative feedback they were exposed to and yet, in reality, much patient feedback was positive. They also discussed the need to counteract the human tendency to focus only on the negative. This need to celebrate what was good and the role that this could play in increasing staff morale was deemed to be critical within the highly pressured environments in which staff worked. In one department, staff referred to the use of positive FFTs in a staff huddle to help improve the mood of the team at the end of a shift. This became an important focus of the design of the prototype toolkit.

Dealing with the emotional response of staff to feedback also became something that the action researchers were acutely aware was one of their key roles. It soon became clear that staff did not have the time or the skills to collate their patient feedback and that they needed support to make sense of it. This task was completed by the action researchers who had by this time taken on a major facilitative role. Based on the co-design process, the presentation of feedback always included a strong emphasis on positive feedback. This helped to offer a form of reassurance to staff that they were doing something right. Despite this, staff focused on the negative because they saw this as the driver for improvement. It was not unusual for staff to become despondent, defensive, upset or frustrated when viewing the feedback. In most cases the feedback was not a surprise, but it upset them to hear that patients were bored and depressed, for example. The feedback served to increase cognitive dissonance,166 an uncomfortable feeling associated with knowing that what you want, or claim, to do is not what you actually do. Staff often knew that patients were not getting the best care, but with the pressures they were under and the resource constraints they experienced, they often felt helpless to address these problems. The facilitators were able to coach staff to work towards small changes, asking them to consider perhaps what they could do tomorrow, and this sense of the possible helped them to move beyond their initial emotional reaction. Facilitators also recognised the importance of celebrating success during the course of the implementation of the toolkit and they used certificates to keep staff on board and to assure them that they were doing well even in the face of adversity. Staff responded very positively to this encouragement and it was clear that positive reinforcement was not something that they routinely experienced in their work. This lends support to approaches for health-care improvement that attempt to identify and reward positive cultures and behaviours, such as Positive Deviance167 and Learning from Excellence. 168

Facilitation, not the toolkit, as the change mechanism

In a systematic review of approaches to using PE data, Gleeson11 reports ‘a lack of expertise in QI and confidence in interpreting patient experience data effectively’. Our evidence strongly supports this finding. Staff on the wards needed support to collate and interpret the data and to test out their ideas in PDSA cycles. They also needed help to access other functions or teams outside their own team to address problems that were not within their immediate control. This is linked to a second tension highlighted within the co-design process between bottom-up and top-down improvement and who owns the toolkit. Although this varied to some extent across the six teams we worked with, the teams came to rely on the facilitating role of the action researchers to guide them through the toolkit and to do some of the work (e.g. collating and thematising the qualitative comments) while they took ownership of the day-to-day work itself. The toolkit document itself acted as a cue to action and might have been a necessary precondition to engagement from the teams. However, it did not on its own support the continued engagement from teams.

For some teams, the facilitative role also helped them make links with other parts of the hospital infrastructure that were necessary to access the data that they needed or to make change happen. Within the Promoting Action on Research Implementation in Health Services (PARIHS) framework, the facilitator role is proposed to be central to the process of getting evidence into practice,169 in this case implementing a PET. The skills proposed by Rycroft-Malone169 and others for successful facilitation are co-counselling, critical reflection, giving meaning, flexibility of role and realness/authenticity. These skills were critical to the success of the current project and it is likely that if we had not had the opportunity to engage in facilitation that came with the AR method that we adopted, the toolkit would have been of little value to the ward teams. Flexibility, patience, creativity, pragmatism, persistence, understanding and responsiveness were the most important attributes demonstrated by our facilitators.

This finding echoes other recent work15 that identified that even when staff valued patient feedback (there was normative legitimacy) and they owned and were committed to a plan for change (there was structural legitimacy), without organisational readiness, teams were rarely able to enact the changes that they wanted to make. Within this project, the action researchers served this function. This research suggests that the most appropriate model for using the toolkit is one where external facilitators, with particular skills in QI and with the ability and legitimacy to draw on the services of other departments in the hospital, may be necessary to support teams in making improvements beyond those within their immediate control. Without this support, the patient voice is only heard by those who already have direct contact with patients and, it could be argued, already know how patients are experiencing their care. Thus, the findings here suggest that a bottom-up approach to data collection that is meaningful to staff and patients is most appropriate. The responsibility for acting on this information must be shared across the organisation, because this approach is most likely to both facilitate organisational learning and support change.

The process evaluation demonstrated that even with effective facilitation three of the teams did not complete each of the six steps from establishing a team through to measuring improvement. The greatest success in terms of improved PE is likely to be achieved when staff are highly engaged, multidisciplinary team structures already exist, extensive organisational support is in place and staff and service pressures are at a minimum. In situations where this was not the case, ward teams were not able to complete the six steps of the toolkit within a year and often had to stop and start, gaining and losing momentum a number of times. Our quantitative analysis, using SPC Charts, demonstrated that for those teams that did complete the entire process it was possible to demonstrate an improvement of 4% in negative PE, with scores dropping from 25% to 21%. Although these improvements were small, the questions in the survey deployed (a short version of the Picker PE survey) contained items that did not necessarily align with the changes that staff were choosing to focus on based on the collection of their own patient feedback.

Robustness of the research and limitations

The strengths and limitations of each of the studies conducted within this programme of work are set out in the individual chapters. Here, we discuss the overall approach. A key strength of this research and the AR design was the flexibility that this approach allowed us. Rather than developing a precisely specified intervention that was handed over to teams and evaluated at a distance, our approach meant that we were able to respond in a flexible way to the constraints of time and resources experienced by staff, as well as their different levels of skill and motivation. We could adapt and iterate the intervention, developing content for the toolkit that addressed existing gaps and needs of the teams. This meant that, even when teams were struggling to engage with the implementation process, they remained engaged in the project through a recognition that we were ‘on their side’ and that our aim was to make the toolkit work for them. Given the level of involvement of our action researchers, it was important that an independent evaluation was conducted alongside, and this did occur in a robust manner. However, it could be argued that being part of the wider research team and sharing an office with the other researchers in the team meant that it was not possible for the evaluator to be entirely objective in their work. This was tempered by meetings of the evaluator with the co-investigator (Laura Sheard) and the PI (Rebecca Lawton) who, being much more removed from the front-line teams, were able to prompt, question and probe so as to understand the emerging themes. This process of discussion about the way that the intervention was working and the role of facilitation led to a conceptual analysis of the nature of logic models for complex interventions of this kind. This was not anticipated in the original proposal and it represents a key strength of the work undertaken here.