Public involvement in research: assessing impact through a realist evaluation

Research brief

The NIHR HS&DR Programme and INVOLVE jointly invited proposals in 2010 to address the gap in evidence around the impact of public involvement in research. 15 A background paper summarised the evidence then available, particularly drawing on the reviews by Staley11 and Brett et al. 12 The invitation expressed three key aims for the research: to collect evidence on the impact of public involvement in research, to identify methods of evaluating this involvement, and to identify effective ways of involving the public in research (implementation). The call was open to a range of methodological approaches.

Realist evaluation framework

Our research design was based on the application of realist theory of evaluation, particularly drawing on the work of Ray Pawson,13,16,17 which argues that social programmes (in this case public involvement in research) are driven by an underlying vision of change – a ‘programme theory’ of how the programme is supposed to work. The role of the evaluator is to compare the theory and the practice: ‘It is the realist evaluator’s task, and the added value of social science, to identify and explain the precise circumstances under which each theory holds.’17 Moreover, the outcomes of social programmes can be understood by identifying regularities of CMO. Thus the key question for the evaluator is ‘What works for whom in what circumstances . . . and why?’17 The realist approach is increasingly used in the evaluation of complex health programmes, and producing useful analyses. 18,19 After our study began, Staley et al. published a paper calling for the application of realist evaluation to the study of the impact of public involvement in research. 20 The development of our realist theory of public involvement in research over the course of our study is described in more detail in Chapter 3.

Our realist theory of public involvement in research was based on the two recently published literature reviews,11,12 which allowed us to identify a number of contextual factors and mechanisms that we believe were intended by policy-makers and other stakeholders in research policy to enable desired outcomes to be achieved. There has not previously been a robust testing of the underlying ‘programme theory’ of public involvement in research; our study was designed to allow an independent prospective testing of this underlying programme theory for the first time. We included an economic evaluation, designed to complement a realist evaluation design, estimating the resources used for public involvement across eight case studies.

Case study sampling

The setting for this project was within organisations hosting health and social care-related research studies (i.e. universities, NHS trusts and third-sector organisations) in the west of England. Our aim was to recruit a methodologically diverse sample of eight case studies which would have significant elements of public involvement during the period January to December 2012. There was no existing database or other source of routinely available data that enabled such upcoming studies to be identified. To meet our aims the studies needed only to be diverse, not representative, so we took the pragmatic decision to sample through our existing knowledge of studies with public involvement in the west of England and to ‘snowball’ through our existing networks, including the People and Research West of England consortium.

We developed a pro forma to identify from network stakeholders upcoming studies they were involved in or aware of with what they identified as ‘significant’ elements of public involvement. Our key inclusion criterion was evidence of some ongoing public involvement in key stages of the research process (design, recruitment, data collection, analysis, dissemination). A key exclusion criterion was that no study would be included unless both the principal investigator (PI) and at least one research partner agreed to take part. In order to identify generalisable regularities of CMO for public involvement in research, we wanted to identify a maximum variety sample of studies in terms of study type, stages of the research process and public involved. In a relatively small-scale study such as this, however, we knew we would not be able to achieve full diversity in all three dimensions. We therefore prioritised diversity of study type, as different study types can drive very different priorities for public involvement (e.g. emphasis on participant information and recruitment in clinical trials). We also prioritised including some studies involving young people and families with children because they make up a substantial minority of health service users but are underrepresented in the literature on public involvement in research. Our case studies are described in Chapter 4.

In deciding the number of case studies to undertake we recognise that there is always a trade-off between the depth of exploration (which suggests a small number of case studies) and identification of regularities (which benefits from a larger number). There are many ways to categorise research studies (e.g. basic science vs. applied, qualitative vs. quantitative, pilot studies vs. full trials, clinical vs. epidemiological, primary vs. secondary data, action research, translational research) and we could not hope to cover the full diversity in our case studies. From previous experience of case study research,21 we believed that eight case studies would enable us both to examine the CMO regularities in depth and to look for generalisable regularities across the case studies. This number of case studies did not enable us to examine all potential types of research study, but did enable us to include the most common, for example qualitative, mixed methods, feasibility and clinical trials. We received agreement from four PIs with appropriate funded studies taking place in the west of England at the application stage of our study, and the final four between approval and the early months of our study.

Case study data collection

The first stage of data collection involved initial mapping of the eight case studies through informal visits, encompassing observation of research settings and team meetings (where possible) and unstructured interviews.

Intelligence from the informal visits, together with previous findings from the two literature reviews and the CMO configuration, was then used to design an interview guide for semistructured interviews with case study project stakeholders. For each case study we aimed to carry out semistructured interviews with approximately five stakeholders (PIs, other researchers, research managers and two research partners) on three occasions over the course of the year of data collection, January–December 2012 (three interviews × five participants × eight case studies = 120 interviews in total during the year). Potential interviewees were identified in discussion with PIs and invitations forwarded via the PI or an administrative member of the PI’s team.

Interviews were broadly structured around our CMO hypothesis. Data collected include measurable elements (e.g. resources allocated for supporting public involvement and actual spend) and stakeholder perceptions (e.g. respective views of researchers and research partners on whether research partner contributions influenced project decisions). In addition, some of the stakeholders were given a resource log to record over 2 weeks, chosen at random, the amount of time spent contributing to a range of activities linked to public involvement in each case study. These were then costed using prices taken from published or recognised sources (see Chapter 7). Interviews were intended to take place at three broadly evenly spaced times over the 12-month data collection period.

In practice the number and timing of interviews varied widely across the case studies for a variety of reasons including delays in research governance approvals, illness among case study participants and research team members, delays in one research project commencing, and general logistical issues. In two case studies it was possible to carry out only two rounds of interviews rather than three, and the total number of interviews completed was 88 with 42 participants rather than the 120 with 40 participants initially envisaged. Table 1 summarises the total number of interviews (and research partner interviews) undertaken in each round across the case studies.

Case study 4 was exceptional in that, unexpectedly, no public involvement activity took place during the year. Thus the PI, a research manager and one research partner were interviewed initially and only the research manager at the end of the year. The other case studies where numbers of interviews were relatively low were case study 6, which started much later in the year than expected and where only one research partner chose to participate, and case study 7, where the research team was relatively small, there was relatively little involvement activity, and illness prevented final interviews with the two research partners. The relatively low completion rates on the initially planned 120 interviews were not a problem in themselves, so much as a symptom of lack of involvement activity for long periods in some of the case studies.

Given the small numbers of research partners overall, and the fact that some case studies targeted particular socioeconomic or age groups, we do not believe it would be meaningful to present demographic data on these participants. Our perception was, however, that our experience echoed other reports that those members of the public who choose to get involved in research tend to have attained a higher educational level than the population as a whole.

We recognise that a few interviewees do not fall easily into the categories of researcher, research manager or research partner, but we have kept to a limited number of categories to ensure anonymity.

Each case study was intended to be conducted by pairing an academic researcher and a research partner, under the overall supervision of the PI and co-ordinated by the research fellow. In one case, for logistical reasons, the research fellow undertook the data collection on his own. In the other cases, interviews were conducted by both an academic and research partner, usually separately but on occasion interviewing together.

The first round of interviews focused particularly on understanding the context of the case study and the mechanisms for public involvement planned for the remainder of the year. The second round of interviews was ‘light touch’, intended to capture developments in public involvement since the first round and to identify members of the research team able to nominate at least one research partner per case study who could be approached to complete the resource logs for economic costing. The final round of interviews focused on capturing outcomes and learning from the year, to enable us to assess how the researchers’ initial intentions and aspirations for public involvement turned out in practice. In addition to the semistructured interviews, a flexible approach to capturing data included observation of meetings where possible and/or other group tasks directly related to public involvement, and collection of project documents related to involvement processes. Observations were carried out in case studies 1, 2, 5, 7 and 8 but were not possible in case studies 3, 4 or 6, either because of internal case study project considerations or because no public involvement activity took place during the study period.

Developing a methodology for the economic costing of public involvement

There has been minimal exploration and there is little evidence for the costs and benefits of public involvement in research. El Ansari and Andersson conclude that analysis of the costs and benefits of participatory activities should form part of an overall evaluation of public participation. 22 They state that, for participation to move forward as a field, a broader ‘set of analytical frameworks is required, which captures the richness and unique qualities of participation, [and] that recognises and values the different perspectives that led to its initial development.’22

Our work here is an attempt to develop an analytical framework of how to assess the economic costs of involvement in research. Planning the budget for public involvement in research at the outset is crucial. The budget needs to include all planned research involvement work to be completed by research partners (for example participating in patient advisory groups or undertaking data analysis) as well as time for academics to facilitate research partners. Two key aspects of budgeting for public involvement are the researcher and research partner relationship and contingency planning. For example, research partners may be asked to contribute their expertise to respond to problems arising during a research project, for example poor recruitment of participants to a study (there were several examples of this among our case studies). These contributions generally arose during the research process and were not foreseen.

Payment and reward issues have generally proved controversial. At our second consensus event our case study participants debated the nature of payment and reward for public involvement vigorously, revealing a wide range of strongly held views on this subject. INVOLVE has developed guidelines on payments for involvement research work to respond to these issues. A recent document outlines the issues to bear in mind in paying research partners, gives examples of payments and provides general tips about issues connected with payment and ‘payments in kind’ that need to be carefully considered by project managers. 23 A range of pay rates for different research activities connected with public involvement are mentioned in INVOLVE documents, including a flat rate payment of £19.40 per hour. 23

Our economic analysis aimed to collect data from each case study team, in order to:

1. identify all activities relating to public involvement

2. measure the amounts of activities using a resource log

3. value or put a price on these activities using prices from published or established sources.

Identifying and measuring involvement activity

To gather data from our eight case studies for our economic analysis, we asked selected members of the case study teams (researchers, research managers and research partners) to log all the resources that were used in public involvement work/activities over a snapshot 2-week period. During the 2 weeks each person recorded/logged:

1. all involvement-related activities

2. length of time spent on each activity.

We asked them to include all activities (or inputs) that were undertaken as contributing to or enabling the central objective (or output) of public involvement in research. A sample log sheet for research partners on 1 day is in Appendix 1. Our ethical approval letter stipulated that research staff within each case study were to nominate research partners to provide our data, so we were dependent on these nominations being made successfully from within our case studies, as we were not able to make direct contact with research partners.

We issued user-friendly guidance for completing our resource log, and supported respondents over a 2-week period by e-mail and providing a telephone helpline. Our guidance document for research partners to complete resource logs is in Appendix 2. Our contact and ongoing dialogue with the academics and research partners who used our guidance and completed our resource logs enabled us to become familiar with how involvement activities were working within each case study from the point of view of both academics and their nominated research partners. These exchanges helped us gain a rounded understanding of the nature and diversity of involvement activity and the relationships and issues within each case study.

Economic valuation/costing of involvement activities

We translated the knowledge we had accrued of each case study into some working assumptions about each one. These assumptions are significant but complex, so we have detailed them in Appendices 3–5.

We then used the completed 2-week resource logs to estimate involvement costs for a projected 12-month period. From there we scaled up the 12-month projected costs to the length of each case study. This enabled us to compare the actual budgeted costs from each grant with the projected costs on a like-for-like basis.

We followed a standard economic approach to treat resource use and prices separately to arrive at a cost.

For example:

Ideally the price applied should come from a published source or the next best alternative, a recognised or established source. There are illustrative examples within INVOLVE guidelines of a range of prices for different research activities connected with public involvement. In our own project we had previously paid research partners at a ‘meeting rate’ of £19.77 per hour, but early in this project it became obvious that most work was being done outside meetings, so a lower ‘research associate’ rate of £14.02 per hour was agreed. Research partners kept records of all their work for the project (including e-mails, collecting and analysing data, and writing) and submitted claim forms regularly. Our project did not have a means of costing researcher time for public involvement activities, as working alongside research partners was a continuous process during our project.

A new set of guidelines from INVOLVE to budget for involvement was incomplete at the time of our analysis, but we saw the draft document, which again gave the example of the flat rate payment of £19.40 per hour for public involvement participation, so we used this price when costing research partners’ activities for our case studies. 23

Reflective practice

Data were captured on our reflective learning on the impact of public involvement in our own study. This was done by facilitating and audio-recording short reflective sessions during team meetings on our own experiences as a project team of academic researchers and research partners working together.

Consensus events

Two consensus workshops were organised as part of our plans to develop and test a theory of public involvement in research. Initially we aimed to hold the first event prior to the first round of data collection to inform the interview schedules for this round. As the project developed, however, we realised that this would not be practical in terms of the length of time research governance approval was taking from some NHS trusts and, more importantly, that an event after the first round would be more fruitful in terms of theory development. Thus, the decision was taken to hold the first consensus event between the first and second rounds of data collection.

At the first workshop we presented an overview of our initial findings from our first round of interviews and visits in the eight case studies. The overview was in the form of 12 statements drawn from our initial mapping of the case studies. The statements identify key contextual factors and mechanisms for public involvement in research that we hypothesised were regularly linked to positive impacts on research design and delivery.

The aim of the workshop was to test these statements with case study participants and steering group members, drawing on their experiences and insights regarding public involvement in research, in order to refine or replace the statements, to inform the next phase of data collection and analysis. The workshop was limited to one afternoon in the hope that this relatively short time commitment would make it more feasible for case study participants to attend.

Nineteen participants took part in the first consensus workshop. Six of the eight case studies were represented. The intention had been that all case studies would be represented by both research staff and research partners in their projects, but it was not possible to achieve this because of participants’ other commitments and some last-minute illness.

Participants first voted electronically on the 12 statements with the choices ‘agree’, ‘disagree’ or ‘abstain’. Participants were then divided into three groups, with each group asked to look in depth at four of the statements, discuss and revise them as necessary and identify any omissions, connections or other comments. The groups then fed back to a plenary session and participated in a final discussion.

A second half-day consensus workshop was held at the end of the third round of data collection. On this occasion all eight case studies were represented with a total of 29 participants. Our emerging theory of public involvement was presented in graphic form in a set of four slides covering different aspects of CMO regularities (field of research, leadership and culture, relationships and structures of involvement). Participants were asked to discuss, amend and comment on A1 printed versions of the slides. The output of the workshop was amended slides with marginal commentary, which were further analysed by the project team and used to form the basis of the analysis of findings presented in Chapter 5 and the development of our theory of public involvement as described in Chapter 8, Our revised theory of public involvement in research.

Case study data analysis

All interview data were transcribed and entered into an NVivo 10 database (QSR International, Warrington, Cheshire, UK). A key team discussion was how to most fully involve our research partners in the analysis of these data given that only one of them had experience of using any version of the NVivo software. The decision was made to offer NVivo 10 training to research partners but not to require this, as some did not feel confident of learning and using the programme effectively in the time available. A manual coding alternative was therefore made available. In order to make this practical, we limited the number of codes we identified to a minimum necessary to allow meaningful analysis. Those team members coding in NVivo 10 were supplied with this coding framework. Those coding manually were given a numerical code to use with transcripts and the coding was entered into the NVivo 10 database by the project research associate. For each case study, at least one transcript was coded independently by a researcher and research partner, and any divergence discussed and a joint approach agreed.

Data analysis focused on identifying CMO regularities across our case studies. From the initial CMO configuration identified in the proposal, with amendments from the first consensus workshop, a coding framework was devised with 38 codes (see Appendix 6) organised into six broad themes: relationships, leadership and culture, field of research, structures of involvement, resources and outcomes. We agreed as a team that the codes were the primary unit of analysis and the themes were provisional. Following coding of data, team discussion lead to the codes being reordered in terms of hypothesised CMO regularities presented in Chapter 5. This coding framework was then validated by one academic team member not involved in the case studies, who independently undertook a framework analysis24 of a sample of transcripts and compared her emerging framework with that drawn from the CMO configuration.

Verification of coding

Initial data were analysed by team members identifying codes within interviews. NVivo 10 software was used for data storage, retrieval, coding, analysis, memo writing and theme building across the CMO approach. This was useful in that data coding and development of findings was a collective team activity but an overall verification process was also established. One senior team member, who had not been party to the initial coding discussions, undertook a second verification to ensure consistency and rigor. The coding verification consisted of five transcripts randomly chosen across the case studies and involved two activities: naive reading and structured coding review analysis.

First, transcripts were repeatedly read, blindly from assigned codes, by the independent reviewer, with memo writing with regard to potential codes. Next, data were coded line by line, and each sentence or group of sentences was given a code using the direct meaning of the text. The second reviewer then read the transcripts with the allocated codes assigned by the initial coder. Similarities and differences were recorded. Comparing and contrasting meanings across and within transcripts through the use of memos was used. There was very high agreement found between the coder and second reviewer, which was a very positive result.

Narrative review

There were many issues of agreement in the broader level of understanding. For example, leadership emerged in all five transcripts, as did common terms such as culture (team and organisational), PI beliefs or senior lead issues. Feeling valued, trust and interpersonal relationships and other ‘emotional’-type codes were allocated by coders in all five transcripts. The second reviewer found similar patterns and this showed good overall broad agreement of coders. Power emerged in three transcripts more clearly and repeatedly but was in all five transcripts in some form.

There were minor issues of differences in coding where coders had consistently coded information in a similar way in terms of the value of public involvement. There was only one research partner who coded public involvement not just in terms of value but in terms of impact.

The thematic analysis of each code across the case studies was then used as the basis for the thematic analysis used to test our theory of public involvement in research as presented in Chapter 5.

Research ethics and governance

The study team took the importance of ethical practice extremely seriously and considered whether it raised any substantive ethical issues. As the study was primarily qualitative and did not involve questions around particularly personal health status or behaviours, we came to the view that it was relatively low risk. However, we recognised that, in asking researchers and research partners from the same studies about what was working and not working in terms of public involvement, we could potentially be raising some sensitive interpersonal relationship issues. We therefore sought to address these issues in our study design, participant information sheets and processes for ensuring confidentiality and anonymity. Following the screening questions on the Integrated Research Application System form, our study was identified as eligible for proportionate review. Ethical approval for the study was therefore sought from the County Durham & Tees Valley Research Ethics Committee prior to the commencement of the study and approval was given with minor conditions in August 2011. An application for a substantive amendment was made in December 2011 to include observation of case study meetings, which had not been included in the original application. Approval was given in February 2012.

Research governance approval, which was sought from the three NHS trusts hosting the four NHS-based case studies, proved much more time-consuming and problematic to obtain than expected. This was particularly because of our desire to enable research partners to participate fully in data collection, which required taking them through the NHS research passport system, something that neither the university nor the NHS trust human resources departments appeared familiar with. Final research governance approval for all three NHS trusts was not obtained until late March 2012, thus delaying our planned date for data collection in some case studies by around 3 months.

Throughout the study we sought to adhere to our ethical and research governance approvals by ensuring informed consent for all participants and fully complying with all conditions of our approvals. All case study participants were sent copies of the report in draft form and invited to comment on how their data had been used and any inaccuracies or other comments on their case studies.

Public involvement in our team

Our aim throughout this project has been to model good practice in our own research while studying the impact of public involvement in our case studies. The project was developed by the SUCIR group at UWE, which had strong service user representation. One research partner co-applicant was the cochair of SUCIR. Three other research partner co-applicants had previously worked with the PI and other academic co-applicants on developing the SUCIR scheme and/or on other research projects.

The four research partner co-applicants were involved in all aspects of the project including design, data collection, analysis and dissemination. The case studies were designed to be undertaken by four subteams, each consisting of one academic researcher and one research partner working together on two case studies. The research partners also formed a separate research partner reference group meeting bimonthly.

The initial intention was for two academic co-applicants with extensive experience in working with young people to recruit and support a young persons’ advisory group to work on the two case studies where participants were young people. In the end it did not prove feasible to recruit such a group and a decision was made to develop an alternative model of involving young people in the project. Two young people, one of whom who had worked on a previous study, were recruited to join the project as research partners. Over time they came to play a similar role to the original four research partners, attending team meetings, research partner meetings and other events, and participating in data collection and analysis in their two case studies.

Research partners were involved in our team’s reflective process on what worked well and what did not work well in terms of our own processes around public involvement. A period of approximately 15 minutes was set aside at the beginning of each team meeting and research partner meeting to share reflections and learning about public involvement in our own project. Research partners have co-authored and presented our outputs at the INVOLVE conference and elsewhere, and have contributed to ensuring that this final report is as user-friendly as possible, and that our wider dissemination plans include outputs specifically designed to be accessible to a wide public. The plain English summary of this report was drafted by research partners. Chapter 6 of this report includes the synthesis, led by one research partner, of the shared reflections on public involvement in our project by both the academic researchers and the research partners.

Case study 1

Case study 2

Case study 3

Case study 4

Case study 5

Case study 6

Case study 7

Case study 8

Influence on research design and delivery

Having hypothesised that influencing research design would be one of the regular intended outcomes of public involvement in our case studies, we found that this was the case in some but not all the case studies; notably, intention to influence design was not as common as an intention to influence the practical aspects of research delivery (e.g. recruitment materials and strategies and data collection tools), which was present in all our case studies. Influencing the design of future studies was a longer-term and somewhat less specific objective in several cases.

There were a number of contextual factors that impacted on the influence members of the public could have on the initial research design. Research funders are often prescriptive in the research they are prepared to fund, outlining specific design requirements and outcome criteria. This had an impact on the degree to which the public could influence an overall design, as reported in case study 1:

Okay. I think they have had more of an impact on the delivery.

Yes.

The initial design had already been written and agreed because it was very influenced by the Department of Health . . . and so it was, it was very controlled . . . so you know it all depends who your funder is.

In addition to the constraints imposed by funders, academic informants often reported managing the public input based on the perceived skill and knowledge of members of the public. This informant suggested that the public required specific knowledge and skills in order to engage in research design:

So with the Big Lottery they are having a bigger input into the design and what we need and what we want to do. So it’s, it’s also the stakeholder involvement and how they matured . . .

Yes.

. . . in their knowledge and skills.

The longer-established and more ongoing the public involvement mechanism, the more this helped support effective public contribution into the development of research design. The advisory group working with case study 8 provided one example of a well-established panel that commented on research projects and activity across this research programme. This group was presented as a powerful mechanism for influencing research project design decisions:

And we often have to spend quite a bit of time explaining to researchers, ‘This is how we do things in [name of programme]. If you want to do the study then it needs to go through the group of participants.’

Yes.

Sometimes they’re going to tell you things that you don’t like. Sometimes they turn round and say, ‘No, that is not okay and that will not be popular.’

Yes.

People will not want to do that. And we always would go with the participant views.

A specific example of the influence this group has had on research design is provided below:

. . . it was quite clear about how that would work and how we would ask those questions and we took it to [the public advisory group] who hated it.

Research manager 1

This advisory group raised new insights for the research team to consider in developing their research design and so had influenced the proposal.

By contrast, newly established and time-limited research partner groups were less involved in influencing research design, but were often able to influence and have impact on the later implementation of the proposed research design. This is evidenced in case study 4, where the input of a patient panel led to changes in the approach to gaining consent from participants:

That was very valuable because that kind of did have an impact on the trial design with that one because it was how we were going to do it. So then with their ideas about that it then dictated our ethics application.

Research manager 2

Influence on recruitment and retention

Recruitment to studies is influenced by many factors; designing good publicity and information sheets is one aspect that public involvement can help with. Participants consistently reported that research partners’ contributions improved recruitment strategies and materials, although these improvements did not necessarily lead to improving participant recruitment to their studies. For example, an advisory group in case study 1 helped develop a poster to support recruitment which was used in all of the trial sites. At a later stage in the trial the group were approached to support the researchers think through ways in which retention might be improved. Their ideas at this stage changed the original strategy:

At some point we had already decided to give out telephone cards to participants for taking part in particular interviews and the [public advisory] group had a look at those cards and they said ‘well it’s not so useful, high street vouchers would be more useful,’ so that kind of input they had.

Researcher 6

An interview recorded at a later stage in the project suggested that recruitment had still been an issue despite the public involvement, though this could be attributed to a number of factors, not merely the recruitment strategies:

It took us longer to recruit; therefore it’ll take longer to follow up for this study.

Researcher 7

Other studies also involved research partners to assist with recruitment. In case study 2 the project was having difficulties in recruitment and sought advice from the advisory group. The group suggested targeting potential participants through the community rather than through NHS services. The parents helped develop fliers to aid recruitment; despite distributing hundreds of these, recruitment remained low:

Clearly the redrafting didn’t do much good, because they didn’t recruit that many parents, so although we put quite a lot of effort into redrafting it, it still wasn’t perfect . . .

Research partner 2

The reasons for low recruitment may not be attributed to the quality of recruitment material alone, and a number of factors related to the project and the target population may have contributed to this outcome.

In case study 5 the local context also impacted on research partners’ efforts to recruit through a programme of local events:

And so we held two events but we didn’t get any people sign up through those events and it was partly due to timing in that we were relying . . . obviously we put word out on our website and through our networks, through our peer specialists, these events are taking place, but we were also relying on the individual teammates and managers and care co-ordinators to tell people about those events and for whatever reason I don’t think the message really got out there.

Research partner 3

Here the research partners’ efforts were thwarted by both a lack of local support from key staff and a lack of people eligible to join the study, due to local authority policy decisions. Recruitment remained problematic despite a number of innovative approaches to recruitment.

In some case studies recruitment and retention strategies developed with public involvement had been more successful. The parents involved in case study 3 made suggestions to aid recruitment to and retention on a training course delivered to parents as part of the feasibility trial. They suggested a number of strategies to help, including making sure a member of the children’s centre, familiar to the parents, was available when the course ran; shortening the length to mirror the school term; and encouraging parents to bring a friend or relative with them to the sessions. They also influenced the scope of recruitment to the study, suggesting that all parents should be invited to the course, not just those with children who had been injured. The research team felt this approach had been successful and intended to use it in the design for a proposed future trial.

Influence on data collection tools and methods

There were a number of examples of research partners influencing the development and use of data collection tools and methods. In case study 6 the research team made a significant change to the order of questions on a questionnaire and the mode of delivery, based on feedback from four patient research partners during a research meeting. One major change was noted:

Yes, so it was going to be one big questionnaire and now we have put in two questionnaires, and that was under the, well it was because of the change, the rephrasing and how they would answer it.

Researcher 8

One research partner in the same study also reported challenging the wording of a question relating to patient recovery, suggesting that patients may see themselves not as ‘recovered’ but as learning to live with the disease, coping with it rather than recovering to their previous level of health. This feedback led to changes in the questions asked, which the research partner felt would have beneficial outcomes for the project:

I think adjusting those questions a little bit is likely to draw out the answers that are really being sought rather than the answers that might have been given otherwise, because they didn’t really understand exactly what was being got at.

Research partner 1

The parent panel in case study 2 influenced the methods of data collection, by commenting during a panel meeting, which resulted in significant design changes:

So they suggested we did this play session instead of direct focus groups and the main headline was, ‘oh, fun drama play session’ . . . We’ve changed to telephone interviews as one of the outcomes of the parent panel recommending that, you know parents who do want to be involved but couldn’t make it to the focus groups.

Researcher 3

This change had been recently implemented and the researchers were unable to comment on the actual impact of the change on the research outcomes. However, it is expected that the change will impact positively on recruitment. Further impact was evidenced in case study 3, where the research team felt members of a parent advisory group added support to their view on where a particular intervention should be delivered:

From a design point of view it’s made a difference in – well it’s reaffirmed our opinion that the setting for delivery of courses should be in children’s centres. The setting for identification of potential participants should be in children’s centres, through children’s centres.

Which was quite different from the plan wasn’t it?

Which is different . . .

Case study 7 also reported changes to a questionnaire design which was influenced by the patient partners’ contributions. These included changes in layout, wording and the inclusion of additional questions. Changes were also made to patient information sheets and letters. Research partner comments at one meeting led to the recognition that a draft questionnaire was being reviewed, rather than the validated version:

Yes because I read through it and basically one question had been duplicated but phrased slightly differently so I told her it should have been this way round and they have now changed it and they are all very impressed with me.

Research partner 4

The advisory panel in case study 8 developed guidelines on the development of information sheets, consent forms and other documentation. These are now provided to all researchers applying to undertake research in the programme, with the intention of helping researchers produce research documents to a specific standard:

I think they have had some real impact in terms of, okay so there are things like the [named] guidelines, so that has gone wider now, so that is not just looking at the information sheets but looking at the consent forms, to look at the invite letters and the reminder letters, and based on the guidance that they gave us, that we then whipped up into a sort of guidance document that fitted with the NIHR guidelines for information sheets and consent forms, we are now taking that and creating templates and a set of guidelines to give to all collaborators when they approach us.

Research manager 4

The experience and insight research partners brought was seen by researchers as valuable, adding an important perspective:

I think I just value their opinion on certain things I guess, especially in terms of how the questionnaire is going to be answered and how long it might take a person to complete . . .

Researcher 8

Case study 5 relied on research partners from a local service-user organisation to gather local intelligence vital to the study:

They are very effective at providing an insight, an on the ground insight, into what’s going on in a way that we wouldn’t have access to especially in the second phase of the study.

Researcher 9

In three of the case studies the researchers suggested it was difficult to separate out the influence of research partners in the development of research design and delivery. The quote below from case study 7 illustrates the difficulties of tracking who is responsible in collaborative teams for the contribution which makes an impact in the research:

Mostly it’s really hard to see what the influence has been because what happens in the research is a synthesis from everybody around the table and you know it’s a bit difficult to say, well, what exactly did [name of research partner] say that made a difference to it . . .

Researcher 5

A number of academics said that they would involve the public again in future research activities and use the learning from their current projects to inform that. Almost universally, research staff stated that involving the public was valuable, was seen as important externally and should be supported in the future:

I think it does add value . . . the ability to involve patients will be a benefit in terms of future research in the sense that we can yes, we have got a patient panel, it’s got this pedigree.

Researcher 10

One academic was sceptical initially of the process of public involvement but was very quickly won over to the benefits:

I think the whole overall process has been helpful, I mean initially I’ve got to be honest and say I thought it was going to be paying lip service to the NIHR and what they were trying to achieve but after the first meeting I think I can quite happily say that was straight out the window.

Research manager 2

Longer-term impact

In several of the case studies participants noted longer-term, strategic or personal impacts of public involvement that might not necessarily contribute directly to the current study, but would contribute to enhancing future research. In case study 2 the team articulated a strategic role for their parent research panel initiative that went beyond contributing to the specific research programme and was intended to help develop an ongoing strategic approach to public involvement for the research group as a whole:

I think we were determined that part of what we would do is work that [role of parents] out really and I think because it is this kind of infrastructure process, part of what we wanted to do was make sure that we had some kind of system that wouldn’t just work for the actual funded project, but it would be something that we could expand and develop into something that works for the whole research unit really . . .

The parent panel is a testing ground for how we involve people in the future and that we start to use them . . . work with them in that kind of broader way so that it’s not just related to that programme of work but others as well. So as I say we’ve taken bids to them that we’re thinking of and got their views about [if] it’s an appropriate one, whether it’s relevant, what parents might think about that, whether the questions would have occurred . . .

Researcher 2

When asked if there were models for this approach elsewhere in the NHS trust, a team member commented:

I think we are quite unique in kind of thinking about it at that strategic level. I think most of the groups we have are very much focused on particular projects whereas we are trying to plan this as a long term, so no, I haven’t seen any other planning or strategy or policy documents, so it’s all kind of feeling my way through it.

Research manager 3

Case study 8 participants commented on the influence of the advisory panel in the development of guidelines that would impact on the processes within the programme. These guidelines had the support of senior programme managers and were being implemented for use by all collaborators:

So like this [named] guidelines is a really good example, I know I keep bringing it up but it is a really good example, because now moving forward we are going to have all our documentation in house in the same format as we would ask collaborators to, all following these templates and these guidelines which I think is the first time we have done that.

Research manager 4

Some case studies reported other intended impacts. Case study 5 planned to involve the research partners in the development of three toolkits to disseminate the study findings, which had the potential for national impact. Research partners from all four local study areas contributed to this through the national public advisory group meetings, organised and facilitated by the research team.

Academics and research partners noted a wide range of potential benefits and impact from public involvement in research on service improvement as well as on research, but were aware of the organisational challenges of achieving this in a meaningful way:

I think there is an organisational issue about how important it is to engage with parents and is it going to be just a tick box exercise? Do you engage parents or is it going to be actually this is really tough? We really want to do this well and we want to design our services with parents informing us of what they want.

Third-sector partner 1

All case study research teams felt they wanted to continue to work with public involvement in future research. Some had long-standing research panels and groups in place that they were continuing to work with, whereas others were considering developing their strategy for public involvement to include the possible use of mechanisms such as formal panels with terms of reference. It was also recognised that investment was needed to support panel development and widen their scope of influence:

I think we want to continue to utilise the panel to . . . you know in terms of validating research questions, I don’t think our panel is developed enough to generate research questions although that may be unfair but I don’t necessarily see that at the moment . . . the more we invest in the panel the more we’ll get out of it because hopefully the people remain relatively constant over a period of time and they possibly understand more about what’s going on and therefore they can make . . . you know a better information base.

Researcher 10

Overall both researchers and research partners felt that research partners brought a crucial perspective to studies which enriched research and research teams:

In the formal governance meetings for the project I’ve certainly found it useful, it’s always useful to have verbal feedback and different points of view being expressed in a constructive environment.

Researcher 7

I think it is making a much richer study, because without their involvement it wouldn’t even be relevant I don’t think.

Researcher 8

For some academics, who were practising clinicians, this impact went beyond the project and into the workplace:

Makes me a better clinician . . .

Researcher 11

In some cases attending external training on public involvement complemented the learning that researchers and research managers gained from their direct involvement activity:

Attending the theory group sessions introduced me to the literature and theoretical discussion on public involvement that in the normal course of events I wouldn’t have been exposed to – and this too may have had an impact on my work with the parent panel.

Research manager 5

Overall, we found consistent evidence that public involvement influences research design and delivery, in particular improving recruitment strategy and materials, and improves data collection tools and methods. Public involvement shaped initial research design only in a minority of our case studies, but there were a number of indications of potentially longer-term influence on future research design, and other intangible impacts such as on researcher learning.

Impact on the public involved

In most of the case studies the primary objective of the public involvement was to improve the quality of the research, particularly around research delivery. Having a positive impact on research partners themselves was not usually an explicit objective of research teams, although it was often recognised that such involvement could be a development opportunity, especially for young people or others who needed to build confidence or develop skills to enhance their career or employment prospects.

Across all the case studies, being involved in research had a range of positive impacts on the research partners, including learning new skills and knowledge, personal development and support, friendship, enjoyment and satisfaction. However, there were a few occasions where research partners were more ambivalent about their involvement.

Developing research skills and knowledge was viewed as a benefit of involvement. One participant commented on the variety of opportunities their involvement brought:

We were invited along to be part of the recruitment interview panel in order to appoint someone to the study, that was not something I would have expected necessarily, but it was quite an interesting opportunity and an interesting way to involve us . . .

Research partner 2

In this example, the public involvement was focused on recruitment rather than data collection. Involvement in some studies enabled research partners to develop dissemination skills, through co-authorship on main papers and contributing to conference presentations.

Involvement in research provided opportunities for development and engagement in stimulating academic work, particularly noted by a respondent in case study 6 who was unable to continue to work:

Having had to give up work because of my condition I do miss the intellectual challenge and so anything like this, which involves me in the academic light again, really is something that I’m very keen to be involved in.

Research partner 1

Taking part in advisory groups led to an increased level of confidence for some research partners. One young group member referred to her experiences gained in the advisory group in her university application, and felt that these skills would be useful at university:

I think definitely when I was younger it helped confidence-wise, just being able to speak in front of people you don’t know and particularly being able to disagree with people politely and not to get frustrated, I think that definitely helped me when I got to uni[versity] as well and I had to talk to irritating people in seminars and so yes, I hope it will help me with my confidence.

Research partner 5

Another research partner noted the personal impact of involvement for some of those participating:

I think the other thing I haven’t said is it has been quite useful in terms of personal contact, if that doesn’t sound selfish.

Research partner 6

Involvement in advisory groups provided research partners with opportunities to learn from others experiences:

One thing that they have said, they really like to come and see what’s going on in other sites, particularly for those sites where they feel like it has been a struggle.

Research manager 6

The struggles (with recruiting study participants) referred to here were also mentioned by a research partner who had felt personally responsible for the lack of progress. The national meeting attended by research partners from the four sites provided a helpful forum for reassurance and support:

Actually we ended up with more participants than [another location, which] has made me feel better.

Researcher 12

One case study reported that research partners had experienced difficulties when other patients made unkind remarks about their involvement in research activity:

About 6 or 7 years ago [name of research partner] brought up a problem, I think it was at one of the training days, that some of the other patients who were sitting in the waiting room who recognise her and a few other people are being sort of involved in the research and things were making nasty remarks about them getting special treatment because they are friends with the doctors.

Researcher 5

While this issue had occurred some time ago, it remained in the thoughts of the researcher and highlights a challenge for some members of the public who become involved in research.

Some of the mechanisms put in place by researchers to support research partners were not used. For example, case study 2 developed a website to facilitate communication between research partners, but they did not use it.

Case study 2 also explored whether payment was an important mechanism to support public involvement, and, for this group, knowing they were making a contribution was a more significant factor:

I’ve been asking parents what they think about payment and whether that was a motivation for becoming involved and actually it’s not really, it’s more about wanting to do something that’s interesting and an interesting subject matter. So I don’t think the payments are particularly . . . well, I think it’s a way of saying you are valued, but the parents don’t seem to see it in that way.

Research manager 3

Providing feedback to research partners which demonstrated that their input had influenced a project was commonly noted as an important motivator:

So again they were quite pleased to see that they had had an input and we had actually listened to them. So you asked about what do we give back to them? But that in itself felt as giving back.

Researcher 6

Only one case study expressed, in an early interview, an explicit objective to promote the development of research partners, who joined the study as delegates from a user-led third-sector organisation:

We are very keen to be not only having users involved but also to be kind of cutting-edge and kind of do something that’s innovative and relevant to the people that we are working with. Because I think it can be relatively easy to involve people who already have skills at a very high level and we wanted to make sure that our approach was inclusive and we allow opportunities for people to learn and develop as part of what we were doing. So we wanted to do something that was innovative and kind of pushing the edges a bit and giving opportunities . . .

Research manager 7

When asked about the impact on the research partners at the end of the data collection period, another researcher in the same project reflected on how difficult it was to assess this:

I don’t know, it depends on what you mean by impact. You know, I think a lot of people are very interested. I am sure they have taken those ideas somewhere else, I am sure they have talked about them with other people, but they were anyway engaged in this kind of work . . . So whether that has impacted beyond what they would have had anyway or maybe it has become part of their other involvement activities I suspect. We haven’t really identified people who have never been involved . . . So it is one of the things that this model does, it brings in people who haven’t had that opportunity before, so in some sense the impact is perhaps less obvious for that reason, it might well be people have gained confidence from it, but I haven’t asked them and it’s very hard to know.

Researcher 6

This was, however, the only case study where research partners reported feeling disappointed or frustrated. The disappointment arose from mismatched expectations about the roles they were to play in the study. The research partners recruited had unusually high levels of experience of working in research, and initially expected to be involved in data collection, which the academics had not planned for or expected. The same research partners also felt frustrated when some distinct local issues hampered recruitment within their study site.

This issue was resolved, and the project continued successfully with the research partners fully engaged. The partners were later encouraged to take on analysis and dissemination work, which they enjoyed. However, this episode illustrates the potentially negative impact of differing expectations or a lack of clarity about roles at the beginning of an involvement partnership.

The example above also exemplifies a more general observation that the case study research teams did not usually have mechanisms to capture the impact of public involvement; this is unsurprising, as many decisions and internal processes in research teams are not recorded or only noted briefly in meeting minutes. Case study 2 was unusual in that it tried out several different mechanisms to try to capture the impact of involvement from the perspective of their parent panel members:

So in terms of actually demonstrating the impact of this involvement from your perspective is . . . to map what the parents are saying in the group discussions and then seeing if we can actually see where that is influencing decisions that are made by the project team?

I think it’s more than that; you’ve got to capture your entry point, where they are now. So hopefully that has been caught and this is why I’m keen that we look at them keeping a ‘diary’ so that we can actually capture how they are feeling, you know what’s changed for them?

Another mechanism later used by case study 2 was cartoon drawing, as observed in one of their parent panel meetings:

Towards the end of the meeting, the PI introduced the idea of ‘cartoon-drawing with speech bubbles’, which the parents took to enthusiastically to ‘capture their journey’. They all said positive things about their participation being more interesting and rewarding than they had expected.

Field notes extract

Outcomes conclusions

Across our case studies the main objective of research teams in involving the public was to improve the practical delivery of research; overall, there was good evidence of some impact by research partners in all of the case studies, except that the impact in case study 4 occurred before the study period. Research partners contributed to a number of aspects of research delivery including improvements to ethical review applications, recruitment materials and strategies, data collection instruments and methods, analysis and dissemination. There was evidence of some longer-term impacts and more strategic impacts, but these were harder to document. A secondary outcome for most of our case studies was the range of positive experiences that research partners reported, with only one negative experience reported early in one case study.

Comparing outcomes with the initial objectives for public involvement was difficult over the relatively short time period of our data collection, particularly for those case studies for which there were substantial further stages of research still to come. Most of the case studies did not themselves have mechanisms to capture the impact of public involvement. The one case study which attempted this found it took some time to develop a mechanism (cartoon drawing) that worked for its parent panel.

The next step in testing our theory must be to turn to the contextual factors and mechanisms we hypothesised were linked to outcomes and assess what these tell us about our theory of public involvement.

Leadership and culture

Our initial theory hypothesised that a number of inter-related contextual factors would influence the extent to which the case studies achieved their intended outcomes. In this section we review the extent to which we observed the anticipated contextual factors of PI belief in public involvement, senior leadership on public involvement, culture and values, and history of public involvement within the wider institution.

Diversity in types of studies

The field of research where the project was situated influenced the range of options available for public involvement and that this happened as an implicit, taken-for-granted choice, rather than as an explicit conscious decision. For example, in one case study there was a predominance of team members coming from epidemiological backgrounds, rather than more applied clinical backgrounds. When the participation lead suggested more diverse methods for the participants to be involved, the executive group did not support the proposal. The reasons for this may have been completely valid in their own terms, but the issue is that the type of study and methodologies chosen at the outset may not lend themselves to the participants being more involved. In another study, a clinical trial testing a set of interventions, a research partner sits on the Independent Data Monitoring Committee, a requirement of trial-based studies. While this research partner was made very welcome by the chair and committee, they still felt that the nature of the discussions was so technical that their contribution was limited. In several cases the research partners came into a study after inception and funding had been agreed, thus limiting their ability to contribute at the design stage. For each of the studies involving a clinical trial, there were long gaps in involvement activity during the data collection phase; in each case there were good reasons for this, but it does again demonstrate that the field of research does shape the opportunities for involvement.

Context conclusions

We have explored our selection of hypothesised key contextual factors for effective public involvement in research through our eight case studies. As we expected, these factors are interconnected and difficult to interpret separately. The impact of PI leadership on public involvement as a contextual factor to put mechanisms of involvement in place, and on intended outcomes, appears to be a critical factor across our case studies, but has followed a somewhat different pathway in each one. In most cases there were clear expressions of personal commitment from the PI; in some cases this was inextricably linked with a wider institutional culture of public involvement. In two cases with relatively new researchers, their personal commitment appeared to be shaped by this wider culture. In only one case was the initial impetus primarily from a funder’s requirement for involvement; but even here the PI was influenced by their experience of it to value public involvement. Thus PI approaches may be shaped by a culture of public involvement or may help shape it. Most of our case studies were based in organisations with a history of infrastructural support for public involvement, but this did not appear as significant an influence as the PI’s personal approach. It would appear that either a committed PI or a wider institutional culture (or both) is necessary and sufficient to engender effective public involvement. Our case studies have helped us develop a more nuanced view of how contextual factors affect public involvement. Overall, our case study data have broadly validated our theory with respect to the role of context in CMO regularities for public involvement in research with the additional recognition of the importance of the field of research.

Relationships

Resources

Resources in the broadest sense was another aspect of the mechanisms of involvement which included budgets for involvement, training and support, defined roles, practical aspects and rewards.

Structures of involvement

In our initial conceptualisation we identified involvement throughout a research project, long-term involvement and linking involvement to decision-making as potential key mechanisms for public involvement in research. In developing our analysis these interconnected factors all appeared to relate to structures of involvement and we explore them here under that overarching theme. Across the eight case studies, structures varied according to which type of activities research partners were involved in. Case studies differed in how actively members of the public were involved in the projects. In four of the case studies, research staff had initiated and recruited members of the public to a study-specific panel or advisory group. In two other studies researchers had accessed an existing service user or community group. In the final two studies research partners worked more individually, with individual research partners at different study sites in one multisite study, and a pair of research partners working with the other project.

Whatever the structure of involvement, research partners were asked for help and advice with similar aspects of the projects, particularly on recruiting participants and the format or wording of documents, such as information sheets. In the longitudinal study involving young people, the group met frequently and regularly, as well as communicating by e-mail and online. In another case study, meetings were held every 2 months; however, the PI felt that even this frequency did not always guarantee full involvement:

Every two months we’re doing it [holding meetings], but the kind of the meetings or the discussion happen here informally, then by the time their [parent panel] next meeting comes around we‘ve already formulated [issues].

Researcher 2

In the other four studies, the groups/panels met less frequently. Involvement was often characterised by short intervals of focused activity, interspersed by longer periods when the group neither met nor contributed to the study’s progress:

That’s what we found with our patient and public involvement group that we seemed to have periods of quite intense activity and then it seems to fall down.

Research manager 2

A different model of involvement occurred in the two case studies where a few individual patient research partners were included as research team members, attending relevant project meetings. However, despite greater direct engagement, the pattern of involvement resembled that of the other case studies; the research partners contributed to the projects only relatively infrequently, depending on the stage of the research. In these two projects and some others, research partners had input into the study design. However, in a few cases, this input did not involve research partners, who subsequently became members of the panel during the conduct of the study:

So I worked together with those two people to develop the proposal, and we also had a service user and researcher working with us at that time who also was involved in developing the idea.

Research manager 7

I would be really surprised if they hadn’t involved people with experience in the design stage, but I don’t know whether they did.

Research partner 3

The difficulty of involving research partners effectively in the early design stages of a study was described by one researcher:

When you respond to a commissioning brief, you don’t have much time and it is very constrained as to what you can do . . . it would have been nice to have been able to have gone to the group before we put the application in, but that wasn’t possible.

Researcher 13

Four studies involved research partners in developing applications for ethical approval. In at least one case, this input was crucial:

As part of their [ethics committee’s] agreement to allow us to have it as a non-consenting study, we had to demonstrate input from our patient panel that they were happy.

Research manager 2

Some researchers expected that, following their experience in current projects, research partners might get involved in designing future or ongoing studies. There was one incidence reported of the design of a study being fundamentally altered following research partners’ input:

Somebody had got funding to do this particular study on friendships . . . for us it was quite clear about how that would work . . . we took it to [the servicer user panel] who hated it, absolutely hated it and just consistently kept saying this is too private, it’s too complicated . . . it ended up being a very, very, very different study to the study that we had first planned.

Research manager 1

Five of the case studies relied quite heavily on research partners to facilitate access to particular communities and to support recruitment of participants. Researchers reported having found this advice invaluable in some instances, particularly where young people were involved:

On a very practical level with the project . . . I came with our own draft design of some leaflet and the group . . . didn’t even want to look at it, just threw it over the side of the chair and said this is far too much text . . . if we sent such a leaflet out to our participants, it comes through the door and it goes straight in the bin.

Researcher 6

Research partners in seven case studies were asked to comment on and help develop the study instruments, including questionnaires and interview guides. This appeared to be standard practice; all the researchers interviewed were keenly aware of the need to ensure that questions asked were both appropriate and intelligible from the respondents’ perspectives. In the eighth study, research partners were asked to help design materials for an educational programme which was being tested.

There was only one case of research partners being actively engaged in data collection, where some study participants belonged to an underserved ethnic minority community. The researchers found that the most effective way of collecting data from this target population was to work closely with two representatives from the community, who then went on to collect data from a group comprising other community members.

In another study, there had been plans to involve research partners comprehensively in data analysis, but logistical problems had prevented this:

It is difficult to know what to do with the analysis . . . They [research partners] don’t have the time to get involved in kind of analysis systematically across the board; so we are using a framework analysis . . . They are identifying what they think the key things are from [specific transcripts], and those will then be collated by us into an overall framework . . . which can then be systematically applied, but that will be done by the team and not by the lay members.

Research manager 6

In two other studies, findings were brought to the public panel/advisory group for discussion. In one of these studies, the PI explained why he considered this to be important:

Getting a perspective on some of the importance and the meaning of some of the results as well, so we will have a view as researchers and academics about what we think is significant; I don’t mean statistically significant . . . I’m also interested in actually tapping into a lay perspective about what might be important messages for the wider community.

Researcher 7

In the two case studies where research partners were considered team members, they were actively involved in analysing qualitative data. This entailed their receiving appropriate support and training in order to perform this function.

Research partners were involved in dissemination of study findings in half of the case studies, although it should be noted that some others were not yet at a stage to disseminate. In one of the studies where involvement in dissemination had not occurred, this omission was recognised by one of the researchers on the project:

We wrote in our protocol that one of the things we would ask our parent advisory group to do was help on disseminating information . . . and we haven’t fulfilled that bit.

Researcher 4

In another case study, it appeared that plans to involve research partners in dissemination could be inconsistent:

One of the things that was decided in a meeting that I was not at, that we would create a book . . . At no point was I asked ‘Can you engage the cohort and find out what they would like from a book?,’ nothing, it was just decided we would have a book . . . I was like ‘Well let’s maybe get some involvement now’ . . . I got it agreed that we would advertise for a participant to become involved in that working group, who would sit on the interview panel to interview the designers and the editors of the book . . . and I got them to agree that we would have presentations as part of that interview process and they would present to a group of participants and they would feed into who we decide to have.

Research partner 12

I’ve taken a couple of [the young people’s advisory group] members to different conferences with me and they’ve presented about [the group] to those conferences . . . They were involved really heavily with the dissemination and doing the posters . . . like revising the lay posters, and then they created their own film of what [the study] is about and they ran with that project completely.

Research manager 4

However, in most of the studies, there seemed to be a clear intention to ensure future research partner involvement in dissemination of the study findings:

The next thing that they will be involved in is in the dissemination of the results of that study. How we do it, how we tell the doctors, because that’s going to feed in to the educational component. So they will have an input into how we disseminate it on more than one level.

Research manager 2

Planned dissemination activities included co-authoring papers for publication, as well as contributing to the developing and presenting posters and papers at conferences.

Research partners were involved in all project stages, from design through to dissemination, in only one of the case studies. Although engagement was sometimes irregular and infrequent, in most case studies there was input from research partners to most research activities, apart from data collection, at least to some extent. Input into design sometimes involved different research partners from those working with the research teams at later stages.

Some informants raised questions about the changing status of research partners who develop considerable experience in research. Some researchers perceived these individuals no longer functioning purely as patients or service users, and felt this might not be beneficial for projects:

Well the naive patients, if they help and then help again and help again, they become sophisticated patients . . . One of the things that we’ve taught ourselves to do is to always try and get two patients involved, an old one and a new one.

Researcher 5

You could argue that once they have sat through six months of the committee they should be off it, because they are already socialised into the way of a committee, and actually they have already stopped being independent.

Researcher 16

In five of the case studies, involvement appeared to be managed in a systematic fashion, although clear systems had not always been included in the study design. One informant considered this a disadvantage:

Perhaps a slightly more structured approach than we had here . . . work with clearer plans before we go in . . . It might have helped the outputs, I mean, it was great that we got [public involvement], but it could have been even better.

Researcher 6

Two of these case studies involved a two-tier model of involvement. In the third-sector case study, the project design incorporated involving service users at both national and local levels. In the other case study, based in the NHS, some research partners belonged to a formally constituted panel, while others were intended to contribute as and when required:

We’re looking at two levels of participation, the first is a panel. So we have 8 to 10 parents who attend bimonthly meetings to review and discuss some of their project issues . . . On a wider level we realise that some parents can’t commit to being on a panel formally . . . but they’re happy to be involved on an ad hoc basis and to review documents as and when they can.

Research manager 3

In practice, this wider group did not get involved over the course of the year, partly because of time and logistical constraints on the research team, thus illustrating that the absence of relationships with research partners may be a barrier to involvement.

In the two case studies where research partners were regarded as members of the research team, structures for their involvement were relatively informal; this may have been a function of their closer inclusion in the team. In the case study where the PI was involving research partners for the first time, the researchers sought to manage the group responsively to the needs of the moment:

It was a relatively new thing so we just had to sort of learn a bit about it and try and understand what it was and how we might deliver it . . . we sort of sit and think, well that would be an interesting thing to discuss with the patient panel, why don’t we ask them about that? . . . we’re sort of coming at it in a slightly more ad hoc way . . .

Researcher 10

In three cases, practices for managing involvement changed during the course of the project. This appeared to happen for a number of reasons, including trying different approaches and finding that some did not work, as well as pragmatic considerations, including staff changes:

We set up the website for them [the parent panel] to use. I think they were very enthusiastic about that at first but actually I don’t think that’s really adding a lot so I think we’re just going to go back to . . . [doing] most things via email and kind of sending things in the post so . . . I think that something that they wanted introduced actually hasn’t been working so we’re going to get rid of that.

Research manager 3

I think originally the concept was that those groups should to a large extent shape what their involvement would look like themselves . . . But that’s not what actually happened, and it might be to some extent that that has happened less and less as the staff involved in the initial conversations left, and as we had over time to just focus on how do you deliver this project.

Research manager 6

In one case study, the long-standing nature of the research programme necessitated changes in the management of public involvement over the course of time, as environments and wider requirements and standards changed, as did the research partners involved.

In five of the case studies, particular individuals held responsibility for co-ordinating involvement. In some cases the role description was drawn up during the study design process, but in others it developed as it became apparent that involvement co-ordination and management were required. One participant clearly described the requirements of the co-ordinator role:

You do have to be organised and . . . do the administration part of it quite well . . . whether you’re able to establish that relationship or not . . . I can’t think how specifically I’ve done that, I’ve just been there every time that the parents have been in, I’ve been around and met them all and got to know them quite well; but maybe, you know, I’m a parent, so I have something in common [with them].

Research manager 4

In the remaining three case studies, co-ordination of involvement appeared to be shared between different members of the project team. In two of these, research partners were themselves project team members.

Two case studies could access their research partners only through external gatekeepers. One of these studies involved underserved and/or ethnic minority communities, and it was felt to be both practically and ethically necessary to approach service users through individuals with whom they already had established relationships. In the other case study, it was felt that research partners might find it easier to refuse researcher requests if these were channelled through an intermediary:

[Name’s] assumed a sort of gatekeeping role . . . all of the enquiries are filtered through [name] . . . I think one of the fears is that actually if you are approaching the parents and the parent advisory group directly they may feel as though they ought to respond . . . or they ought to co-operate, whereas they may feel able to say no more easily to the gatekeeper, and she can then feed it back through.

Researcher 4

There were very few examples offered by informants of specific practices or mechanisms for managing involvement. Only two participants mentioned arranging meetings at a time which would suit research partners. In only two of the case studies were there specific practices for managing and recording relevant meetings:

The stakeholder work package is represented on the project executive [meeting] . . . we meet monthly . . . because there’s always a long agenda, we cycle through different areas to focus upon . . . Every third month we have a focus on stakeholder involvement.

Researcher 7

Each [advisory group] meeting is minuted, so at the beginning of the minutes it sort of represents the feedback from last time.

Research manager 4

A lack of record keeping, coupled with staff changes, led to different assumptions about payment for involvement in one case study:

Where that has been a real challenge is . . . where people are saying ‘Well, this was agreed’ and we have got no record of that agreement, that conversation; and it makes it very, very hard, because people are working to very different assumptions, and I think probably that has been the biggest implication of the staff changes.

Researcher 12

Overall, there were fewer regularities in the structures of involvement than in other mechanisms of involvement we have explored. The degree of involvement through research projects varied, as did the extent of long-term involvement. The informality of much involvement activity made it more difficult to robustly demonstrate links between involvement and decisions made. There were clear links with operational decisions about recruitment materials and data collection tools, but less clear and less frequent links with more fundamental design decisions.

Mechanisms conclusions

The importance of mechanisms to engender and support good relationships between research teams and research partners was emphasised by informants across all our case studies. While recognising the inevitable power inequalities in the relationship, informants highlighted the actions researchers could take to bring greater equality and trust into these relationships. These included taking time to establish trust in relationships, getting the right balance between formal and informal processes (perhaps emphasising particularly the crucial importance of the informal), and giving feedback to demonstrate how researchers were using and valuing research partners’ contributions. Although we had recognised the importance of attitudes of trust and respect towards the public involved in our initial theory, these findings represent an important development, as we had initially seen them as contextual whereas now we recognise them as key mechanisms that need to be actively nurtured by researchers.

Budgeting and formal training did not appear to be major mechanisms contributing to the impact of public involvement in research in our case studies. However, both researchers and research partners supported and valued ‘payments in kind’ to research partners for the contributions. Only two of the eight case study projects paid research partners for their time. In only one of these cases, our mental health case study, was the payment seen as a significant factor. It is important to recognise the range of reasons why payment may have been regarded differently in this case study. Cultural and political values have influenced expectations about payment for mental health service user involvement in both research and service delivery, and relationships between patients and clinicians are often not as trusting as in other health-care fields.

The policy of not paying some research partners may also reflect the fact that in most case studies involvement took place through advisory groups or panels, which took relatively little time and required fewer skills; payment issues are seen differently where research partners play a more substantial role as coresearchers. Payment was acknowledged more widely as a problematic issue in public involvement in research, for a variety of reasons, including the problems it creates for research partners on welfare benefits. Issues connected with payment and payment in kind are very complex, and are explored in two publications on budgeting from INVOLVE. 23,25

Training was seen as crucial in only one case study, where research partners were offered the opportunity to contribute to the analysis of qualitative data. However, we must caution that training may be less needed in cases, such as the majority of ours, where research partners were mainly taking advisory rather than coresearcher roles. More generally, informal support was seen as much more important in engaging and rewarding research partners. The value felt by both researchers and research partners for ‘payments in kind’, such as access to support networks and information, for the contributions of research partners, was significant.

As with resources, structures of involvement appeared less significant than relationships in shaping the impact of public involvement in research in our case studies. Involvement throughout the research project was not found consistently; in most cases there was little early involvement in bid development and research design. In several of the trials, there were long periods of non-involvement during data collection and processing. Long-term involvement was important in some case studies, but others involved new research partners whose involvement was time-limited to the particular study. Informants made differing comments on the benefits and limitations of involving experienced and new research partners. Similarly, informants reported differing practices on how formally defined or informal and organic the research partner role was, and the benefits and limitations of each approach.

Impact on research design and delivery

Our team included six research partners, who were involved in all research tasks alongside the 12 academics, in an integrated way. Almost all case studies were researched by subteams of an academic and a research partner. It was therefore very difficult to identify and separate out the specific impacts of research partners’ many contributions:

Because the research partners have been involved from the beginning and to some extent have been so integral to the team, it is very difficult to pinpoint the contribution they have made. With hindsight, we should have thought much earlier how we were going to capture the impact of research partners within our own team . . . The overall design and research questions were better because of [research partner] involvement, though it is impossible to track the exact pathways of influence.

Researcher 20

This integrated approach contrasts with, for example, academics in other studies revising a questionnaire after comments from a patient panel, which can then be shown to the panel as evidence of their impact. In this project, the questionnaires emerged from group exercises, including team meetings and consensus workshops, which always included at least some research partners. The same applied to most aspects of the project:

In practice I would say [the research partners] contributed at least as much if not more than some of the academic partners.

Researcher 17

I am most aware of the research partner’s contributions in my own case study team. For me, this partnership has validated our decision to involve research partners in all aspects including data collection. There are a number of times that [the research partner] has contributed important insights that relied on being (a) a service user research partner and (b) involved in data collection and therefore experiencing/observing the issues at first hand.

Researcher 20

Individual research partners observed when their perspectives had an impact:

I’ve noticed various things where I feel, if I hadn’t suggested that, things would have just gone a different way and I feel that’s a very identifiable impact that I’ve had, and it’s carried through to having questions in the interview schedule and it’s carried through to it being a code.

Research partner 16

The academic working with the young research partners found it more difficult to pinpoint their impact:

I’d like to think that I could [unpick how their presence impacted] but I’m not sure I can. I’ve found it very enriching and enjoyed working with them, and they gave me a lot of insights. When we did the coding together, on some occasions we all looked at it from different perspectives, from different angles, but it was still a bit of an academic exercise. I think they have contributed an understanding of what is helpful and what isn’t helpful in terms of involving young people in research, which is important.

Researcher 17

There were also some team perceptions of negative aspects of involving research partners:

I suspect meetings go much slower because of the perceived need to be ‘gentle’ on research partners, and not appear to dismiss our views.

Research partner 16

To enable research partners to contribute to coding, we agreed to allow manual coding, which meant that we artificially restricted ourselves to 38 codes; in retrospect I think we thus failed to include some important codes. The whole process of working with research partners is slower for academics than working on their own; this has contributed to our struggling to finish the project within the allocated time and funding.

Researcher 20

Impact on public involved

Some of our research partners were experienced user researchers, but all expressed similar personal benefits to those identified by Staley:11 acquiring new skills and knowledge (or applying them in the case of experienced research partners); personal development; support and friendship; enjoyment and satisfaction; and being rewarded both by being acknowledged and financially. As with Staley’s analysis, for our research partners both ‘personal development’ and ‘enjoyment and satisfaction’ related to making a useful contribution to worthwhile research.

My motivation and reward is that . . . I think research is of a better quality, more applicable, more easy to put into practice . . . if the people it’s about are involved in doing the research . . . the fact that I’ve commented, and seen certain things being taken on board, keeps me on board as well.

Research partner 15

I’m not doing it for money. I’m doing it because it’s mentally stimulating. I actually enjoy it.

Research partner 17

There’s no way I would have done it without getting paid. But I greatly value the experience, too. It’s given me the chance to find a way back into the working world, and I feel able to do much more now than when I started.

Research partner 16

The best bits of the project were being able to work with so many different people and feeling valued as part of the team . . . there are so many aspects of it that I can transfer into my own learning.

Research partner 14

Some of these motivations and perceived impacts, such as learning, mental stimulation and money, applied equally to our academics. But there are other academic imperatives, such as publication and reputation, which were never mentioned by our research partners. This was an important reminder that academics and research partners have different motivations.

Leadership and culture

There was an existing culture of involving patients, users and carers in the Health and Life Sciences Faculty at UWE. Our PI played an important role in developing support for public involvement in research, including setting up the SUCIR initiative (now replaced by People and Research West of England), and leading the development of a system for paying research partners early in the bid process:

I feel he has made a tremendous effort in the extent we are involved, consulted, listened to.

Research partner 16

Relationships

Relationships were important for the whole of the research team, but especially for our research partners. Some research partners felt ‘parachuted in’ to an alien environment, so they relied on good relations, with both academics and other research partners, to feel comfortable:

I think if academics are good at being welcoming and friendly and just having goodwill and warmness . . . it makes it easier to say you don’t understand things . . . it also creates the connection – the academics have got there professionally, whereas, for us, we don’t have a right to be there.

Research partner 13

Because research partners had very different motivations from academics in the team, these motivations could be undermined if relations with academics were not good. It was via relationships with academics that research partners received feedback that their contributions were ‘hitting the mark’:

In team meetings, my perception is people really valued what we said.

Research partner 14

I would like somebody to say, that was helpful, that was good. I would like a bit more of that, really.

Research partner 15

Structures of involvement and resources

The idea for the bid grew out of meetings of SUCIR. Most of the original research partners were co-applicants on the bid, and their early involvement was funded through UWE’s policy of paying for public involvement before external funding is granted. The original research partners were involved in all whole-team meetings. One original research partner, in rotation, attended each steering group meeting. Research partners also met separately, with one or more of the academic staff attending only on request:

Meeting as a research partners group is of value. When you’re involving service users [it’s important] to give them time to begin to get to know each other, to feel less isolated.

Research partner 17

Original research partners were able to be involved in virtually all aspects of the project, including interviews, planning and facilitating at consensus workshops, developing the coding framework, analysis, contributing to report writing and planned dissemination.

Public involvement I’ve seen before has involved the public in quite a small role. This project took it to a different level, which is much harder, and I think research partners and academics were sometimes out of their comfort zone.

Researcher 19

Some research partners decided not to undertake some tasks, where they lacked confidence, skills or interest:

It has become very clear during the project that not everyone wants to do everything – what I’ve learned from that is that those possibilities need to be planned for, or at least borne in mind, from early on, so that managing them doesn’t upset the rhythm of what you as a team are trying to do.

Researcher 18

The initial plan was that young people would be involved via an existing young people’s panel. When that did not work out, two individuals were recruited instead. When one original research partner went off long-term sick, the young people were supported to develop into the research partner role. This process left one academic with reservations:

It was all done pretty much on the hoof, and I think it was probably done on the hoof as well as it possibly could have been done. But I’m still left with an uneasy feeling about why did we perhaps not say, well OK because this has happened [the panel not working out], actually now it would be unethical to suddenly bring young people in?

Researcher 17

The delay in starting involvement, a relative lack of prior research experience and less clearly defined roles meant that the young research partners were never as fully integrated into the team. Nonetheless they did come to some whole-team meetings and research partner meetings, and were involved in some interviewing and coding.

Most of the project’s work was organised around four teams, each of which was responsible for two case studies. In response to various factors over the life of the project, these teams morphed into various shapes, though all but one still included a research partner. One case study, though, was undertaken by the research fellow alone, leading to some research partners feeling disappointed:

To me that went totally against what I thought the other seven projects were doing, which was working with a service user. I didn’t understand, and it went against what I thought this project was about.

Research partner 17

An overwhelmingly positive aspect of team members’ reflection focused on how involvement had been integral throughout the project:

From my perspective we definitely have been involved all the way through it from the beginning.

Research partner 13

It’s been very useful having the service user point of view on an ongoing basis – it’s so easy to start a project intending to give that view equal weight to any other, and then for everything to slip because you don’t meet with/work with service users regularly. I think the important thing is face-to-face regular contact.

Researcher 18

Despite research partners being involved throughout the project, involvement processes did not always work seamlessly:

Writing a protocol for the ethics application commits you to quite a lot of detail . . . I don’t think there was as much time as we might have had to talk about some of that. But I think it is that thing about deadlines and the academics having loads of other things they have to do.

Research partner 13

Support for research partners started out being the research fellow’s responsibility, though research partners felt that at times it was squeezed out by the pressure of other work:

You can have all the policies and the right system in place, but when push comes to shove . . . then all the stuff about nurturing public involvement just goes out of the window because the project has got to be delivered.

Research partner 15

Part way through the project a research associate joined the team, and most research partners felt that helped ensure they got the support they needed. One research partner was ill for 6 months, and she felt her return to the project was not smooth:

Given that research partners have illnesses and limitations, [returning to a project] is something that should be managed a bit better . . . For me it is working now because I was offered the support of someone I’ve worked with for a long time . . . and that’s made it all much better.

Research partner 15

It’s clear there’s a need to have some sort of ‘advanced agreement’ with research partners that lays out the process to follow in the case of significant illness or disability during the project.

Researcher 20

Early in the project, the potential need for mentor support for research partners was identified and one of the academics agreed to take on the role, but this did not work as envisaged:

We identified a senior academic that research partners could go to for support, although this did not work in practice, due I think to her not really having a relationship with the research partners. The one research partner who did want external support went to a member of the project steering group and the project was flexible enough to support this financially.

Researcher 20

Conclusion

The project set ambitious goals for its own public involvement, and it is not surprising that research partners had high expectations. Our collective reflection is that, on balance, the benefits of our own public involvement vastly outweighed the costs, and that it would in practice have been impossible to do this research without such integral public involvement. Involvement by research partners was extensive, running throughout from concept to (planned) dissemination. And it was intensive, often involving research partners in detailed, hands-on work such as interviewing and coding transcripts. They have remained involved up to and including contributing to the revisions to the report, and will be involved in further dissemination.

There are lessons to be learnt. For the young research partners, starting involvement later was not ideal. And there were aspects, such as the support of research partners, where the reality did not always match up with their expectations or needs. As with our case studies, the academic researchers collectively underestimated the time required to support research partners and the challenges of balancing meaningful involvement with other academic pressures. There were other support needs that became clear as the project progressed, for example research partners’ access to computer software and training.

What was the impact of public involvement in our project? There were significant positive impacts on research partners, but also some frustrations along the way. We believe there were also significant positive impacts on the research design and delivery, although these were not always easy to pin down. One concrete example was that research partners as well as researchers used their networks to identify and contact potential case studies, and some of our case studies were recruited from this intelligence.

Our experience was that it was important to build flexibility into the research partner role, and this insight also informed our analysis of the case studies. In our team, some research partners took on all research tasks while others took on more limited roles.

Critical learning that we take from our own experience of public involvement is that, the more integrated the involvement, the more difficult it is to demonstrate impact without a significant investment in resources to capture and analyse the impact. Our reflective processes were very beneficial, and helpfully informed our analysis of our case studies; in retrospect, however, it would have been even more valuable if we had sought and obtained the resources to more systematically capture and analyse the impact of public involvement in our own project as well.

Phase 1

Two health economists drew on experiences of the research partners in our research team to develop a resource log checklist which would be simple to use, but also capable of capturing the diversity of resources incurred by various case studies. Detailed feedback was received from the research partners and revised in the light of their comments and specific feedback (see Appendix 1). Our development process within the project team revealed that ‘on the spot’ help was vital to support capture of resource use data and completion of resource logs. As a result, guidance documents were developed to help the case study researchers and research partners to complete resource logs (see Appendix 2). A telephone helpline was set up to resolve any issues and clarify participant understandings of requirements for completion as they arose.

Phase 2

The project research fellow, who had the most complete overview of the eight case studies, worked with the health economists to identify those researchers best placed to complete each case study resource log. In accordance with our ethical approval these identified researchers also nominated a research partner from within their team to complete a resource log. A health economist then had a one-to-one discussion (either in person or by telephone) with each of our case study informants to provide detailed notes and guidance on how to complete the resource logbook. All participants completing the logbook were also given a helpline e-mail or telephone contact, which was monitored during the data collection phase over a 2-week ‘snapshot’ period during 2012. The snapshot period was selected at random; respondents were also asked to assess how typical or indicative each snapshot was of the overall profile of their involvement-related activity. Their assessments within each case study contributed to the process of developing the assumptions for costing (see Appendices 3–5), which we applied to project the overall costs.

Costing price assumptions for researchers

We did not ask participants about pay grades, but assumptions were made based on job title, for example professor or senior research fellow. If titles were not given, assumptions about salary scales were made based on the description (from the resource log) of the specific task that the member of the research team was undertaking, for example preparing paperwork for an advisory group meeting (administrative role) or preparing slides for presentation at an advisory group meeting (more senior role, such as senior research fellow). Where we were unsure we asked case study PIs to clarify grades from information available. Price assumptions are detailed in Appendices 3–5.

Costing involvement activity of research partners

A flat rate payment rate of £19.40 per hour based on an INVOLVE example23 was the price applied to the resource use estimates of research partners.

Completed resource logs

Our aim was to have at least one completed resource log for both research staff and for nominated research partners (referred to as members of the public in the resource diary) for each of the eight case studies. All eight case studies responded, and response rates for both resource logs were very good.

Research staff completed 12 resource diaries and one interview, giving 14 responses; no one refused to complete a resource diary. There were four completed resource diaries and a summary of costs (via researcher) for research partners plus data from the researcher (case study 4), giving six responses in total. There were no research partner responses from two case studies (case study 1 and case study 8) for lack of a nominee from researchers within the team.

Other realist evaluations of public involvement in research

At the time we began our study, we were not aware of any studies that had applied the realist evaluation approach to public involvement in research. Subsequently, Staley et al. have produced a paper based on Staley’s earlier literature review11 calling for studies of public involvement in research based on the principles of realist evaluation. 20 They argue that studies based on realist evaluation can test the links between CMO more rigorously than existing observational studies. The key difference in their view is that with a realist evaluation the researcher identifies a specific question, develops a hypothesis about how this type of involvement could make a difference in the context of their study, uses this hypothesis to select appropriate methods for capturing impact, and factors in the contextual and mechanistic factors already known to influence outcome. They identify two recent studies of involving peer interviewers which ‘are more in line with the principles of realist evaluation’. 27,28 Neither of these studies was presented by its authors as a realist evaluation, and the first of the two, by Gillard et al. ,27 is of debatable relevance because the authors neither present a hypothesis nor explore the links between CMO. The second study, by Hamilton et al. ,28 is arguably more relevant, although again there is no explicit discussion of CMO. There were, however, five clear related hypotheses to be tested (e.g. that involving peer interviewers would enable participants to be more open and honest about their experience of stigma and discrimination). Implicit in the study are a number of context factors (e.g. shared experience of stigma and discrimination among mental health service users) and a facilitating mechanism of disclosure (about having used mental health services) by the interviewer leading to the outcome of more complete data collection.

Notably for a study exploring user involvement in research, this paper gives no details of the extent of user involvement in the study itself. One of the paper’s authors is a well-known user researcher, but the significance of this is not discussed in the text. Thus we know nothing about how user members of the research team contributed to the study design, methods or outcomes. It is therefore difficult to see connections with our theory, particularly as none of our case studies involved a peer interviewer model. There is, however, a connection with our team experience of research partner coresearchers. In our study team, academic researchers and research partners shared the interviewing of both case study professionals and research partners. Although we did not do an experimental study as Hamilton et al. 28 did, in terms of our reflections on our interview experiences, we did not notice any consistent differences in response or the openness of either professional or research partner interviewees between our academic and research partner interviewers. To that degree, our experiences have been consistent with the findings of Hamilton et al. 28

The importance of principal investigator leadership

Principal investigator leadership is a critical element in our theory of public involvement in research but was not explicitly identified in the reviews of either Staley11 or Brett et al. 12 although it is implicit in a number of the factors they identify. Subsequently, PI leadership has not been highlighted in more recent reviews, conceptual papers or newer empirical work on public involvement in research. The recent paper by Shippee et al. ,29 which includes both a systematic review and a conceptual framework, does not identify PI leadership as an element of their framework. 29 Other recent literature reviews such as Brett et al.’s 2012 updated article on their systematic review30 or Boote et al. ’s31 bibliometric review similarly do not identify it as a factor. There have been several other recent papers offering conceptual frameworks for making sense of public involvement in research32 but these also have not included any reference to or discussion of PI leadership as a factor. Wider conceptual frameworks for public involvement in service improvement33 equally do not have any comparable component of professional leadership in their conceptual models.

There are two possible explanations for this disjuncture. First, we may have been mistaken in identifying PI leadership as a key contextual factor. But this does not seem likely to us, as this factor was identified so strongly across our data. Second, this may be a crucial factor that has not previously been identified simply because no one else has conceptualised it and examined the hypothesis that it is an important factor in effective public involvement in research. Although leadership as a factor is not explicitly named, there is some evidence in the wider literature to support this. Staley et al. report on service user collaboration in mental health research. 34 Although they do not discuss leadership per se, they discuss the closely related concept of the role researchers play in establishing the boundaries of involvement:

Many researchers have much narrower expectations of involvement, consequently limiting its impact. Some saw enhancing the ‘lay-friendliness’ of research materials and processes as the main purpose, while a smaller number realized the value of incorporating the service-user perspective into research design and delivery. A researcher’s understanding sets constraints on what they ask of the service users involved.

Reproduced from Staley K, Kabir T, Szmukler G. Service users as collaborators in mental health research: less stick, more carrot. Psychol Med 2013; 43:1121–5 http://dx.doi.org/10.1017/S003329171200166334

In most health services research, power lies with the PI and other researchers to determine the extent of any involvement, and we would argue that the opening of opportunities for meaningful public involvement by researchers is usefully understood in terms of leadership.

Our conceptualisation is also consistent with recent research about health researchers’ attitudes towards public involvement. For example, Thompson et al. have conducted a qualitative study of health researchers’ attitudes towards public involvement in health research. 14,35 They found researchers held a range of varying constructions of public involvement in research, with a number of benefits being identified but also some feelings of apprehension expressed. 35 In particular, they identified the issue of ‘epistemological dissonance’ and the ‘know–do gap’. 14 This relates to the tensions between researchers’ perceptions of the potential benefits of public involvement in research and their actual practices, which were often mediated by their perceptions of the limitations of involvement. In terms of our theory, these insights into the impact of researchers’ attitudes and behaviours complement our emphasis on the importance of PI leadership. Those PIs with a smaller know–do gap and less epistemological dissonance are more likely to be open to and allocate resources for involvement, thus enabling it to be more effective.

In most health services research, an academic or clinical researcher will be the PI, assembling the research team, and taking a lead in the distribution of roles and resources within the team. Although the PI will be influenced by external factors (e.g. funders’ requirements) and internal team factors (other team members’ experience, expertise and attitudes towards involvement), the PI’s leadership will be crucial in determining where public involvement in the project lies on the spectrum from tokenism to real partnership. The importance of this PI leadership role has not previously been explicitly identified or discussed in the public involvement in research literature, and represents an importance conceptual insight from our research.

We recognise, however, that more work needs to be done to understand and theorise leadership in public involvement. Leadership is more generally a poorly defined and contested concept. Martin and Learmonth among others have cautioned us on the need to critically analyse the discourse of leadership which has arisen in UK health services over the last 10–15 years. 36 In particular they note the way that leadership has been conferred not only on those in positions of formal power but on frontline clinicians and on patients and the public. In the context of research, this emphasises how we need to critically examine the extent to which researchers, research managers, patients and the public, and other research stakeholders share leadership in public involvement in research.

The question was raised in our second consensus workshop of whether leadership on, and a culture of, public involvement within service user groups was an important factor. This was not an aspect that came up strongly within our data except in the one case where research partners from a local service user-led group and researchers initially had mismatched expectations of the role the research partners would play. This is likely to be a more important issue for some service users with long-term conditions, where greater experience and expertise in involvement in research is likely, and for coresearchers. We are not able to comment from our data on leadership and culture within service user groups but it is arguably an important question for future studies.

To date, public leadership in research has been largely unexplored, except in the concept of user-led or user-controlled research (which represents a tiny proportion of funded research). 37 There is public leadership at a policy level within INVOLVE and more widely in the NIHR in terms of public membership of national advisory groups and commissioning boards. However, at the level of individual research programmes or projects, the PI will be key to the extent to which the public are invited in to take on involvement roles in general and leadership roles in particular. In this sense PIs play a key role in ‘constituting the public’38 in public involvement in research, that is shaping the ways in which the public are seen to have a legitimate voice in research. ‘Patient leadership’ has developed as a strong policy theme in health services more generally;39,40 to date the concept has not yet emerged in the research setting, and it is worthy of further exploration.

Other recent models of public involvement in research

Another important test for our theory is how it compares with other recent theoretical frameworks and models of public involvement in research. Over the last 6 years a number of authors have put forward such models including Oliver et al. ,41 Robinson et al. ,42 Morrow et al. ,32 Brett et al. 12,30 and Shippee et al. 29

Based on a systematic review of the literature, Oliver et al. used framework analysis to propose a ‘multidimensional conceptual framework’ for analysing public involvement in research with three critical dimensions: whether lay people were involved as individuals or as members of organised groups; whether public involvement was at the invitation of the research programme or in response to action by the lay public; and the degree to which the public was involved (consultation, collaboration and lay control). 41 The authors state that the framework combines and extends concepts proposed by Arnstein43 (the often-quoted ladder of participation) and Mullen. 44 Both the second and third dimensions of this model relate to the degree of engagement of the researchers (and implicitly the PI) in facilitating involvement, but do not explicitly consider the role of the PI in particular or the question of leadership. On the other hand, there is nothing in this framework inconsistent with our theory. The authors’ reflections on their framework emphasise the greater value and impact of researchers’ engagement with more collaborative approaches, which is consistent with our emphasis on PI leadership on involvement.

Robinson et al. present a simpler, two-dimensional conceptual model, although the sources of this model are less clear; the authors report having completed a literature review, but give few methodological details. 42 This model identifies two axes: the degree of collaboration (running from less collaborative to more collaborative) and the degree of empowerment (expressed in terms of a top-down or bottom-up approach). Like the Oliver et al. 41 framework, this model does not address PI or researcher leadership on involvement explicitly, but its concern with the potential exclusion of the public from the research process and the balance of power between researchers and the public is wholly consistent with our model’s emphasis on PI leadership and on the importance of nurturing interpersonal relationships between researchers and research partners.

A more complex model of involvement is proposed by Morrow et al. 32 drawing on ‘a range of philosophical and social theories of power and empowerment’ (although these are not detailed). This model focuses on both service user factors (what a person feels able to do, whether or not they feel their potential is being fulfilled and their sense of being valued) and research factors (research relationships, ways of doing research and research structures). The model details personal factors for service users, not researchers, so PI or researcher leadership does not feature explicitly in the model. However, there is a focus on research relationships, with key factors identified including ‘allocating sufficient researcher time for two-way communication in the early stages of the research studies’. 32 Thus, this model mirrors our mechanism of a senior lead allocating resources for involvement and the nurturing of interpersonal relationships. Again, there are no factors identified in this model which are inconsistent with ours.

Stewart and Liabo critique the (now superseded) INVOLVE model of levels of involvement (consultation, collaboration and user control) with its implicit hierarchy devaluing consultation. 45 They join critics such as Tritter33 in arguing that such hierarchies, often based on Arnstein,43 fail to reflect the complexity of public involvement. Instead they argue for a model where patient expertise is integrated with researchers’ and policy-makers’ expertise. In this model, patient expertise comprises knowledge of living with illness and receiving care whereas researcher expertise encompasses knowledge of research design, methods, conduct and analysis. We would argue that this model is unnecessarily restrictive in terms of public contribution to research design and delivery, but it is again consistent with our emphasis on the importance of PI leadership and relationships.

The most recent conceptual framework put forward for public involvement in research is based on a ‘metanarrative systematic review and environmental scan’ conducted by Shippee et al. 29 on what they call patient and service user engagement (PSUE) in research. The authors reviewed 202 sources which met their inclusion criteria and claim to have identified 41 which describe ‘some framework, conceptualisation or planning scheme for the parts of the PSUE process’. Although their methodology for the systematic review was clear and appropriate, the methodology for the synthesis of the 41 frameworks was much less clear, and the 41 sources are not identified. Nonetheless, the synthesis led to a two-part framework: (1) the integral components of PSUE and (2) the phases and stages of PSUE in research. In terms of the integral components, the authors identify four essential components: patient and service user initiation, building reciprocal relationships, colearning, and reassessment and feedback. One of the methodological uncertainties about this framework is how the authors arrive at these four categories from the synthesis of the 41 papers, particularly as the four categories are based on 10, 30, 9 and 2 studies respectively. Without further methodological detail it would appear the ‘building reciprocal relationships’ has greater face validity based on 30 sources than ‘reassessment and feedback’ does with just two sources. The importance of building relationships between researchers and research partners is a recurring theme across the frameworks and models discussed above, as it was in our own research and theory building. Again, explicit discussion of PI or research leadership is conspicuous by its absence.

Allocating resources for involvement

In several of the case studies the lack of a formal budget for public involvement in their projects did not appear to be a significant determinant of the extent of their involvement activities or the impact these had on the research project. But it is possible that lack of a formal budget for public involvement impacted on the type of public involvement adopted within case studies and restricted these activities to less active forms. Case studies without a formal budget for public involvement tended to rely on occasional advisory groups to deliver public involvement in research, which is one of the more inexpensive ways to involve the public in research.

How then does our theorising around the allocation of resources for involvement relate to the wider public involvement literature? There has been little or no previous research on resource allocation for public involvement in research. None of the literature reviews or conceptual models discussed above has given more than cursory attention to resource allocation, budgeting or payment issues for public involvement; for example, Brett et al. 30 include ‘funding’ as one of the contextual factors to check when assessing whether the right conditions for public involvement are in place, but do not explore what this might mean in any further detail. Similarly, we have identified no specific primary papers on costing public involvement in research (although we did find one paper on measuring the costs of public participation in health and social care more generally). 22 Our case studies demonstrated a wide variety of approaches to budgeting for public involvement and in the hidden costs that our economic evaluation identified. What this research demonstrates is that resource allocation is an important but often poorly identified or understood element in the causal pathway leading to impact. The challenges of budgeting and payment for involvement is an area much discussed in the field and has been the focus of recent helpful MHRN/INVOLVE guidance on payment and budgeting, including the provision of a new online budgeting tool. 23

Assessing impact

Our starting point around assessing impact drew on the reviews by Staley11 and Brett et al. 12 Brett et al. 30 have subsequently updated their discussion of their review in their 2012 journal paper but their analysis of the state of the art remains substantially the same, that is, that the evidence base is still relatively weak and requires further substantive development. Our experience echoes this. An honest reflection on our case studies is that we did not capture as much evidence of impact as we had initially hoped. This may partly be a result of the limitations of our methodology, but may also simply reflect the reality of relatively limited impact of public involvement in our particular selection of case studies. On the latter point, our case studies were not, in practice, as diverse as we expected when we recruited them, with more adopting an advisory group model and less coresearch than we initially envisaged. We could report only on impacts which we could document or observe, and these were mainly related to improvements in recruitment materials and processes. In terms of methodology, in hindsight we realise that we probably sought funding for too short a period for data collection and did not sample sufficiently the early period of research design, when research partners are more likely to be able to influence design aspects such as research questions and outcomes to be measured. There were, however, institutional factors that militated against capturing impact on the early design stage, including the long lead time between negotiating case studies for inclusion in our funding application and the eventual commencement of our data collection, with related delays in research governance approvals.

Our study is intended as a contribution to a wider methodological debate on how best to capture the impact of public involvement in research, and indeed the extent to which impact can be measured in any robust and meaningful quantitative way,46 and the relative value of quantitative and qualitative assessment. This debate has been developing for over a decade with an early paper from Telford et al. 47 and recent papers by Barber et al. 48 and Wright et al. 49 Nilsen et al. 50 have also produced a relevant Cochrane review which assesses the effects of ‘consumer’ involvement in research as well as in health-care policy, clinical practice guidelines and patient information materials. The most recent contribution has been the Public Involvement Impact Assessment Framework (PiiAF) developed by Popay et al. through a MRC Methodology Research Programme grant. 51 The PiiAF was disseminated too recently to influence our study but potentially offers the most evidence-based and systematic framework yet available for assessing the impact of public involvement in research. There are also two other larger-scale NIHR HS&DR- or INVOLVE-funded studies (one also based on the principles of realist evaluation) with longer time scales which will report in the near future and contribute to this methodological debate.

More recent empirical studies on the impact of public involvement in research published subsequent to the Staley11 and Brett et al. 12 reviews give some possible avenues for future methodological development. For example, Ennis and Wykes used routine data from the MHRN database to identify positive correlations between levels of patient involvement and recruitment success. 52 Staley et al. also utilised the MHRN database to conduct a qualitative follow-up evaluation of impact in adopted studies. 34 The more that meaningful data on public involvement can be routinely built into such research databases, the easier it will be to conduct larger-scale studies of the impact of public involvement in research. Thus, for example, if data on resource allocation for public involvement were more systematically captured, then these data might be correlated with data on participant recruitment to test our theory on the link between resource allocation and outcome.

Models of public involvement and impact

In revising this report we were specifically asked by the reviewers and editors to say more on the impact of different models and mechanisms of involvement in the studies reviewed on research practice and outcome. This comment stimulated us to think about the different ways in which the term ‘model’ is used and the implications for assessing impact. In most of the literature reviewed earlier in this section, ‘model’ is used to describe a conceptual framework for making sense of the overall complexity of public involvement in research. But the term ‘model’ is also used here by the editors and reviewers to describe the particular form or type of public involvement, for example involvement through an advisory group or by coresearchers. In our eight case studies, there were four such forms of involvement: project-specific public advisory groups (four), individual research partners contributing to research teams (two), working with an existing service user group (one) and a long-term programme advisory panel (one). Each of these broad forms included context-specific aspects; thus one of the individual research partner forms was undertaken in a unit with a long-term history and culture of involvement whereas the other was in a research group with relatively little involvement experience. Similarly, some of the project advisory groups brought together members of the public who were not previously known to each other while others drew from a group of service attendees who knew each other but were not a pre-existing user group. In addition, we also had the opportunity to reflect on our own coresearcher type of involvement. Our analysis did not lead us to judge any one of these forms as more likely to lead to desired outcomes; our focus was on the key context and mechanism factors that we assessed as making a difference (e.g. PI leadership and nurturing relationships) which were relevant to all these forms of involvement and which can be applied to a greater or lesser degree in each and any form. The specific form of involvement adopted by any research team will be determined by a number of contextual factors (e.g. field of research, organisational base, type of study and type of study population).