Project objectives

The overall aim of the project was to identify PPI activity across the SPCR and to explore the costs and consequences associated with this. The specific objectives of the project were as follows:

1. Identify from published sources and discussion with PPI representatives the broad types of consequences (intangible) and costs (financial) associated with PPI, from the following perspectives: Patient/Public, Researcher/funder
2. Identify questions from existing sources/previous studies and discussion with PPI representatives which relate to these broad costs and consequences.
3. Develop two data collection tools (surveys) for i) researchers and ii) patient/public representatives which adequately cover the full scope of costs/consequences identified for objective one.
4. Collect information on costs and consequences from SPCR project researchers (electronic survey via email) and from patient/public representatives (postal survey).
5. Categorise and analyse responses from the data collection phase.
6. Supplement/triangulate the data collected from the researcher survey with data on PPI activities drawn from project-related documents such as project proposals, annual and final reports, minutes and observations of meetings with PPI.
7. Develop a framework for presentation of costs and consequences of PPI involvement.
8. Develop an exploratory willingness to pay (WTP) survey to test whether it is feasible to use methods drawn from health economics to understand how people value PPI.

Changes to project since initial approval

Research Plan and Methodology:

Two changes were made to the original research plan because of methodological difficulties with accessing PPI representatives to participate in the patient survey (objective 3, page 2) and PPI meetings for the observation study (objective 6, page 2). As the contact details of PPI representatives involved with SPCR work were not available, we decided to ask Principal Investigators of projects to pass the patient survey to PPI representatives who had been involved in their projects.  Unfortunately, all Principal Investigators who responded to the survey were either unable or unwilling to pass on the postal survey to PPI representatives. This meant that we were unable to gather data on the costs and consequences of involvement from PPI representatives, although members of the Research User Group at Keele were involved in the analysis of the data and the development of recommendations to ensure some representation of the patient perspective.

It is also hoped that the patient survey that we have developed could be used as an evaluation tool to record the costs and consequences of involvement in future projects. The researcher survey was additionally used to sample forthcoming PPI meetings for SPCR-funded projects for the observation study. In the final section of the survey, Principal Investigators were asked to indicate whether they had any forthcoming meetings with patient and public involvement, and if so whether they were happy for two researchers to observe the meeting. Unfortunately, most respondents did not have any meetings with PPI planned, and one respondent was not willing for us to observe their meeting. Again, it is hoped that the development of materials for the observation can be used in future studies.

Brief summary

Background

Involving patients and the public in research is seen as a marker of good research practice and can lead to better research, clearer outcomes, and faster uptake of new evidence (NIHR, 2014). A key principle is active involvement whereby there is “an active partnership between the public and researchers in the research process, rather than the use of people as the ‘subjects’ of research”. This involves “doing research ‘with’ or ‘by’ the public, rather than ‘to’, about’ or ‘for’ the public” (INVOLVE, 2012).  Patients can be involved in many aspects of the research process (INVOLVE 2012) including:

• Identifying and prioritising research topics
• Commissioning
• Design and Management
• Undertaking research
• Disseminating results
• Implementing
• Evaluating impact

There is a small but growing knowledge base about the practice of patient and public involvement (INVOLVE 2010), and how to assess the impact of this (Barber et al, 2011; Wright et al, 2010). Tarpey has outlined motivations for user involvement generally (Tarpey, 2006) and criteria for successful patient and public involvement (PPI) have also been published (Boote et al, 2006; Hanley et al, 2003; McLaughlin 2009). Two systematic reviews of patient and public involvement have been published in the UK (Brett et al, 2010; Staley 2009) and these identify a range of benefits. However, both reviews also noted the immature state of the evidence base and that many studies consist of self-reports as opposed to rigorous independent evaluations (Staley 2009). Recommendations for future research were for:

1. More research to “tell the story of involvement” (Staley 2009). There is an expectation that researchers will have involvement in research and this is reflected in grant application forms. What is less clear is how well this expectation is translated into practice. Are researchers “doing PPI” because funders expect it rather than ensuring PPI is relevant appropriate and meaningful to all parties? There is a need to understand the extent of user-led, collaborative or consultative involvement and the various impacts of these different approaches.
2. Qualitative research that takes explicit account of process and context as well as impacts. This will help to identify how contextual factors enable or hinder PPI processes, activity impact and outcomes (Brett et al, 2010). This builds on earlier recommendations that further conceptualisation of patient involvement in research is required (Boote et al, 2002).
3. Future collaborations with health economists to advance understanding of how to develop economic appraisal of PPI impact (Brett et al, 2010). The recommendation was, therefore, to develop methods of economic appraisal of PPI in research. Putting a price on empowerment or involvement may be viewed by some as problematic, but an alternative view is that the costs of negative impacts (such as patient or carer burnout or disenchantment as a result of tokenistic practices) should be avoided wherever possible. The first stage would be to identify these costs and how and why they occur.

Further, the recent 'Breaking Boundaries' review of PPI published by the NIHR in 2015 has highlighted a need to improve the quality and practice of PPI in research. It is therefore timely to identify PPI activities within research projects funded by the SPCR and to explore the quality and impact of those activities.    

Methods

Research design

1. Documentary analysis

A documentary analysis was conducted on project proposals, reports and other documents containing data on PPI, collected from the SPCR and from researchers who had included PPI within their projects.

2. Researcher and Patient Survey

A cross-sectional survey design was employed. The researcher and patient surveys were developed through a review of the literature, a search for existing questionnaire items on the costs and consequences of PPI, and with patient involvement to ensure that items in the patient survey represented the range of costs and consequences experienced by PPI representatives. The surveys included items aligning with agreed quality indicators for good PPI practice (Boote et al, 2006) to enable the assessment of PPI activity in the School against best practice, and also items on the perceived impact of PPI.  

3. Willingness to pay-type study

A cross-sectional survey design was employed which drew on principles of willingness to pay methods used in health economics. Two hypothetical scenarios were developed with represented lay abstracts of projects submitted for funding to the RfPB scheme. The scenarios were designed to be in scope for RfPB and both cost £250,000. The scenarios were different in terms of topic and research design.  Scenario A outlined a medical record review with the aim of identifying treatments and referrals for depression in patients with osteoarthritis, while Scenario B described a pilot and feasibility trial to develop and test methods for improving adherence to self-monitoring in patients with type 2 diabetes who need to monitor their blood glucose. Our hypothesis was that there was potential for a greater amount of PPI in Scenario B than Scenario A, therefore funders would recognise a need for more funding of PPI in Scenario B than in A. This hypothesis was based on the principle of sensitivity to scope, used in health economics, where the “theoretical consistency” of willingness to pay (WTP) responses is tested, and “WTP should vary appropriately as the size of the good under consideration changes” (Bateman & Brouwer, 2006). We chose not to include any information on planned PPI activities within the two scenarios, as we wished to leave respondents free to consider the level and types of PPI that would be appropriate in each.

Cognitive interviews (Campanelli, 1997) were conducted with a small sample of participants as they completed the survey to explore in depth committee members’ views of funding PPI in health research and views about the application of willingness to pay concepts as presented in the survey.

Sampling and recruitment

Documentary analysis

A request was sent to the SPCR for electronic copies of project proposals, annual and final reports for all projects funded since the inception of the School in 2008. Posters showcasing PPI activities within School-funded projects were collected at the 2014 School Showcase Event and other documents containing data on PPI were obtained from researchers who participated in the survey and reported PPI within their project(s). Publications linked to projects which included PPI activity were also identified through PubMed and searched for information on PPI.  

Researcher and patient surveys

All Principal Investigators of projects funded by the SPCR since its inception in 2008 received an electronic survey via email for each project that they were leading or had led. Project and contact details were taken from a database provided by the SPCR. As the details of PPI representatives involved in SPCR projects were not available, an item was included in the researcher survey to determine Principal Investigators' willingness to pass on the postal survey to patients involved in their projects. PPI representatives involved with the project were consulted about this recruitment strategy and felt that it was acceptable.   

Willingness to pay-type study

Four NIHR Research for Patient Benefit (RfPB) funding panels were selected at random, using a random number generator, from a total of eight regional committees (allRfPB committees excluding West Midlands). All members of these four committees were contacted by email and invited to participate in an online survey about funding for patient and public involvement. Details of membership are available on theNIHR Research for Patient Benefit regional funding committee webpages, and email addresses for clinician and researcher members were obtained online. Contact details were not available online for all lay members, so regional programme managers were asked to pass on the study invitation and documents via email to those lay members who the research team were unable to contact directly.

A total of 80 RfPB committee members were invited to participate, of whom 65 were clinicians or researchers, and 15 were lay members. All members of the West Midlands committee, comprising 14 clinicians or researchers and 3 lay members, were contacted via email and invited to complete the survey in the context of a 'think aloud' interview with a researcher. Again, the regional programme manager passed on the study invitation and documents via email to lay members whose contact details were not available online. Members of RfPB funding panels were selected for thestudy because of their experience in making funding decisions for health-related research projects which include PPI.   

Analysis

Documentary analysis

A data extraction form was developed to record data from project proposals and reports on the following variables for each project, where available: study design, disease/condition, study sample, funding round, whether there was a section dedicated to PPI within the document, whether there was PPI in the development of the project proposal, who was involved if there was PPI in the development of the proposal, whether further PPI was planned, who would be involved if further PPI was planned, whether there was any explanation for a lack of PPI, details of PPI activities planned or conducted, whether reference was made to a budget for PPI, and whether reference was made to rewards and recognition for involvement. Consistency between planned and reported PPI activities was noted, and details of any discrepancies were recorded. Supplementary data were added from four posters exhibited at the annual SPCR Showcase Event, which presented PPI activities within SPCR funded projects. Data were also taken from a small number of PPI-related documents gathered through the researcher survey. Summary descriptive statistics were calculated for all variables and analyses were conducted to explore whether there was a change in PPI activity over time, using project funding round as a proxy measure for time. Sub-group analyses were conducted to examine frequency of PPI and type of PPI activity by study design and disease/condition. Qualitative data on PPI was also extracted from publications linked to projects for which PPI activity was reported.     

Researcher and patient surveys

Descriptive statistics were calculated for all quantitative items in the researcher survey. To address the primary research question of how PPI activity in the SPCR compares with quality indicators of best PPI practice (Boote et al. 2006), the analysis focused on how many projects met at least one item for each quality indicator. For the secondary research questions, descriptive analyses were conducted on types of PPI undertaken in primary care research, costs of PPI, and impacts of PPI on the research process, researcher, and institution. A quality indicator index score was also calculated for each project, based on the number of quality indicators met, and the correlation between this measure and perceived impact of PPI was computed. Responses to all survey items relating to time spent on PPI activities and associated costs and related free text comments were independently categorised by two researchers into costs and consequences of PPI. An overall framework of all potential monetary and non-monetary costs and consequences (benefits) of PPI was constructed and presented as a table of disaggregated costs and consequences for both the research/researcher and patient.

We were not able to conduct analysis on any patient survey data because researchers who reported PPI within their projects were either unwilling or unable to pass the patient survey to those involved.

Willingness to pay-type study

Survey responses from the main survey were combined with those from the cognitive interviews for analysis. Summary descriptive statistics (means and ranges) were calculated for the demographic items and for the two main quantitative outcomes (allocation of funding to PPI for two hypothetical research proposals). Descriptive statistics were compared across the two main outcomes to test the hypothesis that more funding would be allocated to PPI for the second proposal. Qualitative thematic analysis (Aronson, 1994) was conducted on the free text responses within the survey to identify key themes and issues considered by respondents in completing the survey. Audio recordings from the cognitive interviews were transcribed verbatim and analysed thematically. Two researchers independently coded the first two transcripts on a line by line basis and then met to discuss the coding and resolve any differences. The final agreed codes were then developed into a thematic framework, which was subsequently applied in the analysis of the remaining transcripts. This systematic approach to organising qualitative data, known as framework analysis (Ritchie and Spencer, 1994), facilitates the mapping and comparison of themes across different interviews.     

Key findings and Results

Documentary analysis

A total of 200 full project proposals, 233 annual reports and 39 final reports were provided by theSPCR for the documentary analysis. The annual and final reports provided data on 180 projects, and information onPPI was taken from a poster relating to a further project for which reports were not available to the research team. Of the 200 projects for which full proposals were available, there had beenPPI in the development of the proposal for 47 (23.5%) projects. Those involved in the development of proposals were largely patients (40 projects), with limited involvement of health or social care professionals (six projects), the general public (four projects), charities and voluntary organisations (three projects), and advocates (two projects). There was insufficient information available to determine who had been involved in the development of the proposal for one project.PPI activities were planned for 113 (56.5%) of the projects, and of the 87 (43.5%) projects for whichPPI was not planned, an explanation for the absence of involvement was given in only 16 proposals. Plans forPPI predominantly involved patients (92 project proposals), while plans to involve the general public (15 project proposals), health and social care professionals (14 project proposals), charities and voluntary organisations (11 project proposals), carers (eight project proposals) and advocates (five project proposals) were less common. In six project proposals it was not clear who would be involved. Of the project proposals that included plans for PPI, 57 (50.44%) made reference to a budget for PPI and 32 (28.32%) referred to rewards and/or recognition for those who would be involved. However, the research team did not have access to the full costings for each project and it is therefore possible that this represents an underestimation of the number of projects for which PPI was budgeted.

A range of planned PPI activities were outlined in the project proposals, relating to different stages of the research cycle. Plans for involvement in managing research through membership of a project steering committee or management group were most common (51 proposals), followed by plans for involvement in dissemination of project findings (41 proposals). A number of proposals described plans for PPI in designing methods (36 proposals), analysis or commenting on results (36 proposals) and designing study materials (33 proposals). Less common were plans for involvement in writing information leaflets (15 proposals), commenting on the study protocol (nine proposals), developing future research (seven proposals), conducting the research (two proposals) and recruitment of participants (no proposals). For 15 project proposals, the nature of the planned PPI activities could not be determined from the information provided.      

Although there was wide variability in the numbers of projects employing particular types of study design, there were trends in the data for more involvement in the development of proposals with cross-sectional designs (57.14% of seven projects), mixed methods (31.91% of 47 projects), qualitative (25% of 36 projects) and intervention/trial (24% of 25 projects) designs. Involvement in the development of project proposals was absent for projects employing secondary analysis (0% of eight projects) and case control (0% of four projects) designs, and uncommon for systematic reviews/meta-analyses (11.76% of 17 projects). In terms of condition type, involvement in the development of the proposal was more common for projects focusing on cancer (43.75% of 16 projects), renal and urogenital (40% of five projects), reproductive health and childbirth (40% of five projects) and issues of generic health relevance (31.03% of 29 projects). Involvement in the development of the proposal was less frequently reported for projects that focused on cardiovascular (14.81% of 27 projects), stroke (14.29 of seven projects), metabolic and endocrine (7.14% of 14 projects) and neurological (0% of three projects) conditions and multimorbidity (0% of six projects). There was no clear linear trend for an increase in involvement in the development of project proposals over time, using funding round (FR) as a proxy measure for time (FR1 = 11.11%; FR2 = 30.43%; FR3 = 16.13%; FR4 = 22.22%; FR5 = 16.67%; FR6 = 62.5%; FR7 = 35.29%; FR8 = 36.84%).

Plans for PPI within project proposals were more common for methodological (80% of five projects), qualitative (63.89% of 36 projects), mixed methods (63.83% of 47 projects), longitudinal/prospective cohort (62.07% of 29 projects) and intervention/trial (60% of 25 projects) designs, while proposals for case control (25% of four projects), retrospective cohort (30.77% of 13 projects) and individual participant meta-analysis (33.33% of three projects) included plans for PPI less frequently. PPI was also more frequently planned for studies in the fields of neurology (100% of three projects), renal and urogenital (100% of five projects), reproductive health and childbirth (100% of five projects) and an ‘other’ category for projects outside the specific health categories listed in the Health Research Classification System (77% of 31 projects). Plans for involvement were uncommon in proposals focusing on multimorbidity (14.29% of seven projects). A higher percentage of proposals from each of the last three SPCR funding rounds included plans for PPI than from the first three funding rounds, but there was no clear linear increase in planned PPI over time (FR1 = 16.67%; FR2 = 52.17%; FR3 = 45.16%; FR4 = 72.22%; FR5 = 56.25%; FR6 = 100%; FR7 = 70.59%; FR8 = 78.95%). There were 18 further projects for which details of funding round were not available.

In terms of the 180 projects for which annual and/or final project reports were available, 69 (38.33%) projects had been completed, 108 (60%) were uncompleted and this data was missing for three projects. For the projects for which data were available (including the single project for which data were taken from a PPI poster), PPI had taken place in 84 (46.41%), had not taken place in 74 (40.88%), and for 23 projects (12.71%) there was insufficient data available to determine whether PPI had taken place. Where PPI had not been included in the project, a rationale for the absence of PPI was provided for 26 projects. In the case of uncompleted projects, PPI activities were planned for 39 (36.11%) projects, there were no plans for PPI in 50 (46.30%) projects, and there was insufficient information available to determine whether PPI was planned for the remaining projects. Plans for PPI in uncompleted projects mainly involved patients (29 projects), with less planned involvement of charities and voluntary organisations (six projects), the general public (three projects) and carers (two projects). There was insufficient information to identify who would be involved for two projects. Where there were no plans for PPI, a rationale for this decision was provided for nine projects. Examples of these rationales were that the project focused on methodological work or was based on secondary analysis of existing data. There were no references to a budget for PPI in any of the annual/final reports or PPI posters obtained from the SPCR, and only one reference to rewards and recognition for those involved.

The most common types of PPI activity described as having taken place in annual and final project reports were involvement in designing study materials (33 projects), designing methods (25 projects) and managing research through sitting on a project steering committee or management group (17 projects). Involvement in commenting on the protocol (12 projects), developing the research idea (10 projects), dissemination (nine projects), the grant application (eight projects), writing information leaflets (eight projects), analysis or commenting on results (eight projects), developing future research (five projects), conducting the research (one project) and recruitment (one project) were reported less frequently. There were 14 projects for which the exact nature of thePPI activities could not be determined from the information provided in reports. For uncompleted projects (N = 108), plans for involvement mainly related to managing the research (nine projects), analysis or commenting on results (eight projects) and dissemination (five projects). Plans for PPI in designing study materials (three projects) and developing future research (two projects) were less commonly reported and there were no plans for involvement in designing study methods, writing information leaflets, recruitment of participants, conducting the research or commenting on the protocol. There was insufficient information in reports for 14 projects to determine the type of PPI activities planned.    

PPI activities or a lack of PPI reported in the annual/final reports and posters were compared with plans for PPI outlined in project proposals where the proposal and at least one report were available for the same project. PPI, or the absence of PPI, was consistent with plans set out in the project proposals for 65 projects, inconsistent for 70 projects and there were insufficient data to make this judgement for 44 projects. Where reported PPI activity was consistent with that described in the project proposal, for 27 projects this was because there had not been any involvement.

PPI had taken place in a higher percentage of mixed methods (61.54% of 39 projects), intervention/trial (59.09% of 30 projects), cross-sectional (57.14% of seven projects) and qualitative (56.67% of 30 projects) projects relative to other types of study design. PPI activity was comparatively low in methodological (25% of four projects), retrospective cohort (23.08% of 13 projects) and systematic review/meta-analytic (11.76% of 17 projects) designs. However, it is important to note that there was considerable variability in the numbers of projects employing particular types of study design. In terms of the condition of study, PPI had taken place in a higher percentage of projects on cancer (62.5% of 16 projects), reproductive health and childbirth (80% of five projects), mental health (64.71% of 17 projects), and renal and urogenital (60% of five projects) in relation to other conditions. Involvement was less frequent in projects on multimorbidity (37.5% of eight projects), cardiovascular conditions (35.71% of 28 projects), an ‘other’ category for conditions outside the specific health categories listed in the Health Research Classification System (34.78% of 23 projects) and infection (20% of five projects). There was no clear linear trend for an increase in involvement over the successive funding rounds, but PPI was more frequently reported in funding rounds 6 and 7 (FR1 = 18.18%; FR2 = 57.89%; FR3 = 46.88%; FR4 = 52.63%; FR5 = 40%; FR6 = 75%; FR7 = 76.47%; FR8 = 33.33%). There were six further projects for which details of funding round were not available.     

Researcher and patient surveys

A total of 191 surveys were emailed to Principal Investigators and 46 responses were received (response rate 24%). The Principal Investigators who responded to the survey were either unwilling or unable to pass on the patient survey to PPI representatives involved in their projects, so we did not collect any patient data. However, to ensure that the patient perspective was represented in project outputs, PPI representatives involved with the project were asked to attend a data analysis workshop and work with the research team in the co-production of recommendations for improving PPI practice from the survey findings.  

Of the 46 responses received from Principal Investigators, 15 reported PPI activity, most commonly in designing methods (8/15) and developing participant information (7/15). It was less common for patients and members of the public to be involved in suggesting the topic(s) to research for the project (2/15), conducting the research (2/15) and helping with dissemination (1/15). There was variation in how best practice, according to the quality indicators reported by Boote and colleagues (2006) was met across studies. Best practice was more frequently achieved in terms of having a specific budget for PPI (9/15 studies); offering PPI members personal and technical support (13/15); and involving PPI members in advising on recruitment issues (11/15). Fewer studies met best practice for PPI in terms of PPI members advising on informing participants about study progress (1/15); advising on dissemination methods (1/15) or having to access to training (3/15).

Perceived impact was most commonly reported for study processes with most PPI activity (i.e. designing methods and developing participant information). Perceived impact of PPI on the research process and individual Principal Investigators was largely positive and included benefits such as clarity of information, increased recruitment and follow-up rates, validation of findings and more useful outputs for clinicians and patients. The only negative impact reported was the view that a more homogenous study sample may have been recruited, since PPI members encouraged their friends to participate. Despite reported PPI activity in developing the grant application (3/15), managing the research (3/15), and conducting the research (2/15), Principal Investigators reported minimal perceived impact in these areas. There was a moderate positive correlation between quality indicator index score and perceived impact score (r = .50, p = .056), such that greater perceived impact of PPI activity tended to be reported for projects where a higher number of quality indicators for PPI were met.

The most significant cost from a researcher perspective appeared to be the researcher’s time, however reported researcher hours related to PPI was variable, and ranged from 0 to 30 hours as a total across all activities. Few studies were able to confirm that expenses were reimbursed to PPI representatives. A third of the respondents reported difficulty in providing information about PPI in their project(s) (5/15) and almost half (7/15) found it difficult to give information relating to the costs of PPI. Free text responses indicated that the researchers did not keep records of the costs associated with PPI activity in their projects. Responses from the researcher survey provided information for the costs and consequences framework under sub-headings of researcher, research project, research institution and funder. As no responses were received for the patient survey, only costs and consequences identified from the literature could be included in the framework.  

Willingness to pay-type study

29/80 (36%) RfPB committee members responded to the survey, with a further 4/17 (24%) of the West Midlands committee completing the survey via a “think-aloud” interview, giving an overall response rate of 34%. 19 were academic researchers and/or clinicians with 10 lay members and 4 not indicating their background. 7/33 did not give a value for scenario 1, with 9/33 not responding to scenario 2. Of those who did respond, the mean value for scenario 1 was £6,913 (range £0-£25,000) compared with £14,091 (range (£2,500- £50,000) for scenario 2, demonstrating sensitivity to scope. 15/33 respondents gave a higher value for scenario 2, with 7/33 giving the same value for both scenarios and only one person valuing PPI higher for scenario 1. Qualitative analysis of the free text responses suggested that most respondents found the task difficult, but that most responded as intended by the researchers (i.e. based upon an independently formed judgement of the likely role for and extent of patient and public involvement).  Whilst some respondents adopted a detailed ‘micro’ approach to costing specific elements of patient and public involvement, others drew on broad rules of thumb (assigning a particular percentage of the overall budget to PPI) and others still reported more of a “rough estimate” based upon the nature of the project outlined.  Free text responses indicated that the two zero responses were real zero responses and not protest votes. There was evidence that respondents had carefully considered the detail in the abstracts; some respondents suggested that more detail would have been useful, particularly in the case of the first abstract. 

Comparisons were drawn between the values given by researchers/clinicians and those given by lay members. Five responses from researchers/clinicians were excluded because these were presented as a range rather than a single value. Despite two researchers/clinicians giving zero responses, lay members gave a lower value on average for scenario A (mean = £6,083, median = £4,250, range = £2,500 - £12,500) than did researchers/clinicians (mean = £7,656.25, median = £5,500, range = £0 - £25,000), but lay members gave a higher value on average to scenario B (mean = £16,714.29,  median = £10,000, range = £2,500 - £50,000) than researchers/clinicians (mean = £12,866.67, median = £10,000, range = £5,000 - £30,000) and as a group showed greater sensitivity to scope.

The thematic analysis of data from the first two cognitive interviews was used to produce a thematic framework, which was then applied in the analysis of the remaining interview data. Key overarching themes included comprehension, validity and relevance, judgement and decision-making, views of PPI, views on how to value PPI, and views on the costing of PPI. Findings on comprehension of the scenarios were mixed; the academic respondent felt that these were generally well written and provided good descriptions of the proposed research, while the lay respondents commented that the scenarios were vague, included research ‘jargon’ and were difficult to understand. Respondents’ views on the validity and relevance of the scenarios were more similar; there was consensus that the second scenario was more typical of an RfPB proposal and that the overall costs were more justifiable within this scenario. The respondents felt that scenario A was shorter and vaguer than those that they were used to reviewing. The lay members also expressed some concern over the relevance of the instructions with regard to their usual method of assessing proposals, for instance in being asked to make a decision on funding for PPI without information on planned PPI activities.

Only two of the four respondents (one academic and one lay respondent) provided a value for the funding that they would allocate to PPI in each scenario, as two lay respondents felt that it was not possible to make this judgement on the basis of the information provided. One of the respondents who provided a value commented that it was ‘random’ because it was difficult to make a judgement, while the other felt that the judgement was ‘easy’ because of their previous experience. In considering allocation of funding to PPI, past experience of PPI, past experience of funding for PPI, and presumed types of PPI activity either informed responses or were mentioned as relevant factors. Three respondents (one academic and two lay respondents) demonstrated the principle of sensitivity to scope in perceiving more of a role for PPI in the second scenario than in the first: 

“Erm, well it’s [Scenario B] a – it’s – it tells more about – a lot more; it’s a lot more structured, so I think it’s much easier to understand and immediately see where PPI might be involved. (TAO1).

“Yeah, yeah. I would assume there should be higher PPI costs in the second one than the first one, if the first one was just a notes review”. (TA03).

Three respondents explicitly stated that they saw greater opportunity for PPI within the second scenario, for instance in evidence synthesis, designing a topic guide, interpretation of findings and dissemination. In thinking aloud about the amount of funding to allocate to PPI within the two projects, respondents mentioned considerations such as the frequency of PPI activities, the size of the PPI group, expenses, the need for training of PPI representatives and the patient-centeredness of the project. Three respondents felt that more information on the planned PPI activity was required in making a judgement about costs:

 “Erm, and, because I’ve no idea what the PPI plan is in the first place. So I don’t know whether they’ve got a good one, or a bad one. So, I, I don’t feel competent to answer the question, really. So, I’m going to put, ‘Don’t know’”. (TA03),

Two of the lay respondents also felt information on involvement in the development of the project proposals was needed before making decisions on the allocation of funding to further PPI activity.

Broader issues were raised by respondents during the course of the interviews, such as the professionalisation of PPI representatives and the value of PPI contribution in terms of setting the research agenda, asking questions about the allocation of public funding, and ensuring that research is grounded in the real world. Differing views emerged on the most appropriate approach to take in valuing PPI; while one respondent indicated that PPI should be valued for what it costs rather than its importance, two lay respondents expressed the opposing view that PPI should be valued for its contribution and not its financial cost. Two lay respondents also felt that it was not appropriate to view PPI as a separate cost in isolation but rather as integral to the project:

“...that's not the way to look at it you see I think effective PPI is integrated into any trial um or piece of research particularly where it's looking for RFPB or RFPB money.  It's not an add on, it's not a fixed proportion of cost [okay] it's fundamental to how the thing is being thought about and how it's going to be implemented…” (TA02).

Development of recommendations for PPI practice in the SPCR

Findings from the documentary analysis, researcher survey and willingness to pay-type study were collated and discussed with members of the Research User Group at Keele, to ensure that the patient perspective was considered. On the basis of the key findings, a set of 23 draft recommendations was co-produced with members of the Research User Group. Nineteen of these recommendations related to PPI practice within the SPCR, while four were system and process recommendations for PPI. The draft recommendations were refined following feedback from the wider research team and further input from members of the Research User Group, which resulted in a final set of 15 recommendations (please see Appendix 2). The recommendations were also mapped against the INVOLVE (2013) values, principles and standards for public involvement in research, the NIHR INVOLVE (2015) “Breaking Boundaries: Going the Extra Mile” strategic review of public involvement and the SPCR strategic aims for PPI to ensure that they encompassed these important standards and goals for PPI practice.

Expected impact

As stated above, the results of this project have been used to develop recommendations for good PPI practice within the SPCR at both system and practice levels (please see Appendix 2). The recommendations have been developed in collaboration with members of the Research User Group at Keele to ensure that they represent the patient perspective. Implementation of these recommendations will ensure that PPI activities in the School meet quality indicators and that records of PPI activities are kept to facilitate the evaluation of impact. A SPCR wide PPI workshop is being held on October 28^th 2015 to disseminate the findings of the project, discuss the recommendations and future directions for PPI research and PPI in the SPCR. 

Conclusions

Although there is some PPI activity within research projects funded by the SPCR, many of the projects do not include PPI in the development or the execution of the research. Further, findings from the researcher survey indicate that where PPI has taken place, it often fails to meet indicators of best practice. For example, in a number of projects PPI representatives were not offered payment for their time or reimbursement of expenses. Researchers reported difficulty in providing information on PPI and its costs, which suggests that records of PPI activity are not routinely kept. The project has also highlighted the difficulty of accessing patients involved in SPCR projects, as researchers were either unwilling or unable to pass the patient survey to those who had been involved in their research. The finding that reported PPI activities were often inconsistent with those described in research proposals suggests a need to monitor PPI activity within SPCR projects and to provide guidance for researchers in developing realistic plans for PPI. Results of the willingness to pay-type study suggested that respondents placed value on PPI, although allocation of funding served only as a proxy measure of value and was more oriented to the concept of ‘cost’. The finding that most respondents demonstrated sensitivity to scope in allocating higher values to PPI for the second than the first scenario indicated that they considered the appropriateness and merit of PPI within the hypothetical projects. However, findings from the cognitive interviews indicated some limitations of the methodology, such as problems with the clarity and validity of the scenarios and the need for more information in order to make a judgement about funding for PPI. Respondents also indicated that the methodology used may not be the most appropriate way to determine how people value PPI because it is difficult to quantify. Further, isolating the value of PPI from other aspects of the projects was perceived as artificial when PPI should be integral to the research. Alternative methods should be explored in future studies. The set of recommendations for improving PPI systems and practice within the SPCR will be discussed at a forthcoming meeting of the SPCR working party for PPI.  

References

Aronson, J. (1994). A pragmatic view of thematic analysis. The Qualitative Report, 2, 1.

Barber, R., Beresford, P., Boote, J., Cooper, C., & Faulkner, A. (2011). Evaluating the impact of service user involvement on research: a prospective case study. International Journal of Consumer Studies, 35, 609-615.

Bateman, I. J. & Brouwer, R. (2006). Consistency and construction in stated WTP for health risk reductions: A novel scope-sensitivity test. Resource & Energy Economics, 28, 199-214.

Boote, J., Barber, R., & Cooper, C. (2006). Principles and indicators of successful consumer involvement in NHS research: Results of a Delphi study and subgroup analysis. Health Policy, 75, 280-297.

Boote, J.,Telford, R., & Cooper, C. (2002). Consumer involvement in health research: a review and research agenda. 

Health Policy, 61, 213-236. 

Brett, J., Staniszewska, S., Mockford, C., Seers, K., Herron-Marx, S., & Bayliss, H. (2010). The PIRICOM Study: A systematic review of the conceptualisation, measurement, impact and outcomes of patient and public involvement in health and social care research. London: United Kingdom Clinical Research Collaboration.

Campanelli, P. (1997). Testing survey questions: New directions in cognitive interviewing. Bulletin de Methodologie Sociologique, 55, 5-17.    

Hanley, B., Bradburn, J., Barnes, M., Evans, C., Goodare, H., Kelson, M., …Wallcraft, J. (2004). Involving the public in NHS public health, and social care research: briefing notes for researchers. Eastleigh: INVOLVE.

INVOLVE (2010). Invonet Bibliography 3. References on public involvement in NHS, public health and social care research. London.

INVOLVE (2012). Briefing notes for researchers: public involvement in NHS, public health and social care research. Eastleigh: INVOLVE. Retrieved from http://www.invo.org.uk/wp-content/uploads/2014/11/9938_INVOLVE_Briefing_Notes_WEB.pdf

INVOLVE (2013). Values, principles and standards for public involvement in research. Eastleigh: INVOLVE. Retrieved from http://www.invo.org.uk/about-involve/current-work/standards-for-public-involvement-in-research/

McLaughlin, H. (2009). Service user research in health and social care. London: SAGE.

NIHR (2014). NIHR Annual Report 2013/14. Retrieved from http://www.nihr.ac.uk/documents/about-NIHR/NIHR-Publications/NIHR-Annual-Reports/NIHR%20Annual%20Report%202013_2014.pdf

NIHR (2015). Going the extra mile: Improving the nation’s health and well-being through public involvement in research. Retrieved from http://www.nihr.ac.uk/get-involved/Extra%20Mile2.pdf

Ritchie, J., & Spencer, L. (1994). Qualitative data analysis for applied policy research. In A. Bryman & R.G. Burgess (Eds.). Analysing qualitative data (pp. 173-194). London: Routledge.  

Staley K. (2009). Exploring impact: Public involvement in NHS, public health and social care research. Eastleigh: INVOLVE.  

Tarpey, M. (2006). Why people get involved in health and social care research: a working paper. Eastleigh: INVOLVE.  

Wright, D., Foster, C., Amir, Z., Elliott, J., & Wilson, R. (2010). Critical appraisal guidelines for assessing the quality and impact of user involvement in research. Health Expectations, 13, 359-368. 

Plain English Summary

Background

Over recent years, there has been increasing emphasis on involving patients and members of the public in social and health research. The National Institute for Health Research, for example, suggests that patients and the public should be involved in all aspects of research, from identifying research priorities to publicising the results of studies. There are a variety of arguments supporting the importance of involving patients and the public in research, including the wealth of knowledge and experience that they can bring to research, the potential for patient and public input to improve the quality and relevance of research, and the moral obligation to involve patients and the public in publicly-funded research which may impact upon health services. However, more information is needed about the costs of PPI in health research and its impact on researchers, PPI representatives, institutions, funders and research. 

Objectives

The objectives of this project were to identify patient and public involvement (PPI) activities taking place in research within the NIHR School for Primary Care Research (NIHR SPCR), which comprises nine leading University departments for primary care research in England, to explore the costs (financial and non-financial) and consequences of these PPI activities, and to develop recommendations to improve PPI practice within the NIHR SPCR. A further objective of the study was to explore how best to find out about how people value PPI through looking at their willingness to pay for it, and how decisions are made regarding the allocation of funding for PPI activities in health research.

Methods

An electronic survey (questionnaire) was sent to all lead researchers of research projects funded by the SPCR to gather information on PPI activities taking place within the NIHR SPCR and the costs and consequences of those activities. Information on projects and the lead researchers was taken from a project database provided by the SPCR. A paper-based postal survey for PPI representatives involved in SPCR-funded projects was also developed to gather information on the costs and consequences of involvement from their perspective. Further information on PPI activities was drawn from project paperwork, such as project proposals, project reports and minutes of PPI meetings. These were mainly obtained in electronic form from the SPCR but several documents were provided by researchers who completed the survey. A paper-based postal survey for PPI representatives involved in SPCR-funded projects was also developed to gather information on the costs and consequences of involvement from their perspective.

An online survey was sent to all members of four NIHR Research for Patient Benefit (RfPB) regional funding committees, who are involved in making decisions about funding for health-related research. These committees were selected at random from eight of the nine RfPB committees and details of membership were available on the RfPB website. The survey contained two brief hypothetical research proposals written by the research team and asked people to state the maximum amount of the £250,000 budget that they would give to PPI in each project, followed by comments on how they had arrived at those values. We predicted that people would see more opportunities for PPI activities in the second than the first project and so would allocate a higher amount for PPI on average for the second project.  We chose not to include any information on planned PPI activities within the two scenarios, as we wished to leave people free to think about the level and types of PPI that would be appropriate in each.

In addition to the online survey, all members of another RfPB Advisory Committee were invited to complete the survey in the presence of a researcher and to voice their thoughts aloud as they completed the survey, to give the researchers some insight into people's thought processes when responding to the questions. It was hoped that this would help the researchers to determine whether the survey was an appropriate way to gather information on how people value PPI in research.

PPI representatives from the Research User Group at Keele were involved in every stage of this project, from developing the idea in the first place to helping the researchers understand the findings and writing recommendations for improving PPI in the SPCR. PPI representatives will also help to tell others about the findings of the project, for example by sharing these at local Arthritis and Musculoskeletal Alliance (ARMA) patients’ group and Public Involvement and Lay Accountability (PILAR) meetings, and with colleagues in EULAR and INVOLVE

Conclusions

Some research projects funded by the SPCR have included PPI activity, but many projects do not include PPI in the planning or conducting of the research. Findings from the researcher survey indicate that where PPI has taken place, it does not always meet guidelines for best practice. For example, in a number of projects PPI representatives were not offered payment for their time or reimbursement of expenses. Researchers found it difficult to provide information on PPI and its costs within the survey, which suggests that they do not routinely keep records of PPI activity. The project has also shown that it is difficult to contact patients who have been involved in SPCR projects, as researchers were not able or willing to pass the patient survey to those involved in their research. The finding that reported PPI activities were often different to those described in research proposals - and sometimes planned PPI was not conducted at all - suggests a need to keep a check on PPI activity within SPCR projects and to help researchers to make realistic plans for PPI at the outset. Results of the willingness to pay-type study suggested that respondents did place value on PPI, although found it difficult to decide on how much funding to allocate without information on planned PPI activities. However, people did think about the type of PPI activity that would be appropriate for each scenario and drew on past experience to help them make a decision. Findings from the ‘think aloud’ study suggest that the survey could be improved to ensure that it is easier to understand. People also suggested that the survey may not have been the best way to find out how people value PPI because it is difficult to put a figure on this. The set of recommendations for improving PPI within the SPCR will be discussed at a meeting of the SPCR working party for PPI.

Patient and Public Involvement 

PPI has been embedded within each stage of this project. Four patients were involved in a workshop in February 2011 to develop ideas for a grant proposal and were instrumental in forming the idea for this project. Patients at the workshop talked about the need to highlight the hidden costs of involvement, and how things that are taken for granted by researchers (e.g. train travel, talking in a meeting) often have costs and consequences for the patients. A PPI coordinator who attended the workshop stressed the importance of an economic study and felt that this gap in existing knowledge needed to be addressed. The four patients involved in the workshop reviewed the proposal before submission to the SPCR for funding and advised the lead applicant on the lay summary. Two patients also became co-applicants on the grant. Subsequently, patients were involved in the development of the study materials. A meeting was held with members of the Research User Group at Keele in March 2013, in which patients advised on the types of costs and consequences they had experienced through their own involvement, potential recruitment strategies, the consent process for the observation study, the content of the observation guide for PPI meetings, and the content and language of study documents. Three patients also provided a detailed assessment of the content of the patient survey. Following preliminary data analysis by the research team, patients who had been previously involved with the project attended a data analysis workshop in which they looked at the key findings of the documentary analysis and researcher survey with the aim of developing recommendations to improve PPI practice within the SPCR. The workshop was a success and resulted in the co-production of recommendations which are attached to this report. There has also been involvement in the dissemination of project findings, as a patient collaborator co-presented with a researcher at the INVOLVE Conference 2014. Three patient co-applicants were invited to comment on the lay summary for this report. Further involvement in dissemination is planned, as patients will be asked to assist with the writing of further lay summaries of the findings.  Patients have therefore played a fundamental role in shaping the project, from the initial development of the research idea to the dissemination and implementation of findings.This project was funded by the National Institute for Health Research School for Primary Care Research (project number 128)

Impact on policy and practice

The results of this project have provided a detailed account of PPI within the SPCR and have shown the variability of PPI activities in projects to date. In particular, findings have highlighted areas for improvement in PPI within the School. This has led to the development of recommendations for good PPI practice, in collaboration with members of the Research User Group at Keele to ensure that the patient perspective is represented.  Implementation of these recommendations will ensure that PPI activities in the School meet quality indicators and that standardised records of PPI activities are kept to facilitate the evaluation of impact. A meeting/workshop is also planned with patients and other interested parties to disseminate the findings of the project and discuss future directions for PPI research. Further, the surveys developed within the project can be used by the School to collect detailed data on the costs and consequences of PPI in current and future projects.

Key Presentations and publications

Published articles 

Blackburn S, McLachlan S,  Jowett S, Kinghorn P, Gill P, Higginbottom A, Rhodes C, Stevenson F, Jinks C: The extent, quality and impact of patient and public involvement in primary care research: a mixed methods study https://researchinvolvement.biomedcentral.com/articles/10.1186/s40900-018-0100-8

Jinks C, Blackburn S, Chuter A, Gill P, Griffiths M, Higginbottom A, Jowett S, Kinghorn P, McLachlan S, Rhodes C, Stevenson F, Redwood S: Case Study 9: Understanding the cost and consequences of patient and public involvement in primary care research  National Institute for Health Research (2015). Patient and Public Involvement: Case Studies in Primary Care Research

Article Submissions

Blackburn S,  McLachlan S, Chuter A, Gill P, Higginbottom A, Jowett S, Kinghorn P, Rhodes C, Stevenson S, Jinks C (authors tbc): Quality and impact of patient and public involvement in primary care research: Results of a mixed-methods study Target journal: Health Expectations. Planned submission date: Dec 2015

Kinghorn P, Jowett S, Blackburn S, Chuter A, Gill P, Higginbottom A, McLachlan S, Rhodes C, Stevenson S, Jinks C (authors tbc): Can we understand the costs and consequences of patient and public involvement in research? An exploratory willingness to pay type study Target journal: Research Involvement and Engagement or BMC Health Services Research. Planned submission date: Dec  2015

Presentations and Posters

Blackburn S, McLachlan S, Jowett S, Kinghorn P, Gill P, Higginbottom A, Rhodes C, Stevenson F, Chuter A, Jinks C: Quality and Impact of Patient and Public Involvement in Primary Care Research: Results of a Survey of Researchers (Poster presentation) Society for Academic Primary Care Conference 2015, University of Oxford

McLachlan S, Chuter A, Blackburn S, Gill P, Griffiths M , Higginbottom A, Jowett S, Kinghorn P, Redwood S, Rhodes C, Stevenson F, Jinks C: Exploring the costs and consequences of patient and public involvement in research projects within the NIHR School for Primary Care Research (Oral presentation with patient collaborator) INVOLVE Conference 2014: Changing Landscapes, National Exhibition Centre, Birmingham, 26^th - 27^th November 2014

McLachlan S, Blackburn S, Chuter A, Gill P, Griffiths M, Higginbottom A,  Jowett S, Kinghorn P, Redwood S, Rhodes C, Stevenson F, Jinks C: Can we understand costs and consequences of patient and public involvement in Primary Care Research? (Poster presentation) NIHR School for Primary Care Research Showcase Event 2014, University of Oxford 

This project was funded by the National Institute for Health Research School for Primary Care Research (project number 119)

Department of Health Disclaimer

The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the NIHR School for Primary Care Research, NIHR, NHS or the Department of Health.