Project objectives

Aims:

1) To review current guidance on PRO assessment in trials.

2) To determine which PROs are currently assessed in trials within the NIHR HTA portfolio, in particular those based in a primary care setting.

3) To establish the current quality of trial protocols in relation to the PRO information they provide.

4) To identify guiding principles and examples of good practice in protocol design that would facilitate optimal reporting (as identified in the CONSORT PRO extension).

Changes to project objectives

Aims and Objectives

Since initial approval of the project the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) Statement has been published (2013).  The SPIRIT statement provides a guideline for the minimum content of a clinical trial protocol¹.  As a result in addition to assessing the PRO protocol content we decided to assess content against the SPRIT checklist.

Research Plan and Methodology:

To our knowledge this is the first time that protocols have been evaluated against the SPRIT checklist.

Methodology

Please see the previous section for protocol search strategy and study selection.

Two researchers examined each protocol independently to determine adherence to both the SPIRIT checklist and the PRO checklist developed as a result of the systematic review of existing guidelines for PRO Protocol design.

Brief summary

Background

Patient-reported outcomes (PROs), including health-related quality of life (HRQL), symptoms such as pain or fatigue, and health utility, are increasingly assessed in clinical trials as a measure of effectiveness.^1-3 PRO trial data may be used to inform clinical care and decision-making, predict long-term outcomes and influence health-policy; but to do so, as with any trial outcome, they must be collected with rigor. Unfortunately, evidence shows that the quality of PRO data can be undermined in some trials by inconsistencies in data collection⁴ and, in particular, by high rates of missing data⁵; this adversely affects the integrity and usefulness of such data in clinical practice.                    

To help ensure optimal PRO data collection, PRO-specific information should be clearly documented in the trial protocol.^6,7 The trial protocol is the cornerstone of a well-conducted trial, and should provide specific instruction on how to conduct all aspects of the study.⁸ The protocol also allows external funding bodies, regulators, research ethics committees, journal editors, health care providers, systematic reviewers and policy makers to evaluate the design and methods.⁶ Despite the importance of PROs, recent data suggests that some trial staff feel protocols provide little guidance regarding PRO-specific aspects of the trial, leading to ambiguity and the potential for significant inconsistency in the way PRO data are gathered, analysed, acted upon, and reported.^4,9,10

Aims

The recent publication of the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidance aims to promote the inclusion of important general methodological components in trial protocols⁶; however, it does not provide specific guidance related to PROs. It is currently unclear exactly what PRO-specific information should be included in trial protocols.

Aim 1: To address this, we aimed to conduct a systematic review to summarize current PRO-specific guidance for clinical trial protocol developers.

Qualitative evidence suggests that PRO information is frequently omitted from clinical trial protocols, leading to inconsistent PRO data collection and risking bias, however, direct evidence regarding the extent of PRO trial protocol content is lacking.

Systematic Review of guidance for trial protocol writers 

Methods

We searched the MEDLINE, EMBASE, CINHAL and Cochrane Library databases (inception to February 2013) for PRO-specific guidance regarding trial protocol development.  Further guidance documents were identified via Google, Google scholar, requests to members of the UK Clinical Research Collaboration registered clinical trials units and international experts in the field. Two independent investigators undertook title/abstract screening, full text review and data extraction, with a third involved in the event of disagreement.

Key Findings

21,175 citations were screened during the systematic review and 54 met the inclusion criteria. Guidance documents were difficult to access: electronic database searches identified just 8 documents, with the remaining 46 sourced elsewhere (5 from citation tracking, 27 from hand searching, 7 from the grey literature review and 7 from experts). 162 unique PRO-specific protocol recommendations were extracted from the included documents. A further 10 PRO recommendations were identified which related to other supporting trial documentation. Only 5/162 (3%) recommendations appeared in more than half of the guidance documents reviewed, indicating a lack of consistency. 

Conclusions

PRO-specific protocol guidelines were difficult to access and lacked consistency, therefore, they may be challenging to implement in practice. There is a need to develop easily accessible consensus-driven PRO protocol guidance. Guidance should be aimed at ensuring key PRO information is routinely included in appropriate trial protocols, in order to facilitate rigorous collection/reporting of PRO data, to effectively inform patient care.

 

Aim 2: Our objective was to systematically review randomised controlled trial (RCT) protocols including either a primary or secondary PRO outcome, to evaluate the completeness of their PRO-specific content.

Evaluation of the PRO content of clinical trial protocols

Methods

We undertook an electronic search of the UK National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme database (inception to August 2013).NIHRHTA protocols were included if they described a randomized controlled trial that included a primary or secondary PRO (judged by agreement between two independent investigators). Two investigators independently reviewed the content of each protocol, extracting data using a specially developed PRO-specific protocol checklist (developed following a systematic review of PRO guidance for trial protocol writers), alongside the ‘Standard Protocol Items: Recommendations for Interventional Trials’ (SPIRIT) Checklist (a measure of the general completeness of protocols). Disagreements were resolved through discussion with a third investigator.

Key findings

The 75 most recent NIHR HTA trial protocols including a PRO primary/secondary outcome were evaluated. Protocols included a mean of 63% SPIRIT items (n=32, range 4-18, SD 3.56) and 33% PRO checklist items (n=11, range 16-41, SD 5.62). Over half (61%) of the included PRO items were incomplete. Trials with a primary PRO endpoint generally included more PRO checklist items in their protocols, but standards were still poor (mean 43%). PRO protocol content was not associated with general protocol completeness; thus, protocols judged as relatively ‘complete’ using SPIRIT were still likely to have omitted a large proportion of PRO checklist items.

Conclusions

The PRO components of HTA clinical trial protocols require improvement. Information on the PRO rationale/hypothesis, data collection methods, training and management was often incomplete or missing completely. Study findings also suggest there are a number of PRO protocol checklist items that are not fully addressed by the current SPIRIT statement. We therefore advocate the development of consensus-based supplementary guidelines, aimed at improving the completeness and quality of PRO content in clinical trial protocols.

Plain English summary

Quality of life and other patient reported outcomes (PROs) are routinely assessed in clinical trials, providing “the patient voice” in evidence on treatment effectiveness. PROs are relevant to many primary care research questions and in such instances, protocols should include various sections that clarify and specify the detail of which PROs are included; why, how and when they will be measured; and how they will be analysed and interpreted. Both scientific and logistical issues should be addressed to ensure that the PRO data are of a high quality. Preliminary evidence suggests that critical deficiencies in the PRO components of trial protocols are common. The aim of this study is to assess the quality of the PRO components of trial protocols in relation to existing guidance. Ultimately this work will help improve the quality of the PRO information collected from trials so that robust results can help inform clinical care for patients.

Dissemination

Published articles

1. Kyte D, Duffy h, Fletcher B, Gheroghe A, Mercieca-Bebber R, King M, Draper H, Ives J, Brundage M, Blazeby J, Calvert M; Systematic Evaluation of the Patient-Reported Outcome (PRO) Content of Clinical Trial Protocols  PLOSone Published: October 15, 2014  https://doi.org/10.1371/journal.pone.0110229

Public involvement

Patients will be involved in the next stage of the project in developing consensus on best practice guidance and a checklist for protocol writers (in collaboration with the International Society for Quality of Life Research).

Impact

Implications for Practice

PRO-specific protocol guidance was difficult to access and lacked consistency and therefore may be challenging to implement in practice.

HTA clinical trial protocols require improvement, particularly with regard to their PRO components.  There is emerging evidence that these findings may be indicative of PRO protocol content generally.

As a result of the work undertaken we are now able to disseminate the following;

1. Increased knowledge on the current PRO evidence base for Protocol Writers
2. Critical Appraisal of the PRO content of trial protocols by utilising the PRO checklist developed as a result of this work.
3. Understanding of guiding principles and examples of good practice in protocol design that would facilitate optimal reporting (as identified in the CONSORT PRO extension).

Steps already taken

Funding application submitted for the MRC Methodology Research Programme (with patient advocates as co-Is)  to improve the patient reported outcome content of trial protocols (December 2013).  The aim of this work is to build on the work completed above by developing consensus driven PRO protocol guidance supported by online training resources, aimed at improving the completeness and quality of PRO content in clinical trial protocols and ensuring that PROs are collected in a robust way to inform patient care.  

 

This project was funded by the National Institute for Health Research School for Primary Care Research (project number 185 )

Department of Health Disclaimer

The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the NIHR School for Primary Care Research, NIHR, NHS or the Department of Health.