Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study a mixed methods study.

Intervention development undertaken in a pre-trial context can strengthen the relevance of a trial by optimising the intervention and/or its delivery from the perspective of recipients and providers. The rationale for this use of qualitative research fell into three categories:

1. The need to optimise an existing intervention that has some clinical or practice history in order to ensure that the ‘best’ intervention is tested in the trial. Multiple versions of an existing intervention may be in use with no identifiable ‘gold standard’ and the problem may be presented as needing to develop an optimised intervention.

2. The need to adapt an existing intervention for use in a different context:

   1. – A new clinical or practice context, for example a similar programme of care for a different patient group or a hospital service delivered in primary care.

   2. – A different geographical context for which the goal is to adapt a successful intervention developed in one country for use in another country where health services and health-care delivery differ (e.g. Gulbrandsen’s ‘Four Habits’25).

   3. – A different cultural context that may require adapting a mainstream intervention for use with a specific minority group or subgroup of a clinical population (e.g. Nagel’s Torres Strait Islanders28 or Marciel’s adolescents with cystic fibrosis27).

3. The need to develop a ‘de novo’ intervention, based on perceived clinical or practice need and, often, a theoretical understanding of the nature of the required intervention. 26,29

The driver for undertaking qualitative research in this context was not clinical appropriateness, but rather the need to explore the intervention as part of a matrix of health-related social behaviour on the part of patients, carers, health professionals or communities. That is to say, articles described participants in terms of their social and behavioural characteristics and explore experiences and beliefs from the perspective of a community of recipients or carers, a community of professionals delivering the intervention, or a particular context of delivery (primary care, community care, web-based care). Key to this was the way in which the research participants were defined and the issues particular to the group that were predicted to be relevant. Examples of this included women living with human immunodeficiency virus (HIV) who were also fulfilling the role of mothers,24 people with mental health problems who were also part of a minority indigenous group with specific cultural characteristics,28 or physicians who delivered care within the context of a particular national culture and its health-care system. 25

In order to achieve the research goal of identifying the best possible (or most appropriate) intervention for testing in a full trial, the need to attend to the health-related behaviours and beliefs of relevant groups (to be accessed via their experiences and perspectives) was the starting point and rationale for most of the qualitative research undertaken in this subcategory, whether the intervention was being designed de novo, being optimised or being adapted for use in a new context. In addition, some studies reported the importance of qualitative research in the development phase of a trial in terms of the opportunity to engender ‘stakeholder support’ for the trial process. 26–28 This was described as operating both at an individual level (patient and professional) and at a community level. 26–28

1. Be clear that changes were made to the intervention

   • In some cases, the qualitative findings resulted in a demonstrated change being made to the intervention content or delivery. In others, confirmation of acceptability was the real objective of the study. Good practice would entail making a clear distinction between these two aims within articles.

2. Ensure changes to the intervention are grounded in the data

   • When the aim of the qualitative research was to develop the intervention for a planned trial, it was important for researchers to demonstrate the mechanisms by which the qualitative data influenced the final intervention design. For this to happen it was necessary for the changes to be convincingly grounded in the data. This did not always occur, perhaps owing to limited article word count because in papers reporting the development of the intervention alongside the trial results, very little space was allocated to the qualitative component and its impact was usually reported rather than demonstrated.

3. Place within an evaluative framework

   • The MRC framework for the development and evaluation of complex interventions offers endorsement for the practice of undertaking qualitative research to support intervention development. 3 A conceptual framework for the qualitative research, such as the MRC framework, was not used in the majority of articles in this subcategory. One advantage of using such a framework may be to raise the profile and status of the qualitative research contributing to intervention development and to ensure that its contribution is documented. For example, referencing and using the MRC framework, the authors of one paper were able to claim that their goal was the development of ‘the definitive risk-factor management intervention’. 29

A lot of space may be needed within a report to describe in detail the components of a complex or multifaceted intervention delivered to recipients. This may not be possible within the structure of a conventional trial report, or valued enough to ensure that a limited word count includes intervention description. The MRC framework for complex interventions3 supports this process at an early stage of evaluation, but all 10 articles undertook this description alongside the main trial (see Table 5 ). Qualitative research undertaken alongside a trial can provide a rich account of elements of the intervention in detail, enhancing understanding of how the intervention plays out in the experimental delivery context of the trial. In the articles reviewed here, there were two main reasons for using qualitative research:

1. to describe and make available an unfamiliar or complex intervention using the experiences and perspectives of trial participants35

2. to increase understanding of individual components of a complex intervention and make them available for further exploration or testing. 30,32–34

Researchers chose to focus on a particular component of a complex intervention for a number of reasons. Macpherson and Thomas32 focused on an underexplored component, looking at acupuncture practitioners’ intention to deliver self-care advice alongside needling. This article raised a question regarding the integral nature of this aspect of the intervention as delivered and the implications of this for future trials of acupuncture care. Romo et al. 34 explored patient and family experiences of receiving prescription heroin in a hospital setting, and identified the perceived medicalisation of addiction (in contrast to criminalisation) as a component of the intervention in its own right. In contrast, Foster et al. 30 focused on an element of the intervention that was common to both arms of the trial (the use of medication companions) and highlighted the fact that this ‘taken-for-granted‘ component was integral to both home- and facility-based delivery of antiretroviral therapy and that its mechanisms of action would benefit from further research. Qualitative research was also used to explore components of complex interventions relating to what might be described as generic strategies, such as ‘shared-decision making’ in the context of advising about different cancer screening options within a consultation33 or the way health behaviour advice is ‘packaged’, rather than its specific content. 31

Accounts of complex interventions used in trials run the risk of reflecting the research protocol, rather than what was delivered in practice. Quantitative research methods may be employed to verify measurable elements of delivery, but qualitative methods, such as non-participant observation or in-depth interviews, can provide rich descriptive accounts of how the intervention was delivered on the ground during the trial, from the perspectives of recipients or providers. This descriptive function may form part of what is described as a ‘process evaluation’, although none of the articles we describe here used that term.

In each of our examples, the qualitative research appeared to provide an opportunity to focus attention on one or more components of a complex intervention, the significance of which might otherwise remain hidden or over-looked in a pragmatic trial. Although some of these articles discussed their findings in terms of possible mechanisms of action, the main purpose was to provide a rich and detailed account of different components of the intervention, based on the participants’ experiences. Using qualitative research data in this way is sometimes seen as ‘poor’ qualitative research because it does not extend itself into interpretive analysis or theory building. However, this tendency to undervalue the potential contribution of rich description has been recognised,36 especially in terms of its ability to ‘challenge taken-for-granted assumptions about the nature of the setting or group under study’. 37

1. Be clear about the origin of the research question

   • In our sample of articles, the genesis of the qualitative research question was sometimes unclear. The stronger articles gave a clear account of the reason for the focus on particular components and indicated whether this was an emergent issue or one that was identified as a research focus early in the trial.

2. Place within an evaluative framework

   • Describing, elucidating and highlighting hidden or overlooked components of a complex intervention using qualitative methods is a useful research output that has value for future intervention development, but many of the articles lacked a wider framework for presenting this type of analysis. Use of the MRC framework for the development and evaluation of complex interventions was rare and it may be that this framework could help provide this type of research with a useful rationale and contextual reference point.

3. Consider findings in relation to mechanisms of action

   • Descriptive accounts developed through qualitative data may also lend themselves to further development relating to an exploration of mechanisms of action or the explicit development of theoretical concepts underpinning interventions. Some of these articles moved towards this type of analysis, but this was not their aim.

4. Make explicit links to the main trial

   • Links to the main trial tended to be limited to the background and methods sections of articles, with little or no attempt to link this research back to the trial in the discussion or conclusion. This raised concerns about whether or not the research had any impact on the trial and also limited its utility for future trials.

Qualitative research with this focus appeared to be motivated by a desire to acknowledge the role of complexity in many interventions being trialled, owing either to the multifaceted nature of the intervention or to the complex social structures and contexts in which interventions took place. The qualitative research was intrinsically evaluative and entailed an exploration of the intervention using the perspective of the participants and wider stakeholders. When the goal was an explanatory model of the intervention,38,39 the qualitative research was part of a broader mixed methods approach. Alternatively, the focus was on ‘generic’ mechanisms rather than on those deemed to be specific to the trial intervention, in which case the message was less for the trial itself and more for a ‘class’ of intervention, such as ‘self-management’43 or ‘placebo effects’. 41 Additionally, the research was sometimes designed to address a recognised body of theoretical work, such as theories of behavioural change. Researchers undertaking research in this subcategory reported a number of related reasons for doing so:

1. To unpick the complexity of an intervention and develop a theoretically informed model of mechanisms of action by modelling the relationship between the elements.

2. To refine interventions by identifying the most ‘effective’ components from participants’ accounts, with reference to underlying theories from the social sciences.

3. To create conceptual clarity around an aspect or element of the intervention (e.g. patient-centred care).

4. To develop a theoretically informed model of treatment effects from a patient perspective.

5. To better understand elements of the process of care (e.g. modelling the conditions for ‘well-functioning teams’).

6. To develop empirically based and theoretically grounded explanations of wider treatment effects, especially those related to beliefs and behaviour (e.g. placebo effects).

7. To evaluate existing theories of mechanisms of action and develop new insights or models.

8. To develop existing theories of the mechanisms by which generic outcomes (e.g. ‘self-efficacy’) are achieved through the intervention as a whole or through specific elements of the intervention.

9. To identify mediating factors and ‘hidden mechanisms’ relating to the intervention. The work of Ray Pawson was frequently referenced in relation to this type of work. 44

10. To create explanatory models of the impact of the intervention using a mixed methods approach showing how the context and the complex social systems within which an intervention occurs can be an important mechanism of change.

Research in this subcategory ranged from simple linking of existing theories to the accounts given by participants, to the development of sophisticated and complex models of mechanisms of action that may encompass treatment, process and context. Understanding the impact of ‘context’ on the outcomes of an intervention could have particular relevance to the transferability of the results or the external validity of the trial. The value to the original trial was high, especially where unexpected outcomes were observed, or where measured outcomes were uniformly negative but patient satisfaction with the intervention or the process of care was high. The work by Hoddinott et al. 39 was particularly interesting as the intention to develop an explanatory model of the intervention incorporating contextual effects was designed at the outset of the trial in anticipation of expected context-driven variation in outcomes in the different cluster sites in which the intervention was delivered. This work was therefore able to directly inform the interpretation of the trial results. Alternatively, this research was sometimes reported with little reference in the discussion or conclusions of the trial in which the data were collected, with the key message from the article being directed to clinical practice or to the body of literature that would form the theoretical basis of intervention development in the future.

1. Be explicit about messages for transferability of trial findings

   • Successful research in this subcategory was able to generate new knowledge that had the potential for deepening our understanding of how interventions achieve the results observed in the context of a particular trial and how these lessons may be transferred to other contexts. When appropriate, a commitment to taking a wider view by incorporating aspects of delivery and context would help this type of research to contribute substantially to our understanding of the transferability of the trial findings and identify likely barriers to the successful roll-out of practice based on the trial results.

2. Extend current thinking by challenging as well as endorsing theories

   • Research addressing underlying generic concepts and theories underpinning interventions have the potential to inform a wide range of possible interventions in different contexts. Weaker articles missed the opportunity to extend current thinking because external constructs, concepts or theories were employed more for the purposes of referencing or endorsing the findings than for challenging existing ideas or developing new knowledge.

Articles were largely focused on complex interventions with behavioural and/or psychological components. Although often not stated, the rationale for conducting the qualitative research appeared to be a desire to understand the experiences of trial participants in terms of the perceived value of the intervention and its benefits, with an underlying assumption that the quantitative measures employed in the trial outcomes were likely to fail to capture some aspect(s) of the intervention valued by patients or participants. This allowed some researchers to identify benefits that were not anticipated at the outset of the trial (and, therefore, not measured) and others to explore benefits that were anticipated but not measured, either because they were not amenable to measurement or because they were considered peripheral to the primary outcome. For example, Morone et al. 50 undertook a qualitative study expecting to find benefits relating to the intervention that were broader than the trial’s primary outcome of pain reduction. For other researchers, a rich account of the meaning of the experience, which allows them to explore why the intervention is valued by participants, may be used to support the case for further research or the adoption of an intervention in practice. In the article by Dowrick et al. ,46 the exploration of the perceived benefits relating to the management of medically unexplained symptoms identified its potential value as a transferable skill in the management of chronic illness. In the article by Lipman et al. ,48 the exploration of benefits was understood in the context of the lives of the trial participants (lone mothers), the qualitative work undertaken during the trial therefore had the potential to increase understanding of the needs of this group. Finally, data collected relating to the ways in which participants experience benefits from a trial intervention may be used to illustrate quantitative findings and give them more immediacy, or they may be used to explore contradictory findings in the trial outcomes, such as high satisfaction scores alongside small changes in other measured outcomes.

1. Consider using qualitative research at the pilot phase

   • This research was most frequently conducted with the main trial rather than as part of any formal pre-trial or piloting work. This restricted the potential impact of the research on the trial itself. If undertaken pre-trial, unexpected benefits could then be considered as candidates for formal outcome measurement (see subcategory on outcomes Breadth of outcomes to be measured in the trial).

2. Give more consideration of findings in relation to trial results

   • Because most of this research was conducted during the main trial phase, impact was more likely to be achieved in terms of giving the trial results a ‘human context’, making recommendations for future research or suggestions for clinical practice. Opportunities appeared to be missed within these articles for using the findings of the qualitative research to support the interpretation or presentation of the trial results.

3. Give more consideration to messages for implementation in the real world

   • Researchers could have been more explicit about the utility of their findings for encouraging take-up or implementation of an intervention following positive trial results, especially when the intervention was perceived as novel or involved the active collaboration of health staff or patients in its delivery.

The authors of these articles offered a range of reasons for conducting research into the in principle acceptability of the trial intervention. Some started with this as a purpose but for others it emerged as an issue during the main trial and it then became the focus of the article. In each latter case, the statement supporting the need to undertake the study indicated that there was reason to believe that the idea of the intervention might not be received entirely positively by the recipients or the providers. The given rationale for exploring the in principle acceptability of a trial intervention tended to fall into one of three trial-related categories:

1. The need to explore the impact of individual and wider cultural beliefs on behaviours, such as the likely take-up of a trial intervention by participants.

2. The need to explore the potential impact of individual and wider profession/cultural beliefs on the optimal delivery of the trial intervention.

3. The opportunity to understand changes in existing beliefs about acceptability that emerge during the course of the trial and their implications for the trial results and/or for future clinical practice. In these cases, retrospectively reported acceptability in principle may be contrasted with reported acceptability in practice.

The qualitative research aimed to understand the relationship between beliefs and behaviour or choices made prior to, or during, the trial. A minority of the 32 articles explored this issue in isolation and prior to the trial itself and these articles focused on patients or the general population rather than professionals involved in the delivery of the intervention. They were characterised by a cultural distance between the trial organisers and the recipients of the intervention. An example is Zhang et al. ,56 in which the trial lead was based in the USA and the trial intervention was in China.

The potential impact of this use of qualitative research on the trial itself is likely to depend on its timing. When a concern exists about the acceptability of an intervention to be trialled, doing this work prior to the main trial may impact on trial recruitment, long-term participation or adherence to the intervention, as well as the effective or optimal delivery of the intervention in the context of the trial. An example of this is provided by Zhang et al. 56 who explored the acceptability of dietary changes in advance of a planned trial and found strongly held negative beliefs about the acceptability of brown rice in terms of beliefs about taste and quality. They were able to identify nutritional messages that could increase acceptability and encourage participation in the trial, but these did not change the intervention itself. This highlights why this subcategory differs from what we have called ‘intervention development’ because here it is beliefs about the intervention that are important. In the 32 articles, pre-trial work was relatively rare, with most studies being conducted after commencement of the full trial. This may be due in part to our study inclusion criteria, which required the qualitative research to be linked explicitly to a planned or ongoing trial.

The qualitative research was undertaken alongside the main trial for many of our articles. These explored acceptability in principle alongside acceptability in practice and some also addressed the perceived value and benefits of the intervention as part of a broadly specified rationale for the qualitative research such as ‘to explore the experiences of patients participating in the trial’. This limited the potential impact of the research on the trial because it could not shape the intervention delivered in the trial. However, it did have the potential to relate the qualitative findings to the outcome of the trial, poor implementation of the intervention or low recruitment. 53 Mihaylov et al. 53 were able to explain the failure to recruit in a trial of constipation treatment according to the strongly held beliefs about appropriate or favourite existing treatment and a reluctance to change behaviour as required by entering the trial. Buzza et al. 52 described how negative prior beliefs that existed about some aspects of the intervention (e.g. patient activation seen as a threat to professional autonomy) did not appear to impact on behaviour during the trial. Ockleford et al. 54 were able to demonstrate important relationships between acceptance of a diabetic diagnosis and willingness to engage with self-management groups. They suggested that engagement with the intervention programme might help some trial participants who were struggling with accepting this change to their identity.

Many articles concluded with messages for clinical practice, without reference to the trial findings. The trial appeared to present an opportunity to explore issues relating to beliefs and acceptability relevant to clinical care, some relating to the likely success of the intervention in a real-world context, others making explicit suggestions for improving the intervention or its delivery in practice. For example, Reid et al. 55 concluded with the need to address the ‘in principle’ fears and beliefs about opioid analgesics among cancer patients in order to improve clinical care. In this case, the trial was identified as an opportunity to interview patients who were being offered opioids for the first time and found that patients strongly associated them with end-stage care and death and therefore discounted them as helpful pain relief at earlier stages of treatment. A few articles took their discussion beyond the immediate clinical or public health interventions that were the subject of the trial, raising issues about the transferability of the lessons learnt to other clinical contexts requiring behavioural interventions with similar underpinnings and principles for success, for example patient activation strategies. 52

1. Consider the timing of the qualitative research in relation to the trial

   • When a concern exists about the acceptability of an intervention to be trialled, qualitative research is best used prior to the main trial to allow it to impact on trial recruitment, long-term participation or adherence to the intervention, as well as the effective or optimal delivery of the intervention in the context of the trial. If the qualitative research is undertaken within an ongoing trial, the obvious limitation is that, once in the trial, participants have already made a decision that reflects some degree of acceptance and the sample will reflect this range of beliefs. Some designs overcame this problem by exploring in principle acceptability among those invited but declining to participate in the trial. 51,55

2. Be clear about messages for the trial

   • Research in this subcategory had the potential to produce messages of relevance to the trial and beyond in clinical practice or the delivery of population-based public health programmes. Overall, we found a considerable divorce between the trial itself and the qualitative findings. This may have been due to the qualitative articles being published in advance of the trial findings but sometimes the main trial results article did not reference the qualitative research (e.g. Ockleford et al. 54).

The potential value of this use of qualitative research depended on when it was undertaken in relation to the trial. We discussed its potential value at the pre-trial and full trial stages. First, feasibility and acceptability in practice can be explored as part of a pre-trial pilot or feasibility study with the explicit aim of impacting on the way the intervention is delivered in the main trial. For example, the rationale for using qualitative research at this stage was that members of the intervention group were required to make substantial changes to their sexually related behaviour in order to comply with the intervention57 and the use of theoretically informed, but untested, interventions gave rise to practical concerns about the feasibility of delivering them optimally within the context of the rehabilitation clinic. 59 The potential value to the trial endeavour was scope for radical redesign of the intervention, or its mode of delivery, or to raise concerns about the design or value of the main trial being conducted. In this sense, these studies had something in common with those aimed at intervention development; however, intervention development was not an explicit research aim. Second, feasibility and acceptability in practice can be explored during the main trial and indeed the majority of the 83 articles focused on this stage. The impetus for this was not always clear. The initial qualitative research question was ‘to capture the experiences of the participants’ stemming from the recognition that the intervention was either complex or novel or both. In some articles, the focus of the analysis was broad and diffuse whereas in others, the data collected under the banner of ‘participant experiences’ were used selectively in the analysis to address a particular issue arising from the trial conduct or the trial results. The qualitative research was used to describe the intervention itself or introduce it to a clinical community that would not get a full picture of its potential impact on recipients or providers from the main trial results alone. Potential value also lay in providing evidence of acceptability of an effective, but controversial, intervention. 61

A subset of articles presented findings on feasibility and acceptability as part of a broader ‘process evaluation’ that sought to capture and evaluate a range of contextual and behavioural issues relating to the trial conduct and intervention process. Two articles reported qualitative research undertaken after the trial results were known to understand low or variable intervention adoption rates within a trial. The importance of this issue was only identified as the trial progressed. 58,60

1. Identify a specific research question.

   • The research question was very broad in many articles in this subcategory. The apparent lack of focus of some articles may have reflected an underlying methodological tension between a perceived need to identify a priori questions for the qualitative research and a desire to allow emergent issues to be identified during data collection and analysis. However, articles could be strengthened by having a clearer rationale for the analysis, rather than an assumption that patient experiences are important, per se.

2. Be explicit about the message for trials.

   • There was a clear sense that it was important to gain an understanding of the way an intervention was received and/or delivered in practice, but not why this deeper understanding was important to the trial itself. Articles with a narrower, trial-linked focus to the analysis carried a stronger message than those trying to present a broad brush analysis of all the factors (positive and negative) that constituted the experience of delivering or receiving an intervention under trial conditions.

3. Consider the timing of qualitative research in relation to the trial.

   • Surprisingly few studies looked at delivery issues at the pre-trial stage. There is scope for more work to be carried out at that early stage to shape the practical delivery of the intervention and to ensure that the main trial is not jeopardised by a failure to implement the intervention optimally.

The most direct impact of this qualitative research on the trial was interpretation of the trial results. If an intervention was not delivered with adequate integrity (i.e. as planned), the results of that trial could reflect the quality of the intervention as delivered and produce false-negative results. Triallists can use quantitative measures of intervention fidelity, dose and reach, but these articles showed the utility of qualitative methods such as observation and interviews in gauging the way an intervention was delivered or implemented within a trial. Observational data were used to create a checklist of quantifiable actions that comprised adequate delivery, showing variation or deviation from protocol. In contrast, qualitative data comprising first-hand accounts and observations of delivery in context explored questions relating to how or why variations and deviations from protocol occurred. Interventions as planned often included a concrete, measurable element defined in a protocol, but the implementation of this was subject to external influences (context) and individual preferences (behaviours and beliefs) on the part of intervention recipients and/or those delivering the intervention. Using qualitative methods to explore these issues was identified as useful in assisting the interpretation of the relationship between implementation and observed outcomes. 66 Cluster trials in particular were seen as vulnerable to a powerful potential context effect, as local structures and practices were likely to impact on delivery. 64,66 Complex interventions frequently required behavioural changes on the part of recipients and/or those delivering the intervention which were sensitive to individual preferences, beliefs and circumstances. 65 Less complex interventions, such as biomedical or drug interventions, were also acknowledged to have strong behavioural components that required attention. The drug trial reported by Plummer et al. 68 attempted to measure adherence to drug therapy among the trial participants, but recognised the limitations of these data for explaining the observed patterns of adherence. Indeed, the qualitative research raised questions regarding the reliability of the quantitative measures of adherence. 66,68

In a pragmatic trial, it could be argued that some variation in the integrity of intervention delivery would be found in practice and, therefore, acceptable in terms of the external validity of the trial results. However, messages for (clinical) practice could emerge as a by-product of assessing fidelity of implementation that had relevance beyond the trial results in terms of understanding potential moderating factors and identifying those that may be amenable to local modification in order to optimise the effectiveness of the intervention in practice. Thus, a study by Holliday et al. 64 was usefully able to identify variations that the authors considered to be both ‘unacceptable’ and ‘avoidable’ in the future implementation of a schools-based smoking cessation intervention, as well as identifying examples of acceptable, and anticipated, variation.

1. Link findings back to the trial.

   • Despite a strong case being made for the potential of qualitative research with this focus being used to aid interpretation of trial results, we found few examples of this in practice in the qualitative articles here. Some articles announced the intention to do this once the trial results were known and these trial articles were not included here. However, separate publication strategies may militate against this happening in practice. When trials assess interventions that are dependent on successful implementation, building this type of work into the trial design at the outset, with clear strategies for incorporating findings into the trial analysis, if appropriate, may increase the utility of undertaking this type of qualitative research with trials as well as enhance the value of the trial itself in terms of the dissemination of successful practice.

Articles here addressed the broad question of how confident users of research evidence can be that an intervention delivered successfully in a trial will be transferable outside the experimental trial setting. They were linked to the emerging discipline of ‘implementation science’, which recognises that positive trial results will undergo a process of adoption and incorporation into real-world practice. Murphy et al. 37 describe this evidence gap as the result of an inevitable ‘context stripping’ that occurs in a hypothesis-driven experimental research design. They also suggest that qualitative research is particularly useful for putting context back into the picture in terms of behaviours, beliefs and local circumstances that do not form part of the original quantitative trial evidence.

In our examples, the qualitative data were analysed after the results of the trial were known, in order to identify specific factors relating to behaviour, beliefs or context that might impact on the future adoption of the intervention. Cals et al. 69,70 demonstrated how qualitative interview data, collected prior to the trial results being known, were analysed retrospectively and in the light of positive trial findings to address explicit issues relating to the likely adoption of the intervention. A particular strength of this work was its ability to describe the perceived value of the intervention (near-site testing) to a sample of clinicians similar to those who would be considering its adoption in the future. For Carnes et al. ,71 the trial message was that neither oral nor topical administration of non-steroidal anti-inflammatory drugs was superior in controlling knee pain. However, the choice between them still has to be made in everyday clinical practice and, using the qualitative research, these researchers were able to describe how prescribing decisions might be enhanced if patient preferences, and the reasons for these preferences, were incorporated into the prescribing process.

All the trials here were positive. However, when trial findings indicated a lack of effectiveness of an intervention used routinely in clinical practice, qualitative enquiry could be used to explore contextual and behavioural barriers to cessation of that intervention. We found no examples of this type of enquiry in the sample of literature reviewed.

1. Undertake analysis with a clear, trial-related, focus.

   • Qualitative research in this subcategory overlapped substantially with that undertaken to explore the feasibility or acceptability of an intervention and to assess the fidelity of its implementation. However, the strength of research in this subcategory lay in its ability to address focused and practical questions relating to the translation of trial findings into practice.

2. Be clear about the aim of the analysis in the abstract.

   • Articles by Cals et al. 69,70 were examples of how abstracts sometimes describe the aims and objectives of the type of data collected (views and attitudes based on experiences) rather than the rationale for conducting a particular analysis (to identify factors that may support the process by which an intervention found to be effective is adopted in practice). In these articles, data were collected for a process evaluation and then analysed retrospectively to explore particular issues, given the results of the trial. This fact emerges from reading the full article only. The value of research in this subcategory would be more apparent and accessible if a distinction was made between the original (broad) objectives of data collection and the specific aims of the analysis being presented throughout the article, including the abstract.

A considerable amount of research has been undertaken about barriers to trial recruitment and this has been synthesised. 73 Most of the studies in the review by Ross et al. 73 were quantitative and the articles we identified in this subcategory were building on this evidence base. Rather than attempting to identify barriers as in the review by Ross et al. ,73 they focused on strategies for overcoming the barriers,78 filling gaps in the knowledge base such as barriers to recruiting health professionals rather than patients,79 focusing on retention as well as recruitment,77,79 considering barriers in new research environments80 and considering barriers faced by different types of recruiters. 81 Most of the articles included here undertook the qualitative research to help recruitment in future trials. The main goal of De Salis et al. 76 was the use of qualitative research to increase recruitment rates for the specific trial. These endeavours were extremely useful because researchers were building on the evidence base and addressing an issue for improving future trial viability.

1. Consider the depth of the qualitative research required.

   • Some of the qualitative research reported here was descriptive, listing reasons for poor recruitment or retention as if the data had been obtained from surveys. 77,80,81 The authors were able to identify important issues even when they themselves pointed out the limitation of their content analysis that did not ‘allow a deeper analytic approach’. 78 This raises the issue about whether the strength of qualitative research to move beyond accounts and identify underlying issues, as opposed to creating lists of issues, is needed here or whether descriptive qualitative research is enough. There was also an issue about the labelling of some approaches as qualitative research when in some articles it read as if researchers took advantage of information collected as part of the recruitment process to write up about barriers and solutions. This was a useful endeavour but perhaps wrongly labelled as qualitative research.

2. Consider including those who do not participate as well as those who do.

   • Some authors included in their samples people who had not participated or stayed in the trial as well as those who had. This was a useful thing to do but more clarity was needed about the different insights offered by non-participants and participants.

3. Look beyond interviews.

   • Most of the studies included here used interviews. De Salis et al. 76 used observation to identify changes to make to a specific trial and were able to draw on more than accounts.

4. Impact on the current trial

   • The focus was on learning from current trials to inform future trials. De Salis et al. 76 main goal was the use of qualitative research to increase recruitment rates for the specific trial. This is an approach that should be considered more often.

Authors of these articles were usually building on the evidence base about barriers to broadening trial recruitment by focusing on strategies for overcoming the barriers84,88 and understanding how these barriers impeded diverse recruitment. 85 Some of the studies appeared to be driven by the use of community-based trials,86–88 for which there was a need to include hard-to-reach groups such as those in rural areas, with low income, or from ethnic minority groups. The qualitative research by Jaspan et al. 84 had a high utility for the specific trial because it was identified as a way of interacting with the community and seen as a way of communicating and consulting with the community about the planned trial. There was an aspect of action research in the study by Velott et al. 88 in that they were trying to promote recruitment for their specific trial. This endeavour felt extremely useful because it was addressing the external validity of trials but also the viability of community-based trials. These kinds of articles could also challenge the interpretation of trial findings by highlighting problematic recruitment practices. 85

1. Consider the depth of the qualitative research required.

   • Some of these articles described lists of issues,83 which were useful, but some authors went beyond this to develop understanding. Russell et al. 87 used content analysis but embedded it in known literature to move beyond a simple list. While one of the authors was an anthropologist and observation was used, the question was not ‘what are the barriers’ but ‘how do barriers work’, looking for underlying explanatory rather than descriptive issues. 85 These authors identified the desire of recruiters to recruit good patients and described how the definition of a ‘good patient’ in their research setting excluded minority groups. There was also an example here of an approach which was labelled as qualitative research but was a meeting of recruiters to discuss problems. 86 It produced useful insights but there was also lack of credibility owing to lack of detail about who attended, how many attended and who was able to speak at the meeting.

2. Do not be tempted to downplay the strength of qualitative research.

   • There was some evidence of qualitative research being judged from a quantitative perspective in the discussion section of some articles. 87 For example, references were made to small numbers affecting generalisability and how obtaining ‘just perceptions’ would need to be validated. 87 This may have been related to the use of content analysis of the qualitative data in this article, but there is a need to be careful about the inferences drawn from any study and this can be done based on the strengths of the methodology used. In contrast, Joseph and Dohan85 dealt with this extremely well. They recognised that what constitutes a ‘good patient’ is likely to depend on context and that they had studied one context. They attended to the transferability of their findings by requesting that future trial leaders articulate what they think is a good patient to recruit and consider its effect on breadth of recruitment.

3. Be explicit about learning for trials.

   • Russell et al. 87 were explicit about the learning for future trials by producing a checklist and being clear about this in the abstract of the article.

4. Look beyond interviews and focus groups.

   • Focus groups were commonly used in this subcategory. The one study using observation was extremely insightful. 85 This may have been related to the anthropological background of one of the authors rather than the observation, per se.

The articles were about understanding trial participation, focusing on the decision-making processes people went through when asked to join a trial. They focused on sensitive trial recruitment practices, such as when parents were asked to put sick infants in a trial89,92 and cancer trials that recruited patients at the end of their life. 90 Their value was in understanding how trials can be more sensitive to human beings by avoiding misconceptions and improving communication as opposed to improving ethical procedures.

1. Consider the depth of the qualitative research required.

   • This subcategory lent itself to a more in-depth approach to analysis. McCann et al. 91 produced a sophisticated qualitative analysis based on observation and interview to develop the concept of ‘conditional altruism’ rather than simply list themes. Kohara and Inoue90 also provided an in-depth analysis using a grounded theory approach.

2. Be explicit about learning for future trials.

   • Good practice was stating clearly how the findings of a study were useful, and to whom, in both the abstract and body of the article. Some of these articles could have been much more explicit about how triallists or recruiters could use their research, for example, those recruiting for trials could understand how to be more sensitive to the journey a parent is on. 89,92 This contrasted with McCann et al. 91 who placed emphasis on the utility of their research by including a section in the discussion on implications for triallists.

3. Consider including those who do not participate as well as those who do.

   • Different insights can be gained about trial participation by including those who did not join, or who left the trial, as well as those who participated. 89,90,92

4. Consider looking beyond interviews and focus groups.

   • The two articles using observation appeared to be more insightful and took an explanatory rather than a descriptive stance. 90,91

There were two drivers for using qualitative research to explore the acceptability of a trial in principle. The first was concern about the viability of a proposed trial because key service providers or target participants may be unwilling to participate in a trial which required doing something difficult, for example abstaining from sex,97 the randomisation of options may be unacceptable93 or the control controversial (Campbell et al. ,94 with placebo surgery). Qualitative research for this purpose seemed extremely useful by saving time and money and avoiding the low morale that might be involved in setting up a trial that failed. 93,94 Of course, it could be argued that the qualitative research itself cost time and money and that a cost–benefit comparison may be necessary. However, there was utility of the qualitative findings beyond the proposed trial because it identified why there was a problem so that other researchers could consider the viability of related trials. There was also explicit consideration of the implications for wider types of trials, for example placebo trials and surgical trials in general. 94 The second driver was the closure of a trial and how media portrayal influenced the closure or could influence the viability of future trials by raising the issue of these trials being unacceptable in principle. 95,96

1. Be clear about the research questions.

   • Some studies focused only on the acceptability of a trial in principle and offered very clear findings. 93,94 However, some addressed so many issues concerning both the intervention and the trial itself that it was difficult to understand the purpose of the study and, therefore, engage with the findings. 97

2. Focus on in-depth understanding as well as description.

   • The critical discourse analysis by Mack et al. 96 stood out as an exemplar of qualitative research used to understand rather than describe an issue. They offered a sophisticated analysis focusing on how and why the media coverage was persuasive rather than simply describing the media coverage and its effect.

3. Make the implications for different audiences clear.

   • Campbell et al. 94 offered an exemplary approach to thinking about the implications of their research for a number of audiences in both the abstract and the conclusion section of their publication, that is, implications for researchers undertaking trials of surgery and placebo surgery trials as well as those interested in the specific trial under consideration. The longer report format of their publication may have helped by offering more space, which might encourage authors to consider the wider implications of their work. However they should be given full credit for doing this in a thorough and clear manner. Others had limited conclusions and reflections for trials, leaving the reader wondering what action triallists might take in response to the findings. 95

4. Be explicit about the origin of the research question.

   • Some researchers were explicit about being driven by a question they wanted to know the answer to,93,94,96 making it easier to understand the findings and their implications.

One of these studies was undertaken in the context of a feasibility trial, for which the researchers wanted to know whether or not it was feasible to run a trial in a busy emergency environment. 101 The main trial was not feasible owing to difficulties recruiting, ability to consent in a busy ward and emergency situation and ethical issues of carer consent when patients were frail. The other three were undertaken in the context of the main trial, for example to consider the acceptability of the trial design to researchers because the control arm offered a service which the researchers believed did not contain the active ingredient but could have offered what they term placebo effects. 98 It was not clear whether or not this was prompted by the trial results, whereas Thorstensson et al. 100 undertook the qualitative research in response to something going wrong with the trial because a large number of participants were moving from one arm to the other and there was a need to understand why Jonvallen99 did not present an explicit motivation for her study. Using qualitative research to consider the acceptability of a trial in practice felt like a useful thing to do. It identified why the trial was not feasible in a busy emergency situation,101 questioned whether or not some controls offer interventions in themselves,98 identified lessons for future trials of interventions that were likely to have strong user preferences100 and questioned whether or not ensuring compliance with trial protocols could cover up the lack of compliance with the intervention. 99

1. Make the implications for different audiences clear.

   • All the articles would have benefited from making the implications for different audiences clear in both the abstract and the conclusions. This was certainly an issue for Thorstensson et al. ,100 for which conclusions could have been directed at triallists to help them deal with strong preferences for treatments by perhaps using different designs in this situation. In other articles, there was opportunity to maximise the value of the qualitative research by going beyond description of the results in the abstract and including implications. For example, Beal et al. 98 could have discussed whether or not triallists should use a control comparison comprising large amounts of education, what other types of controls might be more appropriate and what types of trials and interventions their findings were applicable to. Instead, it was framed as a general inquiry about placebo effects and trial participation.

2. Do more of this at the pre-trial stage.

   • Only one of the four studies was undertaken at the pre-trial stage so that lessons were for future trials rather than the specific trial with which the qualitative research was undertaken.

Researchers appeared to be driven by concerns that informed consent for participation in trials was problematic in practice. Some were driven by concerns about therapeutic misconception – the conflation of research with clinical care105,108,109 – and whether or not it was actually occurring in a trial. For example, Wasan et al. 109 seemed to be concerned about whether patients joined the trial for clinical benefit when there was no likely benefit. Others were concerned about complicated informed consent processes that appeared to pay no attention to important communication issues relevant to different groups. 107 Gikonyo et al. 104 were part of a Social and Behavioural Research Group independent of the trial, funded to consider a wide range of issues around the trial. They were interested in whether or not everything was working as planned, including informed consent. They produced multiple articles about how to help RCTs work in low-income or diverse communities. Mangset et al. 106 were funded by an ethics group and focused on people who had just had a stroke and for whom informed consent was difficult.

This appeared to be a valuable contribution to the trial endeavour because it challenged ethical assumptions by both ethicists and triallists, showing that informed consent could be more complicated than ethicists present. For example, people do not participate in trials for one simple reason but for a mix of altruism, desire for clinical benefit, access to treatments they do not have access to, or to help health professionals who are there to help patients. 105,108,109 They identified that a benefit of participating in a trial could be better health care when trials are undertaken in contexts in which people pay for health care. 104 This body of research also showed that gold standard informed consent procedures could be problematic and that a simpler informed consent process, which paid attention to language and communication, could improve people’s understanding and, therefore, access to trials. 107 This research also showed that emphasis on formal information giving procedures rather than attention to building relations with a low-income community could be problematic104 and showed the need for individualised rather than standardised informed consent. 106 There was an overlap here with the next subcategory of adaptation of the trial conduct to the local context. As well as considering how to improve informed consent, researchers focused on making trials work for diverse groups, paying attention to local rules and appropriateness104,107 and respect for communities. 104

1. Consider undertaking an in-depth analysis.

   • Many of these articles undertook an in-depth analysis108,109 and benefited from using observation107 or ethnography105 to show insights about complex ethical issues.

2. Be explicit about learning for the trial endeavour.

   • The message for the trial endeavour was very clear in these articles. There was only one example in the six articles where the researchers failed to offer direction and recommendations for that type of trial with that type of patient.

3. Be prepared to challenge standard practice and accepted beliefs.

   • One strength of this set of articles was that the authors were prepared to challenge ethicists’ theories using empirical evidence as well as challenge what is considered to be gold standard ethical practice in trials.

The core activity in both articles appeared to be the establishment of a liaison network and a council/committee of local people to facilitate two-way communication about the trial. The key driver appeared to be ensuring the trial was feasible in the local context. Other aims were at play, with specific reference to increasing participation within the trial,111 empowerment of a community, dialogue between community and researchers so that research is undertaken with people and not on people,110,111 reduction of the probability of negative reactions to the trial111 and good participatory practice. 111

Taking a participatory approach to making trials appropriate to local contexts seemed like a very important thing to do for community-based trials in ‘hard-to-reach’ communities. Shagi et al. 111 made the point that this was not only essential to the successful conduct of their feasibility study for the trial and would also be used in the main trial. Balcazar et al. 110 reported that the process helped disseminate to the community the need for the trial. The two-way communication seemed key to improving the trial quality and trial sensitivity to human beings.

1. Consider the utility of an action or participatory research paradigm/methodology.

   • Researchers used participatory research rather than qualitative methods to improve local appropriateness. This approach could be used in community-based trials as a matter of course.

2. Measure success of participatory approaches.

   • Participatory approaches may be expensive and cost-effectiveness must be considered. Balcazar et al. 110 felt that it was important to take that approach but also say ‘unfortunately, specific data to measure success of this interaction were not collected’. Shagi et al. 111 did collect quantitative data on the goal of the action research, for example, 1600 women were recruited and follow-up rates > 80% were achieved for all routine clinics, but the authors could not attribute this to their participatory approach. If people use a participatory approach they may wish to provide evidence of its success to convince others to use it.

Researchers were not explicit about why they used qualitative research for this purpose. It seemed to be driven by concerns about the viability of the specific trial and future trials: concern that the practitioner community had little experience with trials and thus might be burdened by the trial,112 concern about the viability of trials in the context of the closure of trials in a particular community,115 or concern that practitioners might not be able to practice normally within a trial. 113 A different concern was that trials might impact on the content of the very treatment they were trying to test. 114 It appeared to be useful work for both the specific trial and future trials. For example, Grbich et al. 112 reported that the qualitative research improved communication so that adversaries of the trial turned into advocates, sorted out problems at an early stage, resulted in improvements in some trial procedures and identified issues about neglecting non-trial patients to offer extra care to trial patients. There was also evidence of challenging assumptions. Kyle and Marks-Maran113 challenged what appeared to be strong concern within a community about RCTs and with empirical evidence showed that although that concern existed, there were other positive aspects of RCTs. Paterson et al. 114 challenged the assumption that treatments offered in RCTs reflect practice.

1. Be clear about the rationale for the research.

   • These articles had very helpful messages for trials, but these were not necessarily clear because the rationale for the research question was not clear.

2. Balance the needs of the specific trial and the stakeholders.

   • Grbich et al. 112 had to pay attention to what they could change within the specific trial and what needed to stay the same. They could not change the intervention too much within their trial by dispensing with case conferences because staff did not like them but could change some issues around the delivery of the intervention.

3. Be explicit about the implications for trials.

   • Grbich et al. 112 paid attention to the implications of the qualitative research for both the specific and futures trials, detailing these in a large section in the discussion. Other articles in this subcategory could have been clearer about this.

The researchers did not appear to set out to address the breadth of outcomes best measured in the main trial. As part of the main trial, they collected data on the patient experience of the intervention and identified the opportunity to consider the breadth of outcomes used in the trial compared with the perceived benefits identified within the open comments. 117 They asked all 127 trial participants receiving acupuncture for menopausal symptoms to complete a written statement about changes they attributed to the intervention. The participants identified a variety of health changes that the authors were concerned may not be revealed by limited outcome measures in acupuncture studies. The utility felt high in terms of questioning whether trials could be missing benefits important to patients or perceived as being achieved by patients,117 although the potential value was for future trials.

1. Consider timing in the context of the MRC framework for Evaluation of Complex Interventions.

   • Alraek and Malterud117 took advantage of a data set to explore this issue. However, the utility of using qualitative research for this purpose lies at the early stages of the MRC framework to ensure that an optimum set of measures is used in the full trial.

2. Consider the value of perceived outcomes identified.

   • This approach has the potential to identify many possible outcomes. Alraek and Malterud117 asked respondents to identify ‘even issues that might be considered as less important’. Qualitative research could consider the problem of respondent burden during data collection in trials and give attention to the value of perceived outcomes by participants so that only outcomes valued by participants were measured.

3. Be explicit about the message for the current trial or future trials.

   • Articles could do more than problematise the outcome measurement of the trial and identify explicitly what outcomes trials should measure, or what important outcomes they failed to measure and the implication of this for the conclusions of the trial.

Variation in outcomes measured is likely to occur in any trial. Some variation will be random but some may be explained by differences between individuals receiving the intervention or differences between groups or communities in a cluster trial. Researchers from the two cluster trials were driven to explore variation in outcome because they expected it to occur39 or identified the variation and wanted to understand why it had occurred;118 they used qualitative data collected during the trial. We wondered if this would be driven by null trials but this was not the case because Hoddinott et al. 39 planned to do their analysis prior to the trial and the trail reported by Aagaard et al. 118 trial was positive. The researchers seemed aware of the different contexts in which the intervention operated within the different clusters. Understanding the effect of context on outcome could then lead to understanding the transferability of findings to different contexts, offering textured conclusions from the trial rather than a simple ‘yes/no’ answer. 118 The utility of exploration of variation in outcomes for individual patients was less obvious in the two articles included here and this may have been because they did not absolutely fit this subcategory. 119,120 Both had useful messages for future intervention developers and interestingly, both identified the importance of the individual’s context to making or maintaining successful lifestyle change. The findings could also have been used to interpret the trial results but were not.

1. Pay attention to validity and credibility.

   • Blinding was identified as an important issue in the two cluster trials for which the outcomes of a small number of clusters were compared with qualitative findings from each cluster. Hoddinott et al. 39 undertook seven ‘embedded case studies’ using the clusters as cases and emphasised the importance of building their model of how organisations used the intervention prior to knowing the trial outcomes ‘to minimise bias’. Aagaard et al. 118 identified the importance of blinding the researchers to the seven emergency departments in their study when analysing the qualitative data, but stated it was not possible, and this, therefore, was a limitation of the study. For individual-level studies, it was blinding of individuals during the interviews to their categorisation of ‘benefited from intervention’ that researchers identified as important. 119,120

   • There was a need for transparency when defining an individual as having benefited from the intervention or when producing ratings based on outcomes and processes, stating how this was done and who was involved. Aagaard et al. 118 used their qualitative research to produce a ‘qualitative rating’ on three key issues, including leadership of quality improvement. Then they compared scores on outcomes with scores on processes across the seven clusters. They hypothesised the direction of effect for their three process variables and found it for one, found nothing for another, and found the opposite to the hypothesised direction in the third. The authors drew due attention to the limitations of doing this, rightly reporting that their results were formative, but perhaps still drew too firm a conclusion based on the data they had.

2. Clarify implications for practice.

   • Penn et al. ,119 Peterson et al. 120 and Hoddinott et al. 39 were clear about the implications of their findings, although Aagaard et al. 118 could have considered the implications for transferability of findings and mechanisms of action and Peterson et al. 120 could have considered implications for the specific trial.

3. Consider using ethnography.

   • Hoddinott et al. 39 produced an in-depth and convincing analysis, highlighting the strength of ethnography for that the outcomes would vary and planned to investigate this. This prospective the work they were doing.

4. Plan for variation.

   • Hoddinott et al. 39 expected approach was likely to ensure they had the information and resources they needed to explore any variation.

Researchers did not necessarily plan to explore the accuracy of measures as the aim of their qualitative research pre-trial. It appeared to occur in response to problems arising during data collection and analysis of the trial. An exception was an exploration of the performance of a validated instrument for a particular patient group in a pilot trial in preparation for the main trial. 122 In other studies, issues occurred within the trial and qualitative research was established to explore them, for example, during the trial the research assistants were being asked to clarify how a key outcome measure should be completed. 124 The reason for undertaking the qualitative research was not always clear,123 with some suggestion that a team member had a particular interest and explored it using the trial as opportunity. 126

Even though there was only one example of the planned use of qualitative research on the measures used in the trials prior to the main trial in our six extracted articles, and three in total in this subcategory, there appeared to be merit in making this the planned objective of future qualitative research with trials. The utility felt high by providing extra instructions to allow valid completion of validated measures,126 not using an inappropriate instrument in the full trial,122 and facilitating the development of accurate measures using quantitative data in the full trial. 123 The potential value was improving the internal validity of the specific or future trials.

1. Increase depth of qualitative research.

   • Sometimes the qualitative research was reported briefly in a mixed methods article123,125,127 or was extremely descriptive – almost like a mini-survey. 126 This did not damage the utility of the research but did generate concerns about its credibility.

2. Reflect on the relationship between the qualitative and quantitative research.

   • There was disagreement about two approaches to ensuring accurate measures were taken in RCTs. The first was that having explored accuracy using the qualitative research, then quantitative measures could be used. 123 The second was that qualitative research should be used to collect difficult to measure processes and outcomes within the trial, even if numbers were large and it was time consuming and expensive. 124,125 The best option might be dependent on the situation, with attention needing to be paid to the trade-off between efficiency and accuracy.

3. Link with the current trial or future trials.

   • Messages for future studies and trials were sometimes explicit in the abstract. 123,124,126,127 For example, Penman-Aguilar et al. 123 was clear that they had learnt a lot for the main trial using the qualitative research in their mixed methods study in their pilot trial and developed a measure for the main trial. Farquhar et al. 122 concluded that they would not use the instrument in the main trial, although the abstract focused more on the learning for people using the instrument in general.

4. Timing of research: more pre-trial work needed.

   • In an ideal world, all of this type of work would occur at the pre-trial stage and there were examples of this. 122,123 However, some of the articles used qualitative research during or after the main trial,124–127 so that lessons learnt were relevant only to future trials rather than the specific trial.

Non-completion of outcome measures affects the internal and external validity of a RCT. Nakash et al. 128 undertook a useful study exploring why some participants did not return their postal survey within the RCT. They argued that the usual methodological issues about survey response might not apply within a RCT. Indeed, they found that a lack of understanding of the trial resulted in non-return of questionnaires, identifying ways of improving completion of measures in future trials.

1. Timing of research: more pre-trial work needed.

   • Nakash et al. 128 undertook this useful work during the main trial with lessons for future trials, when it would have been more beneficial at the pre-trial stage in order to impact on the specific trial.

Trials need validated measures and these are not necessarily available for secondary outcomes in a trial. Resources may not be available to develop validated secondary measures prior to the main trial. The two articles included here faced this situation and used a mix of qualitative research before the main trial and quantitative psychometric testing during the main trial to develop validated outcome measures. The potential value was to ensure validated instruments were available for future trials, or to improve the credibility of the trial findings by being able to report that these were based on validated instruments.

1. Describe the qualitative research in more detail.

   • These articles reported mixed methods studies. Articles reporting mixed methods research have space limitations that affect the reporting of details of methods and findings. The authors here chose to limit the space given to reporting the qualitative research, but this can be problematic when the basis of the instrument is the qualitative research.

2. Explicitly consider the utility of instruments beyond the specific trial.

   • Some instruments may be useful for a range of interventions or may be very specific to the intervention for which it was developed. Researchers should be more explicit about the utility of their instrument beyond their specific trial and the extent to which an instrument has been shaped by the needs of a trial, which might make it difficult to use in other trials. 129

The authors did not offer an explicit rationale for using trial participants to explore wider health issues. This appeared to be an opportunistic endeavour, with researchers taking advantage of an identified group of people with a disease, or experiencing a health service, to address a research gap. In some studies, the intention was to collect data on the intervention and other interesting issues arose during the interviews, which then became the focus of an article. 132 In other studies, we could see that the trial offered access to a group of people that might otherwise be hard to identify for qualitative research,135 that the data collected in the trial were rich enough to allow secondary analysis to explore issues of concern to carers133 and that researchers had an unanswered question when the trial ended which was addressed by returning to trial participants. 134

There was no doubt that researchers were attempting to address research gaps and made convincing arguments for the need for their research. These articles contributed to the wider knowledge base in terms of understanding more about users’ experiences of a service136 and offering implications for clinical practice. 132,135 There was even an indication in one article that drawing on the trial as a sampling frame for the qualitative research strengthened the research because of the ability to use the large geographical spread of trial participants which would not be feasible in a separately funded qualitative study. 132 Some articles also offered contributions to the trial endeavour by identifying other potential interventions for testing134 and understanding more about a health problem that was important but was not being measured in clinical trials. 133

The authors themselves often drew attention to concerns, within the discussion section of their articles, related to the generalisability or transferability of their findings beyond the trial. First, health professionals or patients in a trial might not be representative of those in the real world; for example, clinicians in a trial may have had more of an interest in smoking cessation than the general population of clinicians134 or areas participating in the trial may have done so because they had poorer services. 132 Second, being in a trial may have affected the views of participants. 132 Third, the need to fit in with a trial may have affected data collection; for example, the need to collect qualitative data after the last trial outcome measurement may have introduced recall issues for the qualitative research. 132 It was extremely helpful to read researchers’ reflections on the strengths and limitations of this approach to identify suggestions for good practice.

1. State clearly whether or not the intervention and control group are included in the qualitative study and consider the implication of this sampling strategy.

   • In some of the articles, the qualitative research had been undertaken with participants from both arms of the trial whereas, in others, only those offered or completing the intervention had been included. Some authors stated their sampling frame clearly in their methods sections131–133 whereas in other articles we had to work this out ourselves because it was mentioned only in passing in later parts of the article. This information was important for interpretation of findings because experiences in different arms of a trial may not be the same. It may also be important to reflect on sampling from different arms of the RCT both when planning the best sampling strategy for the study and when analysing the qualitative data because one group of authors who used both intervention and control participants expressed concerns about having two groups within their sample with very different experiences. 136

2. Consider the influence of the trial on the findings.

   • In some articles, the researchers described the trial and its participants in some detail. 132–134,136 This offered an in-depth description of the context of the research participants, which is important for the credibility of qualitative research. The detail of the inclusion and exclusion criteria for the trial, by Bair et al., 131 was particularly helpful because of the degree to which this shaped the qualitative sample. Description of the content of the intervention and the control was also important because this affected interviewees’ recent experiences of health care and treatment. The discussion section of these types of articles would benefit from an explicit subsection considering the influence of the trial on findings and interpretation.

3. Consider implications of the findings for the trial endeavour.

   • We acknowledge that the articles considered here may be one of a number written from the qualitative research undertaken with the RCT and that relating findings to the trial endeavour may be better undertaken within other articles. Even so, we think it is important to consider inferences for the trial endeavour because the qualitative study participants were sampled from trial participants. Holtrop et al. 134 did this by showing that the intervention in the trial was one of many influences on the behaviour they wished to change and they were able to identify other future interventions that might be more useful. Some researchers discussed this issue as a brief point in the discussion section of their article rather than in the abstract or conclusion. For example, Cook et al. 133 point out in their discussion that clinical trials are not measuring an important outcome for carers because they do not understand how to measure it. This was an important message for triallists that could have been made more explicitly within the article.

4. Pay attention to the clarity of the key message for different audiences.

   • More attention could have been paid to implications of the findings for different audiences. This seemed to be more important for articles in this category because of the opportunity for confusion. Authors had to describe the RCT and the intervention within these articles and then go on and describe qualitative findings that related to neither of these issues. This could lead to problems understanding the message of the article because it might be unclear if the implications were for the trial intervention or the health condition.