Synthesis for health services and policy: case studies in the scoping of reviews

Challenge 1: clarifying the questions of interest and deciding the most appropriate method of synthesis

The brief topic outline we were given at the start of the project was broadened and expanded during the initial communications with NHS England. They were interested in patient-focused digital innovation in primary care and identified four broad themes and nearly 20 separate questions for which they wanted answers. The questions were wide-ranging, covering issues of contracting and implementation, as well as effectiveness, cost-effectiveness, benefits and risks. They reflected the interests and priorities of the various sections of NHS England involved in implementing ‘Digital-First Primary Care’.

It became clear in these discussions that it was not possible, at that stage, to identify a clear and focused research question (or questions) of high priority to NHS England that was amenable to evidence synthesis. Instead, we identified the need for a responsive and iterative approach to support the needs of the policy-makers.

We adopted a multistage approach to the work. We began by searching the research literature to scope available evidence syntheses. We extracted the key characteristics of all included documents and produced an interactive database of published and ongoing evidence that could be ordered or filtered according to these characteristics, incorporating links to the full-text versions where available. We produced an interim report that summarised the key evidence identified in this scoping exercise, along with an annotated bibliography and the interactive spreadsheet.

We presented the interim report to NHS England and asked them to decide whether it provided the information they needed or if a gap remained to be addressed by further rapid evidence synthesis. NHS England responded with seven questions that reflected specific ‘live’ policy areas in which they were most interested. Therefore, we undertook a second stage and conducted a rapid synthesis of the relevant evidence identified from the stage 1 scoping exercise with the seven research questions identified by NHS England, forming the basis of a thematic framework. Critical appraisal of included evidence was facilitated by relevant assessment tools and reporting standards used to inform judgements about the internal and external validity of included research results presented in the thematic synthesis.

We produced a report that combined both the initial scoping exercise and the rapid synthesis undertaken at stage 2. Again, we presented the findings through a summary report and teleconference to representatives from various NHS England teams (e.g. new business models, primary care, digital and workforce). Following this presentation, and the subsequent discussions, the topic was concluded.

Challenge 2: identifying and incorporating recent and ongoing research

As a result of the initial scoping exercise and discussions with other researchers working in the field, we became aware of recent and ongoing projects, as well as two open NIHR calls for a proposal relevant to the topic. Although we searched for evidence syntheses, we incidentally identified some recent or ongoing primary research studies.

We were keen to ensure that we did not duplicate effort and also to alert NHS England to academic groups actively researching the topic. Therefore, we made contact with the researchers who were eager to engage, providing early sight of their draft reports in confidence to be included in our work. They were also keen to engage further with NHS England to inform ongoing policy work.

Key challenge 1: uncertainty about the type of evidence that would best answer the questions

The first question was whether or not any prior research had focused on patient and carer experiences of the nearest relative provisions within UK legislation. The questions of interest seemed so specific that we doubted if research would address them. Our review group’s contract and work remit extended to the potential for primary research or surveys, where such methods best answer policy-maker or NIHR HSDR questions. However, two early discoveries enabled us to focus on the evidence review component, rather than primary research. First, we identified a 2017 survey of people’s experiences of the MHA by the Mental Health Alliance and Rethink Mental Illness (London, UK). 21 Second, we learned that the independent review team were, themselves, conducting a survey of carer experiences.

It became important to know early on whether the policy customer expected the review team to summarise research evidence or, given the distinct possibility of finding no or very little research, to extend coverage to other relevant non-research forms of evidence in the public domain (e.g. blogs, online discussion forums). Our first teleconference with the policy customer revealed a clear preference for research-based evidence, if available. This was to ‘reduce possible bias’ and achieve a breadth of evidence in the review process, given the other information sources and consultation on which the independent review processes would be drawing.

Therefore, the policy customer gave us a clear and early steer to search for and synthesise research evidence only. Nevertheless, at that time, we were not sure if there would be sufficient, or even any, good-quality UK-based research evidence on patient and carer experiences of the nearest relative provisions. Therefore, we also noted that, later, we might need to extend the scope to grey literature and less research-based sources of evidence, or include evidence from beyond the UK. This highlights pervasive uncertainty about the available evidence for answering alternative potential questions, and the motivation to make best use of a review team’s core skills and resources in responsive, commissioned policy-informing reviews.

Key challenge 2: deliverability within 8 weeks

This was the overarching challenge and a non-negotiable constraint, against which the other challenges and decisions were all weighed. The deliverability of any review would critically depend on the number, richness/quality and, therefore, amenability to formal synthesis of the studies found, and this would not be known until 4–6 weeks before the report submission deadline. Concerns about this were compounded by our awareness that synthesising qualitative research, usually being an interpretative process, takes time and requires a team approach. We had team discussions about the value of also searching for and including survey research (i.e. which may not be qualitative). However, having found enough qualitative research, we did not ultimately search for survey studies. (In any case, such a review would be unlikely to provide more recent and relevant evidence than a reanalysis of the 2017 Mental Health Alliance survey responses.)

Key challenge 3: how old and from which other jurisdictions would evidence be relevant to experiences of the UK Mental Health Act?

As mentioned, this was our first review of an aspect of legislation. Although people in other countries might have experience of equivalent legislation and practices for including family members of close friends in decision-making for people experiencing severe mental distress, only people from the UK (i.e. the jurisdiction of application of the MHA16) would have directly relevant experience of the nearest relative provisions.

Key challenge 4: conducting rapid synthesis of qualitative research evidence

Given the extremely tight timelines, and the researcher-intensive nature of qualitative evidence synthesis, we were very nervous about the possibility of finding too many studies that met our inclusion criteria. This would have risked making the review undeliverable, rushed and of reduced quality, or having to retrospectively exclude studies.

Key challenge 1: reconciling multiple independent stakeholder consultations

The original questions prioritised by the NIHR originated from Health Inspectorate Wales (Merthyr Tydfil, Wales). A teleconference held between the York team and representatives of both Health Inspectorate Wales and Care Inspectorate Wales (Merthyr Tydfil, Wales), provided background and context to their proposed questions. However, before this teleconference could be arranged, the research team had already consulted other key stakeholders about the proposed questions.

Initial contact with representatives from the Professional Standards Authority (London, UK) and Care Quality Commission (London, UK) provided the team with an overview of health-care regulation in the UK and associated research. They expressed a willingness to assist with the proposed work and arranged for researchers to attend (1) the Professional Standards Authority’s Policy and Research Forum and (2) a meeting of the health and social care regulators. The former included representatives from the Professional Standards Authority and various regulatory organisations [e.g. the General Pharmaceutical Council (London, UK), the General Chiropractic Council (London, UK), the Health and Care Professions Council (London, UK), the General Optical Council (London, UK), the General Osteopathic Council (London, UK), the General Medical Council (London, UK), the General Dental Council (London, UK) and the Nursing and Midwifery Council (London, UK)]. The latter included senior managers from the Care Quality Commission, DHSC, the General Dental Council, the Health and Care Professions Council, the Local Government and Social Care Ombudsman (Coventry, UK), the Nursing and Midwifery Council, the General Pharmaceutical Council, the Parliamentary and Health Service Ombudsman (London, UK), the Professional Standards Authority, the Medicines and Healthcare products Regulatory Agency (London, UK), Social Work England (Sheffield, UK) and the General Medical Council. The research team also held a separate teleconference with a representative of the General Medical Council.

In each case, different stakeholders emphasised slightly different areas of interest, deviating to a greater or lesser extent from the original questions. The Health and Social Care Regulators suggested that interdisciplinary regulation of online primary care was the topic of interest. The diverse disciplines and regulatory organisations providing online care provoked questions about effective regulatory oversight and complaints related to care. The potential relevance and utility of the international evidence was particularly emphasised.

The General Medical Council agreed that, even given the differences in the regulatory architecture and frameworks across countries, there was scope to learn from other health and social care systems that face similar issues and potential risks. Of particular interest was multidisciplinary team working: understanding the barriers to, enablers of and issues around responsibility, given joint working and multidisciplinary collaboration.

The representatives of Healthcare Inspectorate Wales and Care Inspectorate Wales described recent policy initiatives to promote the integration of health and social care in Wales and the implications of such policies for regulators. They elaborated on their initial questions to pose a series of fundamental questions about the regulation and inspection of integrated health and social care provision, including what models exist, their effectiveness, the barriers to their implementation and non-legislative means of overcoming these barriers.

Key challenge 2: not all stakeholders are familiar with rapid evidence synthesis

It was clear that stakeholders’ experience and knowledge of rapid evidence synthesis varied widely, particularly in relation to formulating implementable research questions. We wanted to avoid the potential hazard of arriving at a scope that was simply amenable to evidence synthesis rather than reflecting a genuine stakeholder knowledge need.

Key challenge 1: the role of definitions in specifying relevant concepts

Dixon-Woods et al. 34 make the distinction between an integrative synthesis ‘where the focus is on summarising data, and where the concepts (or variables) under which data are to be summarised are assumed to be largely secure and well specified’ and an interpretive synthesis with its concern with ‘the development of concepts, and with the development and specification of theories that integrate those concepts’. The reviews specified by the NIHR HSDR programme typically include at least one concept that has to be ‘discovered’ during the course of the review, whether that be unpacking the constituents of an intervention, the diversity of outcomes or, as in this case, in exploring if a health-oriented concept of access and candidacy could usefully be translated to social care. Elsewhere, Dixon-Woods et al. 31 cite the distinction between the question as an ‘anchor’ [i.e. where four elements of population (or patient type), intervention, comparator and outcomes are prescribed and pinned down] and where the question is a ‘compass’ [i.e. where one element is pinned down (e.g. ‘adult social care’) and the implications of variation in the remaining concepts is explored during the course of the review].

These archetypal uses of review questions are, of course, end points of a continuum of approaches between having highly fixed inclusion criteria and questions and very fluid/flexible criteria and questions. Although the concepts of compass and anchor continue to offer useful heuristics at the question-generation stage, additional conceptual development could usefully explore their further utility and application, as they clearly also link to the core trade-off between the importance of the question(s) and the availability of relevant research to answer it (them).

In reality, when working on a review, we can recognise that all concepts are potentially either ‘compasses’ or ‘anchors’. Furthermore, definitions may be semisecure, such as the working definition provided in Thorlby et al. 32 (see Table 5), or bound by statute, such as the Equality Act30 definitions associated with protected characteristics. Definitions may come from one of many different types of source (Table 6). Our lead reviewer for this project leads for diversity and inclusion at a faculty level and so was already familiar with the scope and variety of terms and concepts within this topic.

Key challenge 2: adding to the value of previous review work

Frequently, the review work that we undertake involves updating or complementing a previous review. In this case, the DHSC had already undertaken a rapid descriptive literature review. Part of our PICOSS challenge was to convey the added value of our preferred form of synthesis for this particular question. This can be conceived as a two-way dynamic flow between review commissioner and review team, in which the commissioner shares details of topic and research question and, in return, the review team seeks to communicate the synthesis methodology. Scoping, therefore, involved identifying existing evidence syntheses (such as Ford et al. 29) that could communicate the potential of our planned review, act as templates for our own methods and outputs, and introduce the methods to be used.

Key to this process is the requirement to be familiar with diverse review methods so that matching between topic/question and methodology can be managed effectively. The emerging science of reviewing research (‘reviewology’) involves familiarisation with different review choices and engaging in an informed negotiation with the review commissioner. Evidence Synthesis Centres, therefore, need to include or have good access to experienced synthesis methodologists to perform this integral function.

Key challenge 3: scope and protocol as a work in progress

As mentioned above, the review team had a final opportunity to determine the scope of the rapid realist synthesis following completion of a process of sensitisation and rapid data extraction that preceded the interim remote meeting with the DHSC staff. The review team prepared and shared 13 candidate programme theories in a technical document, with accompanying illustrative extracts. At a subsequent meeting, the team at the DHSC indicated the relative priority of these programme theories and then confirmed these with policy colleagues. Engaging with such a vivid exemplification of the review scope in practice helped in finally determining the limits of the work. From January 2020 onwards, the review team worked within the scope of each of the five prioritised individual programme theories, selected by DHSC from the initial list, to organise analyse and interpret data within what are essentially ‘mini-reviews’ based on each programme theory. As befits an interpretative review, the parameters of the review product were determined relatively late and from within a wider sampling frame. Furthermore, the limits of the mini-reviews continued to emerge as decisions were made on the scope of each programme theory and their degree of complementarity, synergy or overlap.

Key challenge 1: specifying the phenomenon of interest

Although the initially stated topic made no mention of specific models of social work practice, the earliest communications with the Chief Social Worker for Adults usefully and quickly clarified that one broad ‘model’ of social work practice was of most interest, that is strengths-based approaches. Such approaches had been enshrined in policy as part of the Care Act 2014,36 and were already strongly supported and promoted by the DHSC, the Social Care Institute for Excellence (SCIE) and NICE.

However, from expert advice and reading key reports, it quickly became clear that a strengths-based approach is not a single, well-defined ‘intervention’ or model of practice. Rather, it is a holistic way of working, with a multidimensional set of core ideas and principles about how social care can be effectively and positively provided in a way that respects people’s rights, desires, life and family circumstances, and draws on the capabilities and strengths of the individuals being supported, their families and the communities in which they live. At the same time, we had also been sent a table by the Chief Social Worker for Adults (i.e. the policy customer), which listed various subsidiary approaches, theories and models of practice, on which she had marked ‘those that more clearly fit under strengths-based approaches’.

Therefore, by 3 October 2019, when corresponding with NICE (about key challenge 2), we wrote:

We are aware from our previous scoping searches that there will be few, if any, comparative empirical evaluations of using strengths-based approaches vs. using traditional/usual approaches to social work practice. So we have provisionally, instead, decided to conduct a systematic review of some of the component therapeutic approaches or specific tools that are commonly seen as ways of delivering social work within a strengths-based approach.

Rob Anderson, University of Exeter Medical School, 2019, personal communication

In the likely absence of comparative, empirical evidence of the effectiveness of the overall strengths-based approach (based on our own scoping searches), the team had briefly considered conducting a theory-driven review (e.g. a realist review) to better specify the supposed underlying principles and mechanisms of strengths-based approaches (i.e. how the strengths-based approach improves social work outcomes). However, rather than using a review to better understand the effectiveness of strengths-based approaches from a theoretical perspective, we instead chose a more pragmatic approach to meeting the user’s stated evidence need.

Key challenge 2: rapidly becoming familiar with a new domain of policy and practice

The review team have primarily worked within health care and health policy research, and so the world of adult social care and social workers as a specific professional group was unfamiliar to all members of the review team at the beginning of this review. This meant that more time than usual had to be invested in talking to experts and reading key background reports to get to grips with the language, key concepts and basic context, such as what is adult social care? What is social work and social work practice? What are the types of adults and families in the UK who typically need support from social workers?

Key challenge 3: assuring that we would not duplicate ongoing or recent evidence reviews by others

By following various leads in the literature and a tip-off from our policy customer about ‘a similar piece of work’, we became aware of (1) a recent policy document from NICE and the SCIE,37 which could have been based on a review of research evidence similar to our emerging topic, and (2) another university-based group conducting a review of the evidence relating to strengths-based approaches to social work. We, therefore, investigated both possibilities, through e-mails and telephone calls, to ensure that our planned review of the evidence relating to strengths-based approaches had not already been conducted.

In 2019, NICE and SCIE had jointly published Evidence for Strengths and Asset-Based Outcomes: A Quick Guide for Social Workers. 37 Knowing that NICE evidence products are typically based on commissioned or internally produced systematic reviews, the review team contacted the Quick Guide authors at NICE to find out what evidence this guide was based on. Ultimately, e-mails and telephone conversations with two members of the social care and leadership team at NICE confirmed that no new or specific systematic reviews had been conducted to inform the Quick Guide. Rather, the evidence comprised a collation of evidence-based recommendations drawn from NICE, SCIE and other authoritative sources.

The second potential duplication of our planned review involved a small team based at the University of Kent (Kent, UK) that we had been told was ‘involved in developing a research proposal with others on strengths-based social work, through NIHR’. We spoke with both the principal investigator and the main researcher on this work to gain a clearer picture of their planned work. They confirmed that an ongoing project, conducted by the Adults Social Care Outcomes Unit (a DHSC policy research unit), involved exploring ‘strengths-based practice in social care’.

However, discussions with the lead researcher clarified that this was effectively a scoping review of relevant literature. We concluded that a broader focus and different methods (e.g. no specific study designs or inclusion criteria, etc.) meant that, notwithstanding shared inclusion of some studies, our systematic review would have a sufficiently different focus and methods (i.e. on effectiveness studies, and on specific, selected subapproaches within a strengths-based approach) to be both distinctive and valuable.

Key challenge 4: avoiding an empty review

A frequent criticism of systematic reviews is the apparently high prevalence of so-called ‘empty reviews’. Although empty reviews may give research commissioners impetus to fund more primary research to fill research gaps and give service commissioners greater freedom to encourage service innovations based on experiential knowledge and theory, they typically have less value in the immediate decision-making context of users of rapid reviews.

Key challenge 1: absence of an explicit policy customer or question

With no explicit policy customer, it was not easy to identify a clear and specific clinical or policy need or evidence gap, within a given population. Our initial consultations with the HSDR programme provided insight into specific populations of interest; however, we did not know what evidence already existed for these populations or the scale and importance of the problems relating to their planned hospital admissions. Conversely, several relatively recent, high-quality systematic reviews indicated where there was an abundance of research and also which evidence had already been synthesised. We were also informed by our research commissioners about another review that was being conducted, which focused on ‘early discharge’-style interventions and mainly encompassed hospital-at-home and social care interventions after patients left hospital.

Key challenge 2: lots of previous research and quite a few previous systematic reviews of it – but was there a gap?

Owing to the substantial cost of hospital admissions within all modern health systems, there has been a focus for decades on implementing and evaluating various types of service changes that aim to avoid hospital admissions in the first place (i.e. admissions avoidance) or shorten them by enabling people to be discharged earlier and supported at home or in other non-hospital settings. Increasingly, such intermediate care, hospital-at-home and early supported discharge schemes have also been seen as ways to further policy goals of person-centred care and, relatedly, providing care closer to or in people’s homes.

This has produced a wealth of research and also systematic reviews of that research. However, our research commissioners had felt that an evidence need remained in relation to this topic, but it was not clear if that need had already been met by primary research or syntheses. Early scoping activity centred on three population groups believed, by our commissioner, to exhibit greater uncertainty: (1) people with learning disabilities, (2) palliative care and (3) people with dementia or delirium.

Initial scoping searches of Ovid MEDLINE and Google Scholar identified several systematic reviews of the effectiveness of enhanced recovery after surgery and enhanced recovery programme interventions. However, we found no reviews that specifically focused on the three population groups of interest. We followed up this finding in two different ways. First, we searched for primary studies to ascertain whether or not a review of interventions focusing on these population groups would be feasible. Second, because we were unsure whether or not primary research would have been conducted specifically on these population groups, we considered the possibility of conducting a review of reviews, using subgroup analyses of the population groups of interest within systematic reviews of primary studies of general population groups. We also initiated discussions with external experts, colleagues and stakeholders about our proposed question.

Key challenge 3: defining and operationalising hospital-led ‘organisational interventions’ that aimed to reduce length of stay

Searching for evaluations of hospital-based interventions and hospital-led interventions to reduce length of stay revealed numerous studies that compared specific treatments or procedures. For example, many evaluations compared different drugs or protocols for anaesthesia, laparoscopic surgery with open surgery or different types of postoperative physiotherapy. Therefore, the team needed to distinguish ‘treatment interventions’ from broader ‘organisational interventions’ and changes in how whole teams work to support earlier discharge. In line with Medical Research Council definitions of complex interventions (as comprising multiple components41), checklists for describing components of enhanced recovery after surgery protocols42 and Cochrane Effective Practice and Organisation of Care group checklists,43 we decided to distinguish our interventions of interest as multicomponent interventions, as follows:

• Intervention has multiple components. As defined in the guidance for using the Intervention Complexity Assessment Tool for systematic reviews (iCAT_SR), commissioned by Cochrane, an intervention component is ‘a discrete, active element of an intervention that could be implemented independently of other elements’ (p. 6). 43 Therefore, for inclusion, interventions should contain more than one component (e.g. education and exercise).

Therefore, organisational (multicomponent) hospital-based or hospital-led interventions in older people became the ultimate focus for our systematic review, and distinguished this review from those focusing explicitly on alternatives to hospital care or those focusing on single-component interventions (e.g. laparoscopic vs. open surgery or comparing different regimes for anaesthesia and analgesia for planned surgery). We also excluded strategies that were focused on discharge planning or pre-treatment assessment only (e.g. comprehensive geriatric assessment alone), as these do not affect other stages of the hospital stay.

In the absence of reliable intervention labels, the second challenging aspect of defining our included interventions was how to decide when an intervention aimed to reduce length of hospital stay (and, ideally, in a way that could be plausibly identified from a journal paper’s abstract). Our two main options were (1) to include only those papers that explicitly stated that its intention was to reduce length of stay (i.e. including papers in which length of hospital stay was the stated primary outcome), or (2) to also include those in which the aims implied that a reduction in length of stay was one of the aims [e.g. if the paper stated that it aimed to ‘improve’ or ‘enhance’ recovery (or equivalent language, e.g. ‘accelerate rehabilitation’)].

Key challenge 1: targeting a representative part from a larger ‘whole’

An initial challenge was to select, from a long list of ACSCs on the DHSC website, a suitably productive focus to exemplify and, hopefully, to encapsulate issues for conditions generally. A frequent criticism is that realist syntheses fail to capitalise on the inherent analytical capacity of ‘what works for whom under what circumstances’ by failing to link qualitative insights to ‘what works’ (i.e. the effectiveness information). To avoid this pitfall, the team explicitly constructed a table (Table 10) that mapped the interim conclusions on effectiveness to different ACSCs. It immediately became apparent that cardiovascular or respiratory conditions had been sufficiently explored and adequately supported as effective to make these an appropriate focus for a realist inquiry. A telephone conversation with the technical adviser at the HSDR programme examined the case for each of the candidate conditions, in turn, to finally resolve the review scope. This approach to scoping, which resolves the coverage of a review by optimising both relevance and the likely explanatory power of the evidence, represents a risk-averse strategy when commissioning rapid reviews.

Realist synthesis typically requires prioritisation of programme theory to explore the phenomenon of interest in sufficient depth. The team had to ensure that, having identified review capacity to explore up to three chosen candidate programme theories, they optimised the analytical and explanatory power of their chosen foci. In consultation with the technical adviser from HSDR and the ScHARR Standing Advisory Group on Public Participation, the team selected foci to represent patient, carer and GP perspectives, respectively, within the candidate programme theory.

Key challenge 2: establishing comparability with the target population across diverse study populations

In a realist synthesis, evidence is excluded only if it does not relate to or inform the development of the programme theory. The practical focus of this review required that we exclude evidence with limited transferability to the NHS, such as avoidable admissions in low- and middle-income countries. In line with this decision, we prespecified five countries from which to derive direct evidence to the review, namely the UK, USA, Australia, Canada and New Zealand. These ‘big five’ were selected because of comparable health-care systems to the UK setting. The team continued to engage with the wider evidence base through systematic reviews, opinion pieces and direct reference to individual study reports, particularly where authors connected interventions to the UK context.

Key challenge 3: maximising the utility of review outputs

Much of the emphasis of published descriptions of scoping focuses on determining the conceptual boundaries, methodology options and logistic constraints of an individual review. In comparison, little attention has focused on the role of scoping in shaping the knowledge end product from the very beginning of the process. The scoping process offers an opportunity for commissioners and stakeholders, through their engagement, to exert an early influence on tailoring the final product to their requirements. So, for example, the focus on cardiovascular and respiratory conditions was not determined by evidence on effectiveness only, but also from the commissioner perspective, as articulated by the NIHR programme staff. For the review team, the resultant risk-averse strategy minimises the chances of producing a product that is neither useful nor appropriate. Scoping different options is one way of modelling different review configurations without embarking too far down a particular route.

Unlike other review processes that typically prioritise rigour and reproducibility (e.g. the internationally regarded NICE evidence synthesis products, with their explicit interest in effectiveness and cost-effectiveness), the emphasis of the HSDR programme has generally been to optimise a third ‘R’ of relevance (e.g. encompassing feasibility of implementation, acceptability and sustainability in different organisational contexts). This must continue to be underpinned by the preceding considerations of rigour/reproducibility. Scoping offers an early opportunity to sensitise the context within which the review is eventually situated. In this connection, subsequent attempts by the centres to further improve their working relationships with the patients, public and commissioners is particularly germane. Necessarily, this process of shaping review outputs does not simply involve determining the content, but also entails stimulating receptivity for the form of the final review. Review teams spend a significant amount of time, both in conversations and through the written protocol, cultivating the expectations of the commissioners with regard to less-standard review products, such as realist syntheses.

Key challenge 1: defining ‘relevance’

This review would include evidence from the UK only. This decision was taken to ensure relevance and to assist in managing the large volume of literature on this topic. A key scoping decision is which interpretation of relevance and context makes most sense in relating the study populations to the target population within each review. In some reviews, relevance is defined simplistically, as either geographical or chronological proximity. In other situations, relevance relates to similarities in how interventions work (i.e. conceptual homogeneity). For example, primary care settings for this review represent a ‘first point of contact’ for ID patients or their carers. In either case, the review team has to determine how to incorporate relevance into their review,7 either formatively, by methods used in the review, or summatively, in using one of many tools available for transferability or applicability. 52,53

Key challenge 2: ‘mapping as scoping’

Broad topics are often characterised by uncertainties in relation to the quantity and quality of research. With limited time, the review team must decide how best to optimise the review resource. ‘Empty’ reviews are not helpful to decision-making, and yet overpopulated reviews are likely to either overrun timewise or prove superficial in their analysis and interpretation. A mapping review offers the possibility to offer breadth before the depth of a targeted systematic review. For this mapping review, initial descriptive analysis took place based on title and abstracts only. Subsequently, a subset (of approximately 20%) of the overall studies was sifted from the original retrieval results, supplemented by grey literature searches to form the data set for the targeted review. This approach keeps the scope as elastic as possible until informed decisions on coverage have been made. At this point, a formal protocol is finalised to provide the necessary scientific rigour. A further benefit of this staged approach is that it diversifies review outputs for different target audiences. Those requiring broad coverage can use the mapping review and those requiring a filtered approach that includes quality assessment are presented with a conventional systematic review.

Key challenge 3: building on previous systematic reviews

Although it may seem attractive to build on previous systematic reviews, the complexity of review decisions may offer minimal advantage from starting a review de novo. As illustrated by this review, not only had the preferred terminology changed, but also different fundamental assumptions about how to interpret ‘access’ shaped the subsequent review. Search terms need to be checked for new index terms and for the emergence of new terminology around the user community. Searches must be specified from the date of last search (not from publication date) together with overlap to avoid key items falling down a gap between reviews. Will the original data sources be replicated for fidelity to the original review? In the case of this particular review, the information specialist revisited the original list of sources to modify, and even improve, the original research. They also checked new index terms and terminology to ensure that the strategies continued to meet the requirements of the current review. This tension between comparability and contemporaneity needs to be managed according to the intended purpose of the review. Above all, the review team needs to decide whether to build on the foundation of the previous review, dismantle the previous review and reuse its original studies only alongside recent additions, or simply construct their own review, overlapping with the previous review.

Managing and deciding priorities (consultative issue 1)

This theme captured how the review team manages and negotiates with NIHR, policy stakeholders and other customers to ensure that priorities are addressed within resource constraints. In most review scoping processes, the choices made represent trade-offs between various competing goals, especially between the duration/timeliness and comprehensiveness of a review (i.e. resources) and its rigour and relevance. Where a policy or health service customer of a review is identifiable, it is important to ensure that they are aware of these trade-offs and the scoping decisions that affect them.

A clear example of this need to manage trade-offs comes from the rapid review to inform the independent review of the MHA16 (i.e. case study 2). The initial question was expressed as a policy question rather than a research question, and would conventionally be answered by finding comparative quantitative effectiveness studies. Our preliminary searches revealed few, if any, such studies of this kind. To both reduce and share the risk with the policy customer, the review team expanded their focus and question to patient, carer and professional experiences of the ‘nearest relative’ provisions and, at the same time, made some aspects of the review scope and protocol conditional on how much evidence we found. This process was achieved through close and regular consultation with the policy customer.

These discussions and negotiations also taught the review team about particular aspects of reviewing evidence about legislation (as opposed to health interventions). Legislation, by definition, is jurisdiction specific and so this reduces the value of research evidence about similar legislation in other countries. Similarly, sometimes the legal context changes so significantly at a particular time point that any evidence from before then becomes far less relevant. In this rapid review, this gave us a defensible rationale to exclude evidence from before 1998, which was when the UK’s Human Rights Act22 came into law (and, we were advised, this significantly altered the operation of the MHA16).

The key point, perhaps magnified in the context of such a rapid review, is that trade-offs between rigour and relevance, timeliness and deliverability of the review, and all of these with closeness to the perceived original evidence need, have to be carefully negotiated with and fully involve the policy customer. Where the main policy customer is identifiable, the review team needs to present these choices, and their likely implications for these trade-offs, as far as they can be anticipated in the early review stages. Outlining the available choices is also helpful when a network or group of key stakeholders are the main users or audience of the review, as in the review of the regulation and inspection of health and social care, although it makes the process for agreeing the review’s aims more complicated. This consultative issue, therefore, links to all three interface issues and the two-way discussions between stakeholders and the review team that often take place to identify the important gap in synthesised evidence to balance rigour and relevance and to maximise applicability (for our teams, to the UK health and social care context)

Reconciling different priorities/perspectives (consultative issue 2)

This theme captured how the review team manages potentially competing tensions between what different groups or stakeholders may want to achieve within the overall project remit. The need to reconcile priorities was perhaps most evident in the review of the regulation and inspection of health social care (i.e. case study 3), where the diverse stakeholders included both national service regulatory bodies (such as the Care Quality Commission and Health Inspectorate Wales) and the regulatory bodies of different types of care professionals. The review team noted that ‘the stakeholders emphasised slightly different areas of interest, deviating to a greater or lesser extent from the original questions’.

In the review of hospital-led intervention to reduce length of stay for planned admissions (i.e. case study 6), individual (rather than institutional) stakeholders had different reasons for agreeing on the overall importance of the review topic for older people. One stakeholder noted the overall importance of the goal of reducing the length of hospital stays for older people to reduce the risks of hospital-acquired harms, like falls or infections. Another stakeholder felt that older people may often be excluded from enhanced recovery programmes, perhaps because of perceptions that evidence of benefit was lacking in this patient group. However, a clinical stakeholder was aware of the opposite risk (i.e. older people might be subject to enhanced recovery protocols without evidence that it is safe for them or the protocols may not take account of more complex needs). Therefore, in this case study, the differing stakeholder views did not pull the review scope in different directions, but rather reflected the underlying uncertainty of the initial topic, specifically in relation to older hospital patients (in this case, helping to confirm a narrower focus of shared interest).

Achieving buy-in and engagement (consultative issue 3)

This theme captured how a review team secures input from stakeholders into the scoping and topic prioritisation process. Although the focus is on the initial review stages and on developing a detailed review scope and protocol, we acknowledge that these processes have importance beyond scoping and question formulation.

Although often implicit within our case studies, an important incentive for a stakeholder to engage in question refinement and topic scoping is when their organisation has proposed the need for the review. For example, in the review of the regulation and inspection of integrated care (i.e. case study 3), the topic had originated from Health Inspectorate Wales, and this organisation and Care Inspectorate Wales were strongly engaged with the review throughout. This case study suggests another potential incentive – for buy-in and engagement from equivalent agencies in the different devolved nations of the UK – and so, ultimately, the scoping of this project also involved the Care Quality Commission (for England) and the Professional Standards Authority [for the UK (health professionals) or England (social workers)]. We further speculate that securing the input of lead agencies in relevant jurisdictions may incentivise equivalent agencies in other jurisdictions or subsidiary agencies to ‘not be left out’ and to represent their constituencies.

A review team needs to reconcile different priorities and perspectives held by wider groups of relevant stakeholders, often with equivalent or overlapping remits [see Reconciling different priorities/perspectives (consultative issue)]. Although this could present a challenge to scoping, it may also secure greater engagement, overall, by intentionally using a transparent and collective/group engagement process. Such engagement could be achieved by meeting as a stakeholder group or, as in the review of regulation and inspection of integrated care, by sharing individual stakeholder discussions with the wider group of stakeholders. Engagement could also be achieved by ensuring that stakeholders understand that the initially specified questions may not be the ones that best serve user needs.

In contrast, the realist review of access to social care for ethnic minority and LGBT+ populations (i.e. case study 4) had a single identified stakeholder (the DHSC) and was conducted within the context of a scope that was evolving throughout the review process. Here, early and ongoing engagement seemed to be aided by the DHSC’s close involvement in shaping the core concepts that would provide the ‘anchors’ or ‘compass’ of the review. Similarly, as the review progressed, the DHSC were closely involved in discussing and shaping the emerging programme theories. However, it is unclear from this case study whether the promise of involvement throughout a review process acts as an incentive or a disincentive to close involvement in scoping and question formulation. Conceivably, there may be some policy or service stakeholders who invest time in the early stages of scoping and question specification in the hope that this will mean a need for less subsequent engagement and effort. Overall, it seems that factors that determine the strength of stakeholder engagement will be varied, complex and context specific, and may exist at an organisational level or be steered by the interest and commitment of individual policy-makers or commissioners.

Educating the end-user about synthesis process and products (consultative issue 4)

A related consultative issue in many review scoping processes is how the review team explains to stakeholders about review methodology and the choice of different potential synthesis outputs produced (e.g. given different prioritised questions or different available evidence). This closely relates to the technical scoping issues of choosing the different methods of synthesis (technical issue 1) and balancing the goals of mapping, scoping or reviewing previous research (technical issue 3). This dialogue also strongly influences the consultative issue of managing stakeholder expectations [see Managing stakeholder expectations (consultative issue 5)].

Most service, clinical professional or policy stakeholders are most aware of systematic reviews that use quantitative research, such as clinical trials, to produce integrative syntheses of evidence, for instance overall summaries of effectiveness of well-defined interventions. The review of access to social care for ethnic minorities and LGBT+ people (i.e. case study 4) involved working with the DHSC to show how a realist review might use diverse sources of research evidence and provide richer explanations of how ‘access’ worked or did not work for different groups. More generally, it involved conveying to the end-user that not all types of review have to be prespecified and that the concepts guiding them can evolve during the review, including studies that can expand to enable relevant explanations (in this case, by including refugees and asylum seekers).

Another recurrent feature of HSDR reviews experienced across the teams, and one the stakeholders needed to be aware of, was the two- or three-stage nature of many review projects. They would begin with searches and a type of rapid scoping review, and then work with stakeholders to reassess what kind of synthesis, or perhaps just mapping or description, would be feasible and useful on the basis of the number and types of studies known to exist [this common sequencing is further explored as Mapping versus scoping versus reviewing (technical issue 4)].

Managing stakeholder expectations (consultative issue 5)

This theme captured how the review team communicates what the review project will and will not be able to achieve within the available resources and time frame, particularly when the deliverable is likely to ‘fall short’ of the conventional systematic review goals (e.g. of providing a summary answer to effectiveness or cost-effectiveness questions).

A key expectation is often around the fixed or fluid nature of the review question itself [see Balancing fixed versus fluid questions/components/definitions (technical issue 2)]. In the realist review of access to social care for ethnic minority and LGBT+ groups (i.e. case study 4), uncertainty related to the definition of key terms and concepts, and in the recognition that the review would have to refocus at a later stage (in this case, in a realist review, when a smaller set of programme theories are chosen as the basis for the rest of the review). In addition, with realist reviews, the synthesis and end product comprises a set of refined and empirically grounded programme theories.

It is important to have early conversations with stakeholders about the variety of possible synthesis methods, their relative suitability for answering different types of review question and the expected content and format of their findings [see Choosing the methods of synthesis (technical issue 1)]. At the same time, the review team needs as much clarity as possible about the other PICOS elements that define the boundaries of a review’s scope [i.e. population, intervention (phenomenon), comparator, outcomes and study type]. Perhaps the basic strategy for managing expectations about a review is to develop the review protocol iteratively and collaboratively with the main stakeholders or user.

Identifying the niche/gap and optimising added value (interface issue 1)

This theme captured how the review team positions the intended synthesis product within previous literature or reviews and in addressing users’ specific needs. This builds on the principle that a new systematic review adds most value in situations where:

1. the potential review question matches or is very similar to a current and important policy or health service question

2. a sufficient number of primary research studies have directly or indirectly generated evidence to answer the potential review question

3. no recent, high-quality systematic review has sought to answer the same or a very similar review question.

Where any one of these three conditions is not met, the value of a new, high-quality systematic review is significantly diminished. Many scoping processes can be viewed, in retrospect, as an iterative process of trying to establish these conditions. Nevertheless, the central interaction between the review team and the review users is based around condition (a), whereas conditions (b) and (c) are technical and data driven and do not require much input from review users.

The review of access to health care for adults with IDs (i.e. case study 8) illustrated the complexity of building on or filling gaps between previous systematic reviews. Although, ostensibly, the earlier systematic review had addressed a similar question and ‘simply’ needed updating, concepts and terminology relating to ID and to accessibility had changed considerably. This meant that search strategies had to be redesigned and the scopes of the earlier and the new review carefully reconciled. In addition, the methods and, by current standards, the quality of much older systematic reviews may mean that a review update that replicates the methods of the earlier review would be neither useful to stakeholders nor appealing to academic review teams.

In situations where a recent high-quality systematic review exists, the implied gap or opportunity for justifying a new systematic review may lie in asking different questions within the same topic (intervention, population and outcome) boundaries inhabited by a documented body of research evidence. For example, in the review of interventions to reduce preventable hospital admissions (i.e. case study 7), the earlier (2012) systematic review of effectiveness by Purdy et al. 45 showed where and in which specific patient groups there was sufficient evidence of effectiveness for a realist review to add value. Apparent similarities between previous and currently planned reviews, or apparent gaps/differences between high-quality systematic reviews, requires careful exploration to confirm the existence of genuine gaps and a clear need for an updated or new systematic review.

The review of digital innovations in primary care (i.e. case study 1) illustrated challenges related to identifying the gap/niche and meeting user needs. In this case, the user (i.e. NHS England) had identified numerous needs (nearly 20 questions within four themes), which led to an iterative, multistage approach with interim evidence synthesis products to inform potential next stages. This included an initial report, presenting a scoping review, which helped to focus the final stage synthesis on a narrower and more manageable set of prioritised questions.

Finally, both review teams and review users should be open to the possibility that no clear niche or gap might usefully be filled by a new evidence synthesis within a given topic. Such a situation exists when no primary research addresses the main questions and uncertainties of the policy customer, or where such primary research does exist and has already been summarised and synthesised in a high-quality systematic review. This issue links to earlier consultative issues on managing stakeholder expectations (consultative issue 5) and educating the end-user about synthesis processes and products (consultative issue 4).

Balancing rigour/reliability (reproducibility)/relevance (interface issue 2)

This theme captured how the review team manages potentially competing tensions of scientific quality, confidence in the review output and utility to the intended users within the constraints of remit and resources.

The review of digital innovations in primary care (i.e. case study 1) showed how meeting user’s needs might involve departing from conventional standards of academic rigour and reporting. They chose to meet their user’s needs not by conducting a ‘full systematic review’, but, instead, by adopting systematic review methodology wherever feasible. This decision ultimately meant that their evidence synthesis products were also less conventional, going beyond the summary and synthesis of previously published research within a written report. For example, they produced an interactive resource (i.e. a type of evidence map) for identifying relevant evidence, alongside information on ongoing relevant research and links to academic researchers active in the fast-changing area of service innovation.

Similarly, the evidence map about access to health care for adults with IDs (i.e. case study 8) traded-off the quantity of literature covered against data validity. The team constructed their evidence map using evidence from study titles and abstracts, without verifying the abstracts against the full text of the included sources. Such trade-offs are common when seeking to achieve timely and deliverable review products that still have credible quality.

Transferability/applicability of study evidence to the policy/service/service user context (interface issue 3)

This theme captured how the review team manages the need to provide UK-specific interpretation from evidence bases drawn from other countries and contexts. A key and commonly negotiated choice, evident in several of the case studies, relates to whether to include international evidence or restrict to studies conducted in UK health-care and social care settings.

Case studies 2 and 5 both identified specific policy customers and a phenomenon of interest that was explicitly based in legislation or the application and use of legislation (respectively, the MHA’s provisions for involving trusted relatives or friends in compulsory detention16 and the strengths-based working by social workers as promoted by the Care Act 201436). This meant that applicable evidence was likely to be jurisdiction specific, and studies from outside the UK were likely to be of limited relevance. Both of these reviews were commissioned within tight, externally imposed timelines (2 and 6 months, respectively), adding a further pragmatic reason to consider restricting to UK studies. However, in both cases, this presumption had to be discussed and confirmed with policy stakeholders. In case study 2, to inform reform of the MHA,16 interest in looking at international evidence was balanced by existing variation in both relevant legislation and in related evidence between the UK’s four devolved nations.

Other contexts where international evidence may have less applicability might include accessibility and patient experience, especially when linked to cultural, ethnic or demographic factors (as in case study 4, about access to social care for ethnic minorities or LGBT+ people). In contrast, it is often presumed that hospital-based care systems are similar across high-income countries and, therefore, evidence is applicable between them. For example, in case study 7 (a review of the implementation of interventions to reduce hospital admissions), evidence was sought from the major anglophone high-income countries: the UK, USA, Australia, Canada and New Zealand. However, both case studies (4 and 7) involved realist approaches to synthesis, in which the transferability of evidence between studies, and from studies to the intended review user’s setting, is achieved through identifying shared theoretical mechanisms across different contexts. Therefore, applicability and transferability of research evidence is not a static, inherent property of the evidence itself and where it has come from. Rather, it is judged in relation to the models of causality on which the particular synthesis methods are based. Consequently, some ‘configurative’ methods – like realist synthesis – require studies from diverse contexts to develop their outputs, which is in contrast to conventional ‘aggregative’ methods (like quantitative meta-analysis) that work best when between-study heterogeneity is minimal.

Choosing the methods of synthesis (technical issue 1)

This theme captured how the review team explores different options and makes an informed decision about which type of synthesis product will best meet the needs of the intended users. The steps to arriving at a particular method of synthesis varied considerably between the reviews in our case studies, as did the final type of evidence synthesis delivered. These eight case studies comprised effectiveness reviews (typically, narrative syntheses), realist reviews, scoping reviews, evidence maps and reviews of qualitative evidence. The outcomes of interest spanned effectiveness, cost-effectiveness, patient and carer experiences, accessibility and implementation.

In general, the starting point for choosing the method of synthesis was the review question, based on translating the policy customer’s evidence need into a specific outcome of interest (e.g. ‘what works?’ becomes ‘what is the effectiveness of . . . ?’). Within the review question, the explicit or implied outcome of interest was a key determinant of whether quantitative or qualitative evidence was sought and synthesised. For example, in case study 6, the study of hospital-led approaches to reduce length of stay for planned admissions, the review questions were framed around evaluating the clinical effectiveness and cost-effectiveness of alternative approaches. These questions can be validly and reliably answered using only quantitative comparative evaluations, like RCTs, and evidence synthesis methods, like narrative synthesis alongside meta-analysis, typically using an integrative approach to synthesis (i.e. ideally, to produce a generalisable summary answer to the question).

In case study 1, to synthesise evidence on Digital First Primary Care, the questions were so broad and the potentially relevant research so diverse, that a thematic synthesis approach was adopted, largely to accommodate multiple questions and types of evidence. The York team described this as not being a full systematic review, but a process of transparently identifying, critically appraising and summarising the research evidence relating to these questions.

The second main consideration that drove the choice of synthesis method was the quantity and design of available primary research studies. For example, in case study 2, the focus was on patient and carer experiences (of the ‘nearest relative’ provisions) and the scoping searches mainly identified qualitative research studies. Therefore, some type of rapid/pragmatic qualitative synthesis was required. That the synthesis method cannot always be rigidly specified at the protocol stage is further underlined by case study 4. The Sheffield team reflected that ‘The choice of review synthesis approach is not a decision to be made once the scope has been finalised – it represents an integral part of the scoping process (PICOSS)’ and this underlines a wider blurring of the conventional separation of scoping and protocol agreement and commencement of the systematic review [see Scoping as a continuous process through to shaping the end product (technical issue 5)].

Conversely, certain approaches to evidence synthesis are seen as too labour-intensive and time-consuming to do well in the context of rapid and policy-responsive review teams, notably meta-ethnography for reviewing qualitative evidence (although rapid meta-ethnographies have been conducted). In addition, of the eight case studies presented here, only one (i.e. case study 6, on strategies for reducing hospital length of stay in older people) addressed an effectiveness question where a sufficient number of high-quality randomised studies existed for meta-analysis to be feasible. However, even in this systematic review, the meta-analyses included only a smaller subset of all studies comparing the relevant alternatives.

Balancing fixed versus fluid questions/components/definitions (technical issue 2)

This theme captured the extent to which the question as a whole and/or its individual PICOS components are predefined or if they emerge during exploration of the literature. This balancing act seems to be a dominant trade-off for responsive reviews in the health and social care services and policy field, as exemplified in seven of our eight case studies. This technical issue is linked to the consultative issue of managing stakeholder expectations (consultative issue 5) and the technical issue of determining which review activity, of mapping, scoping and reviewing, will take place. Responsive, rapid reviews for policy-makers are multistage and, unlike RCTs, offer valuable opportunities for review and interim revisions of aims and methods.

A key example of this balancing act is case study 6, in which the plans for evidence synthesis had to be significantly adapted when > 100 effectiveness studies were identified (according to the original inclusion criteria). This led to a revised plan for evidence synthesis that sought to balance rigour and applicability. The review included two evidence syntheses: one of comparative effectiveness studies from the UK only (but of all high-quality comparative designs) and the other of studies from any high-income country, but RCTs only. This hybrid approach meant that review users interested in applicability to the UK could use the first synthesis and those interested in the most scientifically rigorous (internally valid) evidence across diverse contexts could use the second synthesis. As a result, the review avoided the effort of having to data extract, quality assess and synthesise the less rigorous evidence from beyond the UK. The realist review of access to social care for ethnic minority and LGBT+ populations (i.e. case study 4) was also conducted within the context of a scope that evolved throughout the review process.

In short, although review protocols are drafted and initial inclusion criteria and synthesis plans are prespecified (and even published, e.g. in PROSPERO), it is acknowledged that they will likely change. Moreover, even where the question remains fixed, learning and clarity about key concepts and terms within it are expected to emerge throughout the review. These technical aspects, therefore, go hand in hand with interface and consultative issues as the evolving protocol for the review is transparently revised, especially in consultation with the ultimate policy end-user.

Taking stock of (and building on) what studies exist (technical issue 3)

This theme captured how the review team explores the quantity, quality and characteristics of primary studies and/or reviews in determining which output will be both feasible and useful.

Sometimes, as in case studies 7 and 8, a key previous review offers a starting point for the new review. In the review of interventions to reduce preventable hospital admissions, the effectiveness review by Purdy et al. 45 provided a useful sampling frame of medical conditions on which the realist review might focus in depth. In contrast, in their review of access to care for people with IDs, the much older review by Alborz et al. ,49 ostensibly on the same topic, did provide some reusable resources (such as an interview schedule). However, it largely seemed to serve as a marker of how concepts of disability, notions of access and frameworks for understanding the performance of health services have all advanced significantly since 2003, and that the focus and intended value of the new review had to be carefully articulated.

In other reviews, taking stock is directly a process of assessing the diversity and quantity of available evidence and tailoring the planned synthesis to make the best of what evidence is available, in relation to the review’s questions. However, these decisions necessarily occur once searching and screening is mostly complete and so may not strictly constitute the topic-scoping phase.

Mapping versus scoping versus reviewing (technical issue 4)

This theme captured how the review team manages the relationship between exploring the characteristics of the evidence base (mapping), determining the parameters of the specific planned synthesis (scoping) and conducting the synthesis (reviewing), and the extent to which these processes transform into discrete project deliverables. This common three-stage sequence of responsive reviews is partly a response to the risks of overspecifying the planned final evidence synthesis at the beginning (and also relates to technical issue 2 of balancing fixed vs. fluid questions). 11 Although they are conceptually useful, we recognise that, in practice, scoping, mapping and reviewing are processes that necessarily interact and often overlap, as well as being labels applied to review outputs that may signal a formal end or transition of a stage in a continuous process.

Scoping as a continuous process through to shaping the end product (technical issue 5)

This theme captured the extent to which the scoping process is used as an opportunity to precondition the users to the content and form of the final synthesis product. Through their potential engagement, the scoping process offers an opportunity for commissioners and stakeholders to have an early influence on tailoring the final product to their requirements

Therefore, in case study 7, the focus on cardiovascular and respiratory conditions was determined not only by evidence on effectiveness, but also from the commissioner perspective, as articulated by the NIHR programme staff. For the review team, it is also part of a risk-averse strategy that minimises the chances of producing a product that is neither useful nor appropriate. In case study 4, the review team shared a previous similar realist review of access to care for a different marginalised group (i.e. older people in rural areas) to show the end-users the kind of findings that a realist review can produce [similar to the earlier consultative issue of ‘educating the end-user about synthesis process and products’ (consultative issue 4)].

Choosing between general and specific and broad and deep focus (technical issue 6)

This theme captured how the review team decides whether to cover an entire topic or to select one or more subtopics as exemplars of the whole, and the extent which they optimise coverage compared with detail. This includes decisions about whether to primarily describe and not to synthesise studies (as in an evidence map), or to also critically appraise and analyse included studies within a formal synthesis (e.g. a framework synthesis or narrative synthesis).

In the review of experiences of strengths-based approaches to social work (i.e. case study 5), this decision involved a choice between reviewing evidence explicitly about strengths-based practice and working itself, although it is a nebulous and loosely bounded set of principles and concepts, and ‘deconstructing’ or ‘representing’ strengths-based working as a collection of specific programmes and initiatives that are widely seen as fostering or exemplifying strengths-based working. Through tabulating documentary sources and consulting with diverse stakeholders, we ultimately identified 17 subsidiary approaches and programmes that were seen as fostering strengths-based working. Although this made searching for named programmes more straightforward, it did, however, risk treating the 17 approaches/programmes as equally strengths based in approach (i.e. not appreciating the substantial differences between them).

The review of interventions to prevent avoidable hospital admissions (i.e. case study 7) also showed the challenges to review teams of selecting a representative and manageable body of evidence to synthesise from a larger whole. The challenge in this case lay in the plentiful ACSCs and the solution was to choose a smaller selection to be reviewed in detail as useful exemplars for understanding others (e.g. cardiovascular and respiratory conditions). These two conditions had been sufficiently explored, and adequately supported as effective, to make these an appropriate focus for a realist inquiry.