Randomised controlled trials for policy interventions: a review of reviews and meta-regression

Defining ‘policy intervention’

The focus of our investigation was on evaluations of interventions for public policy or service organisation and management that:

• are intended to serve communities or populations

• require more than the efforts of individual practitioners to apply

• are not a one-to-one service.

We have adapted the definition of public health interventions provided by Rychetnik et al. 9 In order to embrace broader public policy, this definition of interventions is paraphrased as:

a set of actions with a coherent objective to bring about change or produce identifiable outcomes. These include policy, regulatory initiatives, single strategy projects or multi-component programmes. Policy interventions are intended to serve communities or populations. They are distinguished from one-to-one services that are for the benefit of individuals.

These interventions require more than the efforts of individual practitioners to be applied. They may include legislation or regulation; setting of policy or strategy at the level of national or local government, or institutions; the provision or organisation of services; environmental modification; or facilitating lay or public delivered support/education. These interventions may fall within public policy for health, education, social care, welfare, housing, criminal justice, transport and urban renewal. 10

Another interpretation of the term ‘policy intervention’ refers to intervening in policy making rather than policy intervention. Devlin et al. 11 considered the parameters of policy-making interventions in relation to service user perspectives on HIV policy. To paraphrase them:

policy interventions seek to influence decision-making . . . and ensure that policy supports or at least does not impede [services]. These interventions relate therefore to local and national policy makers (within governmental and statutory sectors) and local and national resource allocators (for example government departments and local authorities). They can also involve seeking to influence those people or agencies charged with the production and supply of information to support policy development and resource allocation. Therefore, they might also seek to influence applied and academic social researchers, epidemiologists, policy advisors and local public health surveillance personnel (collectively called, the research and policy community).

Examples of interventions that impact on policy-makers and seek to influence their (drafting) policy and legislation might include:

• lobbying government departments, local authorities, research bodies

• taking part in national consultation processes undertaken by policy and lobbying organisations and government

• joining professional associations, research/policy forums

• applying for funding from local authorities or government bodies

• subscribing to information sources of national policy-makers and lobbyists.

This distinction between collectively effecting change through setting policy, and collectively effecting change through implementing prior policy decisions is apparent in the policy analysis literature. 12 Harrison12 describes policy as a process, rather than simply as an output of a decision, or an input to management. The policy process begins within the arena of political science with setting agendas around problematic issues, and progresses to designing and evaluating efforts to solve these problems. Thus, a ‘policy intervention’ may be either a method for influencing the policy-making process, or a method for influencing the policy implementation process.

Randomisation and effect sizes of clinical interventions

The RCT is widely regarded as the design of choice for evaluating the effectiveness of clinical interventions in health care, as it can provide the most internally valid estimate. The main benefit of the RCT is the use of a randomisation procedure that, when properly concealed, ensures that the subjects receiving the treatment and control are equal with respect to all conditions except for receiving the treatment or the control. With sufficient sample sizes, and a truly random generation of the allocation sequence, comparison groups should on average be equal with respect to both known and unknown prognostic factors at baseline. 34 RCTs also have written protocols specifying, and thus standardising, important aspects of participant enrolment, intervention, observation and analysis. 35

Our knowledge of the importance of certain design features of RCTs has been derived primarily in the field of clinical health-care interventions. 2,36,37 Meta-epidemiological techniques have successfully been used to investigate variations in the results of RCTs of the same intervention according to features of their study design. 38 Substantial numbers of systematic reviews of RCTs have been identified, and results compared between the trials meeting and not meeting various design criteria such as proper randomisation, concealment of allocation and blinding. These comparisons have then been aggregated across the reviews to obtain an estimate of the systematic bias removed by the design feature. 2,37 The results have been shown to be reasonably consistent across clinical fields, providing some evidence that meta-epidemiology may be a reliable investigative technique. 39

The use of meta-epidemiology has also been extended from the comparison of design features within a particular study design to comparisons between study designs. A recent Health Technology Assessment report reviewed eight such examples:40 seven considered medical interventions, while one considered psychological interventions. The conclusions of these reviews varied, partly due to variations in their methods and rigour but also because of limitations in the meta-epidemiological methods used. The only robust conclusion that can be drawn is that in some circumstances the results of randomised and non-randomised studies differ, but it cannot be proved that differences are not due to other confounding factors. The key lessons that can be learned from this work are:

• The identification and selection of comparisons of randomised and non-randomised evidence should be systematic. This will not overcome the problem of selective publication of primary studies (if studies with positive results are more likely to be published, regardless of design, meta-epidemiological reviews will find designs showing intervention effects in the same direction if not of similar magnitude) but should at least ensure that all available comparisons are included regardless of whether designs show similar or conflicting results.

• To reduce confounding from factors other than lack of randomisation, randomised and non-randomised studies should be assessed for differences in the participants, interventions and outcomes. The possibility of temporal confounding of study types (NRSs typically being performed prior to the RCTs) should also be assessed.

• The similarity of randomised and non-randomised studies should be assessed for differences in study methods other than allocation. Discrepancies and similarities between study designs could be partly explained by differences in other unevaluated aspects of methodological quality of the RCTs and/or the NRSs, such as blinding or intention-to-treat analysis.

• Sensible, objective criteria should be used to determine differences or equivalence of study findings as these can have a large influence on the conclusions drawn. The amount of data available is also important; for example in one review, for each intervention five RCTs on average were compared with four NRSs. Hence the absence of a statistically significant difference cannot be interpreted as evidence of ‘equivalency’, and clinically significant differences in treatment effects cannot be excluded. 40

These previous investigations also suggest that there may be variability in the direction of bias introduced when randomisation is not used. Selection bias is commonly thought of as resulting from the systematic selection of either high or low risk participants to receive an intervention. This would lead to the intervention group being ‘heavily weighted by the more severely ill’41 or alternatively including those least likely to suffer adverse consequences from an intervention (less severely ill). If in fact selection bias arises due to haphazard variations in case-mix, there will be a mixture of under- and overestimates of the treatment effect. The results might all be biased, but not all in the same direction. 40 In these circumstances, an increase in the heterogeneity of treatment effect (beyond that expected by chance) rather than (or as well as) a systematic bias would be expected. Deeks et al. 40 suggest that a formal statistical comparison should aim to compare the heterogeneity in treatment effects, and not just the average treatment effects between randomised and non-randomised groups.

Randomisation and effect size of policy interventions

The effects of policy interventions have been assessed through the use of RCTs, nRCTs and other study designs. The choice has been influenced by the relative rigour of the designs and the feasibility in the circumstances of applying prospective designs and random allocation of interventions. The weight given to each of these influences (rigour and feasibility) when embarking on policy evaluations may be driven by philosophy, as much as by research evidence.

Although studies largely from clinical areas have identified detailed design features of rigorous RCTs that reduce systematic bias in estimating effect sizes,39 meta-epidemiological investigations of medical and psychological interventions concluded that it is less clear what influences the differences in results drawn from randomised and non-randomised studies, as results of NRSs sometimes, but not always, differ from results of randomised studies of the same intervention. 40 There is growing evidence in the meta-analytic literature that even strong quasi-experimental designs assessing criminology are more likely to report a result in favour of treatment and less likely to report a harmful effect of treatment than randomised studies. 42

Chapter 2 considers the methodologies appropriate for investigating the extent and possible causes of such differences.

Prior research

From the outset we were aware of a similar methodological study by Deeks et al. 40 This study did not have any inclusion or exclusion criteria regarding type of intervention, other than they had to have ‘intended effects’. Two chapters of that report are particularly relevant to our work:

• Chapter 3: a review of eight ‘meta-epidemiological’ reviews that reviewed comparisons of RCTs and NRSs in which the original authors had specifically set out to examine the similarity/differences in results according to randomisation. Deeks et al. 40 reviewed interventions that were almost exclusively therapeutic in nature (i.e. aimed to treat or cure disease), but a handful related to the organisation of care and to educational interventions and were eligible for our study.

• Chapter 5: a review of existing systematic reviews that included randomised and non-randomised studies. This chapter included ‘policy interventions’ as well as clinical/medical interventions. We have already drawn on the reviews within this chapter to outline the range of ‘policy interventions’ for this study. The study reported here extends the work of Deeks et al. 40 by comparing the results of randomised and non-randomised evidence from a wider range of policy interventions.

Resistance to randomised controlled trials

As the purpose of this methodological study is to resolve questions about how essential RCTs are in areas where they are less readily available, we anticipated finding relevant studies (randomised and non-randomised) in areas where there has been some but not complete resistance to RCTs. 14,43,44 These include circumstances in which:

• it is difficult to stop contamination between intervention group(s) and control(s) (e.g. community wide interventions)

• benefit may derive in part from an individual or group actively seeking to participate in the particular intervention (e.g. peer support provided by patient organisations)

• randomisation is not feasible (e.g. legislation)

• interventions are multicomponent (e.g. a combination of health service initiatives, face-to-face health education in schools and in the community plus mass media).

Within this study we shall explore whether differences other than the presence or absence of randomisation could account for any variation that might be found in results, and comment on the extent to which resistance to RCTs is justified.

Complex interventions and their relationship with policy interventions

Although not directly relating to policy interventions, the Medical Research Council (MRC) framework for the development and evaluation of RCTs for complex interventions to improve health seemed to capture the essence of what we have been discussing. However, the MRC definition of complex interventions includes interventions that may be delivered by individuals (e.g. the different social/educational/treatment aspects of physiotherapy distinguish physiotherapy as a complex intervention) (see Appendix 1). We noted that in other less clinical areas, ‘multicomponent’ interventions was the term of choice, although this usually included multipractitioners too.

Level of policy making

Discussion distinguished policy interventions at different levels (national, regional, community and institution). These distinctions appeared to translate poorly to policy evaluations at these different levels. In particular it was noted that an evaluation of institution-wide policies may precede or follow national endorsement of a policy; the report may not clearly acknowledge which of these circumstances prevail, and whether it is the former or the latter may make little difference to the methodological challenges of evaluation. When a definition of policy intervention was applied to a set of evaluations (see below) it confirmed the observation above that social interventions as programmes, when evaluated, can inform policy, but that some ‘are’ policy. 7

Implementation and its relationship with policy

We envisaged many clinical interventions also being ‘policy’; for instance, prescribing aspirin following a heart attack. In order to avoid replicating methodological research in the clinical area, we distinguished between, for example, a trial of aspirin treatment for heart attack (a trial of a clinical intervention) and a trial of methods to encourage greater use of aspirin treatment for heart attack (a trial of a social or educational intervention to increase uptake), and included the latter but not the former. We included other similar interventions such as interventions to increase the uptake of vaccination or screening. This scope made reviews conducted by the Cochrane Effective Practice and Organisation of Care review group particularly relevant.

Developing operational definitions

Applying operational definitions

Overall, the definitions and categories as described above (see Developing operational definitions) have proved possible to apply in a way that is consistent between two reviewers working independently. Limitations to the work done so far include characteristics of the studies appraised – EPPI-Centre reviews tend to focus on policy level interventions as described here, and so the tests done so far can only have limited powers to test the discriminatory powers of these tools for including or excluding policy interventions. However, the distribution of types of policy interventions that appeared in each EPPI-Centre review varied, and discriminating between types of policy interventions appeared practical.

In developing our data sets we coded as policy evaluations those:

• in which there was an explicit directive/policy for the intervention OR

• beyond the capacity of individual providers (in terms of their roles/skills, resources, or authority).

These inclusion criteria match the focus of the commissioning brief on policy/management interventions which was ‘those interventions that are not confined to an individual practitioner . . . examples would include peer-led teaching and health promotion in schools’.

Explicit directives or policies include named policies such as national government legislation or programmes or, less formally, explicit collective action plans in which non-researchers had been involved in the decision-making. Collective action plans could be explicit either from descriptions of planning processes or from descriptions of the products of their planning processes, such as guidelines sponsored nationally or regionally by professional organisations or charities, or commercially available curricula.

Operating the second inclusion criterion requires a judgement about the roles, skills, resources and authority of individuals. For instance, distributing fruit to children at school would be judged a policy intervention because it would be beyond the authority/resources of an individual practitioner on the grounds that we do not expect teachers to pay for it out of their own pockets and if the school were to have a budget for it, it must also have a policy for it.

In addition to inclusion/exclusion criteria, we anticipated being able to separately identify and analyse evaluations of interventions that operate at different levels (national, regional, community, or institutional level) and in different policy sectors (housing, transport, health, crime and justice, etc.) or across policy sectors.

Resampling studies data

We had access to two trials of policy interventions in which data were suitable for resampling studies.

Similar studies drawn from systematic reviews

We sought readily available systematic review data stored on EPPI-Reviewer, software for storing and analysing data about primary research for inclusion in systematic reviews. This source includes data from reviews of health promotion (conducted by or in collaboration with the EPPI-Centre published 1999–2004) and education (conducted by the EPPI-Centre or by review groups supported by the EPPI-Centre published before June 2004).

Of the 24 education systematic reviews, only seven included RCTs and NRSs; these included policy interventions of Interactive Communication Technology (ICT) for literacy (three reviews), out-of-home integrated care and education, paid adult support in mainstream schools, supporting pupils with emotional and behavioural difficulties in mainstream primary schools, and personal development planning for improving student learning. Between them they included 32 RCTs and 82 NRSs. This was considered too few studies to analyse further considering their diversity.

There were nine systematic reviews conducted by the EPPI-Centre between 1996 and 2004, and one Cochrane review conducted using EPPI-Centre software. These reviews all included both randomised and non-randomised studies. These reviews were of health promotion, with studies often conducted in educational settings. Between them they addressed workplace health promotion, peer-delivered interventions, mental health, physical activity (two reviews), healthy eating (two reviews), cervical cancer and sexual lifestyle; (nine reviews with a total of 206 studies). See Appendix 2 for summaries of these reviews.

Meta-epidemiological data

The studies described above, when combined as 206 controlled trials of health promotion, also provided a suitable data set for meta-epidemiological investigations.

Another data set suitable for this approach was also available from ongoing work by Colorado State University reviewing and synthesising the past 20 years of research and advancements in the area of transition for youths with disabilities (www.ncset.org/publications/viewdesc.asp?id=714). These data from 126 studies are also held on EPPI-Centre software.

Reviews of reviews

For a review level analysis, data were sought from methodological studies and systematic reviews of randomised and non-randomised studies using systematic search strategies, selection criteria and data extraction procedures.

Baseline characteristics

Groups in nRCTs may differ at baseline for a number of reasons. Recipients of the intervention may have self-selected, or those who declined to participate may have been assigned to the control/comparison group. Alternatively, recruitment may have favoured those most amenable to participation or those in most need, or excluded older people or those with comorbidities. Well-conducted RCTs, with their standardised procedures for recruitment and data analysed according to the intention to treat rather than receipt of the intervention, are more likely to have more equivalence between groups. Non-equivalence at baseline may influence the calculated effect size and variance.

Attrition

Attrition rates may be linked to the quality of the evaluation. Higher attrition may be expected in community and home settings than in organisational settings where it is easier to employ randomisation and good follow-up. High attrition may also be associated with losing a disproportionate number of people who are socially disadvantaged and more resistant to interventions, and hence lead to a misleadingly large effect size. 49,50 For this reason, high attrition may be associated with both lack of randomisation and higher effect sizes. Potential technical solutions in primary research include greater investment in recruiting and retaining participants, perhaps using the NHS number for tracking. Other solutions for primary studies and systematic reviews include adjusting for attrition, assuming those lost to follow-up have poor outcomes.

Theoretical underpinnings

Policy interventions pose serious challenges to evaluations of effectiveness because they may be large and difficult to replicate consistently, and have diffuse boundaries. Evaluation methodologies differ in their responses to such challenges. RCTs, and systematic reviews of RCTs, are the methods of choice employed by public health physicians wishing to elucidate causal effects in variable circumstances. These methodologies emphasise the need to reduce bias by employing randomisation, preferably with blinded allocation to treatment and outcome measurement, minimising attrition, and analysing according to the intended treatment. Examples include community interventions for preventing smoking in young people,51 computerised support for prescribing practice52 and day care for preschool children. 53

By contrast, many social scientists are known to employ different theories from experimentalists evaluating policy interventions. The relatively new profession of health promotion specialists21 has favoured ‘an approach to evaluation that implicitly acknowledges the need for outcome data but explicitly concentrates on process or illuminative data that helps us understand the nature of that relationship’. 54 The centrepiece of the health promotion paradigm is the concept of empowerment – enabling people to increase control over, and to improve, their own health. Empowerment claims to attribute responsibility to people not for the existence of a problem, but for finding a solution to it. The goal is then ‘full and organised community participation and ultimate self-reliance’. 55 This approach is endorsed by Arblaster56 in a systematic review of the effectiveness of health service interventions aimed at reducing inequalities in health. The review concluded that characteristics of successful interventions specifically aimed at reducing health differentials include ensuring interventions address the expressed or identified needs of the target population, and the involvement of peers in the delivery of interventions. The tradition of community development rests heavily on public involvement, and we expect community-based interventions to include the public more often in identifying the aims of the intervention, and/or participating in its development. Examples include impact evaluations of a large-scale social marketing initiative to encourage fruit and vegetable consumption57 and of bar-based, peer-led community-level intervention to promote sexual health among gay men. 58 With this understanding we anticipate evaluations of community development to be more theoretically informed from the tradition of social science, and less subject to randomisation. This expectation is supported by a comparison of the CONSORT (Consolidated Standards of Reporting Trials) statement (a checklist and flow chart, to help improve the quality of reports of RCTs) and the TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) statement. The TREND statement, unlike the CONSORT statement, seeks information about theories used in designing behavioural interventions. 36,59

In summary, the public health approach has tended to emphasise randomisation but not public involvement or community-based approaches, compared with health promotion where the reverse is so; with the expectation within public health that randomisation leads to more conservative estimates of effect size and the expectation within health promotion that public involvement leads to interventions with greater effect sizes.

Similarly, divergent views about appropriate methods for evaluating interventions are found in the areas of social welfare44 and education25 where The Centre for Evidence-Based Social Services (www.ex.ac.uk/cebss/introduction.html) and the Evidence-based (www.cemcentre.org/ebeuk/) Education Network UK stand out from many of their British professional colleagues in social welfare and education, respectively, as advocates for randomised evaluation.

Setting and boundaries of the intervention

In the area of public policy, community-wide interventions, or regional/national interventions may pose challenges in being less easily manipulated for the purposes of evaluation than individual or institutionally based interventions. Standardised implementation of interventions may be more difficult across large communities, regions or whole countries than in single organisations and therefore may be less effective or more variable in effectiveness.

Interventions with a broader reach (communities, regions, nations) have more diffuse boundaries than those set within institutions. Randomisation is less often applied to community, regional or national interventions. Clustered trials are more appropriate for these and some organisational level interventions, in order to reduce the likelihood of participants experiencing comparison interventions to which they have not been allocated. However, clustering reduces the power of a trial, so clustered evaluations are less likely to show statistically significant effectiveness. The solution is to increase the size of the trial, yet recruitment can be particularly challenging in community settings. Moreover, attrition may be greater in larger scale interventions, where tracking of individuals is more difficult than within an organisation (see Attrition).

Providers of the intervention

Community development and peer delivery specialists value health promotion theory and process evaluations more than RCTs. These interventions may therefore be found to have less randomisation. Theories underpinning community development and peer delivery anticipate more effective interventions through their greater relevance. Characteristics of interventions effective for reducing health inequalities include community commitment and peer delivery. 56 Examples in the area of smoking cessation include the non-randomised evaluation of the Wessex Healthy School Award where the intervention was delivered by the school community. 60

In contrast, clinicians design and deliver their own interventions and evaluations, and work in a culture that favours randomisation; for instance, many of the smoking cessations’ interventions for pregnant women are delivered by health professionals and evaluated by RCTs. 61

Also, working in the area of criminology, Lipsey and Wilson62 have shown a statistical association between randomisation and effect size in ‘demonstration’ projects in which the researcher had greater control of both the intervention and randomisation. Our data set of health promotion evaluations provides an opportunity to test this association in another area.

Choice of outcome domains

The impact of health education has traditionally been considered in terms of changes in knowledge, attitudes, behaviour and health. Kirkpatrick’s63 hierarchy of outcomes from the policy area of professional training presents the higher level outcomes (health and behaviour) as harder to attain than lower level outcomes (knowledge and attitudes). The choice of outcomes may be strongly influenced by the intervention setting. Other broader health behaviour theories present knowledge and positive attitudes as necessary but not sufficient for improved behaviour and health in many theories of health behaviour. 64 Thus there is support from these two different policy areas of professional training and health behaviour change for the argument that outcomes in the domains of knowledge, attitudes, behaviour and health are successively more difficult to influence and therefore associated with lower effect sizes.

The choice of outcomes may be strongly influenced by the intervention setting. For instance, the measurement of any health outcome may be easier in a clinical setting where randomisation is also more readily acceptable by staff. Following this argument, evaluations with health outcomes are more likely to be associated with patient populations than community populations. For instance, generating evidence about smoking cessation in pregnancy lends itself to short-term health outcomes such as birth weight, gestational age at birth, perinatal mortality, method of delivery, and measures of anxiety, depression and maternal health status in late pregnancy and after birth as seen in a systematic review of 64 RCTs (51 RCTs and six clustered RCTs). 61 Such data can be easily collected in a clinical setting where randomisation is also feasible. Similarly, a review of smoking cessation for hospitalised adults where nine of the 17 included studies (16 RCTs, one quasi-RCT) measured death of the patient as well as abstinence from smoking. 65 Both these reviews included randomised or quasi-randomised trials.

In contrast, a review of community interventions for preventing smoking in young people51 found that over half the controlled trials were non-randomised, and outcomes were restricted to knowledge about the effects of smoking, attitudes to smoking, intentions to smoke in the future and smoking cessation.

Choice of outcome measures

The choice of particular outcomes or measures may be associated with randomisation, because some outcomes are more feasible in a clinical setting where randomisation is also readily accepted. For instance, both cluster or individual randomisation and the use of clinical outcome measures requiring blood tests may be difficult to impose elsewhere. For example, cotinine tests as markers for smoking cessation may be more common in clinical settings, and provide smaller effect sizes than unconfirmed reports of non-smoking. For instance, only one study out of 17 (6%) in a review of community interventions for preventing smoking in young people51 used cotinine measurements to confirm non-smoking. Similarly, a review of community interventions for adults found 10 of 32 studies included serum thiocyanate analysis and one included cotinine analysis to validate smoking status, giving a total of 34% of trials with biochemical validation of smoking cessation. 66 In contrast, in a review of smoking cessation during pregnancy, when women often attend clinics or hospitals, 35 of 47 (74%) included studies with biochemically validated cessation as an outcome. 61 The implications of this are that ‘hard’ outcomes (in this case, validated smoking cessation) and randomisation are both easier in a clinical setting where clinical tests and access to medical records are easier. A parallel argument might be made in education sector evaluations where self-reported understanding may be less reliable than examination results, so assessing learning within schools may be more objective. In both cases, evaluating interventions within an institution where testing is routine and randomisation is easier leads to more objective findings.

Sample size

Larger sample sizes are more likely to be found in nRCTs or natural experiments that can provide convenience samples on a large scale. Smaller sample sizes, more commonly found in RCTs, are more likely to lead to spurious results, with those reporting positive findings more likely to be published.

Control group

Control groups are always found in RCTs, but only sometimes in NRSs. The lack of a control group may result in a misleading effect size as the study does not take into account possible simultaneous influences.

Blinding

Patients who know that they are on a new, experimental treatment are likely to have an opinion about its efficacy, as are their clinicians or the other study personnel who are measuring responses to therapy. These opinions, whether optimistic or pessimistic, can systematically distort both the other aspects of treatment and the reporting of treatment outcomes, thereby reducing our confidence in the study’s results. In addition, unblinded study personnel who are measuring outcomes may provide different interpretations of marginal findings or differential encouragement during performance tests, either one of which can distort their results. 67 Blinding of community interventions is more challenging, and therefore linked with other characteristics of community interventions such as fewer randomised evaluations.

Follow-up

Longer term follow-up may be easier within institutions, where randomisation is also easier. However, long-term follow-up exposes interventions to additional scrutiny in terms of sustainability or maintenance of effect, and may reveal declining effect sizes. Alternatively, long-term follow-up is also associated with greater attrition which in turn is associated with greater effect sizes.

Clustering

Clustered trials may be ‘natural experiments’ without randomisation, particularly if there are few clusters. Natural experiments may be more likely to have enthusiasts supporting the intervention, and non-enthusiasts supporting the comparisons, and therefore lead to greater effect sizes. This introduces bias from lack of blinding (see above).

Quality of reporting

The quality of reporting of some aspects of evaluation may be associated with researchers’ disciplines. In particular, triallists, who more often use randomisation, may also be more likely to report pre- and postintervention data. Natural experiments in particular may face challenges in collecting pre-intervention data because researchers have less influence. They may be invited to evaluate a policy intervention only after implementation has begun (for example, evaluation of Sure Start Plus, a UK Government pilot initiative to support pregnant young women and young parents under 18 years of age). 68 Although the CONSORT and TREND statements both encourage the reporting for baseline data for RCTs and non-randomised designs respectively, the CONSORT statement, introduced in 1996, has had much longer to influence the reporting of RCTs than the TREND statement, introduced in 2004, has had to influence non-randomised trials. Thus, a greater proportion of randomised studies than NRSs might be expected to report baseline data.

By influencing the reporting of studies, both statements have been able to influence the quality of the design and conduct of studies, and may be expected to reduce bias and, consequently, effect sizes.

Reporting of pre-and postintervention data precludes effect sizes inflated by differences between groups.

Type of studies

After sifting we identified one review of within-study comparisons of randomised and non-randomised participants,70,71 six single meta-analyses42,46,72–75 and one review of meta-analyses. 76 These eight studies form the basis of this chapter. Their scope and methods are described in Table 2.

We also identified nine single within-study comparisons of randomised and non-randomised participants, seven of which were included in the review of such comparisons by Glazerman et al. 70,71 and consequently are not discussed any further (see Methods of analysis). In addition, six small concurrent comparisons of randomised/non-randomised study participants were identified81–86 and a number of commentary papers were found. These weaker sources of evidence were not tabulated.

Scope of the studies

The scope of the studies varied in terms of the interventions included, the populations included and the methodological focus. The topic areas examined by the eight studies varied. They included interventions for preventing juvenile delinquency,42,46 treatment of alcohol abuse73,87 and psychological interventions. 74–76 There was an overlap with the health field, with Heinsman and Shadish72 including interventions in the mental health and health-care fields.

In terms of the type of interventions evaluated, there was a strong focus on psychotherapy and psychosocial programmes, partly reflecting the specialist interests of the authors, many of whom were common to more than one publication (For example, Heinsman and Shadish, 1996;72 Shadish, 1997;88 Shadish, 2000;75 Lipsey, 2003;46 Wilson and Lipsey, 200176). Some studies were broad in their inclusion of interventions, such as Weisburd et al. ,42 who included interventions aimed at communities, families, schools and labour markets.

In terms of scope, many of the studies had the primary aim of comparing effect sizes between randomised and non-randomised evaluation designs. In addition, some also examined the relationship between other modifiers and effects. For example, Shadish and Ragsdale74 also investigated the influence of sample size, attrition, type of control group, publication status and a number of other factors. In other studies the focus was broader, with randomisation only one variable of interest among many. For example, in the review by Lipsey46 [described in further detail in Reviews of meta-analyses of randomised and non-randomised studies (n = 6)], page 71 stated that the aim was to ‘illustrate the hazards and complexities of investigating moderator variables in meta-analysis’. Wilson and Lipsey76 in their review of meta-analyses [described further in Synthesis of meta-analyses (n = 1)] examined the association between a vast range of population, intervention and study, variables and effect size variance. Shadish et al. 75 [described in further detail in Reviews of meta-analyses of randomised and non-randomised studies (n= 6)] had a slightly different focus, examining the characteristics of studies judged to be ‘clinically representative’, in terms of relationship to study design and effect size (see Effect modifiers).

Where random/non-random allocation was the key issue investigated, some studies sought to delineate the influence of different types of NRS on study outcomes, rather than analysing all NRSs as one homogeneous group. For instance, Weisburd et al. 42 classified five types of study. These included (1) correlational studies where an intervention is measured in only one group; (2) studies where a clear temporal sequence can be observed between an intervention and an outcome; (3) studies in which one group receives an intervention, compared with another group that does not; (4) studies in which intervention and comparison groups are compared, with other mediating factors controlled for/or with a matched comparison group; and (5) RCTs. Type (1) was considered non-experimental, type (2) was a ‘stronger non-experimental/weaker quasi-experimental’, types (3) and (4) were considered quasi-experimental and (5) was an RCT. The authors reported mean effect sizes for each of these types, and also separately compared the quasi-experimental studies (3 or 4) and the higher quality quasi-experimental designs (4) with the RCTs (5).

The rationale and context for the studies varied. For example, Weisburd et al. 42 posed the question of whether the type of research design used to evaluate a crime and justice intervention influences its conclusions. Writing from the perspective of the Campbell Collaboration Crime and Justice Coordinating Group, they acknowledged the gold standard status of the RCT for assessing effectiveness, but noted the lack of well-conducted RCTs in the crime and justice area and that systematic reviews that restrict their inclusion criteria to RCTs may be unrealistic. Their central question, therefore, was ‘What are the potential shortcomings of including NRSs within systematic reviews?’. Specifically, ‘Are they likely to over or under-estimate the effects of interventions?’.

In contrast, Moyer and Finney87 were interested in the generalisability of RCTs in the field of alcohol treatment. They suggested that participants recruited into RCTs may be more likely to benefit than those who receive treatment under more typical ‘real life’ conditions, which tend to prevail in NRSs. The study therefore investigated the extent to which RCTs had been used in the field, whether participants differ between the two designs, whether the interventions are implemented differently between the two designs; and whether ‘post-treatment’ functioning of participants between the designs differ, controlling for differences in participant characteristics and other methodological features.

In summary, while one of the central aims of each study was to examine the influence of randomisation on intervention effects, each did so from a variety of different perspectives.

Methods of analysis

As discussed in Type of studies, three main types of study were used to examine the influence of design and other characteristics on intervention effects.

Effect modifiers

Selection of systematic reviews

Systematic reviews meeting the following criteria were eligible for inclusion:

• completed or published between 1999 and 2004 (limit applied to try and ensure the inclusion of reviews with up-to-date methods)

• evaluated a policy intervention (see Chapter 1, Defining ‘policy intervention’)

• included both randomised and non-randomised studies and have estimated intervention effects separately according to design (or provide sufficient data to allow us to do so)

• used quantitative synthesis (meta-analysis).

Reviews which either estimated intervention effects separately according to design but did not quantitatively synthesise the studies or used quantitative synthesis (meta-analysis) but did not estimate intervention effects separately according to design were excluded from the main analysis but are discussed briefly in Additional policy intervention reviews.

A flowchart of the inclusion process is presented in Figure 1. Each review potentially meeting inclusion criteria was screened by one reviewer using a predefined electronic form, and checked by a second reviewer. Disagreements were resolved by consensus, with reference to a third reviewer if necessary.

FIGURE 1.

Flow chart of the inclusion process.

Literature searches and data sources

There are probably even fewer definitive terms available for ‘systematic reviews’ or equivalent in the wider literature than there are for primary evaluation or trial designs. Ideally, to ensure that the searches retrieved relevant references, the strategy would have included terms for ‘review’ or at least for ‘literature review’. 89 However, attempts at searching using these terms on electronic databases (e.g. MEDLINE) produced an unmanageable number of references.

We therefore decided to search for all available references relating to policy interventions from databases that exclusively contain citations to reviews and systematic reviews. All reviews potentially relating to policy interventions available on the following databases were obtained: Centre for Reviews and Dissemination DARE (Database of Abstracts of Reviews of Effects); the Cochrane Database of Systematic Reviews (CDSR), the Campbell Collaboration’s Database, C2 RIPE (Register of C2 Systematic Reviews of Interventions and Policy Evaluation); the EPPI-Centre DoPHER; and the (former) Health Development Agency Evidence Base Database.

A number of test searches were then completed in databases with a focus beyond health to estimate the feasibility of attempting a comprehensive search of the non-health literature for systematic reviews. The searches were restricted to very specific terms for ‘systematic review’, without using proximity operators, and were further restricted by date range (2003–4).

The following free-text terms and indexed keywords (if available) were used: meta-analysis, meta-analysis, systematic review, systematic overview, collaborative review, integrative research, integrative review, research integration, narrative synthesis, evaluation synthesis, meta synthesis, realist synthesis, descriptive synthesis, explanatory synthesis and pool data.

The following databases were searched:

• ASSIA

• BEI

• CareData

• ERIC

• HDA HealthPromis

• PAIS International

• SIGLE

• SSCI

• Sociological Abstracts.

The results of the test searches confirmed that it would not be worthwhile conducting thorough, comprehensive searches of databases with a focus beyond health for systematic reviews. For the year 2003–4 alone, a precise search without ‘literature review’ identified 2494 records, and a more sensitive search with ‘literature review’ added identified 45,596 records. Full details of the search strategies can be found in Appendix 3.

Data extraction and analysis

A data extraction form for recording relevant information from each systematic review was designed and piloted. Full systematic reviews were pre-screened independently by two reviewers. Those meeting the inclusion criteria had data extracted by one reviewer and the completed data extraction forms checked against the full paper by a second reviewer. Any disagreements were resolved by consensus or by referral to a third reviewer if necessary.

The following information was extracted from each review:

• details of literature search used to identify studies for inclusion, including databases searched, years and whether details of the search strategy were available

• method of assessing studies for inclusion

• details about quality assessment

• number of studies/participants per design

• method of synthesis used

• results according to study design and other quality features.

Review methods and results were tabulated and discussed narratively. They were first classified into three groups according to the authors’ judgement regarding the equivalence or otherwise (‘similar’, ‘not similar’ or ‘mixed’) of the results of RCTs and NRSs. We focused on the following key methodological aspects:

• Whether authors attempted to examine similarities or differences in the following aspects across study designs (or whether they provide sufficient study details to allow us to judge): study populations; interventions used (design and provider of intervention); design of evaluation; outcomes assessed; study dates (if NRSs have largely been conducted before RCTs they may be more likely to show positive effects, i.e. their positive results leading to the RCTs being commissioned in the first place); and other aspects as recorded by review authors.

• Whether authors tried to assess heterogeneity either across or within study designs, using statistical methods or other approaches to identifying heterogeneity.

• What criteria were used to establish equivalence (or otherwise) of the results of RCTs and NRSs and whether these criteria were sensible and objective.

Details about these items were recorded so that any observed differences in the results of randomised and non-randomised studies could be considered and were not simply attributed to lack of randomisation.

Description of reviews

Sixteen reviews met inclusion criteria (Table 5 and Appendix 4).

Review results: where authors judged results from RCTs and NRSs to be ‘similar’ (n = 5)

Five reviews are included in this section: Cameron et al. ,99 Kwan and Sandercock,101 Langhorne et al. ,102 Tobler et al. ,107 and Wilson et al. 105 (Table 6, Appendices 4.1–4.13).

Review results: where authors judged results from RCTs and NRSs to be ‘non-similar’ (n = 8)

Eight reviews are included in this section: Cambach et al. ,98 Davis and Gidycz,90 Griffith et al. ,100 Jacobs et al. ,92 Mullen et al. ,93 Oliver et al. ,103 Smedslund et al. ,104 and Wilson et al. 96 (see Table 6, Appendices 4.5–4.7).

Review results: where authors judged results from RCTs and NRSs to be ‘mixed’ (n = 3)

Three reviews are included in this section: Guyatt et al. ,91 Thomas et al. ,94 and Wilson et al. 97 (see Table 6, Appendices 4.8–4.10). In these reviews, similarity and differences between results of RCTs and NRSs varied across outcomes.

Additional policy intervention reviews

Inclusion criteria

This investigation of the effects of randomisation in evaluations of policy interventions was not as straightforward as the investigation by Deeks et al. 40 for evaluating NRSs in health care. In their investigation many of the included reviews specifically aimed to investigate differences between RCTs and NRSs. If we used similar inclusion criteria to such reviews in the field of policy interventions, we would only have included reviews that were already included in the report by Deeks et al. 40 Most of the reviews included in our investigation did not have a stated intention of investigating differences between RCTs and NRSs. Most did not even have the stated intention of separating RCTs and NRSs in the analysis. Those that did have this intention did not always give a rationale for doing so. When a rationale was stated, it was either in order to separate more methodologically sound study designs (randomisation being one indicator of quality, but often not the only one), or to investigate potential moderators of effect. Potential moderators included randomisation and other features of study design, also aspects of the intervention, participants and outcomes measured.

Searches

Searching by study design is problematic even in MEDLINE: indexing of RCTs in MEDLINE by publication type and medical subject heading has improved in recent years but is still inadequate. Searching for NRSs is much more difficult; there are many study designs that could be classed as non-randomised and there is little definitive terminology. Comprehensive and consistent indexing according to study design is lacking. Databases beyond MEDLINE very often have poor indexing by study design, and these problems of definition become more pronounced in databases that are non-health related. Non-health databases in addition rarely have a thesaurus of keyword terms included; the records often lack an abstract and a number of databases only have rudimentary search capabilities. There are probably even fewer definitive terms available for ‘systematic reviews’ or equivalent in the non-health literature than there are for trial designs. Ideally, to ensure that the searches retrieved relevant references, the strategy would have included terms for ‘review’ or at least for ‘literature review’. However, attempts at searching using these terms produced an unmanageable number of references.

Results judged similar

Even in the reviews for which authors judged results from RCTs and NRSs to be ‘similar’, pooled results of NRSs tended to be more positive than RCTs in two of five reviews and more negative in one. Heterogeneity was assessed separately by design in three of the five reviews in this section: there was greater heterogeneity among RCTs in one review and among NRSs in another. Other potential confounders or moderators of effect in reviews in this section were population variables (three reviews) and intervention variables (two reviews). The two reviews that carried out secondary analysis of moderators of effect including randomisation concluded that other potential sources of bias influenced the results of the review more than randomisation.

Results judged not similar

In the eight reviews for which authors judged results were ‘not similar’ between RCTs and NRSs, six found that NRSs had more positive results than RCTs and two found that RCTs had more positive results than NRSs. In one review it was unclear which was more positive. Only three reviews in this section assessed heterogeneity by design; one found more heterogeneity in NRSs and one found more heterogeneity in RCTs. The third found no significant heterogeneity in either group.

Other potential confounders or moderators of effect in reviews in this section were population variables (four reviews), intervention variables (three reviews) and study design/methodological variables (three reviews). No review found that random assignment had a strong effect on outcomes – population variables seemed to be more important.

Potential confounding variables in the three reviews in which results of RCTs compared with NRSs were judged to be ‘mixed’ included participant and methodological variables. Wilson et al. 97 found that other methodological variables were more likely than random assignment to influence outcome.

One possible reason for other variables influencing outcomes more than study design could be that in the reviews we found, randomised and non-randomised study designs have been used in different types of populations/settings/interventions (i.e. the review authors have not set inclusion criteria restricting these other variables). In theory, if a randomised design is chosen, potential confounding variables should be distributed evenly between groups. In reality we cannot confirm this because we do not have reviews in which randomised and non-randomised designs have been applied to the same population/intervention/setting, so we cannot compare them. There is too much heterogeneity between the included studies to isolate the effect of randomisation. This reflects the broad nature of many systematic reviews of policy interventions compared with reviews of more tightly defined health-care interventions (e.g. pharmacological interventions). And so, while within a review it might appear that other potential moderators of effect have a stronger effect than randomisation, within a single RCT this would hopefully not be the case. We cannot confirm or refute this based on the work we have done.

The results of this investigation show us that the ‘state of the art’ in terms of systematic reviews of policy interventions does not yet answer the question of whether RCTs and NRSs are of similar validity in evaluating policy interventions. While it can be argued that RCTs can sometimes be difficult and/or unethical to conduct in certain settings, and that results are not always generalisable to ‘real life’,15,16 it can also be argued that NRSs can be subject to so many biases as to make it doubtful whether it is useful to include them in systematic reviews at all. 148 It seems clear that further investigation should be carried out in the form of properly conducted systematic reviews of policy interventions that include both RCTs and NRSs with the pre-stated objective of investigating differences between them. Methodological studies need to be indexed much more comprehensively in electronic databases.

Criteria used to judge equivalence of RCTs and NRSs

As in the study by Deeks et al. ,40 the manner in which results were judged to be equivalent between study designs varied between the reviews (Table 7). In the majority (13 out of 16) of the included reviews, the criteria used to judge the equivalence of the results were not described. None of the five reviews in which the authors judged the results of RCTs and NRSs to be similar described how it reached such a judgement.

Two of the eight reviews in which the authors judged the results of RCTs and NRSs to be dissimilar described their criteria for equivalence. One93 began by using the chi-squared statistic to identify potential moderating variables. Both calculated a measure of variance between groups (Qb) which was used to estimate the magnitude (or strength of contribution) of the potential moderator variables to the overall effect size. In one of the reviews100 95% CIs were then calculated for each potential moderator variable and non-overlapping CIs were taken to indicate relative strength of moderator or predictor variables. In the other review93 a contribution of 5% or more by a proposed moderator variable to the overall heterogeneity surrounding the effect size was defined as a substantial contribution.

One of the three reviews in which results were judged to be mixed defined the criteria used to judge equivalence between the groups. 91 This consisted of a p-value generated from a z-score based on the null hypothesis that there were no differences between results of RCTs and NRSs. This technique, while being less subjective than simply using authors’ judgement, does not examine the relative contribution of potential moderator variables other than randomisation to the overall effect size.

Given these findings it seems important to note that sensible and objective criteria to judge equivalence or otherwise of results of RCTs and NRSs should be included and applied in systematic reviews that include both study designs. These should be explicitly defined in the review protocol and, where possible, should use methods that take the effects of potential moderating variables other than randomisation into account.

Recommendations for research

Systematic reviews should be carried out with the intention of investigating differences in effects of policy interventions between RCTs and NRSs. Sensible and objective criteria that are supported by empirical evidence should be used to judge equivalence or otherwise of results of RCTs and NRSs in these investigations. Not only should the magnitude of the pooled effect estimates be compared between RCTs and NRSs, but also the variability associated with them. Methodological indexing terms should be developed to enable more fruitful searching of health and non-health electronic databases.

Creating randomised and non-randomised trials

In order to replicate, as far as possible, circumstances that might lead to the creation of non-randomised trials by researchers working ‘in the field’, we created non-randomised trials based on the area in which participants lived. Each area had a number of participants who received the intervention, and a number who did not. Each area could therefore be considered to be a mini-RCT. We had six such areas in Trial 1 and four in Trial 2 (after combining two small areas). We were then able to create non-randomised comparisons by comparing the people who received the intervention in one area with people who did not in other areas. Thus we were able to create 30 non-randomised trials from Trial 1 and 12 from Trial 2. We used all individuals in the selected areas, in contrast to Deeks et al. 40 who drew randomised samples in the selected areas, because our RCTs had far fewer participants than those of Deeks et al.

It is important to note that the average intervention effect in the mini-NRSs must equal the average intervention effect in the mini-RCTs. Our focus will therefore be on comparing the standard deviations of the intervention effects in mini-NRSs and in mini-RCTs. Our design does not allow for the possibility that real NRSs might induce bias by (consciously or not) assigning interventions to ‘more promising’ areas.

Methods for analysis

We had three questions to answer in our re-analysis of the two trials:

1. Do the results of the non-randomised trials differ from those of the randomised trials?

2. Does matching areas on baseline characteristics enable non-randomised trials to approximate the results of the randomised trials?

3. Can adjusting for baseline characteristics in the analysis enable non-randomised trials to approximate the results of the randomised trials?

For the first of the above analyses, we calculated the log odds ratios of the outcomes of interest in all randomised and non-randomised trials. In Trial 1 we had 30 non-randomised and six randomised trials. In Trial 2 we had 12 non-randomised and four randomised trials.

In the second analysis we matched the intervention areas with control areas on baseline characteristics of the study participants. This gave us six non-randomised trials from Trial 1 and four non-randomised trials from Trial 2.

In the third analysis we took each possible comparison to create non-randomised trials, and adjusted for baseline differences in the analysis using logistic regression.

To answer our research questions in each of the above analyses, we tested for differences in the variances of the randomised and non-randomised trials using an F-test. Because this test wrongly assumes that all mini-NRSs are independent, the p-values produced are likely to be too small.

The results of the first re-analysis can be derived algebraically (see Appendix 5). A key quantity turns out to be a correlation coefficient r, derived by computing the observed log odds in each arm in each area, and forming the correlation between the log odds in the intervention arm and the log odds in the control arm. In the appendix we show that the standard deviation of the NRSs is greater than the standard deviation of the RCTs whenever r is greater than 0. It follows that we can test whether the standard deviation of the NRSs equals the standard deviation of the RCTs by testing whether r = 0.

Identification and description of policy intervention evaluations within reviews

Predetermined inclusion criteria and descriptive codes were applied to studies previously reviewed in depth. Reviewers inspected abstracts (where these were available) and previous coding for each study. Previous coding of the EPPI-Centre data set described outcome evaluations according to a standardised keyword system developed by the EPPI-Centre149 covering the type of study (e.g. outcome evaluation, survey, case–control study); the country where the study was carried out; the health focus of the study; the study population; and, for reports describing or evaluating interventions, the intervention site, intervention provider and intervention type. (These studies were also sometimes further classified with review-specific codes.) In addition, extracted data described in detail the population, development and delivery of the intervention, research design and methodological attributes, and the type of outcomes measured (when relevant). Where necessary, reviewers referred to the full reports of these evaluations. Standardised coding and extracted data were also available for inspection as part of the Colorado data set.

The inclusion criteria distinguished policy intervention evaluations from evaluations of other kinds of intervention. Descriptive codes for this study classified evaluations according to our typology of policy interventions (setting of policy/strategies, legislation/regulation, provision/organisation of services, environmental modification, facilitating lay/public delivered support/education); the presence or absence of an explicit collective plan of action; the level at which policy was implemented (international, national, regional, community or institution); the attrition rate; and evaluation designs (RCT or other evaluation design) based on Deeks et al. ’s40 coding framework.

Initially, reviewers worked separately on a subset of studies in the reviews so as to quickly assess the likely availability of policy intervention evaluations and to see how easy/useful it was to apply the inclusion criteria and descriptive codes. This subset was chosen so that it spanned a range of social and organisational settings. The reviewers’ independent responses were compared, discrepancies discussed, and amendments made relating either to descriptions of individual studies or to the definitions of the terms describing policy interventions. Ultimately, screening and coding for each outcome evaluations in the reviews was carried out independently by two reviewers, with discrepancies resolved through consensus.

The results were tabulated to describe the balance of policy intervention evaluations and other intervention evaluations found in each review, and the type of policy interventions (setting of policy/strategies, legislation/regulation, provision/organisation of services, environmental modification, facilitating lay/public delivered support/education), the level at which they operate (national, regional, community or institutional), and whether they were evaluated with an RCT or another design.

Analysis of each study

Analysis combining studies: potential confounders

Any differences between the effect sizes of randomised and non-randomised studies may be explained by other variables that are also related to effect size. For instance, ‘hard outcomes’ provided by clinical data may be more easily obtained in trials set in clinical establishments where randomisation is also more acceptable to the community of researchers and practitioners. In contrast, ‘soft outcomes’ such as self-reported behaviour may be more optimistic and more commonly relied upon on in community settings where randomisation is less acceptable to the practitioners and researchers.

The first stage of analysis was therefore to test for associations between randomisation and any attributes of policy interventions or their evaluations where a theoretical argument may be mounted, or for associations that have been shown in other empirical studies, including results reported in Chapters 3–5. In order to explore these associations between randomisation and other attributes, we cross-tabulated the attribute of interest against study design and tested for statistical relationships using the chi-squared test. (Statistical tests for this analysis were carried out in EPPI-Reviewer.) For some dimensions, e.g. intervention provider, the studies could have more than one attribute. In order to avoid recounting any studies that had more than one attribute in the chi-squared test, the ‘count’ for each study was calculated as one divided by the number of times the study appeared in the test. So, for a study with two different intervention providers, the value of the study was 0.5 in the two cells in which it appeared.

Analysis combining studies: comparing effect sizes of randomised and non-randomised studies

The second stage of analysis investigated the differences between the observed effect sizes of randomised and non-randomised studies within the same systematic review. Because we had data from several systematic reviews, we also wanted to investigate whether or not any differences between the observed effect sizes of randomised and non-randomised studies were consistent across systematic reviews. The analysis was based on the estimated effect size for each study in each outcome domain and allowed for random error in those estimated effect sizes, as expressed by the standard errors. We have controlled for the variation that might be introduced by longer term follow-up by always selecting the outcomes measured as soon as possible after the intervention.

Baseline characteristics

We considered baseline characteristics to be an important variable to explore because of the possible impact of differences in groups on the intervention and evaluation. Groups may differ at baseline because: recipients of the intervention have self-selected or those who declined to participate have been assigned to the control/comparison group; or recruitment favoured those most amenable to participation or those in most need, or excluded older people or those with multiple disadvantages (comorbidities in health, multidimensional identities in social research). Non-randomised controlled trials are more likely to have more heterogeneous populations and non-equivalence between groups. Heterogeneity and non-equivalence at baseline may influence the calculated effect size and variance.

We found that nRCTs were far more likely to report that they had non-equivalent groups at baseline than RCTs in both the EPPI-Centre data set (p = 0.0002) and among the Colorado studies (p < 0.001). However, in neither the EPPI-Centre nor the Colorado studies did this difference translate into an association with effect size.

Attrition

Higher attrition may be expected in community and home settings than in organisational settings and may be expected in transient populations (e.g. commercial sex workers, asylum seekers and socially excluded people). It is easier to employ randomisation and have good follow-up for trials carried out in organisations. High attrition may be associated with losing a disproportionate number of socially disadvantaged people who are more resistant to health promotion/public health initiatives.

We did not find that attrition rates were associated with study type in either of our data sets and, using meta-regression, we also found that different attrition rates were not associated with different effect sizes.

Theoretical underpinnings of the intervention

Logically, interventions underpinned by theory should be more effective. The lack of theoretically based policy interventions has been noted in the fields of changing professional practice and suggested as an explanation for the lack of effective interventions. 151

In another field, we understand that public health triallists value experimental methodologies more than do health promotion specialists, who place more emphasis on involving the community in developing and delivering the intervention, and we expected to find that experimental methodologies in public health are associated with randomisation. 21 We also expected to find that health promotion is associated with community development but not randomisation.

However, we found in the EPPI-Centre data set that RCTs were more likely to have stated or recognisable theories than nRCTs (p = 0.0011), and there were no significant differences among the Colorado studies (p = 0.6930). The presence, or absence, of a theoretical framework was not associated with effect size in either data set. The differences between the results for the Colorado and EPPI-Centre studies may be connected with different data extraction questions. The EPPI-Centre question allows reviewers to infer the theoretical framework, whereas the Colorado question asks whether the authors stated what their framework was.

Public involvement in developing the evaluation

Empowerment theories attribute responsibility to people not for the existence of a problem, but for finding a solution to it. The goal of ‘full and organised community participation and ultimate self-reliance’55 is a feature of social work such as community development and youth work, rather than a feature of public health and randomised experiments. Successful interventions specifically aimed at reducing health differentials include ensuring interventions address the expressed or identified needs of the target population and the involvement of peers in the delivery of interventions. 56

Because of its different data extraction strategy, the EPPI-Centre data set had more relevant information in this area. We found no association between the people who identified the aims of the intervention and whether or not random assignment was employed, or between this variable and the effect size reported by studies (p = 0.7660).

We also found no relationship between the use of needs assessments and whether or not a study employed randomisation (p = 0.3198); and for outcome domain knowledge, attitudes and health state, the use of needs assessments also had no relationship with effect size. However, for behaviour outcomes (the outcome with the most data) we found that, by 0.171 of a standard deviation (95% CI 0.049 to 0.293), interventions based on needs assessments did worse than those which were not. The addition of interventions based on needs assessments to the multivariate meta-regression did not change our findings with regard to whether RCTs have different effect sizes to nRCTs.

We had a similar result when examining the issue of whether or not lay people were involved in developing the intervention. Approximately the same proportions of randomised and non-randomised studies had involved lay people in developing their interventions and those studies which did not involve lay people had better results by 0.210 of a standard deviation (95% CI 0.036 to 0.383) for the behaviour outcome domain. The other domains did not show any significant difference. Including this variable in the multivariate meta-regression appeared both to strengthen the importance of study type and lay involvement in the model.

As part of our coding for policy interventions, we also collected data on whether or not interventions described specific collective plans of action to achieve the intervention’s goals and also on whether the intervention involved the facilitation of lay/public delivered support or education. Non-randomised controlled trials were significantly more likely to have explicit action plans (p = 0.0416) among the EPPI-Centre studies, but were not associated with effect size. There were insufficient data available in the Colorado data set to make a judgement about explicit action plans. There was also no association between lay/public support and type of study or between lay/public support and the effect size of the intervention.

Setting and boundaries of the intervention

Interventions with a broader reach (communities, regions, nations) have more diffuse boundaries than those set within institutions. We expected to see randomisation applied less often to community, regional or national interventions. Clustered trials are more appropriate for these and some organisational level interventions. Attrition may be greater in larger scale interventions, where tracking of individuals is more difficult than within an organisation (see Attrition). Clustering reduces the power of a trial, so clustered evaluations are less likely to show effectiveness. Standardised implementation of interventions may be more difficult across large communities, regions or whole countries than in single organisations and therefore may be less effective.

We collected data on whether there was an explicit formal record of the policy (at institutional, community or regional level), the level at which policy was being enacted (international, national, regional, institutional, community) and whether an intervention was delivered to recipients individually, but we did not find any relationship between these factors and study design or randomisation.

We did find in the EPPI-Centre studies, however, that only nRCTs allocated people by region, with RCTs much more likely to allocate individuals. The relationship between study design and unit of allocation was significant (p = 0.0117) and, when comparing individual assignment with assignment by group, studies that allocated by group had smaller effects on attitudes than those allocating by individuals by 0.281 of a standard deviation (95% CI 0.035 to 0.526). The Colorado studies and the other outcome domains in the EPPI-Centre data set showed no significant differences.

The question on unit of allocation was categorised into individuals, family, group/class (e.g. tutor group), institution, community and region. We then carried out a meta-regression on this variable and found that, for attitudes (p = 0.012) and behaviour (p = 0.033), the size of the allocation unit was negatively correlated with effect size – i.e. that the larger units of allocation had smaller effect sizes [by –0.081 (95% CI –0.144 to –0.018) and –0.051 (95% CI –0.099 to –0.004) respectively]. We then added type of study into the regression finding that this strengthened the size and statistical significance of the associations – both for allocation unit and study type.

We also looked at differences between interventions sited in institutions and those located in the community. RCTs and nRCTs used both settings as much as one another and, possibly looking at the unit of allocation issue from another angle, we also found that community interventions had smaller effect sizes than institutional interventions by –0.159 of a standard deviation (95% CI –0.273 to 0.046).

Provider of the intervention: community/peer provider/practitioner

Community development and peer-delivery specialists value health promotion theory and process evaluations more than RCTs so these interventions may be found to have less randomisation. Theories underpinning community development and peer-delivery anticipate more effective interventions through their greater relevance, and there is empirical evidence to support this. 56

To explore the influence of different types of people delivering or providing interventions, we categorised the intervention providers into community, lay, researcher and practitioner providers. Both RCTs and nRCTs used the same ranges and ratios of providers. Community providers had a significantly worse impact on knowledge (SMD = –0.404, p = 0.003) and behaviour (SMD = –0.247, p = 0.000), and the direction of the other two outcome domains was also negative. Lay providers, often peers, had more mixed results: better than the other providers in changing health states (SMD = 0.276, p = 0.000), similar for influencing attitudes and behaviour, but worse for knowledge (SMD = –0.236, p = 0.024). There were no significant results in either direction for practitioner providers.

Researcher provider

Researchers have more control over the intervention and evaluation and so, theoretically, the researcher would therefore be better able to randomise. Interventions will be found to be more consistently implemented by enthusiasts, and therefore be more effective.

This factor was explored using the same categorisation as ‘practitioner’ (see above). Judging by effect sizes, researchers appear to be better providers for influencing behavioural outcomes (SMD = 0.22, p = 0.045) and about the same as other providers for the other influencing outcome domains.

Choice of outcome domains

Health outcomes are more readily measured in clinical settings; clinical settings are more likely to mount RCTs, and have clinical providers and long-term follow-up. With regard to clinical outcomes being more resistant to change, the health state domain has the lowest overall effect size of 0.123 (95% CI 0.060 to 0.185), compared with 0.251 (95% CI 0.201 to 0.302) for behaviour, 0.306 (95% CI 0.193 to 0.418) for attitudes and 0.449 (95% CI 0.356 to 0.543) for knowledge. Meta-regression also suggests this ordering of outcomes is significant (p = 0.000).This ordering of effect size in relation to domains supports the hierarchy of outcomes proposed by Kirkpatrick63 and Munro et al. 64

Choice of outcome measures

Clinical outcomes are more commonly found in clinical settings. The choice of ‘hard’ or ‘soft’ outcomes can be associated with randomisation. If clinicians favour RCTs, clinical outcome measure may be associated with greater randomisation. Clinical outcomes will be found to be more resistant to change than ‘softer’ outcomes such as reported behaviour

In terms of the choice of outcome measure, there is some suggestion in the EPPI-Centre data set that RCTs use clinical tests more than nRCTs, but this is not quite statistically significant (p = 0.0849).

Sample size

Sample size may be related to the choice of study design. Logically, larger sample sizes may be more likely in nRCTs and smaller sample sizes are more likely to give spurious results; of these, those with positive results are more likely to be published.

As suggested in the above hypotheses, the larger units of allocation provide smaller effect sizes. However, when we regress sample size and effect size in the EPPI-Centre data set, we find no association – although the ‘direction’ is that smaller samples have larger effect sizes (p = 0.221). The Colorado data set has the same characteristics, with a suggestion of smaller effect sizes in the larger studies which does not quite reach statistical significance (p = 0.058). When the two data sets are combined, we have a set of studies with a much larger spread of sample sizes, and the association between sample size and effect size is more pronounced (p = 0.03).

Control group

Control groups are always found in RCTs, but only sometimes in NRSs. We expected to find that the use of a control group leads to smaller effect sizes than are found in uncontrolled evaluations.

As the EPPI-Centre data set did not contain any studies without control groups, this analysis could only be conducted with the Colorado studies. A simple comparison of studies with control groups against studies without does not identify any significant differences. However, combining RCTs with nRCTs conceals differences within the studies that have control groups. The Colorado data set has large effect sizes for RCTs, medium effect sizes for nRCTs and large effect sizes for the non-control group studies. This means that we might expect differences between the nRCTs and the non-control group studies but not between the RCTs and the non-control group studies. If the above hypothesis is correct – that uncontrolled evaluations have larger effect sizes (and the comparison between nRCTs and non-control group studies is consistent with this: p = 0.014) – then the question that remains is why do the RCTs in the Colorado data set have such large effect sizes?

Blinding

Blinding of participants, recruiters, intervention providers and outcome assessors to the intervention allocation is easier for some interventions with randomisation. Poor concealment, common in nRCTs, will be associated with greater effect sizes.

We assessed blinding in three ways in the EPPI-Centre data set: allocation concealment, participant awareness and outcome measurement. RCTs were significantly more likely to use allocation concealment (p = 0.000) and blinded outcome measurement (p = 0.002) than nRCTs. However, nRCTs were equally likely to conceal allocation from the intervention participants. These results are based on small numbers of RCTs and nRCTs (fewer than 40). As our regression analyses are by outcome domain within each review, there were insufficient studies to carry this out to assess blinding.

The Colorado data set did not record concealment of allocation.

Follow-up

Length of follow-up periods is linked with study design; long follow-up may be easier within institutions, where randomisation is also easier. We expected that long follow-up would be associated with declining effect size.

Despite the expectation that longer follow-ups would lead to smaller effect sizes, we did not find any evidence of this in either data set. There was also no association with effect size.

Clustering

Clustered trials with few clusters are more likely to be ‘natural experiments’. Natural experiments do not include randomisation. Natural experiments are more likely to lack blinding and to have enthusiasts supporting the intervention and non-enthusiasts supporting the comparisons, and therefore lead to greater effect sizes. Testing this hypothesis was beyond the capacity of this project.

Quality of the reporting

Quality of reporting specific elements of a study is associated with researchers’ disciplines. Better reporting (of pre- and postintervention data) may be seen to be associated with triallists who also support randomisation. 72 Reporting of pre- and postintervention data precludes effect sizes inflated by differences between groups.

We collected data on a range of aspects of reporting quality in the EPPI-Centre data set: pre-intervention information on sociodemographic variables; pre-intervention data on outcome variables; names of measurement tools; postintervention data on outcome variables; whether there were any obvious shortcomings in the numerical reporting; and whether the study was replicable based on the report. None of the categorical answers was found to be associated with study type: RCTs seem to be as well (or as poorly) reported as nRCTs. Some statistically significant results were found when relating the above factors with effect size, but no obvious pattern emerges:

• Studies that provided full information on pre-intervention sociodemographic variables had higher effect sizes for knowledge: 0.291 of a standard deviation (95% CI 0.030 to 0.551); this variable was not significant in uni- or multivariate meta-regression.

• Studies without obvious shortcomings in their reporting have better results for knowledge by 0.434 of a standard deviation (95% CI 0.261 to 0.607), but no difference for other outcome domains; when combined in a multivariate meta-regression with study type, this variable was significant (p = 0.001) and moved study type from p = 0.183 to p = 0.053.

• Studies giving enough information or providing a further source of information on evaluation design are not as effective as those that do not at changing people’s knowledge by 0.433 of a standard deviation (95% CI 0.046 to 0.821); this variable was not significant in uni- or multivariate meta-regression.

• Studies that do not give sufficient information to ensure that the content of the intervention is replicable do better in the attitudes domain by 0.559 of a standard deviation (95% CI 0.371 to 0.747) than those that do give sufficient information. When combined in a multivariate meta-regression with study type, this variable was significant (p = 0.000) and moved study type from p = 0.059 to p = 0.163.

We examined the Colorado data set on two aspects of reporting quality: replicability and the naming of measurement tools. Neither of these issues was seen to be associated with type of study and, again, we had a scattering of statistically significant results without any clear pattern:

• The meta-regression comparing those studies that rated highly on replicability with those that rated poorly suggests that effect size decreases as replicability reduces, but the result is not quite statistically significant at p = 0.052.

• Studies that named their measurement tools did significantly worse than those that did not by 0.3319 of a standard deviation (95% CI 0.5283 to 0.1355).

Knowledge

The result of exploring the outcome domain knowledge with relation to study type suggested that there was a non-significant effect in favour of smaller effects for RCTs: –0.275 (95% CI –0.641 to 0.091). After taking all the above factors into account in the multivariate regression, the meta-regression now suggests that there is a significant effect in favour of smaller effects for RCTs: –0.269 (95% CI –0.465 to –0.073), p = 0.007.

Attitudes

The previous analysis suggested that RCTs had significantly smaller effect sizes: –0.166 (95% CI –0.319 to 0.012). After taking all the above factors into account, the meta-regression suggests that the direction and quantity of the effect is the same, but it is no longer significant (p = 0.115).

Behaviour

The previous analysis suggested that RCTs had significantly smaller effects: –0.111 ( 95% CI–0.199 to –0.023). The meta-regression suggests that the amount by which nRCTs overstate their effects is slightly larger: –0.192 ( 95% CI–0.330 to –0.053), and the statistical significance of this has increased (p = 0.007).

Health state

The previous analysis suggested that there was very little difference between RCTs and nRCTs: –0.084 (95% CI –0.234 to 0.066). The multivariate meta-regression confirms this (p = 0.611) with the direction of effect now marginally in favour of larger effects in the RCTs.

What is a complex intervention?

Complex interventions are built up from a number of components, which may act both independently and interdependently. The components usually include behaviours, parameters of behaviours (e.g. frequency, timing) and methods of organising and delivering those behaviours [e.g. type(s) of practitioner, setting and location]. It is not easy to define precisely the ‘active ingredients’ of a complex intervention. For example, although research suggests that stroke units work, what, exactly, is a stroke unit? What are the active ingredients that make it work? The physical setup? The mix of care providers? The skills of the providers? The technologies available? The organisational arrangements?

Health services have to evaluate a wide array of existing and newly proposed complex packages, so that the service can learn what is effective about any given intervention so that it can be more widely applied throughout the service. Some complex interventions are intended as improvements in the form of direct interventions at the level of individual patient care; for example, a novel form of cognitive behavioural therapy. Other interventions, although ultimately intended to improve patient care, are actually delivered in the form of an organisational or service modification; for example, the introduction of a physiotherapist or Parkinson’s disease nurse into primary care services. A third type of complex intervention is further removed again from individual patient care, although ultimately intended just as much to impact there, when an intervention is targeted on the health professional; for example, educational interventions in the form of treatment guidelines, protocols or decision-aids. Finally, many complex interventions are delivered at a population level; for example, in the form of media-delivered health promotion campaigns.

Effectiveness of interventions in the workplace: a review

This systematic review aimed to evaluate the potential for using the workplace as a setting for improving adult health. 152 This was seen as an important potential argument owing to the large number of people it would be possible to access, the probable high levels of participations and peer support, and the likely low level of attrition.

Most studies identified were targeted at individuals with varying degrees of environmental modification. No clear evidence of effectiveness was found for type of intervention, topic of intervention or interventions delivered by a particular category of people. Trends of effectiveness were found for comprehensive programmes that combined screening and risk assessment with a range of education programmes, and/or environmental changes. However, these were found in few studies, so replicability cannot be relied upon. Least effective were the weight-loss programmes combining education and financial incentives. No conclusive evidence was found for the effectiveness of peer support.

Due to the general low level of methodological evaluation, data supporting workplace site interventions are not definitive. However, suggestions were made for future research. These included the suggestion that employees should be involved at all levels in the planning and implementation of the activity, the intervention should be supported by top management should be tailor-made to the characteristics of the group. Finally, the quality of reporting should be higher; evaluation in particular should be included in the interventions and a range of outcome measures should be included.

A review of the effectiveness and appropriateness of peer-delivered health promotion interventions for young people

This systematic review synthesised evidence to examine the claim that the peer-delivered approach is a more appropriate an effective method of promoting young people’s health (aged 11–24 years) than more traditional approaches. 153

The most common focus for the outcome evaluations was drugs (including alcohol and smoking), and for the process evaluations it was sexual health. Of the methodologically sound interventions most found clear effectiveness for behavioural outcomes. Studies comparing peers and teachers found equivocal results with regard to effectiveness.

Overall, the review found some evidence to support the effectiveness of peer-delivered health promotion for young people. More than half of the sound studies showed a positive effect at least on behavioural outcome. However, this may be in part because of the scarcity of sounds studies, and lack of good reporting. This report does not encourage peer-delivered health promotion, because there is relatively little sound evidence to support this intuitively appealing idea.

Some suggestions made for future research included implementing health promotion on the basis of a thorough assessment of both self-defined health needs and young people’s views on what would be most effective.

Interventions for encouraging sexual lifestyles and behaviours intended to prevent cervical cancer

This review aimed to determine the effectiveness of health education interventions to promote sexual risk reduction behaviours among women in order to reduce transmission of human papillomavirus. 154 Studies were included if they evaluated educational interventions targeting women only, and measured the impact on either behavioural or clinical outcomes. This was the first review to address cervical cancer prevention in terms of sexual behaviour risk reduction.

All of these included outcome evaluations had the primary aim of preventing HIV and other sexually transmitted diseases rather than preventing cervical cancer. Each of the methodologically sound studies showed a statistically significant positive effect on sexual risk reduction, typically with increased use of condoms for vaginal intercourse. This positive effect was found to be sustained up to 3 months after intervention.

The review drew the following conclusions: that educational interventions targeting socially and economically disadvantaged women including sexual negotiation skill development encouraged at least short-term sexual risk reduction behaviour. This has the potential to reduce the transmission of human papillomavirus and thus possibly reduce the incidence of cervical carcinoma. Health education interventions in which factual information was presented alongside skill development and motivation building was found to achieve short-term increases in reported condom use for vaginal intercourse.

Suggestions for further research included the need for greater attention to gender and culture issues, for interventions need to be sensitive to local culture and context in order to enable women to identify with the health education messages, and that interventions that address power imbalances in relationships are essential for successful implementation. It was also suggested that longer term interventions may show greater effects.

Young people and mental health: a systematic review of research on barriers and facilitators

This systematic review aimed to synthesise evidence on barriers and facilitators of good mental health in young people (aged 11–21 years). 155 A search of systematic reviews was carried out. The findings from this suggested that interventions to promote self-esteem, and those to prevent suicide, were limited in their effectiveness. Some effective interventions were those that addressed young people’s concerns about teachers, parental divorce, bereavement and peer rejection.

The in-depth review of outcome evaluations also revealed limited effects for the promotion of self-esteem. Interventions targeting depression also showed no long-term effects, despite some short-term increases in knowledge about symptoms. The in-depth review of young people’s views revealed some unexpected findings: that mental health tended to be equated with mental illness and thus not be relevant to the respondents; that young people had surprisingly sophisticated understandings of coping strategies and had wide concerns about mental health. The findings also suggested how irrelevant many health promotion materials are to young people’s worries.

The synthesis found some major gaps in the research, including interventions that address young people’s concerns about workload, academic achievement, future unemployment, violence and bullying, and physical appearance (among others). These were suggested as possible future directions for research.

Young people and physical activity: a systematic review of research on barriers and facilitators

This systematic review aimed to synthesise the evidence assessing barriers to and facilitators of physical activity among young people (aged 11–16 years), especially those pertaining to socially excluded groups. 28 This was deemed an important subpopulation as particularly low levels of physical activity, which have been linked with other health-damaging behaviours, are found within this group.

The in-depth review, following a wide mapping stage, comprised 12 outcome evaluations which assessed the effect on health behaviour of interventions aimed at a community/society level. Most were delivered in a school or other educational setting by teachers.

A review of views’ studies was also conducted. Some important barriers to physical activity identified were those related to the self and other people (e.g. incompetence, self-consciousness), practical and material resources/circumstances such as lack of money, and the school (negative physical education teachers). Facilitators identified included activity being good for losing weight, and parental support. Many suggestions were made about how to increase levels of activity by young people, like making activities more affordable and emphasising their ‘fun’ side.

These views and a cross-studies synthesis identified many matches between concerns and evaluations. A need was identified for greater concentration on non-traditional activities, such as aerobics, and evaluations specifically targeting young women.

Young people and healthy eating: a systematic review of research on barriers and facilitators

This systematic review aimed to synthesise the evidence from outcome evaluations of interventions, and from the views of young people (aged 11–21 years) to inform readers on barriers to and facilitators of healthy eating. 27

Most evaluated interventions were carried out in school, so potentially a large proportion of socially excluded young people were missed. This subpopulation was specifically addressed by less than a quarter of identified studies.

The in-depth review focused on interventions aiming to make a change at a community or society level. Most were based in primary or secondary school settings and were delivered by teachers. Some school-based settings were found to be effective at increasing knowledge and improving health behaviour.

The views’ study identified the following barriers to healthy eating: costs and wide availability of/preference for fast food. Facilitators were given as a reduction in costs of healthy food, better availability of healthy food and family support, among others.

Most of the gaps identified in the research by the synthesis were found to have been addressed. However, a need for better nutrition information was suggested, and in general more rigorous research in this area is needed.

Children and physical activity: a systematic review of barriers and facilitators

This systematic review aimed to describe the number, types and quality attributes of existing research studies on the barriers to and facilitators of physical activity among children aged 4–10 years. 156 Both a views’ study and an evaluation of relevant interventions were carried out.

The interventions investigated were extremely diverse, making it difficult to assess patterns of effectiveness. Most were school-based, all involved parents to varying degrees, but some aimed to tackle sedentary behaviour, and others to increase participation in physical activity.

Studies on children’s views about physical activity were scarce. The five that were found focused on barriers to physical activity for children. Twenty distinct barriers were identified, which followed three main themes: preferences and priorities (e.g. a preference for doing other things), family life and parental support, and restricted access to opportunities for participation. Fourteen facilitators to physical activity were also identified, which clustered around the following themes: aspects of physical activity that children value, family life and parental support, and greater access to opportunities for participating in physical activity.

The synthesis found that few health promotion evaluations targeted physical activity outside the physical education lesson, and that children’s views rarely informed the development of interventions. These were identified as possible directions for future research.

Children and healthy eating: a systematic review of barriers and facilitators

This systematic review aimed to find out what was known about barriers to and facilitators of healthy eating in children aged 4–10 years. 94 The review was the first systematic review to rigorously integrate the findings from a meta-analysis with a qualitative systematic review.

The in-depth review focused on the barriers to and facilitators of children’s consumption of fruit and vegetables. Studies that measured fruit and vegetable outcomes were included, as were those studies that examined children’s own perspectives on food and eating.

A substantial amount of research was identified. The relevant outcome evaluations were largely school-based, and often combined learning about the health benefits of fruit and vegetables with hands-on experience. Some interventions also included types of environmental modification, and some targeted multiple outcomes (for instance, body mass index, knowledge, fat intake, physical activity.) These types of interventions were found to have a small but significant positive effect. The larger effect sizes were associated with targeted interventions for parents with risk factors for cardiovascular disease, and with those that focused purely on trying to increase fruit and vegetable increase, rather than ‘diluting’ the effect by promoting, for example, physical activity.

The main messages from this section were that promoting health eating can be an integral part of a school curriculum, and that effective implementation requires skill, time and support from a wide range of people.

The views’ studies comprised eight studies involving children aged 5–11 years and their mothers. From these analyses, the following issues were identified: children do not see it as their role to be interested in health; children do not see health-related messages as relevant or credible; children understand fruit, vegetables and confectionary very differently; children want to exercise choice over their food, eating is valued as a social occasion, and children note the distinction between advice proffered and observed adult behaviour. The synthesis of views and outcome evaluations revealed that opportunities for developing interventions to increase children’s consumption of fruit and vegetables included branding fruit and vegetables as tasty, rather than healthy, and making health messages credible for children.

Suggestions for further research included the need to target interventions towards socially excluded groups and reducing health inequality in general.

HIV health promotion and men who have sex with men: a systematic review of research relevant to the development and implementation of effective and appropriate interventions

This review aims to systematically pull together findings from studies of MSM’s views and integrate them with findings from effectiveness studies. 157 The views’ studies focused especially on young men (aged 16–25 years), men who sell sex to other men and HIV-positive men. The outcome evaluations all had a comparison or control group, discussed interventions delivered during or after 1996 and measured as the outcome of most importance sero-discordant or unknown status unprotected anal intercourse (sdUAI).

The meta-analysis of the outcome evaluations suggested that counselling or workshops based on cognitive–behavioural techniques for MSM at high risk appear to be more effective at reducing the number of men reporting sdUAI than standard counselling. However, this effect was only found where men were recruited from clinic attendance lists, as opposed to adverts or outreach. The narrative synthesis found no evidence for any effect of interventions targeting sdUAI, although none of the evaluations reported knowledge/awareness/attitudes at sufficient quality to be able to assess these effects. Peer-delivered community-based interventions appeared to have no effect on sdUAI.

Suggestions for future research included supporting future interventions such as counselling based on cognitive–behavioural techniques, and that further rigorously conducted and reported primary and secondary research was required on views of all groups of MSM.

Literature search to identify methodological studies comparing RCTs with nRCTs beyond health (see Chapter 4)

It was felt that, although the research team had an extensive knowledge of the existing methodological literature comparing the results of RCTs with nRCTs in health, not enough was known about similar methodological studies in the non-health field (social care, education, criminal justice, housing, etc.). An attempt was therefore made to search non-health databases to identify this methodological literature.

Database searches

Although the searches appeared to be fairly straightforward methodological research comparing two types of study design, a number of problems were encountered.

In the first instance, searching by study design is problematic. Indexing of RCTs in MEDLINE by publication type and medical subject heading has improved in recent years, allowing for more accurate searching, but there is still room for improvement. Searching for NRSs is more difficult. There are many study designs that could be classed as non-randomised, and there is as yet little definitive terminology. It is difficult for the indexer to be sure what type of study design has been used, and so comprehensive and consistent indexing according to study design is lacking. These complications are a major issue in MEDLINE; databases beyond MEDLINE often have poorer indexing by study design. These problems of definition and indexing become more pronounced in databases that are non-health related. As well as poor indexing, there is rarely a thesaurus of keyword terms included, the records often lack an abstract, and a number of databases have only rudimentary search capabilities.

Initial attempts at searching for a combination of terms (indexed and free-text) for RCT with nRCT proved unsuccessful. The number of records retrieved was unwieldy, and the percentage of useful studies within these results was very small. A third search facet was introduced for ‘outcomes’ (bias, effect size, overestimated results, etc.) in an attempt to refine the strategy. This helped a little but not significantly.

The final search strategy used few terms for RCT (and did not include broader terms for ‘study design/methodology’), reduced the non-random terms by rejecting actual study design types (observational, longitudinal, cohort study, cross-sectional, case series, etc.) and used simply ‘non-random’ and equivalent terms (non-experimental, pseudorandom, semi-random, etc.) alongside a precise ‘outcomes’ facet, where fewer search terms were used or aligned using restrictive proximity operators.

It was decided that a more sensitive search using ‘non-random’ study design names, and broader terms for research design should also be completed. This set of results could be referred to in the event that little of relevance was identified in the precise search results.

The following databases were searched:

• ASSIA

• AEI

• BEI

• CareData

• Dissertation Abstracts

• EconLIT

• ERIC

• IBSS

• ISI Proceedings: Social Sciences and Humanities

• PAIS International

• PsycINFO

• SIGLE

• SSCI

• Sociological Abstracts

Literature search to identify systematic reviews with RCTs and nRCTs in the non-health-care setting (see Chapter 5)

It was decided that it would be easier to retrieve all of the references available on review specific databases rather than attempt any searches for reviews with both RCTs and nRCTs included. All of the reviews available on the following databases were obtained: DARE, CDSR, DoPHER and the Health Development Agency Evidence Base database.

There are probably even less definitive terms available for ‘systematic reviews’ or equivalent in the non-health literature than there are for trial designs. Ideally, to ensure that the searches retrieved relevant references, the strategy would have included terms for ‘review’ or at least for ‘literature review’. However, attempts at searching using these terms produced an unmanageable number of references.

A number of test searches were completed in broader non-health databases to estimate the feasibility of attempting a comprehensive search of the non-health literature for systematic reviews. The searches were restricted to specific terms for ‘systematic review’, without using proximity operators, and were further restricted by date range (2003–4).

The following free-text terms and indexed keywords (if available) were used: meta analysis, metaanalysis, systematic review, systematic overview, collaborative review, integrative research, integrative review, research integration, narrative synthesis, evaluation synthesis, meta synthesis, realist synthesis, descriptive synthesis, explanatory synthesis, pool data.

The following databases were searched:

• ASSIA

• BEI

• CareData

• ERIC

• HDA HealthPromis

• PAIS International

• SIGLE

• SSCI

• Sociological Abstracts

The results of the test searches confirmed that it would not be worthwhile conducting thorough, comprehensive searches of non-health databases for systematic reviews.

Searches

The databases searched are listed below with the dates searched and the number of records retrieved. The search strategies used in ASSIA have been listed in full. Full details of the other search strategies used are available from the Centre for Reviews and Dissemination at the University of York.

Methodological studies (see Chapter 4)

Systematic review test searches (see Chapter 5)

Appendix 4.1: Description of included reviews

Appendix 4.2: Similarity of RCTs and NRSs, where authors judged results to be SIMILAR

Appendix 4.3: Results of RCTs and NRSs, where authors judged results to be SIMILAR

Appendix 4.4: Results of any investigation of heterogeneity (where RCTs/NRSs judged SIMILAR)

Appendix 4.5: Similarity of RCTs and NRSs, where authors judged results to be NOT SIMILAR

Appendix 4.6: Results of RCTs and NRSs, where authors judged results to be NOT SIMILAR

Appendix 4.7: Results of any investigation of heterogeneity (where RCTs/NRSs judged NOT SIMILAR)

Appendix 4.8: Similarity of RCTs and NRSs, where authors judged results to be MIXED

Appendix 4.9: Results of RCTs and NRSs, where authors judged results to be MIXED

Appendix 4.10: Results of any investigation of heterogeneity (where RCTs/NRSs judged MIXED)

Appendix 4.11: Table of systematic reviews of policy interventions reporting narrative syntheses of both RCTs and NRSs

Appendix 4.12: Results of systematic reviews of policy interventions reporting narrative syntheses of both RCTs and NRSs

Appendix 4.13: Summary table of policy intervention reviews that pooled randomised and non-randomised designs in a meta-analysis